Managing Simultaneous Learning and Productivity in Software Development

The abundance of opportunities to learn new things is one of my favorite aspects of being a Software Developer. However, software development in professional settings is often severely time-constrained, so I rarely find enough time to gain an expert-level understanding of the numerous concepts I encounter during the course of my work. These time constraints typically affect how deeply I am able to absorb new technical concepts and still maintain high levels of productivity. Whenever I am faced with a situation that calls for balancing time, learning, and action, my inclination is to formulate and apply some sort of structure to the situation. This post describes the framework I use to manage the simultaneous productivity and meaningful learning in time-constrained contexts.

I like to think about the learning process as a series of questions and knowledge levels:

• Level 0: What is it called?
• Level 1: What does it do?
• Level 2: What are some alternatives and why does this one exist?
• Level 3: How does it work/How does it do what it does?

I often use this framework to calibrate how deeply I need to understand new technical concepts within the context of a specific task and associated time constraints. I consider levels 0 and 1 to be “shallow” understanding. Level 2 is neither shallow nor deep, but often sufficient to satisfy my curiosity and help me make more informed choices. Level 3 represents deep understanding. This learning framework may be generalizable to other subject areas and professions, although I use it most often within the context of software development.

Now let us work through a scenario where this framework may be useful. Consider the function below (written in Python): 

def first_n_numbers(n):
    num = 0
    while num < n:
        yield num
        num += 1

The first time I saw a Python function like the one above, the yield statement seemed rather strange to me. What thought process did I follow to learn more about this strange yield statement? You guessed it! 

Level 0: What is it called?

This level of understanding focuses on concepts and terminology. I consider this to be the most fundamental level of understanding one can achieve about a given subject. Knowing the proper terminology that applies to a particular subject of interest is an invaluable foundation for seeking further information.

In the context of our example with the Python function and yield statement, a quick google search for the phrase “python yield statement” tells me that this construct is called a “generator function”, and is related to iterables, iterators, and loops in Python. If I am really in a hurry and the nature of my task allows, I may keep all these related terms in mind for later, stop at this level of inquiry, and move on with the task. More often than not, I will proceed to level 1 to gain a better understanding of what these “generator functions” do.

In some scenarios, one may already know the right terminology for the subject of interest. For example, if I were reading an article or a book on functional programming and came across the word combinator, it would be rather obvious what exact term I need to search for to acquire additional information. In such a scenario where the precise name of a concept is obvious, I’ll immediately begin further learning at level 1.

Level 1: What does it do?

This level focuses on understanding the immediate effect(s) of using or adopting a particular tool, concept, or programming construct. I typically stop at this level when my top priority is to complete a task, rather than to gain a deep understanding of the particular subject of interest.

In our generator function example, this level of understanding focuses on figuring out what generator functions do when you use them. A little digging on the interwebs tells me that:

• A generator function i.e. a function with a yield statement, acts like an iterator. Therefore, such a function can be used as part of a for loop as in for i in first_n_numbers(10). 
• The yield statement in a function provides a result to the caller without destroying local variables.
• Each call to a generator function resumes execution from where it left off after the most recent call.
• Generators produce iterables, but they are not collections and thus have no length.

For programming constructs, it is often helpful to find a few code examples at this level to help illustrate proper syntax and semantics.

Level 2: What are some alternatives and why does this one exist?

This is where learning starts to go beneath the surface. This level focuses on identifying alternative ways to achieve the same or similar objectives, and understanding how the subject of interest compares to the identified alternatives. Knowledge acquired at this level helps me to make informed decisions regarding when to apply the subject of interest or choose some other alternative that might be less elegant but more widely understood.

So what are some alternatives to using generator functions as defined in our example? At this level, I will discover that instead of using a generator function with a yield statement, we could:

• Write a function that constructs and returns a list in memory.
• Create a class with the generator pattern and use an instance of the class as an iterator.

So why would one choose to use a generator function instead of one of the other alternatives outlined above? If the objective is to produce something that can be iterated over, why not just construct a regular Python list, return the newly constructed list and be done with it? It is at this level that I will discover that:

• Writing a function that constructs the full list in memory does not scale well. If each item in the list happens to take up a lot of space, or we need to generate a stupendously large number of items, we could run out of memory real quick.
• Creating a class with the generator pattern helps us deal with the performance issue, but it involves quite a bit of boilerplate code and convoluted logic relative to using a generator function. The boilerplate becomes more of an annoyance if we have to create many such classes with the generator pattern.
• Generator functions are a convenient shortcut to building iterators that are memory-efficient with clear natural logic and without annoying boilerplate code.

With this level of knowledge, I am able to understand when it is beneficial to use a generator function with a yield statement and when it is not. I might choose to use a simple list or some other simple iterable, rather than blindly use a generator. A simple iterable in memory would be a reasonable choice in scenarios where I have a small constrained list of items and do not need to worry about memory and performance. 

Ultimately, I have found the knowledge gained at this level to be incredibly valuable for communicating with others and explaining the rationale behind my decisions. Communication and informed decision making is a key ingredient in successful engineering teams. This level of understanding could help foster useful conversations, particularly in code reviews.

Level 3: How does it work/How does it do what it does?

I consider this to be the deepest level of understanding in my framework, assuming the other levels have been sufficiently explored. I typically resort to this level when my primary goal is to teach a class or write technical documentation about a given topic. Most of the time, I do not reach this level of understanding, as the time and effort required often outweigh the immediate benefits. This level is potentially infinite and requires an understanding of how things work at a level beyond what most situations require. For technical concepts in software engineering, many universities spend a significant amount of time teaching concepts at this level and things can get quite arcane pretty fast.

For generator functions, it is at this level that I will seek to understand all of the following:

• Python interpreter internals
• Python bytecode
• Stack frames
• Heap memory

I have not yet taken the time to fully understand generators at this level as I imagine it is going to require quite a bit of time and dedication to learn all the other associated concepts that make generators work the way they do. For anyone interested in understanding generators at this level, the “How Python Generators Work” section of this AOSA book chapter might help. Good luck, and have some Ibuprofen or Tylenol close by just in case you get a headache. :)

In this post, I have outlined the framework I use when I need to balance the tradeoff between learning new technical concepts and rapid progress toward the completion of a software development task. I use this framework to calibrate how deeply I need to understand new technical concepts in time-constrained situations that require simultaneous productivity and learning. This is a daily reality for me as a professional Software Developer. I hope others find it useful, enlightening, or at least amusing.