Once upon a time, I worked at a company that put me on, to be euphemistic, a challenging project. To be unkind, perhaps: the Hell Project. This project was one of the most dysfunctional and most hated at the organization and even the most senior people interacting with it were overwhelmed at the task of righting the ship. This wasn't a junior varsity team either - the company in question had been around for well over a decade and was considered a heavyweight in its space. It was the sort of project that books are written about. In this case, at least, it yielded a short essay.
When presented with such a problem I will doggedly pursue it with all of my energy, becoming all-absorbed in fixing what's broken. However, as is wont to happen in such a situation, I was dirt honest about how I felt about the project and spoke about it in a very uncharitable manner - and it had a predictable effect on the morale of the team. I was eventually confronted about this.
I try to be open-minded and reflective of feedback that I receive from members of my team despite my knee-jerk hatred of criticism. This is because, deep down, I know that criticism is part of the feedback loops that we all need in order to iterate on our approaches and self-improve. I reflected on this feedback, and what I found on inspection was that it was true - and in fact this realization led to ideas that fundamentally changed the way I thought about the project.
I had been studying some philosophy during that time 1 in an effort to more fully understand who I was and what I wanted out of the world and out of my career. I was a STEM major in college, choosing to follow my values of practicality (Holbrooks are obscenely practical) over what I perceived as bikeshedding questions of mere academic value. However, as I grew older I found philosophy to be surprisingly practical. Philosophy doesn't simply help answer whether morality is relative or absolute (I know what's moral in my heart either way), but also questions with immediate relevance to the software engineering industry. For instance: How should your machine learning model be trained? This is one of the more obvious ones.
One of the philosophers I learned about is Georg Hegel, an early 19th century German idealist philosopher. Hegel is relatively unapproachable - it seems like he was overall an impenetrable writer - but deeply insightful, and highly influential in the field through the 1800s and beyond to today.
One of Hegel's strongest contributions to philosophy was popularizing what we now call Hegelian dialectics 2. Dialectics are a little tough to wrap one's head around (at least for me) and it took me a few tries to start feeling confident about them such that I could apply the idea to my world. For instance, when my coworkers pointed out my negativity, I realized that this project - and in fact all legacy systems - can be understood in the context of these Hegelian dialectics.
For our purposes, I don't equate a legacy project with one that isn't being actively worked on, nor one that's strictly being sunsetted. In Working Effectively with Legacy Code, Michael Feathers describes legacy projects as projects which are tough to change, links this idea to missing/ineffective testing, and then writes a book about object oriented testing. It's one of my favorite programming books, and while I think testing isn't the whole story - this legacy project had a ton of tests - it's certainly true that a "legacy" project by similar definition can be refactored incrementally into a "current" project. It's these projects which I think fit this pattern.
All Hegelian dialectics follow the form of a particular type of narrative, much like how a story might follow the Hero's Journey. The acts in this story take a three part form. Hegel called these stages the abstract, negative and concrete; though most people today talk of thesis, antithesis and synthesis. We can see these in the story of a legacy project.
Thesis: The project you're working on was written by rational, well-meaning people, who made the decisions they made for valid reasons. The vast majority of engineers take pride in their work, want to create quality code, and most importantly want to work on an impactful project that's fun to add features to. We all endeavour to build the best systems we can. If nothing else, this project has stood the test of time, ultimately doing what's been required of it, often for years.
Antithesis: The project you're working on is a legacy project. Doing anything with it is excruciatingly slow-going and it's way too easy to accidentally break something. The abstractions are clunky and indirect. Components are tuned in ways that don't make any sense. Every day you work on this project is a day where you curse your forebearers's names - if you even know what they are.
In a Hegelian dialectic, these two components exist in stark tension with each other. The thesis narratively stands for a sort of hypothesis, a model of the world. The other name from Hegel's writing - abstract - hints at how this component forms an initially naive model. The antithesis - or negative - meanwhile stands as an angry refutation of that model. Our initial thesis - that this is how you make good software - seems to be provably Bad.
These dual components of this present reality exist as a semi-stable equilibrium. As you try to refactor, rewrite or otherwise address the legacy system, you run into snags. Perhaps touching that bizarrely configured tunable somehow breaks the batch jobs. "If I were the engineer that wrote this I would have simply written good code instead of bad code," you find yourself saying, even as you lose your boots to the quicksand. Yet, it continues to bring you pain.
Hegel, it seems, conceived of history as a never-ending series of these dialectics playing out in real life - much like the refactors playing out every day in our offices. In this conception of history, societal progress occurs by truly understanding the contradictions and resolving these duels into a single unified, cohesive whole which cancels the negative while retaining the strengths of the initial thesis. The situation transcends its initial state into a new, better world. I also get the impression that Hegel would have said these effects are roughly proportional - that is, the more stark the contrast and the more intense the conflict, the more satisfying (and perhaps beautiful) the synthesis will be.
In a different role, I led the rewrite of ingestions and pipelines supporting a medium sized media company's Amazon affiliate program. What existed when my team onboarded was in a sad state. Developers who had worked on the core service consistently said mean things about it and went out of their way to not work on it - and it had suffered as a result. Many of the components were strangely broken, some inaccurate. We knew immediately when we saw them that we wanted to push them into the sea as quickly as possible.
It was initially a project we thought we could sunset mercilessly within a few sprints, but we quickly discovered that the issue was more complex than we initially thought. As I pored over the source code to the core service I began to become very worried about the scope of work and rightly so - what we had originally planned to take a few weeks turned into a long-term major set of rewritten and extended pipelines that we developed and maintained over the course of multiple quarters.
We of course fixed and overhauled almost everything. Yet as we pulled metrics into our core BI tool, added systems health and monitoring, generated more accurate numbers and even built A/B testing capabilities into our commerce metrics, a few critical aspects of the old designed remained: The format we used when generating tracking codes, the language and database server for hosting the web-site metrics and the obscure API we were using to pull reporting were all retained in the final design. As time progressed, I gained not just an understanding of the negative aspects of this project's antithesis, but also a greater appreciation for some of the oldest abstractions in the codebase.
The evolution and resolution of the Hell Project is another story, but that affiliate program system ended up being one of my favorite projects that I'd worked on over my career. However, in the world of the Hell Project I was letting my frustrations in that present moment emphasize the negative, not just emotionally but in terms of this refactor dialectic as well. What we need to remember when we work on these legacy projects is that it's inevitable that the dialectic will find synthesis: one way or another, the system will be refactored, and this refactoring will necessarily find unity between the system that exists and the system that the engineers wish they had instead.
Casually! You'll notice that most of the links in this post are to Philosophy Tube and Wikipedia. ↩
More astute readers will quickly realize (or remember) that Marx and friends took and ran with dialectics in a somewhat bizarre direction. This spin on the base theory is interesting follow-up reading, but not necessary for this post. ↩