Module Driven Development - Scaling strategy for your next project

This article was originally posted on papers.adro.codes.

TLDR; Make the move towards developing feature-rich reusable modules instead of once-off components with undetermined dependencies. Module Driven Development aims to move the focus away from code briefly and focus on the cohesion of the codebase.

---

Cohesion

Cohesion is the action or fact of forming a united whole. We measure cohesion as high/low and strong/weak in software development, aiming for high/strong cohesion is preferable, and is often a sign that your software is reusable, reliable and understandable. When focusing on this trait of software development we create fewer dependencies between the pieces of our software, allowing for modifications in one part that don't cause adverse effects on another.

Knowing this, we as developers aim to implement practices and coding methodologies to improve cohesion, often if not always only focusing on our code and not the structure of our codebase. Module Driven Development (MDD) aims to move the focus away from code briefly and focus on the cohesion of the codebase. Taking a step back and addressing the structure of our folders/files, implementing a consistent structure that is predictable can and will provide lasting benefits to any sized project.

Weak Cohesion

Spoilt for choice without direction. This is the world that many developers find themselves in, especially within the front-end community. One of the most popular UI libraries React is one of these choices and one that is made over 15 million times a week, according to the npm package page. Yet, if you were to look for guidance on a project structure you'll find a similar amount of articles discussing the topic. Why is that? React is un-opinionated meaning, it doesn't enforce any structure on you, it is up to the developers and teams to decide their approach. In my experience, this works great when working in small 1-2 person teams, but not 5-10 person teams across time zones. In team sizes like these, the reusability, reliability and collective understanding suffered.

One piece of famous or infamous advice was;

Move files around until it feels right

In many ways, I agree with the underlying idea, don't overcomplicate too quickly and develop the structure as your application grows. I've followed this advice to some degree on many of my projects and everything worked out however, returning to those projects, later on, revealed a problem with this approach. It was vastly different to the project I am working on now, moving through the project was sluggish and inefficient.

The cohesion was weak.

Through this realisation, I started working on a new approach. One where I respect the advice of moving files around until it feels right, but with the confidence that returning to the project in the future will feel as familiar as the project I am working on in the present.

Strengthen Cohesion

Analysing the problem

We've identified that the approach of "move files around until it feels right" can lead to weak cohesion. That is, it can lead to a codebase that isn't scalable or understandable. Below is a representation of a common project structure for many React (or Front-end in general) projects (left) and the location of various user related components and functionality (right).

At first glance, this appears to be manageable. Different parts are separated into logically categorised folders. However, if you have worked in a codebase similar to this, you'll be aware of the challenges as the project scales. Every new component required adds at least 1 more file/folder to 2 or more of your root folders. Soon, you'll run out of names for files, each folder will contain 50+ files ranging from User related components to your Footer. Additionally, some knowledge sharing is required to define a container component, or your styles will slowly migrate to the respective component folder.

All of these problems and more I have faced coming into large projects that use these techniques. The best I can do is hope that the changes I am required to do only span 1 or 2 files across as many folders, although there is usually still time required to find the start of the breadcrumb trail to follow.

Analysing this structure, we can say that the structure is influenced by the file type. It is grouping .css, .d.ts, .stories.tsx, .gql etc files into separate folders to establish a separation of domains. We can also identify potential scaling concerns, as more pieces are added, these domain folders will continue to grow, increasing the time needed to ramp up development on an existing feature. Finally, identifying cross-component dependencies becomes harder as there isn't a single-entry point that can be easily identified for imports.

With these points in mind, we will look at and compare the proposed structure used for Module Driven Development.

Proposing MDD

Below is the representation of the UserProfile when organising the files using Module Driven Development.

We are still separating different parts of the component into ui, styles etc folders, with one key difference. They are all contained within the UserProfile module. In the next section, I will be outlining some of the principles of MDD, for now, I want to compare this structure to the problems we identified with the previous project structure.

• "...the structure is influenced by the file type", "grouping... files into separate folders to establish a separation of domains."
  • We are still separating by file type (within a module) however, the key difference here is we are establishing separation through product/project domains rather than the files. These domains are influenced by the problem domain your project is operating in and create a connection between the modules and your requirements.
• "...as more pieces are added, these domain folders will continue to grow, increasing the time needed to ramp up development on an existing feature."
  • A project will always have a growing file structure. However, with MDD the growth is contained to the top-level modules folder and any module groupings. Rather than spread across several folders.
  • The more important benefit here is in regard to updating existing features. Since a feature would be contained within a module, it is much easier to locate the module and be presented with the full picture. It will also lower the cognitive load to scroll through or search for files across your entire solution.
• "identify cross-component dependencies"
  • This issue now becomes trivial. If you're using something like path aliases with Typescript, you'll be able to do a global search for @module/UserProfile and find all imports referencing that module.
  • Additionally, this provides the benefit of being in control of your module API. If your entry is at @module/UserProfile, you can decide what is exported and what isn't. Allowing for both public and private components/functionality. Using this approach, you will be able to identify when a "private" piece of code is needed in another module. This could signal that some refactoring is needed to make a piece of code more globally accessible.

With a fairly simple restructure of our project, we've been able to address each of the major problems identified with the previous approach. The codebase is now easier to navigate and understand. We've strengthened the overall cohesion.

Let's continue and look at 4 principles that I recommend for MDD.

1. Think narrow and broaden your horizon
2. Continuous refactoring
3. Importance of consistent structure
4. Modules can have modules

Principles

Think narrow and broaden your horizon

It is often tempting to identify something as generic and implement it in a common utilities or lib folder. This principle goes against that way of working slightly, instead of prematurely extracting functionality, first place it within the module that is being worked on, this will:

1. Encapsulate all the functionality within the module and,
2. Allow modules to be easily transferred to new projects.

This does open the question of, what if a piece of functionality is needed in another module. That leads to the second principle of Continuous refactoring.

Note: there is a very high chance that your project will still contain a utilities or hooks module. The difference here is in the contents of the folder. Instead of containing ALL utilities or hooks, the code in this folder would have been identified as functionality that is generic and needed in several places. Once again, making tracking dependencies easier.

Continuous refactoring

When a piece of existing functionality is identified as something generic, minor refactoring can take place to move the functionality to a commonplace. I am a big advocate for "leave it cleaner than when you found it". Having the chance to do a bit of refactoring allows you as a developer to revisit old code and clean it up while extracting the functionality. Not only does this keep your codebase healthy, but you may also stumble on existing bugs that haven't been reported yet.

Importance of consistent structure

The modules you write are only as good as their structure. This may take some work with your team to identify the structure that works for them however, the importance here is, to keep it consistent, not just within the current project but within future projects. This will give any team member a basic understanding of where to locate code if they are new to a project. It will also aid new team members, once they are onboarded into one codebase, they can comfortably navigate any other codebase. Below outlines a structure that I use on projects:

• ui - a collection of UI (React, Vue, etc) components. You can also place stories here if you're using Storybook.
• service - a collection of hooks used throughout the components. This can range from utility hooks to hooks that perform API calls.
• context - define Context/State for the module (Read more about Micro Contexts for details on module-based context).
• common - define common types, utilities and constants for the module.
• styles - any styling required for the components.
• integration/data-access - define GraphQL queries, and data transformation needed for the component.
• spec - any tests that are required. It is up to you whether to place this in an enclosing folder. For example, ui/spec, common/spec.

Modules can have modules

When it makes sense, or to create further domain separation, a module can have nested modules. Above we defined a UserProfile module, this would make more sense to be a part of a user or authentication module, allowing you to group together Login, Sign up, User profile, User Settings, etc in one folder.

The useful approach here is, that the top-level authentication module can share the consistent folder structure as above allowing all nested modules to share common pieces of functionality with each other, for example, the AuthenticationContext.

Strong Cohesion

We've identified what is causing weak cohesion in a codebase and addressed these concerns with the techniques and principles of Module Driven Development. The techniques allow us to structure projects in a predictable, scalable and understandable way, allowing the codebase to stay consistent across project boundaries.

The cohesion is strong.

Case study

The theory of a methodology and the utilisation of one are two different things. I would like to present a case study of a recent JAMStack project I was a part of. Although the client has to remain anonymous, I will describe the problem tackled and how implementing MDD allowed the codebase to be used by multiple businesses owned by the client.

The client and requirements

The client was a health and fitness brand with two separate businesses, a personal training business and an education business. They needed to continue being separate websites, but ideally share the same components and CMS to allow for a consistent user interface and experience, while allowing staff to have a consistent way to update both websites.

From these requirements, we knew that reusability across the projects would be a necessity. With that, cross-component dependencies should be kept at a minimum to allow them to be shareable without needing to share unrelated code. Unfortunately, the codebases needed to remain separate in this case and we weren't able to utilise a package distributor like npm. We also had to keep the CMS instances separate, so transferring components between websites was a bit manual, which is why the cross-component dependencies would need to be kept to a minimum.

Applying Module Driven Development

We started with identifying the different sections of the website and categorised them into several modules, some of these were top-level modules, while others could be grouped into a more generic module, for example, a banner module which included 4 sub-modules.

The authoring experience allowed the client to construct pages using "Lego Blocks", allowing them to stack sections of a page on top of each other. Each section had it's own Model or Content Type associated with them. Not only does this allow for the configuration of the component to be easily located, but the integration into the CMS could also be included in the module itself.

This allowed us to create a system where modules could be included in the data querying process. Once the data has been fetched, we can loop through the received data, identify the module the data belongs to and render the component to the page.

This central "opt-in" system allowed components to be registered for requesting data and rendering and since both the integration and UI were contained within a module, if a component was developed specifically for one of the businesses, we would be able to copy over the module, register the module and have it immediately available for the client to use in their pages.

This approach allowed the first website to be completed and live within ~2 months, with development on the second taking less than half that. The majority of the effort was spent working with the client to re-develop the IA (Information Architecture) and identifying a handful of modules that were required for the new business, a few of them being ported over to the original project once development was completed.

The most encouraging part for me was, I had little to no input on the second project, however, I was tasked with porting over those selected components. I received a ticket with the title "Move over components X & Y to project one". The following were the steps I took:

1. Cloned project two.
2. Located the module folders for X & Y.
3. Copied them over to project one.
4. Configured the CMS for project one.
5. Registered the components.
6. Pushed the changes.
7. Moved the ticket to review.

This process took less than an hour to complete. I would call that a win for using the Module Driven Development methodology. The consistent codebase across both projects allowed me to very efficiently navigate the second project, the developers on the second project followed the patterns outlined in project one and created two modules with zero outside dependencies (other than common UI components), making the porting process effortless.

There was, however, one aspect of the project that was unfortunate, the need for separate codebases. Ideally, if you have a client with multiple businesses that should share components and functionality, keeping the code in one codebase would be preferable.

Conclusion

We've been able to define a very important metric that we measure software against, cohesion, and apply that metric to a common practice in the Front-end development world. Through that lens, we were able to identify several key problems that impact these can have on the scalability and developer experience of the codebase. Through utilising the Module Driven Development methodology, we saw these problems lessen, strengthening the cohesion of our codebase, and finally, we looked at a case study where these techniques allowed for rapid development and share-ability between projects.

I hope that you are able to take some of these concepts and apply some of them to your workflow as an engineer. One of the best parts of this methodology is, it does not require a full refactoring of a project. Because we focused on the folder structure employed, you are able to create a modules folder and place future features in their own module, employing the continuous refactoring approach means as older features are worked on, they can slowly be migrated to a module of their own.

Cover Image Credit: Mourizal Zativa