Hello, my name is Dmitriy Karlovskiy and I... love MAM. MAM owns Agnostic Modules, saving me the lion's share of the routine.

Agnostic Module, unlike the traditional one, this is not a file with a source code, but a directory inside which there can be sources in a variety of languages: program logic in JS/TS, tests for it in TS/ JS, component composition on view.tree, styles on CSS / CSS.TS, localization on locale=*.json, pictures, etc., etc. If desired, it is not difficult to fasten support for any other language. For example, Stylus for writing styles, or HTML for describing templates.

Dependencies between modules are tracked automatically by analyzing source codes. If the module is included, then it is included in its entirety - each module source code is transpiled and gets into the corresponding bundle: scripts - separately, styles - separately, tests - separately. For different platforms - their own bundles: for the node - their own, for the browser - their own.

Full automation, no configuration and no boilerplate, minimum bundle sizes, automatic dependency pumping, development of hundreds of alienated libraries and applications in one codebase without pain and suffering. Wow, what an addiction! Get pregnant, nervous, children away from monitors and welcome to the submarine!

Philosophy

MAM is a bold experiment in radically changing the way we organize and work with code. Here are the basic principles:

Conventions instead of configuration. Smart, simple and universal conventions allow you to automate the entire routine, while maintaining convenience and consistency between different projects.

Infrastructure separately, code separately. It is not uncommon to develop dozens or even hundreds of libraries and applications. Do not deploy the infrastructure for build, development, deployment, etc. for each of them. It is enough to set it once and then rivet applications like pies.

Don't pay for what you don't use. You use some module - it is included in the bundle with all its dependencies. Do not use - does not turn on. The smaller the modules, the greater the granularity and the less redundant code in the bundle.

Minimum redundant code. Breaking code into modules should be as easy as writing all the code in one file. Otherwise, the developer will be lazy to break large modules into small ones.

No version conflicts. There is only one version - the current one. There is no point in spending resources on maintaining older versions if you can spend them on updating the latest one.

Keep your finger on the pulse. The fastest possible feedback regarding incompatibilities will not allow the code to go bad.

The easiest way is the surest. If the right way requires extra effort, then be sure that no one will follow it.

Imports/Exports

Open the first project we find using the modern module system: Module less than 300 lines, 30 of them are imports.

But that's just flowers: A function of 9 lines requires 8 imports.

And my favorite: Not a single line of useful code. 20 lines of swapping values ​​from a bunch of modules into one, so that then import from one module, not from twenty.

All this is a boilerplate, which leads to the fact that developers are too lazy to separate small pieces of code into separate modules, preferring large modules to small ones. And even if they are not lazy, then either a lot of code is obtained for importing small modules, or special modules that import many modules into themselves and export them all in a crowd.

All this leads to low code granularity and bloated bundle sizes with unused code that is lucky enough to be near the one that is being used. At the very least, they try to solve this problem for JS by complicating the build pipeline by adding the so-called "tree-shaking" that cuts out the excess from what you imported. This slows down the building, but cuts far from everything.

Idea: What if we don’t import, but just take and use, and the bundler will figure out what needs to be imported?

Modern IDEs can automatically generate imports for the entities you use. If the IDE can do it, then what prevents the builder from doing it? It is enough to have a simple naming and file arrangement convention that is user-friendly and machine-readable. PHP has long had this standard convention: PSR-4. MAM introduces the same for *.ts and *.jam.js files: names starting with $ are Fully Qualified Name of some global entity whose code is being loaded along the path obtained from FQN by replacing separators with slashes. A simple two-module example:

my/alert/alert.ts

const $my_alert = alert // FQN prevents name collisions

my/app/app.ts

$my_alert( 'Hello!' ) // Yep, dependent on /my/alert/

A whole module from one line - what could be simpler? The result is not long in coming: the ease of creating and using modules leads to minimizing their size. As a consequence - to the maximization of granularity. And like a cherry - minimizing the size of bundles without any tree-shaking.

A good example is the JSON validation module family /mol/data. If you use the $mol_data_integer function anywhere in your code, the bundle will include the /mol/data/integer and /mol/data/number modules, on which $mol_data_integer depends. But, for example, the bundler will not even read /mol/data/email from the disk, since no one depends on it.

Cleaning up the mess

Since we started kicking Angular, we will not stop. Where do you think the applyStyles function declaration is to be found? Never guess, in /packages/core/src/render3/styling_next/bindings.ts. The ability to put anything anywhere leads to the fact that in each project we observe a unique file layout system, often defying any logic. And if in the IDE the "jump to the definition" often saves, then viewing the code on the github or reviewing the pull request is deprived of such an opportunity.

Idea: What if entity names strictly match their location?

To place the code in the /angular/packages/core/src/render3/stylingNext/bindings.ts file, in the MAM architecture you will have to name the entity $angular_packages_core_src_render3_stylingNext_applyStyles, but, of course, no one will do this, because there are so many unnecessary things in name. But you want to see the names in the code short and concise, so the developer will try to exclude everything superfluous from the name, leaving only the important: $angular_render3_applyStyles. And it will be located accordingly in /angular/render3/applyStyles/applyStyles.ts.

Note how MAM exploits developers' weaknesses to achieve the desired result: each entity gets a short globally unique name that can be used in any context. For example, in commit messages these names allow you to quickly and accurately capture what they are about:

73ebc45e517ffcc3dcce53f5b39b6d06fc95cae1 $mol_vector: range expanding support
3a843b2cb77be19688324eeb72bd090d350a6cc3 $mol_data: allowed transformations
24576f087133a18e0c9f31e0d61052265fd8a31a $mol_data_record: support recursion

Or, let's say you want to search all mentions of the $mol_wire module on the Internet - make it easy thanks to FQN.

Cyclic dependencies

Let's write 7 lines of simple code in one file:

export class Foo {
    get bar() {
        return new Bar();
    }
}

export class Bar extends Foo {}

console.log(new Foo().bar);

Despite the cyclic dependency, it works correctly. Let's split it into 3 files:

my/foo.js

import { Bar } from './bar.js';

export class Foo {
    get bar() {
        return new Bar();
    }
}

my/bar.js

import { Foo } from './foo.js';

export class Bar extends Foo {}

my/app.js

import { Foo } from './foo.js';

console.log(new Foo().bar);

Oops, ReferenceError: Cannot access 'Foo' before initialization. What kind of nonsense? To fix this, our app.js needs to know that foo.js depends on bar.js. So we need to first import bar.js, which will import foo.js. After that, we can already import foo.js without error:

my/app.js

import './bar.js';
import { Foo } from './foo.js';

console.log(new Foo().bar);

What browsers, what NodeJS, what Webpack, what Parcel - they all work crookedly with cyclic dependencies. And it would be nice if they simply forbade them - they could immediately complicate the code so that there were no cycles. But they can work fine, and then bam, and give an incomprehensible error.

Idea: What if when building we just glued the files together in the correct order, as if all the code was originally written in one file?

Let's split the code using MAM principles:

my/foo/foo.ts

class $my_foo {
    get bar() {
        return new $my_bar();
    }
}

my/bar/bar.ts

class $my_bar extends $my_foo {}

my/app/app.ts

console.log(new $my_foo().bar);

All the same 7 lines of code that were originally. And they just work without additional shamanism. The point is that the faucet understands that the dependency of my/bar on my/foo is more rigid than that of my/foo on my/bar. This means that these modules should be included in the bundle in this order: my/foo , my/bar , my/app.

How does the bundler understand this? Now the heuristic is simple - by the number of indents in the line in which the dependency was found. Note that the stronger dependency in our example has zero indentation, while the weaker dependency has double indentation.

Different languages

It just so happened that for different things we have different languages ​​for these different things sharpened. The most common are: JS, TS, CSS, HTML, SVG, SCSS, Less, Stylus. Each has its own system of modules, which does not interact with other languages ​​in any way. Needless to say about 100,500 types of more specific languages. As a result, in order to connect a component, you have to separately connect its scripts, separately styles, separately register templates, separately configure the deployment of the static files it needs, and so on and so forth.

Webpack thanks to loaders tries to solve this problem. But his entry point is a script that already includes files in other languages. What if we don't need a script? For example, we have a module with beautiful styles for signs and we want them to have one color in a light theme and another color in a dark one:

.dark-theme table {
    background: black;
}
.light-theme table {
    background: white;
}

At the same time, if we depend on the theme, then a script must be loaded that will install the desired theme depending on the time of day. That is, CSS actually depends on JS.

Idea: What if the module system was independent of languages?

Since the module system is separated from languages ​​in MAM, dependencies can be cross-language. CSS may depend on JS, which may depend on TS, which may depend on other JS. This is achieved due to the fact that dependencies on modules are found in the sources, and the modules are connected as a whole and can contain source codes in any languages. In the case of the themes example, it looks like this:

/my/table/table.css

/* Yep, dependency on /my/theme */
[my_theme="dark"] table {
    background: black;
}
[my_theme="light"] table {
    background: white;
}

/my/theme/theme.js

document.documentElement.setAttribute(
    'my_theme' ,
    ( new Date().getHours() + 15 ) % 24 < 12 ? 'light' : 'dark' ,
)

Using this technique, by the way, you can implement your own Modernizr, but without 300 unnecessary checks, because only those checks that your CSS really depends on will be included in the bundle.

Lots of libraries

Usually, the entry point for building a bundle is some file. In the case of Webpack, this is JS. If you develop a lot of alienable libraries and applications, then you need a lot of bundles as well. And for each bundle you need to create a separate entry point. In the case of Parcel, the entry point is HTML, which applications will have to create anyway. But for libraries, this is somehow not very suitable.

Idea: What if any module can be build into an independent bundle without prior preparation?

Let's build the latest MAM project builder $mol_build:

mam mol/build

Now let's run this builder and have it build itself again to make sure it's still able to build itself:

node mol/build/-/node.js mol/build

Although, no, let's ask it to run the tests along with the build:

node mol/build/-/node.test.js mol/build

And if everything went well, publish the result to NPM:

npm publish mol/build/-

As you can see, when building a module, a subdirectory named - is created and all build artifacts are placed there. Let's go through the files that can be found there:

• web.dep.json - all information about dependency graph
• web.js - bundle of scripts for browsers
• web.js.map - sourcemaps for it
• web.esm.js - it is also in the form of an es-module
• web.esm.js.map - and sourcemaps for it
• web.test.js - bundle with tests
• web.test.js.map - and for sourcemap tests
• web.d.ts - bundle with types of everything that is in the script bundle
• web.css - bundle with styles
• web.css.map - and sourcemaps for it
• web.test.html - entry point to run tests for execution in the browser
• web.view.tree - declarations of all components included in the view.tree bundle
• web.locale=*.json - bundles with localized texts, each detected language has its own bundle
• package.json - allows you to immediately publish the built module to NPM
• node.dep.json - all information about dependency graph
• node.js - bundle of scripts for the node
• node.js.map - sourcemaps for it
• node.esm.js - it is also in the form of an es-module
• node.esm.js.map - and sourcemaps for it
• node.test.js - the same bundle, but also with tests
• node.test.js.map - and sourcemaps for it
• node.d.ts - bundle with types of everything in the script bundle
• node.view.tree - declarations of all components included in the view.tree bundle
• node.locale=*.json - bundles with localized texts, each detected language has its own bundle

The static is simply copied along with the paths. As an example, let's take an application that outputs its own source code. Its sources are here:

• /mol/app/quine/quine.view.tree
• /mol/app/quine/quine.view.ts
• /mol/app/quine/index.html
• /mol/app/quine/quine.locale=ru.json

Unfortunately, in general, the builder cannot know that we will need these files at runtime. But we can tell him this by putting a special file next to it:

/mol/app/quine/quine.meta.tree

deploy \/mol/app/quine/quine.view.tree
deploy \/mol/app/quine/quine.view.ts
deploy \/mol/app/quine/index.html
deploy \/mol/app/quine/quine.locale=ru.json

As a result of building /mol/app/quine, they will be copied to the following paths:

• /mol/app/quine/-/mol/app/quine/quine.view.tree
• /mol/app/quine/-/mol/app/quine/quine.view.ts
• /mol/app/quine/-/mol/app/quine/index.html
• /mol/app/quine/-/mol/app/quine/quine.locale=ru.json

Now the /mol/app/quine/- directory can be placed on any static hosting and the application will be fully functional.

Target platforms

JS can be executed both on the client and on the server. And how cool it is when you can write one code and it will work everywhere. However, sometimes the implementation of the same thing on the client and server is radically different. And I want that, for example, one implementation is used for the node, and another for the browser.

Idea: What if the purpose of the file is reflected in its name?

MAM uses a tagging system for filenames. For example, the $mol_state_arg module provides access to user-defined application settings. In the browser, these parameters are set through the address bar. And in the node - through the command line arguments. $mol_sate_arg abstracts the rest of the application from these nuances by implementing both options with a single interface, placing them in files:

• /mol/state/arg/arg.web.ts - implementation for browsers
• /mol/state/arg/arg.node.ts - node implementation

Sources not marked with these tags are included regardless of the target platform.

A similar situation is observed with tests - you want to store them next to the rest of the source, but you don't want to include them in the bundle that will go to the end user. Therefore, tests are also marked with a separate tag:

• /mol/state/arg/arg.test.ts - module tests, they will be included in the test bundle

Tags can also be parametric. For example, each module may come with texts in a variety of languages ​​and must be included in the appropriate language bundles. The text file is a regular JSON dictionary named with the locale in the name:

• /mol/app/life/life.locale=ru.json - texts for Russian language
• /mol/app/life/life.locale=jp.json - Japanese texts

Finally, what if we want to bundle files side by side, but want the builder to ignore them and not automatically include them in the bundle? It is enough to add any non-alphabetic character at the beginning of their name. For example:

• /hyoo/toys/.git - starts with a dot, so the builder will ignore this directory

Versioning

Google first released AngularJS and published it to NPM as angular. Then he created a completely new framework with a similar name - Angular and published it under the same name, but already version 2. Now these two frameworks are developing independently. Only one has API-breaking changes between major releases. And the other has between minor. And since it is impossible to put two versions of the same dependency at the same level, there can be no question of any smooth transition when two versions of the library coexist in the application for some time.

It seems that the Angular team has already stepped on all possible rakes. And here are some more: the framework code is divided into several large modules. At first they versioned them independently, but very quickly they themselves began to get confused about which versions of the modules are compatible with each other, to say nothing of ordinary developers. Therefore, Angular switched to end-to-end versioning, where the major version of the module can change even without any changes in the code. Maintaining multiple versions of multiple modules is a big challenge for both the maintainers and the ecosystem as a whole. After all, a lot of resources of all community members are spent on ensuring compatibility with already outdated modules.

The beautiful idea of ​​Semantic Versioning is shattered by the harsh reality - you never know if something will break when you change a minor version or even a patch version. Therefore, many projects fix a specific version of a dependency. However, such a fix does not affect transitive dependencies, which may be pulled by the latest version when installing from scratch, and may remain the same if they are already installed. This confusion means that you can never rely on the fixed version and you need to regularly check compatibility with up-to-date versions of (at least transitive) dependencies.

What about lock files? If you're developing a dependency-installable library, the lockfile won't help you because it will be ignored by the package manager. For the final application, the lock file will give you what is called "build reproducibility". But let's be honest. How many times do you need to build the final application from the same sources? Exactly once. Receiving an output that does not depend on any NPM, a build artifact: an executable binary, a docker container, or just an archive with everything you need to run the code. I hope you don't do npm install on prod?

Some people find the use of lock files to ensure that the CI server collects exactly what the developer committed. But wait, the developer himself can simply build it on his local machine. Moreover, he must do this to make sure that he did not break anything. Continuous Integration is not only and not so much about build, but about checking the compatibility of what one developer wrote with what someone else wrote. The concept of CI is to detect incompatibilities as soon as possible, and as a result, to start work on their elimination as early as possible.

With fixing versions, dependencies go bad very quickly, creating even more problems for you than they solve. For example, once in one company they started a project on the then current Angular@4 (or even 3). The framework was developed, but no one updated it, because "this is not included in the scope of the task" and "we did not take this into the sprint". A lot of code was written for Angular@4 and no one even knew that it was not compatible with Angular@5. When Angular@6 loomed on the horizon, the team decided to take the update of this dependency into a sprint. The new Angular required a new TypeScript and a bunch of other dependencies. We had to rewrite a lot of our own code. As a result, after 2 weeks of the sprint, it was decided to postpone the update of the framework until better times, since the business value will not create itself until the team returns the technical debt taken with, as it turned out, infernal interest.

And the icing on the cake of the versioning rake is the spontaneous appearance in the bundle of several versions of the same dependency, which you learn about only when you notice an abnormally long loading of the application, and climb to figure out why the size of your bundle has grown by 2 times. And everything turns out to be simple: one dependency requires one version of React, another requires another, and a third requires a third. As a result, as many as 3 React, 5 jQuery, 7 lodash are loaded onto the page.

Idea: What if all modules have only one version - the latest one?

We fundamentally cannot solve the problem of incompatibility with updates. But we can learn to live with it somehow. Having recognized attempts to fix versions as untenable, we can refuse to specify versions at all. Each time you install any dependency, the most up-to-date code will be downloaded. The code that is currently maintained by the maintainer. The code that all other consumers of the library now see. And all together solve problems with this library, if they suddenly arise. And not so that some have already been updated and are struggling with the problem, while others have a hut on the edge and they do not help in any way. And the help can be very different: start an issue, explain to the maintainers the importance of the problem, find a workaround, make a pull request, fork in the end if the maintainers completely scored on support. The more people experiencing the same pain at the same time, the sooner someone will be found who will eliminate this pain. It brings people together to improve a single codebase. At the same time, versioning fragments the community into a bunch of different versions in use.

Without versioning, the maintainer will get feedback from its consumers much faster and either release a hotfix or simply roll back the changes to better work them out. Knowing that a careless commit can break the build to all consumers, the maintainer will be more responsible for making changes. Well, either no one will use its libraries. And then there will be a request for a more advanced tooling. For example, this one: a dependency repository sends notifications to all dependent projects that a commit has appeared in a feature branch. They check the integration with this feature branch and if problems are found, they send details about them to the dependency repository. Thus, the library maintainer could receive feedback from consumers even before merging his feature branch into the master. Such a pipeline would be very useful for versioning too, but, as you can see, in the NPM ecosystem nothing like that is still not common. All because there is no urgent need for it. Rejection of versions forces the development of the ecosystem.

But what if you still need to break backward compatibility, but you don’t want to break the build for everyone? It's simple - create a new module. Was mobx, became mobx2 and change the API in it as you want. It would seem that this is the same versioning, but there is a fundamental difference: since these are two different modules, they can both be installed at the same time. In this case, the latest implementation of mobx can be implemented as a lightweight adapter to mobx2, which implements the old API based on it. In this way, you can smoothly transition between incompatible APIs without bloating the bundle with duplicate code.

The lack of versioning has another unexpected effect. Having found a dependency, the bundler always knows which version to install - the latest one. That is, to use a snippet from the Internet of the form:

const pages_count = $mol_wire_sync( $lib_pdfjs ).getDocument( uri ).document().numPages

You don't install the mol_wire_sync and lib_pdfjs modules, choosing the appropriate versions for this snippet:

npm install mol_wire_sync@1.0 lib_pdfjs@5.6

All you need to do is write code and all dependencies will be installed automatically on build. But how does the builder know where to get which modules? Everything is very simple - not finding the expected directory, it looks at the *.meta.tree files, where it can be indicated which directories from which repositories to take:

/.meta.tree

pack node git \https://github.com/hyoo-ru/mam_node.git
pack mol git \https://github.com/hyoo-ru/mam_mol.git
pack lib git \https://github.com/hyoo-ru/mam_lib.git

This is a fragment of the root mapping. In the same way, you can move any submodules of your module to separate repositories, giving the fractal mono-poly-repository.

Integration with NPM

MAM is a completely different ecosystem than NPM. However, trying to move code from one system to another is counterproductive. Therefore, we are working to ensure that using modules published in NPM would not be too painful.

If you need to access an already installed NPM module on the server, you can use the $node module. For example, let's find some free port and set up a static web server on it:

/my/app/app.ts

$node.portastic.find({
    min : 8080
    max: 8100
    retrieve : 1
}).then( ( ports : number[] ) => {
    $node.express().listen( ports[0] )
})

If you just need to include it in the bundle, then everything is a little more complicated. That's why the lib package has appeared containing adapters for some popular NPM libraries. For example, here is what the pdfjs-dist NPM module looks like:

/lib/pdfjs/pdfjs.ts

namespace${
    export let $lib_pdfjs : typeof import( 'pdfjs-dist' ) = require( 'pdfjs-dist/build/pdf.min.js' )
    $lib_pdfjs.disableRange = true
    $lib_pdfjs.GlobalWorkerOptions.workerSrc = '-/node_modules/pdfjs-dist/build/pdf.worker.min.js'
}

/lib/pdfjs/pdfjs.meta.tree

deploy \/node_modules/pdfjs-dist/build/pdf.worker.min.js

I hope in the future we will be able to simplify this integration, but so far so.

Developer environment

To start a new project, you often have to set up a lot of things. That is why all sorts of create-react-app and angular-cli appeared, but they hide their configs from you. You can of course eject and these configs will be moved to your project. But then it will become tightly tied to this ejected infrastructure. If you develop many libraries and applications, you would like to work with each of them in a uniform way, and make your customizations for everyone at once.

Idea: What if the infrastructure is separated from the code?

The infrastructure in the case of MAM lives in a separate repository from the code. You can have multiple projects within the same infrastructure.

The easiest way to start working with MAM is to fork the repository with the underlying MAM infrastructure, where everything is already set up:

git clone https://github.com/eigenmethod/mam.git ./mam && cd mam
npm install
npm start

The developer's server will rise on port 9080. All that remains for you is to write code in accordance with the principles of MAM.

Get your own namespace (for example - acme) and write links to your projects in it (for example - hello and home ):

/acme/acme.meta.tree

pack hello git \https://github.com/acme/hello.git
pack home git \https://github.com/acme/home.git

To build specific modules, just add the paths to them after npm start:

npm start acme/home

It is quite difficult to translate an existing project on these rails. But to start a new one - that's it. Try it, it will be difficult, but you will like it. And if you encounter difficulties, write us telegrams. And follow the news on Twitter, I have a lot more to tell you.