DEV Community

Anton Golub
Anton Golub

Posted on • Updated on

config: apply extends

#javascript #config #showdev

As we know, the forms and approaches to describing configuration files are extremely varied. I discussed this in the previous note. But there are several techniques and practices that may one day become standards.

One of this – the extends directive. Many tools provide this feature, but it works a little differently in each place. For example, tsconfig applies deep merge to compilerOptions, while prettier concatenates overrides array sections, commitlint joins plugin, etc. So the authors have to implement these nuances on site every time, and it's tiring and annoying a bit. I believe we will come up with a more efficient way to handle this routine.

Proposal 

import { populate, populateSync } from '@topoconfig/extends'

const tsconfig = await populate('tsconfig.json', {
  compilerOptions: 'merge'
})
const prettierCfg = populateSync('.prettierrc.json', {
  overrides: 'merge'
})
Enter fullscreen mode Exit fullscreen mode

20 minutes adventure

At first glance, the extends may seem like a simple combination of Object.assign and fs.readFile. Well, technically it is, but not quite so. Let's see how the real scenario unfolds. To simplify the details, we will use pseudocode.

The starting point:

input = {
  foo: 'bar',
  extends: './base.json'
}
Enter fullscreen mode Exit fullscreen mode
  • pick extends 
resource = input.extends
Enter fullscreen mode Exit fullscreen mode
  • load the referenced ./base.json 
extra = JSON.parse(fs.readFileSync(resource, 'utf8'))
Enter fullscreen mode Exit fullscreen mode
  • merge input and extra 
result = {
  ...input,
  ...extra,
  extends: undefined,
}
Enter fullscreen mode Exit fullscreen mode

How about ./base.js? Reasonable request.

  • provide js files support 
input = { extends: './base.js' }
Enter fullscreen mode Exit fullscreen mode

require hurries to the resque:

load = (ref) => isJs(ref) ? require(ref) : JSON.parse(fs.readFileSync(ref, 'utf8'))
Enter fullscreen mode Exit fullscreen mode

./base.mjs?

  • Ooops. Here the API switches to async mode to handle mjs/esm inputs. Of course, we can patch require with esm package, or load module via a process.spawnSynced nodejs. We can, but we won't 
load = isJs(ref)
  ? (await import(ref))?.default
  : fs.promises.readFile(ref, 'utf8')
Enter fullscreen mode Exit fullscreen mode

What if the value is passed without an extension, like base?

  • add support for shared npm configs
  • provide configurable resolve 
resolve = (ref, cwd) =>
  ref.startsWith('.') || ref.startsWith('/') || path.extname(ref)
    ? path.resolve(cwd, ref)
    : ref
Enter fullscreen mode Exit fullscreen mode

But what if extends is a nested object? It definitely is.

  • apply everything recursively: 
handle = input = ({
  ...input,
  ...handle(load(input.extends)),
  extends: undefined,
})
Enter fullscreen mode Exit fullscreen mode

In this case, circular references are something to pay attention to.

  • add index to avoid infinite loops 
memo = new Map()

memoizedLoad = ref => {
  if (memo.has(ref)) {
    return memo.get(ref)
  }
  memo.set(ref, load(ref))
  return load(ref)
}
Enter fullscreen mode Exit fullscreen mode

Relative resource path should use the closest parent dirname as a base to work properly.

  • resolve cwd for each extends reference 
base = path.resolve(process.cwd(), _opts.cwd ?? '.')
[cwd, _config] = typeof config === 'string'
  ? [path.resolve(base, path.dirname(config)), path.basename(config)]
  : [base, config]
Enter fullscreen mode Exit fullscreen mode

Multiple extends maybe?

  • let extends hold several references 
input = {
  foo: 'bar',
  extends: ['./extra.json', './mixin.js']
}
extras = [input.extends].flat().map(load)
Enter fullscreen mode Exit fullscreen mode

How to read toml, yaml, ini, env? Obviously, custom formats are required too.

  • provide parse option 
parse = (ref, contents) => {
  if (ref.endsWith('.yaml') || ref.endsWith('.yml')) 
      return parseYaml(contents)
  if (ref.endsWith('.json')) 
      return JSON.parse(contents)
  throw new Error(`Unsupported format: ${ref}`)
}
Enter fullscreen mode Exit fullscreen mode

How to resolve value conflicts: merge / override?

  • provide rules to handle prop-specific join strategies 
rules = {
  '*': 'merge',
  foo: 'override'
}
rule = getRule(propPath, rules)
result[key] = isObject(value) && rule === 'merge'
  ? extend({
    sources: [value],
    rules,
    prefix: p,
    index
  })
  : value
Enter fullscreen mode Exit fullscreen mode

What about arrays?

  • implement array merge as concat 
if (rule === 'merge') {
  arr.push(...sources.flat(1))
} else {
  arr.length = 0
  arr.push(...sources.slice(-1).flat(1))
}
Enter fullscreen mode Exit fullscreen mode

It would be nice to avoid mutations.

  • introduce clone 
safeLoad = compose(memoize, clone, load)
Enter fullscreen mode Exit fullscreen mode

Conclusion

Summarizing:

  • search extends statements
  • load & parse referenced resources
    • support async and sync flows
    • use cloning to avoid mutations
    • .js, .cjs, .mjs, .json
    • provide support for custom formats
    • resolve cwd for the nested resource by its closest parent resource
    • support custom resolvers
  • walk recursively
    • handle circular references
  • provide sync and async API
  • assemble the result
    • handle arrays / plain objects
    • apply prop-specific merge

Additionally:

  • build as cjs and esm
  • provide types
  • handle windows specific issues
  • add dozens of tests
  • handle different runtimes specifics (node, deno, bun)

There is an obvious benefit to implementing this algorithm on your own – it clarifies and makes controllable every aspect of config processing in place.
However, we do not always have enough time to do this exercise. Therefore, maybe someone will find this shortcut useful:

Top comments (0)

Read next

jennschiffer profile image

This month we're snug as a bug under a Glitch-powered rug

jenn schiffer -

emmabase profile image

How to Implement pagination on API Endpoint Using Nodejs and TypeScript

Emmanuel Eneche -

mjgs profile image

Latest Newsletter: Good Output (Issue #159)

Mark Smith -

ramunarasinga profile image

Tips from open-source: An Object with Map and Set.

Ramu Narasinga -