DEV Community

loading...
Cover image for From Callback Hell to Callback Heaven

From Callback Hell to Callback Heaven

Eugene Ghanizadeh
Mainly programmer, (sometimes) designer, (sometimes) product manager, (used to be) HR-ish guy, (used to be) a teacher.
・7 min read

Remember the time when a lot of JavaScript code looked like this?

router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req), (err, file) => {
    if (err) res.status(500).send()
    else {
      parse(file, (err, parsed) => {
        if (err) res.status(500).send()
        else db.insert(parsed, err => {
          if (err) res.status(500).send()
          else res.status(200).send()
        })
      })
    }
  })
})
Enter fullscreen mode Exit fullscreen mode

For those fortunate enough to not remember those days, this was called a callback hell, for obvious reasons. Fortunately, we have moved on, and these days equivalent code most likely looks something like this:

router.put('/some-url', async (req, res) => {
  try {
    const file = await fs.readFile(filePath(req));
    const value = await parse(file);
    await db.insert(value);
    response.status(200).send();
  } catch {
    response.status(500).send();
  }
})
Enter fullscreen mode Exit fullscreen mode

Of course now that we have async/await and Promises, it is easy to blame the callback hell era on lack of syntactic features of JS at the time and move on. But I do think there is value in reflecting back, analyzing the core issue, how it was solved, and what's to learn from it all.


The Issue

Lets look back at the overall structure of the hellish example above:

doX(args, (err, res) => {
  if (err) { ... }
  else {
    doY(args, (err, res) => {
      if (err) { ... }
      ...
    })
  }
})
Enter fullscreen mode Exit fullscreen mode

A glaring issue here is that most of whats on the screen is dedicated to not-really-important stuff:

doX(args /*, (err, res) => {
  if (err) { ... }
  else {*/
    doY(args /*, (err, res) => {
      if (err) { ... } */
      ...
    /*}*/)
  /*}*/
/*}*/)
Enter fullscreen mode Exit fullscreen mode

For comparison, this would be the equivalent structure for the modern not-hellish version:

/* try { */
  /*await*/ doX(args)
  /*await*/ doY(args)
  ...
/*} catch { }*/
Enter fullscreen mode Exit fullscreen mode

The commented bits in both versions are indicative of the same things: doX() and doY() are async functions, and also there might be some errors. In the hellish version though, you need to spend much more space for these side notes, which results in a far less readable code.

👉 Note that we could trim the boilerplate and restructure the code into a more readable format without additional syntax too. Historically speaking, that IS what happened, in the form of Promise libraries (which then got standardized and a bit of more love with some syntactic support):

doX(args)
.then(() => doY(args))
.then(() => ...)
.catch(() => { ... })
Enter fullscreen mode Exit fullscreen mode
doX(args)
/*.then(() =>*/doY(args)/*)*/
/*.then(() =>*/.../*)*/
/*.catch(() => { ... })*/
Enter fullscreen mode Exit fullscreen mode

An important difference between this code and the hellish code is that in the hellish code, important stuff and boilerplate stuff are extremely intertwined, while with promise libraries, they are neatly separated, making the code easier to read even in cases where amount of boilerplate is almost the same:

// without promises:
doX(args/*, (err, res) => { ... }*/)
Enter fullscreen mode Exit fullscreen mode
// with promises:
doX(args)/*.then(() => { ... })*/
Enter fullscreen mode Exit fullscreen mode
// with async/await:
/*await*/ doX(args)
Enter fullscreen mode Exit fullscreen mode

Promises also provide other important features that help with async programming ergonomics, most importantly:

  • Promises are automatically flattened when chained.
  • Promises are shared.

However, I think these properties, although beneficial, aren't as important as the aforementioned separation. To illustrate that, lets create an experimental promise library that just does the separation and nothing else, and see how it fares.


The Experiment

So initially, we started with functions looking like this:

doX(args, (err, res) => {...})
Enter fullscreen mode Exit fullscreen mode

The callback here is the main boilerplate (and namesake of our hell), so easiest separation is to take it out of argument list of doX(), and put it in a deferred function instead:

doX(args)((err, res) => {...})
Enter fullscreen mode Exit fullscreen mode

☝️ This is basically a change in how doX is implemented, from this:

function doX(args, callback) {
  // do stuff
  // maybe do more
  callback(undefined, 42)
}
Enter fullscreen mode Exit fullscreen mode

To this:

function doX(args) {
  // do stuff
  return callback => {
    // maybe do more
    callback(undefined, 42)
  }
}
Enter fullscreen mode Exit fullscreen mode

In other words, we just changed the convention from:

accept a callback as last argument

to:

return a function that accepts a callback as argument

Our separation convention seems to not have helped much on its own, as we still have the same amount of boilerplate. However, it did open the door for simplistic utilities that help us take away the boilerplate. To see that, let me first introduce the pipe() utility:

function pipe(...cbs) {
  let res = cbs[0];
  for (let i = 1; i < cbs.length; i++) res = cbs[i](res);
  return res;
}
Enter fullscreen mode Exit fullscreen mode

Simply put, this:

pipe(a, b, c, d)
Enter fullscreen mode Exit fullscreen mode

equals to this:

let x = a
x = b(x)
x = c(x)
x = d(x)
Enter fullscreen mode Exit fullscreen mode

In the not so distant future, pipe() might even get incorporated into JavaScript itself, which would look like this:

a |> b |> c |> d
Enter fullscreen mode Exit fullscreen mode

Anyways, the pipe() operator allows us to neatly transform the function returned by the (new convention) doX() (which remember, is a function accepting a standard callback), without having to write callbacks manually. For example, I could create a then() utility as follows:

export function then(f) {
  return src => {
    src((err, res) => {
      if (!err) f(res)
    })

    return src
  }
}
Enter fullscreen mode Exit fullscreen mode

and with these utilities, my async code would transform from this:

doX(args)((err, res) => { ... })
Enter fullscreen mode Exit fullscreen mode

to this:

pipe(
  doX(args),
  then(() => { ... })
)
Enter fullscreen mode Exit fullscreen mode

or better yet (with pipeline operator incorporated):

doX(args) |> then(() => { ... })
Enter fullscreen mode Exit fullscreen mode

which looks a lot like a standard promise library:

doX(args).then(() => { ... })
Enter fullscreen mode Exit fullscreen mode

I could also create a simple catch() utility:

function catch(f) {
  return src => {
    src((err) => {
      if (err) f(err)
    })

    return src
  }
}
Enter fullscreen mode Exit fullscreen mode

Which would give me async codes like this:

doX(args)
|> then(() => doY(args))
|> then(() => ...)
|> catch(() => { ... })
Enter fullscreen mode Exit fullscreen mode
doX(args)
/*|> then(() =>*/ doY(args) /*)*/
/*|> then(() =>*/ ... /*)*/
/*|> catch(() => { ... })*/
Enter fullscreen mode Exit fullscreen mode

Which is as succinct as promise libraries, with almost no effort. Better yet, this method gives us extensibility as well, as we are not bound to a set Promise object and can create / use a much wider range of utility functions:

function map(f) {
  return src => cb => src((err, res) => {
    if (err) cb(err, undefined)
    else cb(undefined, f(res))
  })
}
Enter fullscreen mode Exit fullscreen mode
function delay(n) {
  return src => cb => src((err, res) => {
    if (err) cb(err, undefined)
    else setTimeout(() => cb(undefined, res), n)
  })
}
Enter fullscreen mode Exit fullscreen mode

and start getting a bit wild:

doX(args)
|> then(() => doY(args))
|> map(yRes => yRes * 2)
|> delay(200)
|> then(console.log)
Enter fullscreen mode Exit fullscreen mode

Real-life Examples

Ok so it seems like a simple change of convention allowed us to create utilities and libraries that provide the same convenience provided by promise libraries (and almost similar to async/await syntax). To get a better perspective, let's look at real-life examples. For this purpose (and mostly out of curiosity), I have created an online playground with an implementation of our experimental lib.

First off, let's take a look at our original example, which looked like this in its most hellish version:

router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req), (err, file) => {
    if (err) res.status(500).send()
    else {
      parse(file, (err, parsed) => {
        if (err) res.status(500).send()
        else db.insert(parsed, err => {
          if (err) res.status(500).send()
          else res.status(200).send()
        })
      })
    }
  })
})
Enter fullscreen mode Exit fullscreen mode

This is how the modern JavaScript version looks like:

router.put('/some-url', async (req, res) => {
  try {
    const file = await fs.readFile(filePath(req));
    const value = await parse(file);
    await db.insert(value);
    response.status(200).send();
  } catch {
    response.status(500).send();
  }
})
Enter fullscreen mode Exit fullscreen mode

And this is how our new callback convention code looks like:

router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req))
  |> map(parse)
  |> flatten
  |> map(db.insert)
  |> flatten
  |> then(() => res.status(200).send())
  |> catch(() => res.status(500).send())
})
Enter fullscreen mode Exit fullscreen mode

The convention gets us pretty close to the convenience of async/await. There is a little nuance though: see the flatten utility used twice in the middle? That is because unlike promises, our callbacks are not flattened while chaining. We assumed that parse() is also async, i.e. it also returns a promise-ish. map(parse) then maps the result of readFile() to a new promise-ish, which should be flattened to resolved values before being passed to db.insert(). In the async/await code, this is done by the await keyword before parse(), and here we need to do it with the flatten utility.

P.s., the flatten() utility is also pretty simplistic in nature:

function flatten(src) {
  return cb => src((err, res) => {
    if (err) cb(err, undefined)
    else res((err, res) => {
      if (err) cb(err, undefined)
      else cb(undefined, res)
    })
  })
}
Enter fullscreen mode Exit fullscreen mode

Let's also take a look at another example: here, we want to fetch some Pokémon info from PokéAPI and log its abilities:

fetch('https://pokeapi.co/api/v2/pokemon/ditto')
|> map(res => res.json())
|> flatten
|> then(res => console.log(res.abilities))
Enter fullscreen mode Exit fullscreen mode
async(() => {
  let res = await fetch('https://pokeapi.co/api/v2/pokemon/ditto')
  res = await res.json()
  console.log(res.abilities)
})()
Enter fullscreen mode Exit fullscreen mode

Conclusion

So to recap, these seem to have been the main issues resulting in callback hells:

  • Lots of boilerplate code
  • Boilerplate code severely intertwined with important code

As per our little experiment, addressing the second issue in the simplest manner (just separating boilerplate code and important code with no other change) was pretty key: it allowed us to bundle boilerplate code into small utility functions and reduce the ratio of boilerplate code and important code, making it (almost) as convenience as a heavy-handed solution such as adding new syntax to the language itself.

This notion is particularly important: you might have ugly implementation details and boilerplate that you cannot get rid of, but you can always bundle it together and separate it from actual important code, and doing this even in the simplest manner opens the door for turning a hellish situation into a heavenly one.

It is also notable that the same methodology is applicable to other, similar problems we are facing today. While we have (mostly) solved the issues of asynchronous functions, newer constructs such as asynchronous streams (which are like async functions but with many, possibly infinite outputs instead of one) keep creeping into our toolbox and demand similar problem solving.

P.s. the name callback heaven actually comes from the callbag specification, which is like our new callback convention but for streams instead of promises. If you enjoyed this post, be sure to check it out as well.

Discussion (6)

Collapse
zaboco profile image
Bogdan Zaharia

Really nice idea. I guess that, since the map |> flatten pattern is pretty common, a flatMap would come in handy.
Also, somehow related is this article series. The link is to the last post, which links to the first two.

Collapse
loreanvictor profile image
Eugene Ghanizadeh Author

yes, I think thats what actual libraries like RxJS typically have, and for that particular reason (though I do like the explicitness of flatten operator in callbags for example).

Collapse
mathias54 profile image
Mathias Gheno Azzolini

This article is amazing. Thank you soo much.

Collapse
cbejensen profile image
Christian Jensen

By the end, the code looks very similar to RxJS code

Collapse
loreanvictor profile image
Eugene Ghanizadeh Author

Yeah that is kind of the intention here.

Collapse
bryantobing12 profile image
Bryan Lumbantobing

leave a trace here

Forem Open with the Forem app