Powerful operators for effective JavaScript: "map" and "flat map" (not exclusive to arrays)

You have an array. Given that array's elements, you want to create an entirely new array, with the data being different from the original. For-loops have been historically the construct used for such a task.

But in this article, I aim to introduce to you to two operators, that, when composed, can yield highly expressive code, and potentially improve your productivity. These operators are map and flatMap.

Along with the map and flatMap operators, I aim to have you think about where data originates from, and how they are stored, and how one can use map and flatMap to derive richer data.

I also aim to show you that map and flatMap can be used with just about any types that "hold" data.

By composing these operators, you will be able to work with clean and rich data, that are distinct from the source data, and allow you to quickly re-think how your application uses it.

Synthesizing a new array

You have an array of objects, and each object represents a comment. Each comment object has a date field.

However, that field is a string, and—as the name would imply—represents the date.

// Dummy data.
//
// Array of comments.

const comments = [
  {
    content: 'This is awesome',
    date: '2019-10-12'
  },
  {
    content: 'This is rad',
    date: '2019-11-05'
  },
  {
    content: 'I like your post!',
    date: '2020-01-12'
  },
  // ...
]

Given this array, you want to generate a whole new array of objects, with the date field converted to a JavaScript Date object.

In older versions of JavaScript, before the map method was added to arrays, for-loops were useful.

It will involve initializing an empty array, iterating through the previous array, and pushing the new object into the new array.

// In a much, much older version of JavaScript, this is what people used to do.

const commentsWithDate = [];

for (let i = 0; i < comments.length; i++) {
  const currentComment = comments[i];

  commentsWithDate.push({

    ...currentComment,
    date: new Date(currentComment)

  });
}

Iterating through an array is a very common task. With a for-loop, it involves initializing a number to 0, checking that it's less than the length of the array, and incrementing it. This becomes repetitive, and possibly error prone.

Thus, the map method was added to JavaScript (eventually, iterables became a thing. It didn't only become an idiom, but an important part of JavaScript. And eventually, for-of-loop was also introduced). Replacing the above with an invocation of map would look like so:

// The following code achieves exactly the same outcome as the above for-loop
// example.
//
// The code below is using array.map for creating a new array.

const commentsWithDate = comments.map(comment => {

  // Rather than push to a new array, just return the new object, and it will
  // be appended into the new array for you.
  return {

    ...comment,
    date: new Date(comment)

  };

});

Bear in mind that the concept of map is not exclusive to arrays.

Any container types (even if the container type holds, by definition, only a single value) may have map implemented for it. More on this later.

Exercise

1. Look for code that you've written, or code on GitHub that you find that synthesize new arrays. Are they pushing to arrays for synthesis, or are they using map? If they are pushing to arrays, try to see if you can refactor it to use map.
2. Try to imagine any container types, other than arrays (hint: JavaScript's Map and Set collections are such types). Try to implement some map function for them

Joining things

Let's say you are re-implementing the ubiquitous instant messaging app, Slack.

Slack has a feature where you can view all (unread) messages, across all channels.

Let's re-implement that feature. But we'll keep it simple. We will only implement the ability to view all messages (whether read or unread), across all channels, at a glance.

This is what the array object will look like:

// Dummy data

const channels = [
  {
    tag: 'watercooler',
    messages: [
      {
        body: 'What\'s for lunch, today?',
        timestamp: '2020-03-01T01:42:17.836Z'
      },
      {
        body: 'I don'\t know. Let\'s have some tacos',
        timestamp: '2020-03-01T01:42:48.922Z'
      },
    ]
  },
  {
    tag: 'development',
    messages: [
      {
        body: 'Is anyone willing to get into a pair-programming session?',
        timestamp: '2020-03-01T01:43:09.339Z'
      },
      {
        body: 'Hit me up in five minutes. I may be able to help.',
        timestamp: '2020-03-01T01:44:00.590Z'
      },
    ]
  },
  {
    tag: 'product',
    messages: [
      {
        body: 'Does anyone have the analysis of last week\'s A/B test?',
        timestamp: '2020-03-01T02:04:41.258Z'
      },
      {
        body: 'It\'s in the Dropbox Paper document, titled "A/B test 2020-02',
        timestamp: '2020-03-01T02:04:49.269Z'
      },
    ]
  }
]

The channels variable is an array, that has objects, that each object has a field messages, which are the messages.

A solution would be to iterate through each channel, then iterate through each messages per channel.

const messages = [];

for (const channel of channels) {
  for (const message of channel.messages) {
    messages.push(message);
  }
}

If you wanted to avoid pushing to an array, one can use flatMap.

The flatMap method joins all arrays returned by the mapping function.

const messages = channels.flatMap(channel => {
  return channel.messages
});

Formality of flatMap

Given some container type (such as an array), there exists a method called flatMap, which accepts a callback. That callback accepts a value of the type that the container type holds. The callback returns another container, whos values may not be the same as the original container type.

someContainer.flatMap(theValue => {
  const somethingDifferent = transform(theValue);

  return createSomeContainer(theValue);
});

flatMap will unwrap each item in the container type, and invoke the callback with the value. The callback will then return a container, and flatMap will unwrap the value, and return an entirely new container.

Exercise

1. From the above message retrieval example, modify flatMap code so that messages have a date field that is a Date object representation of timestamp (note: just calling new Date(message.timestamp) will suffice)
2. Look for code that you've written, or code on GitHub that you find that synthesize new arrays. Are they pushing to arrays for synthesis, or are they using flatMap? If they are pushing to arrays, try to see if you can refactor it to use map.
3. Try to imagine any container types, other than arrays (hint: JavaScript's Map and Set collections are such types). Try to implement some flatMap function for them

Why not mutate the original array?

One of he most commonly touted benefits of using map and flatMap is that it avoids mutations. Many would say that mutations are major source of software faults. That's one reason.

Another reason is that although the source data has everything we need, certain aspects of our applications may require it in specific formats. And it's not just one or two aspects of our applications, but possibly dozens. Mutating the source can result in application code that is hard to manage.

Thus, you don't mutate, but synthesize. The original data remains untouched, but aspects of your code can still benefit from getting the data in the format that it expects.

Promise's then method are like map and flatMap

The then method in promises act like both map and flatMap.

Let's say you issue a REST API call to get an article. You can invoke then to gain access to the retrieved article. But, you can derive an entirely new promise, by returning an entirely new object.

Thus, we are using then like map.

getArticle().then(article => {
  return {
    ...article,
    date: newDate(article.date)
  };
}); // Returns a promise, with a comment that has a `Date` object attached.

Using then like flatMap, if you wanted to get comments from an article, you'd invoke it like so.

// Here are using `then` like flatMap.
getArticle().then(article => {
  const commentsPromise = getComments(article.id);

  // Here, we are using `then` like `map`.
  return commentsPromise.then(comments => {
    return comments.map(comment => {
      ...comment,
      date: new Date(comment.date)
    });
  });
});

Exercise

1. In the second promise example, refactor it so that the second then is no longer nested inside the callback
2. What if you wanted to not only return just the comments, but have the comments embedded in the post object. Would you be able to remove the nesting? If not, why?

Functors (those that work with map) and monads (those that work with flatMap)

A container type that works with the map function is a functor. Both arrays, and promises are examples of functors.

A container type that works with the flatMap function is a monad. Both arrays, and promises are examples of monads.

Actually, you can turn just about any container type in a functor and/or a monad.

Iterables as functors (map) and monads (flatMap)

Just as a primer, arrays are iterables. As iterables, you can splat them into arrays and function parameters, as well as iterate through them using for-of.

// Some silly array example.
const arr = [ 1, 2, 3 ];

for (const el of arr) {
  // `el` should be an element of arr
}

// Splat into an array
const newArr = [ ...arr ];

// Splat into function parameter
someFunction(...newArr)

But remember: all arrays are iterables, but not all iterables are arrays.

And hence you don't enjoy the luxuries afforded to you by JavaScript arrays.

However, they are a container type that you can extract their values from.

Iterables don't natively have any map or flatMap function defined. Fortunately, we can define them ourselves.

We will use generator function for that. Generators return iterables.

With generators, we can use the yield keyword to simplify our lives.

function * map(iterable, callback) {
  for (const value of iterable) {
    yield callback(value);
  }
}

Likewise, for flatMap.

function * flatMap(iterable, callback) {
  for (const value of iterable) {
    for (const subIterable of callback(value)) {
      yield value;
    }

    // Actually, we could have just used `yield * callback(value)`, but those
    // who are not initiated with generators, this may be a bit much to digest.
  }
}

Then, perhaps we have posts stored in something other than an array, but is iterable, we can map each value to get the date.

const commentsWithDate = map(comments, comment => {
  return {
    ...comment,
    date: new Date(comment.date)
  }
});

// Note: `commentsWithDate` is an iterable; not an array

Also with the example of getting messages from channels, we can do the following:

const messages = flatMap(channels, channel => {
  return channel.messages;
});

// the above variable will now be a flat iterable of messages; not channels

And, if we are to take the above example, and have it so that the messages have a date field:

const messages = flatMap(channels, channel => {
  return map(channel.messages, message => {
    return { ...message, date: new Date(message.timestamp) };
  });
});

// the above variable will now be a flat iterable of messages; not channels

A helper library: IxJS

Above, I have introduced map and flatMap for iterables.

One problem with the above, however, is it requires that we pass in the iterable as a first parameter. Composing map and flatMap results in nesting, and renders it rather difficult to follow logically which operations are happening in what order.

IxJS introduces an iterator object that exposes a pipe method. This way, you are able to compose map and flatMap.

Here's what the above code would look like with IxJS

const results = from(channels).pipe(
  flatMap(channel => channel.messages),
  map(message => ({ ...message, date: new Date(message.timestamp) }))
);

Other examples: RxJS and observables

Eventing primer

If you wanted to listen in on user mouse clicks, you can attach an event listener (in the form of a callback function) to the mouse click event.

Below is an example of listening to a click events on a button.

button.addEventListener('click', event => {
  alert(`Button clicked at coordinate (${event.screenX}, ${event.screenY}).`);
});

The paradigm applied in the application programming interface (API) above is that the button itself is an event emitter. It's an event emitter because it exposes a method named addEventListener, and you attach an event listener, provided some event tag (in this case, the 'click' event).

An alternative paradigm that browsers could have opted for instead, is that the button holds, as a property, an object that represents an event emitter (also referred to as an event stream).

So here's what the API could have looked like, had browsers instead opted for the paradigm that I am proposing:

button.clickStream.subscribe(event => {
  alert(`Button clicked at coordinate (${event.screenX}, ${event.screenY}).`);
});

In the last example, clickStream is an object. And since it's an object that has the single responsibility of notifying click events, we can grab a hold of it, and apply all sorts of operators of our choosing.

Of course, in the context of this post, it's a perfect candidate for map and flatMap.

Observables as functors (map) and monads (flatMap)

Earlier, I mentioned that you can think of promises as a container type. But if you are familiar with them, they are what are often returned after some asynchronous call (such as AJAX request, file read, etc.).

Nevertheless, it helps to think of them as container types; they asynchronously "hold" a value, that is exposed through callbacks. This is related to how then can act both like map and flatMap.

RxJS introduces a notion called "observables". Observables differ from promises in that promises represent a single instance of a value, where as observables represent a stream of values.

Like promises, we can treat observables like container types.

RxJS observables have a pipe method, that you can apply the map and flatMap functions to.

Chat application notification example using RxJS observables

Throughout this post, I made repeated references to instant messaging (IM) applications. This is because IM apps are very event-driven.

The two events that we will concern ourselves with is

• when a contact logs in
• when a contact sends a direct message

For our IM app, we will have a server. We interface with it via a REST or GraphQL API (detail doesn't matter), as well as WebSocket for streams of data. It's through WebSocket that we will be subscribing to events.

Through WebSocket, our server supports subscribing to these two events, for now:

• users coming online
• which user sent us a message (note: when the user logs out, the subscription closes)

First, we want to listen in on events of a user logging in.

Below is a simple listener to our server for that very event.

const socket = new WebSocket(`${endpoint}/log-ins`);

socket.on('message', (data) => {
  // Do whatever, here.
});

For the purposes of this article, I want to make use of RxJS, as much as possible.

With RxJS, we can convert an event emitter into an observable.

Let's convert the above socket message event into an observable.

import { fromEvent } from 'rxjs';

const socketMessageStream = fromEvent(socket, 'message');

Next, we will filter for only valid JSON messages, and convert them to valid JavaScript objects.

We will peruse the RxJS filter operator.

The purpose of the filter operator is to generate a new stream, for all events that only test true, according to a callback (that callback has a fancy term, and it's called a predicate).

Let's create an RxJS operator that will filter only for valid JSON, and transform them into objects.

/**
 * Operator for filtering out invalid JSON, and converting the messages to
 * objects.
 */
function onlyValidObjects(source) {

  return source.pipe(

    // Get all valid messages that can be converted to an object.
    filter(message => {
      try {
        JSON.parse(message.toString('utf8'));
      } catch (_) {
        return false;
      }
    }),

    // Convert those messages to 
    map(message => JSON.parse(message.toString('utf8')))

  );

}

Next create a stream of only valid events.

import { filter, map } from 'rxjs/operators';

// All messages are valid objects.
const validEvents = socketMessageStream.pipe(onlyValidObjects);

Next, we filter exclusively for messages that are login events.

import { filter } from 'rxjs/operators';

// loginStream will only have events that will exclusively hold log-in events.
const loginStream = socketMessageStream.pipe(
  filter(message => {
    return message.type === 'login';
  })
);

Finally, from the login stream, get a new stream of all new messages from all logged-in users.

For every login, this requires listening to new messages, from every user that logs in.

Listening to new messages from every new login can be their own event stream. This results in "nested" event listeners.

This is now a call for flatMap. The following demonstrates the use of flatMap to flatten out the messages stream.

import { flatMap } from 'rxjs/operators';

const messageStream = loginStream.pipe(
  flatMap(user => {

    const instantMessages = new WebSocket(`${endpoint}/messages/${user.id}`);
    return fromEvent(instantMessage, 'message').pipe(onlyValidObjects);

    // Note: we don't have to worry about logging out, since `instantMessages`
    // closes the moment the user logs out.

  })
);

Now, with messagesStream, we are free to interpret the stream however we wish.

The first of which is to send a push notification to the browser.

messageStream.subscribe(message => {
  notify(`${message.user.name} sent ${message.body}`);
});

// You can implement the `notify` function by following the "Example" section on
// MDN.
//
// https://developer.mozilla.org/en-US/docs/Web/API/notification

If we're using React, we should be able to add a counter.

Below is a React component that subscribes to the message stream, and for every message, increment the counter.

import React, { useState, useEffect } from 'react';

/**
 * A counter that increments on every new message.
 */
function Counter() {

  const [ count, setCounter ] = useState(0);

  useEffect(() => {
    const subscription = messageStream
      .subscribe(() => { setCounter(count + 1); });

    return () => {
      subscription.unsubscribe();
    };
  }, []);

  return (
    <div>
      Notifications {count}
    </div>
  );

}

Exercise

1. As is evidenced by filter, the map and flatMap operators aren't the only RxJS operators. Consider looking into using other operators from RxJS, and see what you can eventually build (hint: consider the scan operator)
2. Just like RxJS observables, map and flatMap aren't the only operators for arrays. Look at the MDN documentation on arrays, and explore all the other operators available.

Takeaways

• rather than relying on loops and actively mutating data, map and flatMap can be composed to synthesize new data
• these operators allow you to iterate quickly by limiting yout work mostly from synthesized clean but rich data
• map and flatMap can be applied to more than just arrays. As long as there exists some data holder, you can extract it, and apply operators to them, and derive new data
• streams are such examples of data holders (even if data is not yet available). RxJS re-imagines how events should be handled. Thus, as demonstrated above, map and flatMap were applied to

In Closing

With map and flatMap, you are given two very powerful operators to work with data, without mutating them.

Hopefully, you will now be able to quickly iterate on new features for your JavaScript-related products (and even in other environments other than JavaScript).