DEV Community

Cover image for What is the best solution for removing duplicate Objects from an Array?
Raffaele Pizzari
Raffaele Pizzari

Posted on • Updated on

What is the best solution for removing duplicate Objects from an Array?

(check out my blog)

There are many solutions to this problem but I wouldn't say that one is better than the others.

In this article I will just go through 2 approaches:

  1. Using the standard built-in object "Set"
  2. Using the method "reduce()" of Array ( Array.prototype.reduce() )

Set

From MDN Web Docs:

Set objects are collections of values. You can iterate through the
elements of a set in insertion order. A value in the Set may only
occur once
; it is unique in the Set's collection.

Equality comparison

It looks like that Set it exactly the tool we need, but what does it mean "may only occur once"?

According to the documentation, Set uses the SameValueZero algorithm for the value-comparison operations, which means that it can determine whether two values are functionally identical in all contexts (0 and -0 are considered equal).

In other words, it's very similar to "===" (strict equality) with one exception: comparing NaN with NaN would return a truthy value.

Basic use case

Let's assume that we have this array:

const myArr = ['a', 'b', 'c', 'b', 'b', 'd'];
Enter fullscreen mode Exit fullscreen mode

and we want to remove the duplicates.

Since the Set() constructor accepts an iterable as parameter (new Set([iterable])) and returns a new Set object, we can do the following:

    const mySet = new Set(myArr); 
Enter fullscreen mode Exit fullscreen mode

mySet is now an instance of Set containing the following values:

'a', 'b', 'c', 'd'
Enter fullscreen mode Exit fullscreen mode

Since the expected result we were looking for is an Array, we still have to convert the Set back to an Array.
We can easily perform this task spreading (...) the mySet values in a new Array:

const uniqValuesArray = [...mySet]; // [ 'a', 'b', 'c', 'd']
Enter fullscreen mode Exit fullscreen mode

That's it :)

Complex object use case

The original question was: What is the best solution for removing duplicate objects from an array?
In the previous example we just used some used some string values.

Let's try to use this Array of Objects:

   let myArr = [
       {id: 1, name: 'Jack'},
       {id: 2, name: 'Frank'},
       {id: 1, name: 'Jack'},
       {id: 3, name: 'Chloe'}
    ];
Enter fullscreen mode Exit fullscreen mode

We could try to use the same approach and create a new Set(myArr) from myArr, but in this case the comparison algorithm will consider every element of myArray unique, since the "SameValueZero algorithm" doesn't perform a deep object comparison:

    {id: 1, name: 'Jack'} === {id: 1, name: 'Jack'} // false
Enter fullscreen mode Exit fullscreen mode

But what if we "prepare" our data and transform the object into something that fits better with the algorithm?

Let's create a new Array and fill it with the JSON-serialized version of the Objects:

    let myArrSerialized = myArr.map(e => JSON.stringify(e));
Enter fullscreen mode Exit fullscreen mode

So we will have:

    ["{\"id\":1,\"name\":\"Jack\"}",  "{\"id\":2,\"name\":\"Frank\"}",  "{\"id\":1,\"name\":\"Jack\"}",  "{\"id\":3,\"name\":\"Chloe\"}"]
Enter fullscreen mode Exit fullscreen mode

Where:

    "{\"id\":1,\"name\":\"Jack\"}" === "{\"id\":1,\"name\":\"Jack\"}" // true
Enter fullscreen mode Exit fullscreen mode

Great. Now we have an array of values that fit with our purpose and the default Set's comparison algorithm.

Now we can continue as we did in the previous example:

    const mySetSerialized = new Set(myArrSerialized);

    const myUniqueArrSerialized = [...MySetSerialized];
Enter fullscreen mode Exit fullscreen mode

But we need a new step at the end: we have to transform the serialized objects back to Objects:

    const myUniqueArr = myUniqueArrSerialized.map(e => JSON.parse(e));
Enter fullscreen mode Exit fullscreen mode

That's it again :)

Summarising in a function

    const removeDuplicatesFromArray = (arr) => [...new Set(
      arr.map(el => JSON.stringify(el))
    )].map(e => JSON.parse(e));
Enter fullscreen mode Exit fullscreen mode

Array.prototype.reduce()

The "reduce()" approach is also a good practice.
In the following example we consider "duplicates" two objects that share the same value of a specific key.

Let's work with this Array:

    let myArr = [
       {id: 1, name: 'Jack'},
       {id: 2, name: 'Frank'},
       {id: 3, name: 'Jack'},
       {id: 4, name: 'Chloe'}
    ];
Enter fullscreen mode Exit fullscreen mode

The value Object {id: 1, name: 'Jack'} and {id: 3, name: 'Jack'} have different ids but the same name's value. That's why we consider them duplicates and we want to keep just the first of them.

Reducer

How Array.prototype.reduce() works isn't part of this post. If you don't know it, I recommend you to have a look to the documentation

This will be the reducer:

    const reducer = (accumulator, currentValue) => {
      if(!accumulator.find(obj => obj.name === currentValue.name)){
        accumulator.push(currentValue);
      }
      return accumulator;
    };
Enter fullscreen mode Exit fullscreen mode

Basically we perform one simple check:

    !accumulator.find(obj => obj.name === currentValue.name)
Enter fullscreen mode Exit fullscreen mode

We iterate over the given array.
Then, element by element, we check if we have already pushed in the accumulatoran Object with the same value of the name property.
If no element matches the condition we push the current element in the accumulator otherwise we just skip the step.

So we just have to apply the reducer we just created to the Array and initialize the accumulator with an empty array:

    myArr.reduce(reducer, []));
Enter fullscreen mode Exit fullscreen mode

Summarising in a function

    const removeDuplicatesFromArrayByProperty = (arr, prop) => arr.reduce((accumulator, currentValue) => {
      if(!accumulator.find(obj => obj[prop] === currentValue[prop])){
        accumulator.push(currentValue);
      }
      return accumulator;
    }, [])

    console.log(removeDuplicatesFromArrayByProperty(myArr, 'name'));
Enter fullscreen mode Exit fullscreen mode

Let's combine both these approaches

As Andrea Giammarchi (ehy, thank you!) pointed out, it's even possible to combine both solutions!
A premise is needed.
As a second paramater, Array.prototype.filter() accepts the value to use as this when executing callback.

let newArray = arr.filter(callback(element[, index[, array]])[, thisArg])
Enter fullscreen mode Exit fullscreen mode

Now we can explore the new solution:

const by = property => function (object) { 
  const value = object[property]; 
  return !(this.has(value) || !this.add(value));
};

const myFitleredArr = myArr.filter(by('name'), new Set);
Enter fullscreen mode Exit fullscreen mode

Let's read it line by line:

const by = property => function (object) { 
Enter fullscreen mode Exit fullscreen mode

This is test function we'll pass to a filter() method in order to test/filter each element of the array.

 const value = object[property];
Enter fullscreen mode Exit fullscreen mode

Assign to "value" the value of the given object's property.

 return !(this.has(value) || !this.add(value));
Enter fullscreen mode Exit fullscreen mode

Return true to keep the element, false otherwise.
Remember that "this" in our example will be "new Set".
If the Set doesn't already have the given value it will return true and add the value to the collection.
If the Set already has the given value, then it won't keep the element.

In this example is possible to reuse the given Set, the one we pass as second parameter to the method filter().

If you don't need to reuse it you can create a new Set everytime:

const by = property => {
  const set = new Set;
  return obj => !(set.has(obj[property]) || !set.add(obj[property]));
};
Enter fullscreen mode Exit fullscreen mode

About this post

I'm am running a free JavaScript Learning Group on [pixari.slack.com] and I use this blog as official blog of the community.
I pick some of the questions from the #questions-answer channel and answer via blog post. This way my answers will stay indefinitely visible for everyone."

If you want to join the community feel free to contact me:

Top comments (8)

Collapse
 
webreflection profile image
Andrea Giammarchi • Edited

You could also combine both solutions.

const by = property => function (object) {
  const value = object[property];
  return !(this.has(value) || !this.add(value));
};

const myFitleredArr = myArr.filter(by('name'), new Set);

This way you could reuse over and over the given Set, but you could also make it a one-shot each time.

const by = key => {
  const set = new Set;
  return obj => !(set.has(obj[key]) || !set.add(obj[key]));
};
Collapse
 
pixari profile image
Raffaele Pizzari

This is a very good hint.
I'll edit soon the post adding this solution with some explanations.

Thank you, Andrea!

Collapse
 
pixari profile image
Raffaele Pizzari

Done :)

Collapse
 
webreflection profile image
Andrea Giammarchi

Thanks, but there is a typo apptets instead of accepts, and I think you should explain the one-off approach too, where you don't need to pass any Set.

Passing new Set is not super handy if you never reuse that Set, so that in such case not passing a second argument to filter(...) provides a better UX.

Thread Thread
 
pixari profile image
Raffaele Pizzari

Thanks again, Andrea!

Collapse
 
vargavince91 profile image
Vince Varga

Hey Raffaele, Great post!

As this post is quite detailed and the main audience is people who learn JavaScript, I just wanted to mention about the JSON method that, for JSON. stringify() , the order matters.

JSON.stringify({id: 1, name: 'Jack'}) === JSON.stringify({id: 1, name: 'Jack'})
true
JSON.stringify({id: 1, name: 'Jack'}) === JSON.stringify({name: 'Jack', id: 1})
false

Though I'm sure you know it, it might be relevant to readers (or on the other hand, it might be too much information and out of scope for the article :))

Collapse
 
pixari profile image
Raffaele Pizzari

Thank you for pointing this out!

I'm sure it would be pretty useful for developers who are learning JS and aren't very confident with string comparison.

Thank you again :)

Collapse
 
aditprayogo profile image
Aditiya Prayogo

You could also use array.find

const duplicateNote = notes.find((note) => note.title === title)