DEV Community

Cover image for JavaScript: How to Remove Duplicate Values from Arrays

JavaScript: How to Remove Duplicate Values from Arrays

Will Harris on July 03, 2020

Originally posted on Will's blog In a previous post we saw how to determine whether a JavaScript array contains duplicate values. Today, I want...
Collapse
 
miketalbot profile image
Mike Talbot ⭐ • Edited

Surely Set must be faster - the whole thing is an O(N) operation with Set, but it's getting towards O(N**2) with anything that has to scan like includes or filter... (The reduce version is better than O(N**2) as it is only having to test the currently created unique list)

I decided to see just how different it is. In this example on jsperf, you can see that the "reduce" is around 180x slower and the indexOf even worse at almost 380x slower. Also the version of "reduce" I've used is slightly faster than your one shown, because it doesn't make an array on each loop and just modifies the one being created.

Collapse
 
will_devs profile image
Will Harris

Thanks for throwing together that jsperf test Mike!

Set does appear to be much faster, which is fantastic because the syntax is so much easier to grok (for me, at least).

Collapse
 
miketalbot profile image
Mike Talbot ⭐

Yes, nice that good syntax has great performance haha! Not a normal happy thing that I find! (Like the immutability is a nicer syntax but comes with a penalty).

Collapse
 
evanlk profile image
Evan Kennedy • Edited

The approach below is O(n) and should work without ES6. The issue with some of the approaches above is they are using Array.indexOf or Array.includes, which is basically another loop through the array. You basically take a performance hit when the searched values are located near the end of the array. The only downside to the method below is that it's memory intensive for large arrays. You're essentially giving up memory for performance.

const originalArray = [1, 2, 3, 4, 1, 2, 3, 4];

let key = {};

const uniqueArray = originalArray.filter((item) => {
  if (key[item]) return false;

  key[item] = true;
  return true;
});
Collapse
 
willsmart profile image
willsmart • Edited

A similar function using Set is

function withoutDuplicates(inputArray) {
  const trackerSet=new Set();
  return inputArray.filter(
    v => !trackerSet.has(v) && trackerSet.add(v)
  ) // add returns the Set instance, which is truthy. You may wish to add a more explicit truthy return
}

These have the main advantage over Array.from(new Set(inputArray)) in that there's no need to iterate the Set after filling it with values, but I've done a little profiling and my snippet has pretty much the same characteristics and timing as Array.from(new Set(inputArray)) (just a little bit slower), even for very large arrays.

tbh, I think the big win is that they aren't leaning on JS Set's latent insertion-order stability, making the code more WYSIWYG, and more portable since in many other C-style languages Sets don't have that guarantee.

Collapse
 
will_devs profile image
Will Harris

Thanks Evan! That's also a great way to approach the problem.

Collapse
 
jaymcconnon profile image
James McConnon

Awesome article. First i am seeing sets. Also glad to read comment below that its runtime is better than expected because to me it is more readable. Also i think allot of the array prototype functions can make the code harder to read in the long run. Assuming you know a bit of background on sets, that last example is so concise!

That being said, could you explain how the ... spread operator works with it? Trying to get my head around that at the moment.

Collapse
 
will_devs profile image
Will Harris

Thanks James!

Because the Set object implements the iterator property, we can use the ES6 spread syntax to cast the values from the Set into an array.

It really is doing the same thing as the second part of that example, where I called Array.from().

Collapse
 
jaymcconnon profile image
James McConnon

Thankyou that makes sense. Spread operator has been a tough one to learn for me. I always feel like i kind if have an idea of what it is doing in one situation but struggle to use it myself.

Collapse
 
savagepixie profile image
SavagePixie

Sets don't come with the same prototype functions as arrays. They have a forEach method, but no map or filter, for example (I think). The spread operator and square braces there are simply extracting the values from the set and turning them into an array.

Collapse
 
felipemfp profile image
Felipe Pontes

Set works great when you are dealing with primitive values. But if you are working with some list of items (e.g. list of comment authors in this post), you're better off using a key mapping approach:

const mapping = {}
const unique = array.filter((item) => {
  if (item.id in mapping) return false
  mapping[item.id] = true
  return true
})
Collapse
 
killshot13 profile image
Michael R.

Is there any way to count how many times the duplicate object(s) appears in the array before eliminating said duplicates using this method? Asking for a noob trying to solve an annoying kata. :)

Reference link

Collapse
 
miggu profile image
miggu

What is quadratic time?

Collapse
 
ky1e_s profile image
Kyle Stephens
Collapse
 
jesuscovam profile image
jesuscovam

too much coolness in one function

Collapse
 
__bangash profile image
shahid
Collapse
 
caoshouse profile image
caoshouse • Edited

let originalArray = [1, 2, 3, 4, 1, 2, 3, 4];
for(var i=0,obj={};i<originalArray.length;i++){
obj[originalArray[i]]=null
}
let noDuplicates=Object.keys(obj)

Collapse
 
4umfreak profile image
Mark Voorberg • Edited

Same using .reduce()...

let myArray = [1,2,2,"hello world",4,5,5, {moo: "cow"}];
let myObject = myArray.reduce((acc, x) => {acc[x]=0; return acc}, {});
let noDuplicates = Object.keys(myObject);