DEV Community

Cover image for 1 line of code: How to remove all duplicates from an Array
martin krause
martin krause

Posted on • Updated on

1 line of code: How to remove all duplicates from an Array

const removeDuplicates = arr => [...new Set(arr)]; 
Enter fullscreen mode Exit fullscreen mode

Assumes that the given argument is an Array and removes duplicate entries, keep in mind that it works only for entries with primitive values (string, number, bigint, boolean, undefined, symbol, and null). Preserves the order of the entries and returns a copy of the array.


The repository & npm package

You can find the all the utility functions from this series at github.com/martinkr/onelinecode
The library is also published to npm as @onelinecode for your convenience.

The code and the npm package will be updated every time I publish a new article.


Follow me on Twitter: @martinkr and consider to buy me a coffee

Photo by zoo_monkey on Unsplash


Discussion (11)

Collapse
michaelcurrin profile image
Michael

Worth explaining this separately when used with array instead of sets

const x = ["abc", "def"]
const y = [...x]
// ["abc", "def"]
Enter fullscreen mode Exit fullscreen mode

And modifying x and y won't affect the other.

Then bringing in set

const x = ["abc", "def", "abc"]
const y = new Set(x)
// Set(["abc", "def"])
const z = [...y]
// ["abc", "def"]
Enter fullscreen mode Exit fullscreen mode

And the the original post does that in one line in a function

Collapse
michaelcurrin profile image
Michael

Also worth adding that you might want to stop once you have a set and not convert it back to an array.

Why?

Lookup time from a set is constant to check if a value is in a set while in an array of 1000 elements worst case you have to go through 1000 elements until you find the one at the end.

Secondly when you convert from array to set, the order will be lost. Sets are inherently unordered. So when you convert from set to array you will not get the order of the original array. So you might as well keep the set.

Collapse
qm3ster profile image
Mihail Malo

It's actually specced to preserve insertion order: developer.mozilla.org/en-US/docs/W...

(...spread operator calls y[Symbol.iterator], which is specced to be the same as what is returned by .values() on Set)

Thread Thread
michaelcurrin profile image
Michael

Oh interesting, in Python it is unordered.

Collapse
martinkr profile image
martin krause Author

Thank you for the explanation.

Collapse
captainyossarian profile image
yossarian

I think it worth saying that it works only on primitives

Collapse
jfbrennan profile image
Jordan Brennan

What do you mean by "primitives"? Set will take any iterable, including custom iterables.

Collapse
humansprout profile image
Erik Waters • Edited on

I think they mean it won't behave as expected if the array contains similar objects that have different references

const originalObject = {}
const boundToOriginalObject = originalObject
const newObjectThatLooksLikeOriginal = {}
const arr = [originalObject, boundToOriginalObject, newObjectThatLooksLikeOriginal]
const dedupe = [ ...new Set(arr)]
dedupe // [{}, {}]
// deduping removes the duplicate reference but not similar objects
const mutation = {a: 1}
Object.assign(dedupe[1], mutation)
originalObject // {}
newObjectThatLooksLikeOriginal // { a: 1 }
// this proves which references remained in the deduped array prior to mutation
Enter fullscreen mode Exit fullscreen mode

It might be helpful to clarify that dedupe will remove duplicate bindings but not similar objects with a different binding

Collapse
martinkr profile image
martin krause Author

Thank you for your contribution. I'll add it to the description.

Collapse
michaelcurrin profile image
Michael

I just learned of an alternative for set to array

const myArr = Array.from(mySet1)
Enter fullscreen mode Exit fullscreen mode

And if you want to iterate over a set without converting to an array.

for (const item of mySet1) { 
  console.log(item)
}
Enter fullscreen mode Exit fullscreen mode

developer.mozilla.org/en-US/docs/W...

Collapse
qm3ster profile image
Mihail Malo

Array.from goes through iterator protocol for Set and Map, so it's exactly equivalent to the [... sugar.