DEV Community

Cover image for 1 line of code: How to remove all duplicates from an Array
Martin Krause
Martin Krause

Posted on • Edited on

1 line of code: How to remove all duplicates from an Array

#javascript #webdev #productivity #codequality 
const removeDuplicates = arr => [...new Set(arr)];
Enter fullscreen mode Exit fullscreen mode

Assumes that the given argument is an Array and removes duplicate entries, keep in mind that it works only for entries with primitive values (string, number, bigint, boolean, undefined, symbol, and null). Preserves the order of the entries and returns a copy of the array.

The repository & npm package

You can find the all the utility functions from this series at github.com/martinkr/onelinecode
The library is also published to npm as @onelinecode for your convenience.

The code and the npm package will be updated every time I publish a new article.

Follow me on Twitter: @martinkr and consider to buy me a coffee

Photo by zoo_monkey on Unsplash

Top comments (10)

Collapse
 
michaelcurrin profile image
Michael Currin •

Worth explaining this separately when used with array instead of sets

const x = ["abc", "def"]
const y = [...x]
// ["abc", "def"]
Enter fullscreen mode Exit fullscreen mode

And modifying x and y won't affect the other.

Then bringing in set

const x = ["abc", "def", "abc"]
const y = new Set(x)
// Set(["abc", "def"])
const z = [...y]
// ["abc", "def"]
Enter fullscreen mode Exit fullscreen mode

And the the original post does that in one line in a function

Collapse
 
michaelcurrin profile image
Michael Currin •

Also worth adding that you might want to stop once you have a set and not convert it back to an array.

Why?

Lookup time from a set is constant to check if a value is in a set while in an array of 1000 elements worst case you have to go through 1000 elements until you find the one at the end.

Secondly when you convert from array to set, the order will be lost. Sets are inherently unordered. So when you convert from set to array you will not get the order of the original array. So you might as well keep the set.

Collapse
 
qm3ster profile image
Mihail Malo •

It's actually specced to preserve insertion order: developer.mozilla.org/en-US/docs/W...

(...spread operator calls y[Symbol.iterator], which is specced to be the same as what is returned by .values() on Set)

Thread Thread
 
michaelcurrin profile image
Michael Currin •

Oh interesting, in Python it is unordered.

Collapse
 
martinkr profile image
Martin Krause •

Thank you for the explanation.

Collapse
 
humansprout profile image
Erik Waters • • Edited

I think they mean it won't behave as expected if the array contains similar objects that have different references

const originalObject = {}
const boundToOriginalObject = originalObject
const newObjectThatLooksLikeOriginal = {}
const arr = [originalObject, boundToOriginalObject, newObjectThatLooksLikeOriginal]
const dedupe = [ ...new Set(arr)]
dedupe // [{}, {}]
// deduping removes the duplicate reference but not similar objects
const mutation = {a: 1}
Object.assign(dedupe[1], mutation)
originalObject // {}
newObjectThatLooksLikeOriginal // { a: 1 }
// this proves which references remained in the deduped array prior to mutation
Enter fullscreen mode Exit fullscreen mode

It might be helpful to clarify that dedupe will remove duplicate bindings but not similar objects with a different binding

Collapse
 
captainyossarian profile image
yossarian •

I think it worth saying that it works only on primitives

Collapse
 
martinkr profile image
Martin Krause •

Thank you for your contribution. I'll add it to the description.

Collapse
 
michaelcurrin profile image
Michael Currin •

I just learned of an alternative for set to array

const myArr = Array.from(mySet1)
Enter fullscreen mode Exit fullscreen mode

And if you want to iterate over a set without converting to an array.

for (const item of mySet1) { 
  console.log(item)
}
Enter fullscreen mode Exit fullscreen mode

developer.mozilla.org/en-US/docs/W...

Collapse
 
qm3ster profile image
Mihail Malo •

Array.from goes through iterator protocol for Set and Map, so it's exactly equivalent to the [... sugar.

Read next

charan_sajjanapu profile image

Evolution of Web Tech and Browsers

Charan Sajjanapu -

akpvt profile image

CMS: Payload CMS

akpvt -

jesterxl profile image

Thoughts on ThoughtWorks Radar 2024

Jesse Warden -

sammybarasa profile image

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory on Digital OceanDroplet

Sammy Barasa -