DEV Community

Removing duplicates from an array

Svein Petter Gjøby on December 08, 2019

Originally posted on javascript.christmas Knowing more than one way to solve a given problem can help you write more readable code. Let's look at ...

Read full post

Charles Assunção • Dec 8 '19 • Edited

Nice article. :)

I guess on a daily basis I would go with Set approach for simplicity. I just wanted to add that in a interview I would probably try to come up with an O(n) solution, so something like:

function removeDuplicates(arr){
    let map = {};

    return arr.reduce((acc, curr) => {
        if(!map[curr]){
            map[curr] = true;
            return [...acc, curr];
        } else {
            return acc;
        }
    }, []);
}

Oliver Anteros • Dec 8 '19 • Edited

const removeDouplicate = (arr) => 
    Object.keys(
        arr.reduce((res, val) => ({
            ...res, 
            [val]: true
        }, {})
    );

Regarding comments about returning a string array independent of the input data:

const removeDouplicate = (arr) => 
    Object.entries(
        arr.reduce((res, val) => ({
            ...res, 
            [val]: val
        }, {})
    ).map(([_, val]) => val);

Regarding comments about creating N number of objects:

const removeDouplicate = (arr) => 
    Object.entries(
        arr.reduce((res, val) => (
            res[val] = val,
            res
        ), {})
    ).map(([_, val]) => val);

This will work for all arrays as long as they only contain primitive values. Seeing as res[{ foo: 42 }] will access the index res['[object Object]'] no matter the object.

Charles Assunção • Dec 9 '19

Yeah, fthis works but you are "mutating" the data itself

Mahendra • Dec 9 '19 • Edited

One more solution..

let obj = {}
let tempArray = []
for(let i=0;i<arr.length;i++){
if(!obj.values.includes(arr(i)){
tempArray.push(arr[i])
}else {
obj[arr[i]] = arr[i]
}
}
console.log(tempArray)

Jilles van Gurp • Dec 9 '19

It starts with realizing that arrays are blocks of memory and that technically you are creating a new array with some of the old values as you can't update the size of an array after you create one.

So, you are copying data to a new array instead of modifying an existing one. There's no way around having to iterate over your data at least once. However, the difference is in the price of your contains operation. I suspect/hope that the set might do some optimal things here (like using a hash function). Of course then having to use the spread operation might mean you are copying twice; which is of course not ideal. Your other two options probably have O(n) complexity for the contains check.

So if you do this a lot, use a Set as your primary data structure instead of converting to and from arrays because that isn't free.

Andy • Dec 8 '19

Before reading your article, I asked myself how I'd do it, and intuitively I came up with the Set solution. So I couldn't agree more with you on the solution you have chosen

Svein Petter Gjøby • Dec 8 '19

Great that you came to the same solution!

Charles Assunção • Dec 9 '19

Not really, I am keeping the same entry as the original array and not converting it to a string, what can be even trickier if your elements is an object or another thing. Returning an array of strings you are not preserving the data.

Manoj Kumar • Dec 9 '19

If you're lazy like me :D and already using Lodash in your project, you can do the following:

const array = [1, 1, 1, 3, 3, 2, 2];
_.uniq(array);