DEV Community

Cover image for JavaScript - remove duplicated lines
Dirask-JavaScript
Dirask-JavaScript

Posted on • Originally published at dirask.com

JavaScript - remove duplicated lines

Hello there! πŸ‘‹ 😊

In this article, I would like to show you two ways to remove duplicated lines from some text in JavaScript. πŸ” 

Before we start, I would highly recommend you to check out runnable examples for the solution on our website:
JavaScript - remove duplicated lines

We're gonna use two methods:

  • filter()
  • reduce()

filter() based example

This approach uses a functional programming pattern.

On the text I've used set of operations to remove duplicated lines:

  • split() method to split text into lines that take a newline character in 4 different variants which is a universal approach,
  • filter() method that creates new array of elements tested with provided function,
  • join() method toΒ mergeΒ items back.

Practical example:

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = (text) => {
    return text.split(newLineExpression)
        .filter((item, index, array) => array.indexOf(item) === index)
        .join('\n');
};

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c
Enter fullscreen mode Exit fullscreen mode

You can run this example here

reduce() based example

This approach was created to show that it is possible to get the same effect as in the example above with reduce() method.

Practical example:

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = (text) => {
    const blocker = {}; // prevents lines dupplication
    return text.split(newLineExpression)
        .reduce((result, line) => {
            if (blocker.hasOwnProperty(line)) {
                return result;
            }
            blocker[line] = true;
            return result + line + '\n';
        }, '');
};

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c
Enter fullscreen mode Exit fullscreen mode

You can run this example here

If you found this solution useful you can react to this post or just leave a comment to let me know what you think. Thanks for reading and see you next time! 😊


Write to us! βœ‰

If you have any problem to solve or questions that no one can answer related to a React or JavaScript topic, or you're looking for a mentoring write to us on dirask.com -> Questions

Top comments (3)

Collapse
 
bytebodger profile image
Adam Nathaniel Davis • Edited

This is all you need:

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = text => [...new Set(text.split(newLineExpression))].join('\n');  

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c
Enter fullscreen mode Exit fullscreen mode
Collapse
 
bytebodger profile image
Adam Nathaniel Davis • Edited

In JavaScript, nearly any time that you are wanting to de-dupe an array (of any kind), the very first tool you reach for should be Set()

Collapse
 
diraskjavascript profile image
Dirask-JavaScript

Great solution! Thanks for the tips. 😊

Some comments may only be visible to logged-in visitors. Sign in to view all comments.