Getting rid of duplicate content in Magento

Before we start getting rid of duplicate content, we should outline its general nature.

In accordance with Google, duplicate content is:

• identical headings,
• title tags,
• similar content on two or more web pages,
• copy-pasted meta tags (e.g. meta description and alt tags for images).

The most important thing to understand about duplicates is that you can't get rid of them completely, so it's OK to have some in your Magento store.

If you don't misuse duplicate content, you don't have to worry about Google penalizing you in search results - it will just decide which of two similar pages the user will see, and make the other one invisible.

That being said, let's see how duplicates appear in Magento 1 & 2 and how you can minimize their quantity. Actionable tips with code examples are included.

Why Duplicates Appear and How to Combat Them

It will be convenient if we split duplicates into user- and machine-generated ones.

User-generated duplicates appear when vendors either don't want to or simply cannot write unique descriptions for each item they sell.

In this case, it's a good idea to motivate users to leave reviews of your products, let them fill your website with original content for you. It would be awesome if they could attach pictures of received goods to their reviews.

What is more, new customers will see the reviews, and this way you will gain their trust. Websites with high customer trust level are ranked higher in Google search, so more users are naturally attracted to such stores.

Now let's move on to machine-generated duplicates and see how you can avoid them.

1. Magento Multistore

It's awesome you sell and ship goods internationally, it's cool you decided to go for regional targeting. In this case make sure you deliver unique content for each region, especially for the ones that speak the same language.

Also, use language metadata to help Google indicate that identical content targets different types of users.

2. Long and short product descriptions in Magento

This one is nice and easy. Just make sure short and long descriptions of each item on your store are different. This move will improve both SEO ranking and customers engagement.

3. Indexing Methods

If you do not use 301 redirects, your Magento store will be available both at www.name.com and name.com. It's not a secret these two domains will be different websites for Google, so you want to let it know that www.name.com and name.com should be treated as one website.

Go to Google Search Console and add BOTH websites to it. Do not forget about such an important detail as the protocol. It's absolutely necessary to add addresses either with http:// or https:// appellation.

A piece of advice: use name.com as the preferred address and make certain 301 redirect points to it.

So, how to create the above mentioned 301 redirects?

For Apache, it's better to edit .htaccess or httpd.conf file. Go to your root directory, open .htaccess, and add a new line on the top of the others (if there are any of course) so it looks like this:


RewriteEngine on
RewriteCond %{HTTP_HOST} www.yourstore.com [NC]
RewriteRule .*$ http://yourstore.com/$1 [L,R=301,NC]


This function is applicable both to Magento 1 and 2.

For Nginx edit server.rewrites so it looks like this:


if ($http_host ~* "?!www\).*$") {
return 301 https://www.$http_host$request_uri;
}


This function is applicable to Magento 1 and 2 as well.

For Varnish put this code to public.rewrites to prevent the caching issue.

Let's get back to the protocols again, shall we?

You want to use rel=canonical to prioritize pages, piece together HTTP and HTTPS pages and get rid of duplicates. So here's a quick & simple guide to making rel=canonical pages in Magento.

For Magento 1 go to System > Configuration > Catalog > Search Engine Optimization. In the fields "Use Canonical Link Meta Tag for Categories" and "Use Canonical Link Meta Tag for Products" choose "yes".

For Magento 2 stores go to Stores > Configuration > Catalog > Search Engine Optimization and choose "yes" for the same field

4. Trailing slashes in URLs

In accordance with web server setup, your Magento store will either default to name.com/link/ or to name.com/link URL.

The difference is hardly noticeable for people but both URLs look different to Google.

The most valuable recommendation here is to not make any changes to core Magento files.

Here we should work with 301 redirects again.

For Apache open .htaccess file which can be found in root Magento directory and make sure it looks like this:


RewriteCond %{request_method} GET$
RewriteCond %{REQUEST_URI} !/downloader.*$
RewriteCond %{REQUEST_URI} .+/$
RewriteRule .+$ %1 [L,R=301]


The fix for Nginx would be altering server rules. Add this piece of code to nginx.conf:


if ($request_method != POST) {
set $rew "A";
}
if ($http_x_requested_with != XMLHttpRequest) {
set $rew "${rew}B";
}
if ($rew = "AB") {
rewrite /(.*)/$ $scheme://$http_host/$1 permanent;
}


5. The Same Product Is in Two or More Categories

It's obvious that this issue results in lots of duplicates. The most simple solution most Magento experts recommend is to use only one category for one product. If for any reason you can't avoid using multiple categories for one item, have a look at these instructions.

Magento 1 users should make sure catalog comparisons and categories are disallowed from appearing in Google search index:


Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/


Magento 2 users want to go to Content > Design > Global Design Configuration and add new lines to your robots.txt:


Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/


6. Robots.txt

Implement the given trick with robots.txt on your Magento store to eliminate duplicates.

These lines will help you to limit search results (which are not so much of a trouble yet they lead to duplicates too), use them for Magento 1.


Disallow: /catalogsearch/result/
Disallow: /sendfriend/


These lines will remove filters and reviews from search results:


Disallow: /review/
Disallow: /?category=
Disallow: /&category=
Disallow: /?price=
Disallow: /&price=
Disallow: /?mode=
Disallow: /&product_list_mode=


To restrict search results for a Magento 2 store you should prevent them from indexing, and the best practice is to go to Content > Design > Global Design Configuration.

And these lines will help you fix duplicate content in search results:


Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/


Also, the second version of Magento generates links to tags and reviews that lead to - we know the answer - more duplicate content.

Here is what you add to robots.txt to fix it:


Disallow: /tag/
Disallow: /review/
Disallow: /?dir
Disallow: /?dir=desc
Disallow: /?dir=asc
Disallow: /?limit=all
Disallow: /?mode*


Long Story Short

Duplicate content is inevitable, do not be afraid of having some in your Magento store in a fair amount. Just do your best in minimizing it and enjoy high conversion rates with these simple steps.