DEV Community

JEFF
JEFF

Posted on

robots.txt

robots.txt As a beginner in react , we all must have come across this robots.txt file when we create our react application. Most of us never bothered to know what it was and delete it , thinking it as a unnecessary file in our react app . Yaa I've also done it 😂 😂.

Now lets know what it is .
*Robots.txt file is a file located on your server that tells web crawlers which pages they can and cannot access. *

The pages we blocked , when the web crawler crawls in it , we get soft 404 as error.

The two action we can do is

  1. Allow: /
  2. Disallow: /

we can allow and disallow the pages to access.

Adding robots.txt file in your application , positively increases your app's SEO Performance . So Know the stuff and increase your apps SEO Performance right away with robots.txt . ☮️

Top comments (1)

Collapse
 
sarathchan profile image
sarathchan

Informative 👍