How to implement a robots.txt file in a Nuxt project — Nuxt 2.10

A robots.txt file helps us to control the way in which Google and other search engines explore and index our content.

The first thing that a robot does when gets to your site it’s to check if a robots.txt file exists and if yes, they examine it to understand how to crawl it.

It’s just a simple public text file in which we can tell the crawlers to which parts of our site should or shouldn’t be crawled and indexed by allowing and disallowing pages.

@nuxtjs/robots it’s a Nuxt module that injects a middleware to automatically generate a robots.txt file.

@nuxtjs/robots it’s a Nuxt module that injects a middleware to automatically generate a robots.txt file.

Requirements:

• Nuxt
• npm or yarn
• Node

To begin with, we’ll install the module:

npm i nuxt-robots-module

Once it finishes, we are ready to go.

We’ll add it to our nuxt.config.js file in the modules array and give some options to it. As an example, I’ll use top-level options disallowing only the /user URL for all user-agents:

export default {
  modules: [
    '@nuxtjs/robots'
  ],
  robots: {
    UserAgent: '*',
    Disallow: '/user',
  }
}

But what if we want to disallow different URLs for different user-agents?
We can transform our options object into an array and add as many user-agents and urls as we want:

export default {
  modules: [
    '@nuxtjs/robots'
  ],
  robots: [
    {
      UserAgent: 'Googlebot',
      Disallow: '/user',
    },
    {
      UserAgent: '*',
      Disallow: '/admin',
    },
  ]
}

Now let’s say that we don’t want DuckDuckBot user agent to crawl any URL that we could have after /admin.

We can achieve by adding a /* to the URL like this:

export default {
  modules: [
    '@nuxtjs/robots'
  ],
  robots: [
    {
      UserAgent: 'Googlebot',
      Disallow: '/user',
    },
    {
      UserAgent: '*',
      Disallow: '/admin',
    },

  ]
}

And finally, to disallow several URLs for the same user-agent we'll do it by passing an array to the Disallow property.
The final code will look like the following:

export default {
  modules: [
    '@nuxtjs/robots'
  ],
  robots: [
    {
      UserAgent: 'Googlebot',
      Disallow: ['/user', '/admin'],
    },
    {
      UserAgent: '*',
      Disallow: '/admin',
    },

  ]
}

That’s all! You can build your application and check the file in /robots.txt URL.