DEV Community

Esty (c)
Esty (c)

Posted on

Serving sitemap.xml and robots.txt with ASP.Net core MVC

Robots.txt is a text file which is used to tell Search Engines which URLs are allowed or disallowed for them to index. The file must be located at https://www.example.com/robots.txt.

Some example of robots.txt:

Blocking all content from all Search Engines:
User-agent: *
Disallow: /

Allow all content for all Search Engines:
User-agent: *
Allow: /

Sitemap.xml is an XML file that contains all URLs of a website that you want Search Engines to index. Sitemap XML file’s helps Search Engines find and index site quickly. The sitemap.xml file default location is: https://www.example.com/sitemap.xml
Example of sitemap file:

<?xml version="1.0" encoding="utf-8"?>    
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">    
  <url>    
    <loc>https://www.codingwithesty.com/</loc>    
    <lastmod>2020-05-07T13:32:30+06:00</lastmod>    
    <changefreq>daily</changefreq>    
    <priority>1.0</priority>    
  </url>    
  <url>    
    <loc>https://www.codingwithesty.com/search-engine-optimization-library-for-dot-net-code-developers</loc>    
    <lastmod>2020-05-07T13:32:30+06:00</lastmod>    
    <changefreq>daily</changefreq>    
    <priority>0.8</priority>    
  </url>    
</urlset> 

In this article we will learn how to create simtemal.xml file and routing simtemal.xml in asp.net core. We also create routing for robots.txt file.

Package Installation

Create a new Asp.Net core project. Go to Nuget Package Manger install AspNetCore.SEOHelper package

Create sitemap.xml file:

AspNetCore.SEOHelper provides SitemapNode class for each URL and CreateSitemapXML
() helps to create sitemap.xml file. Copy to following code in your controller:

public class HomeController : Controller
    {
        private readonly ILogger<HomeController> _logger;
        private readonly IWebHostEnvironment _env;

        public HomeController(ILogger<HomeController> logger, IWebHostEnvironment env)
        {
            _logger = logger;
            _env = env;
        }


        public string Index()
        {
            var list = new List<SitemapNode>();

            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.8, Url = "https://codingwithesty.com/serilog-mongodb-in-asp-net-core", Frequency = SitemapFrequency.Daily });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.8, Url = "https://codingwithesty.com/logging-in-asp-net-core", Frequency = SitemapFrequency.Yearly });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.7, Url = "https://codingwithesty.com/robots-txt-in-asp-net-core", Frequency = SitemapFrequency.Monthly });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.5, Url = "https://codingwithesty.com/versioning-asp.net-core-apiIs-with-swagger", Frequency = SitemapFrequency.Weekly });
            list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.4, Url = "https://codingwithesty.com/configuring-swagger-asp-net-core-web-api", Frequency = SitemapFrequency.Never });

            new SitemapDocument().CreateSitemapXML(list, _env.ContentRootPath);
            return View();
        }

    }

CreateSitemapXML Method takes two parameters, the first parameter is SitemapNode list and the second parameter is the directory path where the sitemap.xml file will be stored. In this case we provide root directory of our project.

Routing sitemap.xml

We have already created sitemap.xml file in our asp.net core project. Now we want to route this sitemap file like: https://example.com/sitemap.xml

Add app.UseXMLSitemap middleware to the Configure method

   public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.UseStaticFiles();

            app.UseXMLSitemap(env.ContentRootPath);


           app.UseRouting();
            app.UseEndpoints(endpoints => { endpoints.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}"); });
        }

Now run your application and navigate to https://hostname:port/sitemap.xml

Routing robots.txt

Routing robots.txt in similar to previous one. Just add this app.UseRobotsTxt middleware to your configure method.

  public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.UseStaticFiles();
            app.UseRobotsTxt(env.ContentRootPath);

            app.UseRouting();
            app.UseEndpoints(endpoints => { endpoints.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}"); });
        }

UseRobotsTxt takes a directory path parameter. If there is no robots.txt file in the directory, it will show default text disallow all

Now run your application and navigate to https://hostname:port/robots.txt

Thanks a lot for reading. Write in the comment box in case you have any questions.

Reference

Codingwithesty.com
AspNetCore.SEOHelper/github

Top comments (2)

Collapse
 
vlatro profile image
vlatro • Edited

Hi Esty,

I'm currently using your package succesfully.
But now I want to create multiple sitemaps in my root directory instead of 1.

I can't find a way to do that. Any thoughts?

Regards,
Gert

Collapse
 
isatindersingh profile image
Satinder Singh

Nice package. I tried to create sitemap using razor pages as not aware of this package. Also written article on it, hope you find it useful codepedia.info/aspnet-core-create-...