Robots.txt is a text file which is used to tell Search Engines which URLs are allowed or disallowed for them to index. The file must be located at https://www.example.com/robots.txt.
Some example of robots.txt:
Blocking all content from all Search Engines:
User-agent: *
Disallow: /
Allow all content for all Search Engines:
User-agent: *
Allow: /
Sitemap.xml is an XML file that contains all URLs of a website that you want Search Engines to index. Sitemap XML file’s helps Search Engines find and index site quickly. The sitemap.xml file default location is: https://www.example.com/sitemap.xml
Example of sitemap file:
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.codingwithesty.com/</loc>
<lastmod>2020-05-07T13:32:30+06:00</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://www.codingwithesty.com/search-engine-optimization-library-for-dot-net-code-developers</loc>
<lastmod>2020-05-07T13:32:30+06:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.8</priority>
</url>
</urlset>
In this article we will learn how to create simtemal.xml file and routing simtemal.xml in asp.net core. We also create routing for robots.txt file.
Package Installation
Create a new Asp.Net core project. Go to Nuget Package Manger install AspNetCore.SEOHelper package
Create sitemap.xml file:
AspNetCore.SEOHelper provides SitemapNode class for each URL and CreateSitemapXML
() helps to create sitemap.xml file. Copy to following code in your controller:
public class HomeController : Controller
{
private readonly ILogger<HomeController> _logger;
private readonly IWebHostEnvironment _env;
public HomeController(ILogger<HomeController> logger, IWebHostEnvironment env)
{
_logger = logger;
_env = env;
}
public string Index()
{
var list = new List<SitemapNode>();
list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.8, Url = "https://codingwithesty.com/serilog-mongodb-in-asp-net-core", Frequency = SitemapFrequency.Daily });
list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.8, Url = "https://codingwithesty.com/logging-in-asp-net-core", Frequency = SitemapFrequency.Yearly });
list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.7, Url = "https://codingwithesty.com/robots-txt-in-asp-net-core", Frequency = SitemapFrequency.Monthly });
list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.5, Url = "https://codingwithesty.com/versioning-asp.net-core-apiIs-with-swagger", Frequency = SitemapFrequency.Weekly });
list.Add(new SitemapNode { LastModified = DateTime.UtcNow, Priority = 0.4, Url = "https://codingwithesty.com/configuring-swagger-asp-net-core-web-api", Frequency = SitemapFrequency.Never });
new SitemapDocument().CreateSitemapXML(list, _env.ContentRootPath);
return View();
}
}
CreateSitemapXML Method takes two parameters, the first parameter is SitemapNode list and the second parameter is the directory path where the sitemap.xml file will be stored. In this case we provide root directory of our project.
Routing sitemap.xml
We have already created sitemap.xml file in our asp.net core project. Now we want to route this sitemap file like: https://example.com/sitemap.xml
Add app.UseXMLSitemap middleware to the Configure method
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseStaticFiles();
app.UseXMLSitemap(env.ContentRootPath);
app.UseRouting();
app.UseEndpoints(endpoints => { endpoints.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}"); });
}
Now run your application and navigate to https://hostname:port/sitemap.xml
Routing robots.txt
Routing robots.txt in similar to previous one. Just add this app.UseRobotsTxt middleware to your configure method.
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseStaticFiles();
app.UseRobotsTxt(env.ContentRootPath);
app.UseRouting();
app.UseEndpoints(endpoints => { endpoints.MapControllerRoute( name: "default", pattern: "{controller=Home}/{action=Index}/{id?}"); });
}
UseRobotsTxt takes a directory path parameter. If there is no robots.txt file in the directory, it will show default text disallow all
Now run your application and navigate to https://hostname:port/robots.txt
Thanks a lot for reading. Write in the comment box in case you have any questions.
Top comments (2)
Hi Esty,
I'm currently using your package succesfully.
But now I want to create multiple sitemaps in my root directory instead of 1.
I can't find a way to do that. Any thoughts?
Regards,
Gert
Nice package. I tried to create sitemap using razor pages as not aware of this package. Also written article on it, hope you find it useful codepedia.info/aspnet-core-create-...