An introduction to IndexNow and why you should care

"Content is king" wrote Bill Gates in 1996, and in 2022 content is still king. But how do search engines discover content, new or updated? Their bots look for it every day and all day, whether you published or not, but that doesn't mean the content you want found, is found.

According to Cloudflare Radar, these last 7 days, 32% of traffic was originated from bots. Bots who are crawling the web all the time, to make sure they got the "freshest" content to serve to their customers.

But, what if you could just tell search engines where to look? Well, you can. Using Google Search Console you can manually request your URLs to be indexed. You can also submit URLs to Bing and Yahoo through Bing Webmasters, and there's a similar process for some other search engines. But those are a lot of clicks, keystrokes, and painstakingly waiting for their slow web consoles. This manual labor does not scale.

Luckily, some search engines provide developers with an interface to programmatically request URLs to be crawled. But every search engine that does offer this, has their own specific API. Bing offers the Bing URL submission API, Yandex offers recrawling through the Yandex Webmasters API, and so do some other search engines. Google also offers a submission API but for some odd reason, only for submitting pages about job postings or broadcasting events.

IndexNow protocol to the rescue

If you're not excited about developing and maintaining an integration with every search engine, join the club! Luckily, a new protocol is trying to save the day: IndexNow. IndexNow allows software developers to work with a single simple protocol, for all search engines who adopt it.

What's more, when you submit URLs to one search engine, that search engine will inform all other search engines with IndexNow support. That means you only have to make an API call to one IndexNow service, not to every search engine's IndexNow service.

How to implement IndexNow

1. Create a key between 8 and 128 hexadecimal characters. Allowed characters: lowercase characters (a-z), uppercase characters (A-Z), numbers (0-9), and dashes (-). You can choose this key yourself, or you can generate a valid key on Bing's IndexNow site for your convenience.
2. Upload a text file to the root of your website. The name of the file is your key with the ".txt" extension , and the contents of the file is also your key.
3. Submit HTTP requests to any IndexNow service:
   • IndexNow.org: https://api.indexnow.org/indexnow
   • Bing IndexNow: https://www.bing.com/indexnow
   • Yandex IndexNow: https://yandex.com/indexnow

You can send HTTP requests 2 ways. The first and simplest way is to send a GET HTTP request with two querystring parameters:

• url: this is the URL you want search engines to crawl
• key: this is the key you generated earlier

This will submit a single URL to IndexNow.

To send multiple URLs, you can use the second way. The second way is to send a POST HTTP request with the following JSON body:

{
    "host": "example.org",
    "key": "db132248f7cb4d309ee9d2aa54131af6",
    "keyLocation": "https://example.org/db132248f7cb4d309ee9d2aa54131af6.txt",
    "urlList": [
        "https://example.org/url1",
        "https://example.org/folder/url2",
        "https://example.org/url3"
    ]
}

Here's what the JSON properties should be:

• host: The domain name of your website. The host must match the URLs you are submitting.
• key: The IndexNow key you generated.
• keyLocation: Where you host your key text file. This property is optional if you store the key text file at the root of your website.
• urlList: An array of URLs you want to submit to search engines for crawling.

IndexNow adoption

IndexNow is a collaboration between Bing and Yandex. Depending on your target audience, you may or may not care at all about these two search engines. Yandex is a very popular search engines if your target audience is located in eastern Europe. Bing on the other hand, has 2.86% (2021) market share globally, which is a low percentage, but a low percentage of a large number of people. Bing and Yandex combined have a small market share, but hopefully more search engines adopt IndexNow. Google, the biggest search engine with 91.94% (2021) market share, is also testing IndexNow as part of become more environmentally friendly.

In addition to search engines, there are others that have started adopting IndexNow. Cloudflare uses IndexNow in its new "Crawler Hints" feature that sends hints to search engines when your content changes. In November, Cloudflare and Microsoft announced that 60,000 websites opted into the feature in less than two months. Microsoft reported that eBay, LinkedIn, MSN, GitHub, and Bizapedia are moving from Bing's submission API to IndexNow. Additionally, there are CMS's, CDN's, and SEO companies that are adopting IndexNow, hopefully with more to come.

Reducing manual labor and environmental impact

IndexNow is still relatively new but is getting a lot of interest in the search engine community. Hopefully, you can soon submit your content once to IndexNow, and stop manually submitting those URLs to every search engine console manually. And instead of search engines having to crawl the web over and over to look for new content, the content can be delivered directly from the source, reducing the environmental impact of crawlers.

Lastly, stay tuned if you're interested in learning how to adopt IndexNow using C# and .NET.