loading...

Devs, make sure your page is searchable!

iggredible profile image Igor Irianto Updated on ・2 min read

The Issue

Ever since launching my site, https://irian.to/, I found that I couldn't google any page on my site. For example, when I google "irian.to redux" (my redux article), only site home page showed on google result. (This was done 2/1/2020.)

irian.to redux search no-show

The Solution

I went to Google Search Console, then I checked URL inspection and entered "https://irian.to/blog/redux-101". Google notified me that it didn't exist.

irian.to redux is not valid url

To make that page searchable, I needed to tell google to index that page. Here's how you can do it:

  • Visit search.google.com
  • Click your domain (you need to add it if you haven't. Check here to learn how)
  • URL Inspection
  • Enter the site URL

You can request for indexing ("Request Indexing"). I usually "Test Live URL" before doing that to make sure that it returns green.

irian.to redux testing live url and requesting to crawl

In this case, it does.

When requesting google for reindexing, it may take a few days. Google also tells you that:

URL was added to a priority crawl queue. Submitting a page multiple times will not change its queue position or priority. Learn more.

I checked my google search again on 2/3/2020 and it showed up on search.

irian.to redux is now found on google

The Conclusion

Moral of the story? If you own personal site, make sure people can google your page! πŸ‘€

Added 2/4/2020
Thanks to helpful inputs in comments (check below πŸ‘‡), here are more things you can do to help with your site's discoverability!

Adding sitemap.xml

Per suggestions below, it is best to add sitemap if you have many pages. For more information on sitemap, check this sitemap 101 article or moz article on search engines (go to sitemap section).

Thankfully, there are many sitemap generators. I created mine with Nuxt sitemap-module.

Testing your site index

How to quickly test indexed pages?

If you do site:yoursite.com (in my case, I did site:irian.to), google will show all results google have.

If you see screenshot below, you'll see I have 68 results (at the moment of writing), which is about right.

Search indexed pages

Posted on by:

iggredible profile

Igor Irianto

@iggredible

Husband πŸ‘«. Programmer πŸ’». Vim πŸ€“. Web Development should be explained as simple as possible, but no simpler πŸ’‘

Discussion

markdown guide
 

Hey Igor,

You can google this "site:your_site" to see the indexed psots by google.
For example, site:ajeet.dev

site-indexing-status

 

On a side note if you tried
site:facebook.com -www

You would get all the subdomains that are indexed by google.

 

Oh nice! What's the difference between googling with site: and without it? (What is "indexed" search?)

 

Quoting from this site: shoutmeloud.com/google-crawling-an...
"In layman’s terms, indexing is the process of adding web pages into Google search."

If you use "site:your_site", you will get a list of your indexed posts/pages/categories/etc that are indexed by google.

You should also read this article - how search engine works :)

moz.com/beginners-guide-to-seo/how...

Just did this today at work. We were researching something and my coworker suggested using "site:further_query" method. Much thanks for sharing Ajeet!

 

This seems like an appropriate place to remind folks that they're able to use canonical URLs when posting on DEV!

From the DEV FAQ:

https://p78.f0.n0.cdn.getcloudapp.com/items/yAuLONYD/Image+2020-02-04+at+5.51.41+PM.png?v=9b95576b7153223502f65fffff78ebd3

If you are sharing any work on DEV that you previously posted elsewhere, it's a good idea to add "canonical_url: blogname.com/postname" to the Front Matter of the post if using the "basic markdown" editor. If using the "rich + markdown" version of the editor, just click on the three-dots menu when writing your post and you'll see the option to input your canonical.

By the way, if you're not quite sure which editor version you're using or didn't even realize that there are different editor versions, 🀯 just go here in your settings and poke around!

 

Very nice!! I wasn't aware of this feature either. I am going to go back to every single blogs that I cross-posted and adding this.
You are awesome. So glad you posted this. 😎

Btw for those who don't know canonical url: yoast.com/what-is-a-canonical-url/

 

No problem! Happy to help. πŸ˜€

And good thinking on providing the explanation of what a canonical url is β€” definitely important for anyone who plans on cross posting to be aware of this!

 

Google can also discover your site on its own if another site links to it. This + submitting my sitemap initially is usually enough for my site.

If you're cross-posting content to DEV from your blog you could try adding an "Originally posted on " line to the DEV post.

 

I knew that more cross-referencing (backlinking?) helps with SEO but didn't know that it helps with unindexed files discoverability. Lots of good insight here πŸ‘

 

You should submit a sitemap.xml file to Google, rather than manually adding each page.

 

Agree with Patryk. Generate a sitemap and submit to google console periodically. Sitemaps looks like this: ajeet.dev/sitemap.xml

 

Is it a good practice to update sitemap and submit to google every time I add new page?
If not, what's a good frequency?

Normally, as long as Google knows to index your site and follow links, it shouldn't really be necessary, but it doesn't really hurt. It's just a PITA to do it manually.

You can always submit a sitemapindex to Google, and generate a new sitemap every time you add a page (I presume you mean blog posts). Then Google will pull your sitemap periodically, and should update its index without intervention.

If you can automate the sitemap generation as part of your workflow (e.g. in the save signal on a Post model in Django), that would require basically zero manual intervention.

 

Thank you for an easy to follow guide! πŸ™ŒMaking my site searchable didn't cross my mind when I created my website recently! I managed to set up Google indexing on my own site successfully.

Regarding the sitemap, I submitted it specifically to the Google Search Console here:

Google Search Console, Sitemaps section

When submitting the sitemap a popup told me that Google would index the sitemap automatically and would let me know if there were any problems indexing the sitemap in future.

 

Glad you found this helpful πŸ‘

 

The method you described is useful also, especially if the page has not been indexed already.