DEV Community

loading...

Devs, make sure your page is searchable!

Igor Irianto
Bad puns, web development, creating stuff.
Originally published at irian.to ・Updated on ・2 min read

The Issue

Ever since launching my site, https://irian.to/, I found that I couldn't google any page on my site. For example, when I google "irian.to redux" (my redux article), only site home page showed on google result. (This was done 2/1/2020.)

irian.to redux search no-show

The Solution

I went to Google Search Console, then I checked URL inspection and entered "https://irian.to/blog/redux-101". Google notified me that it didn't exist.

irian.to redux is not valid url

To make that page searchable, I needed to tell google to index that page. Here's how you can do it:

  • Visit search.google.com
  • Click your domain (you need to add it if you haven't. Check here to learn how)
  • URL Inspection
  • Enter the site URL

You can request for indexing ("Request Indexing"). I usually "Test Live URL" before doing that to make sure that it returns green.

irian.to redux testing live url and requesting to crawl

In this case, it does.

When requesting google for reindexing, it may take a few days. Google also tells you that:

URL was added to a priority crawl queue. Submitting a page multiple times will not change its queue position or priority. Learn more.

I checked my google search again on 2/3/2020 and it showed up on search.

irian.to redux is now found on google

The Conclusion

Moral of the story? If you own personal site, make sure people can google your page! πŸ‘€

Added 2/4/2020
Thanks to helpful inputs in comments (check below πŸ‘‡), here are more things you can do to help with your site's discoverability!

Adding sitemap.xml

Per suggestions below, it is best to add sitemap if you have many pages. For more information on sitemap, check this sitemap 101 article or moz article on search engines (go to sitemap section).

Thankfully, there are many sitemap generators. I created mine with Nuxt sitemap-module.

Testing your site index

How to quickly test indexed pages?

If you do site:yoursite.com (in my case, I did site:irian.to), google will show all results google have.

If you see screenshot below, you'll see I have 68 results (at the moment of writing), which is about right.

Search indexed pages

Discussion (18)

Collapse
ajeet profile image
Ajeet Yadav

Hey Igor,

You can google this "site:your_site" to see the indexed psots by google.
For example, site:ajeet.dev

site-indexing-status

Collapse
chandrasd profile image
Chandra Dasari

On a side note if you tried
site:facebook.com -www

You would get all the subdomains that are indexed by google.

Collapse
iggredible profile image
Igor Irianto Author • Edited

Oh nice! What's the difference between googling with site: and without it? (What is "indexed" search?)

Collapse
ajeet profile image
Ajeet Yadav

Quoting from this site: shoutmeloud.com/google-crawling-an...
"In layman’s terms, indexing is the process of adding web pages into Google search."

If you use "site:your_site", you will get a list of your indexed posts/pages/categories/etc that are indexed by google.

You should also read this article - how search engine works :)

moz.com/beginners-guide-to-seo/how...

Thread Thread
iggredible profile image
Igor Irianto Author

Just did this today at work. We were researching something and my coworker suggested using "site:further_query" method. Much thanks for sharing Ajeet!

Thread Thread
ajeet profile image
Ajeet Yadav

Your welcome 😊

Collapse
michaeltharrington profile image
Michael Tharrington (he/him)

This seems like an appropriate place to remind folks that they're able to use canonical URLs when posting on DEV!

From the DEV FAQ:

https://p78.f0.n0.cdn.getcloudapp.com/items/yAuLONYD/Image+2020-02-04+at+5.51.41+PM.png?v=9b95576b7153223502f65fffff78ebd3

If you are sharing any work on DEV that you previously posted elsewhere, it's a good idea to add "canonical_url: blogname.com/postname" to the Front Matter of the post if using the "basic markdown" editor. If using the "rich + markdown" version of the editor, just click on the three-dots menu when writing your post and you'll see the option to input your canonical.

By the way, if you're not quite sure which editor version you're using or didn't even realize that there are different editor versions, 🀯 just go here in your settings and poke around!

Collapse
iggredible profile image
Igor Irianto Author • Edited

Very nice!! I wasn't aware of this feature either. I am going to go back to every single blogs that I cross-posted and adding this.
You are awesome. So glad you posted this. 😎

Btw for those who don't know canonical url: yoast.com/what-is-a-canonical-url/

Collapse
michaeltharrington profile image
Michael Tharrington (he/him) • Edited

No problem! Happy to help. πŸ˜€

And good thinking on providing the explanation of what a canonical url is β€” definitely important for anyone who plans on cross posting to be aware of this!

Collapse
downey profile image
Tim Downey

Google can also discover your site on its own if another site links to it. This + submitting my sitemap initially is usually enough for my site.

If you're cross-posting content to DEV from your blog you could try adding an "Originally posted on " line to the DEV post.

Collapse
iggredible profile image
Igor Irianto Author

I knew that more cross-referencing (backlinking?) helps with SEO but didn't know that it helps with unindexed files discoverability. Lots of good insight here πŸ‘

Collapse
patryktech profile image
Patryk

You should submit a sitemap.xml file to Google, rather than manually adding each page.

Collapse
ajeet profile image
Ajeet Yadav

Agree with Patryk. Generate a sitemap and submit to google console periodically. Sitemaps looks like this: ajeet.dev/sitemap.xml

Collapse
iggredible profile image
Igor Irianto Author

Is it a good practice to update sitemap and submit to google every time I add new page?
If not, what's a good frequency?

Thread Thread
patryktech profile image
Patryk

Normally, as long as Google knows to index your site and follow links, it shouldn't really be necessary, but it doesn't really hurt. It's just a PITA to do it manually.

You can always submit a sitemapindex to Google, and generate a new sitemap every time you add a page (I presume you mean blog posts). Then Google will pull your sitemap periodically, and should update its index without intervention.

If you can automate the sitemap generation as part of your workflow (e.g. in the save signal on a Post model in Django), that would require basically zero manual intervention.

Collapse
ericathedev profile image
Erica Tanti

Thank you for an easy to follow guide! πŸ™ŒMaking my site searchable didn't cross my mind when I created my website recently! I managed to set up Google indexing on my own site successfully.

Regarding the sitemap, I submitted it specifically to the Google Search Console here:

Google Search Console, Sitemaps section

When submitting the sitemap a popup told me that Google would index the sitemap automatically and would let me know if there were any problems indexing the sitemap in future.

Collapse
iggredible profile image
Igor Irianto Author

Glad you found this helpful πŸ‘

Collapse
colinmcdermott profile image
Colin McDermott

The method you described is useful also, especially if the page has not been indexed already.

Forem Open with the Forem app