DEV Community

Discussion on: How do "public but secret" URLs work

Collapse
 
aurelkurtula profile image
aurel kurtula

But would it not be visible to search engine crawler and found that way? Is there something stoping the crawlers, or that's not how they work.

Thread Thread
 
nektro profile image
Meghan (she/her) • Edited

1) Search engine crawlers only follow links, which there are none unless the OP puts them out there,

2) and you could also have server code that is something along the lines of

if (header('User-Agent').indexOf('Bot') > 0) {
    // respond with blank or 404
}