DEV Community

Cover image for I created my own search engine
schBenedikt
schBenedikt

Posted on

I created my own search engine

I have created my own search engine and I'll tell you how it works.

First of all, I created my own web crawler. It runs on Python and stores various meta data in a mysql database.

GitHub logo SchBenedikt / web-crawler

A simple web crawler using Python that stores the metadata of each web page in a database.

web-crawler

A simple web crawler using Python that stores the metadata of each web page in a database.




The search engine, which I created with Bootstrap, then retrieves the results from mySQL. The bottom right always shows how long the query took.

GitHub logo SchBenedikt / search-engine

The matching search engine to my web crawler.




Please note that no robots.txt files are currently taken into account, which is why you cannot simply crawl every page.

What do you think of the project?

Feel free to write it in the comments!

Image description

Image description

Image description

Top comments (7)

Collapse
 
waleedboss7 profile image
Waleed

Coolđź‘Ťđź‘Ť

Collapse
 
aminnairi profile image
Amin

Pretty daunting task to create a search engine but I welcome the effort and wish you the best.

Any reasons not to use a NoSQL or a Graph database for this kind of project? Not criticizing your work, just curious.

Collapse
 
schbenedikt profile image
schBenedikt

I don't have any experience with the other database systems yet, so it's easiest to use this one. Do you have any other recommendations?

Collapse
 
aminnairi profile image
Amin

I think you might like MongoDB, plus it is a great fit for fast read/write access and you get to host your database for free on MongoAtlas in the cloud (their free tier is very generous).

Good luck for your project!

Thread Thread
 
schbenedikt profile image
schBenedikt

Since I have my own server and domain, I am not currently looking for a free provider. If MongoDB is really faster, I will definitely come back to it. At the moment, however, I would like to improve the search algorithm.

Collapse
 
big_ok profile image
Big Ok

How many rows of data have you stored so far?

Collapse
 
schbenedikt profile image
schBenedikt

It's about 44000, but I'm not going to expand the list because it's still under development at the moment.