I have created my own search engine and I'll tell you how it works.
First of all, I created my own web crawler. It runs on Python and stores various meta data in a mysql database.
SchBenedikt
/
web-crawler
A simple web crawler using Python that stores the metadata of each web page in a database.
web-crawler
A simple web crawler using Python that stores the metadata of each web page in a database.
The search engine, which I created with Bootstrap, then retrieves the results from mySQL. The bottom right always shows how long the query took.
SchBenedikt
/
search-engine
The matching search engine to my web crawler.
search-engine
The matching search engine to my web crawler.
Features
- Like Projects (The most liked pages are displayed at the top)
- Display of the search speed.
Please note that no robots.txt files are currently taken into account, which is why you cannot simply crawl every page.
What do you think of the project?
Feel free to write it in the comments!
Top comments (7)
Coolđź‘Ťđź‘Ť
Pretty daunting task to create a search engine but I welcome the effort and wish you the best.
Any reasons not to use a NoSQL or a Graph database for this kind of project? Not criticizing your work, just curious.
I don't have any experience with the other database systems yet, so it's easiest to use this one. Do you have any other recommendations?
I think you might like MongoDB, plus it is a great fit for fast read/write access and you get to host your database for free on MongoAtlas in the cloud (their free tier is very generous).
Good luck for your project!
Since I have my own server and domain, I am not currently looking for a free provider. If MongoDB is really faster, I will definitely come back to it. At the moment, however, I would like to improve the search algorithm.
How many rows of data have you stored so far?
It's about 44000, but I'm not going to expand the list because it's still under development at the moment.