loading...

E-Books Web Crawling Application with Recommendation System

extremedie profile image Koay Yong Zhuang ・2 min read

My Final Project

Finally finished my last semester of my degree, this article is about my final year project (FYP). It is an mobile application that allows users to search for the web crawled e-books from a few websites. All the e-books are free to download. There are also recommendation systems to recommend users some e-books.

How I built it ?

There are 4 main components for this FYP as shown below:

Web Crawler

3 e-books websites are being scraped by using the following Python technologies:

  1. Scrapy
  2. Scrapyd
  3. Pymongo

Mobile Application

The mobile application is being built with Flutter. Here are just a few UI designs.

Alt Text

^this is the Library Screen^

Alt Text
^this is the Explore Screen^

Alt Text
^this is the Story Screen^

Alt Text
^this is the Profile Screen^

Back-end for Mobile Application

The back-end is used to handle the actions of the mobile application such as login, register, search for books and etc.
It is built with Node.js + MongoDB.


Recommendation Systems

The recommendation systems are being built with the following Python technologies:

  1. flask
  2. pandas
  3. numpy
  4. gensim
  5. sklearn

There are something one might need to take care for is the cold start problem of the recommendation system. In order to solve it, the users will be prompted to select e-books and genres that they like.

There are 2 recommendation systems, the first one is a content-based recommender after users have selected their favourite e-books and genres.
The 2nd one is being used for collaborative filtering (CF) when users have rated a few e-books. The 2nd one is based on k-NN algorithm.


Website

I've also created a website with React to promote this application and for users to download the apk file.


Additional Thoughts / Feelings / Stories

In short, I am really happy that I am able to complete my FYP. Hopefully that I could get a really good grade for it.

Thank you for reading !!

Discussion

markdown guide