DEV Community

Cover image for Installing Scrapy in Fedora34 with pypy( pypy3 )
Amit Kumar Malaker
Amit Kumar Malaker

Posted on

Installing Scrapy in Fedora34 with pypy( pypy3 )

Pre-context:

The thing is I love fast things. And I am sure the reader also pursue same nature like me. Hence, pypy3 is the choice.
Fedora is nice but for me it is new and new thing is not always hassle free so to make your life better this instructions will be life saving.

First install pypy3 and respective pip pkgmanager:

sudo dnf install pypy3
wget https://bootstrap.pypa.io/get-pip.py -o get-pip.py
pypy3 get-pip.py
Enter fullscreen mode Exit fullscreen mode

Then install the below dependencies which are the root cause of scrapy installation failures. So do it carefully.

sudo dnf install pypy3-devel
sudo dnf install libxml2-devel
sudo dnf install libxslt-devel
sudo dnf install @development-tools
sudo dnf install libffi-devel
install them if you find them: [zlib1g-dev, libssl-dev] (optional)
Enter fullscreen mode Exit fullscreen mode

Now create virtual environment:

pypy3 -m venv venv
Enter fullscreen mode Exit fullscreen mode

Activate the virtual environment:

source venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Now install scrapy:

pypy3 -m pip install setuptools
pypy3 -m pip install scrapy
Enter fullscreen mode Exit fullscreen mode

Hopefully scrapy will be installed successfully.
Now create scrapy project.

scrapy startproject demo
Enter fullscreen mode Exit fullscreen mode

Create a spider:

cd demo
scrapy genspider demo demo
Enter fullscreen mode Exit fullscreen mode

In fine, this is the way to have our beloved scrapy working in fedora.
If you have any problem you can comment below. I will try my best to resolve it.

Top comments (0)