Why I HAD To Scrape The Web For Some Data

#career #programming #beginners #webscraping

It all started on a cold night.....JK 😉

It all started when I decided that I want to make my own project so that I will check if developing websites is something that can interest me.

But I didn't want to make a todo or a movie site, I wanted to do it about something that I can connect with.

I thought: if I'll feel connected to the topic of the project it will be a good foundation for the motivation I'll need to harness for this mission.

And...I was right [insert self pat on the back emoji here]

So the first thing that popped in my head was PLANTS! 🌱

Why? 'cause I've always liked growing them!

Ok...everything was going according to plan.
I decided the project will be an Ecommerce store for plants.

The name of the store is grow. ... yes, I thought about it myself... thank you.

The only thing that was missing was the data.
I thought: "No problem, I'm sure there are a tone of APIs waiting for me in the world wide web"

Little did I know I was wrong, all I could find was a bunch of photos of roots for Machine Learning projects to Identify a sick plant, which don't get me wrong, sounds really cool, but I needed DATA.

At the time, I looked at a lot of online plant stores to draw some inspiration.
And then it came to me! I'll just take the plants from one of the websites I found!

But that will be too tedious to do one by one....
So I'll throw some code at this problem and build a bot to scrape it for me!

I searched how to do that,
that's where I found out it's called 🔥Web Scraping🔥.

What's web scraping?
Web scraping is the process of extracting data from a website in an automated fashion.

After a small search I came across an article that suggested Selenium and python. (in hindsight they weren't the best tools for the job, but it was still fun learning about this whole new world called web scraping).

OK OK STOP! Where is the code though?

The Bot's Code can be found here (it's not perfect but it did the job).

The full project can be found on my github ⭐️

Here is a little snippet of the scrape :
Snippet of the scrape

This is how I felt when I saw my script moves through the pages by itself - invincible

** All of the scraped data is for personal use only, and no plants were harmed in the process.

Conclusions:

Choose a topic that you enjoy for your personal projects, it'll take you far.
Don't stress over the technologies, if you enjoy programming you'll enjoy and learn from every project (so much!)
There will always be some obstacles in the way of creating something, our job is to make it happen nonetheless.