Overview

OpenSea has a lot of NFT pictures. It's a good example of learning how to scrape. Compared to other typical static websites, pictures in OpenSea are dynamic loading along with the endless scroll. Pictures are loading if they are on the screen and will be gone
when they are out of the screen.

Clicknium

Clicknium is a Python automation library and very easy to do this kind of scraping work. It's more than a Python library but also provides a smooth and interactive coding experience with VS Code plugins and browser plugins.

Requirement

• Windows7 SP1+
• Python 3.7+
• VS Code

How to

Let's scrape Okay Bear as an example.

Picture info

1. Open Okay Bear's page with Chrome.
2. Press F12 to open the DevTools, click the mouse icon, and then click the NFT picture. In the following pic, we can see:
3. Image alt can be used as picture name
4. Picture src and srcset with different sizes.

Capture Pictures

Capturing similar elements is quite easy for this scenario:

1. Click the capture button in VS code extension to start the recorder.
2. Click Similar elements .
3. Ctrl + Click on the first pic and then capture the second.

You can also use this way to capture the price.

Handle scroll to load pictures

Clicknium provides API to control the mouse and keyboard. In this Scenario, Page-down is the easiest way. More about HotKey

clicknium.send_hotkey("{PGDN}")

Download picture

use requests(pip install requests)

Source code:

Github: OpenSeaScraper

import requests
from time import sleep
from clicknium import clicknium as cc, locator
def main():
    cc.chrome.open("https://opensea.io/collection/okay-bears")
    nftDic = {}

    while True:
        newDic = {}
        finish = True
        sleep(4)
        nfts = cc.find_elements(locator.sample.opensea.img_okay_bear)
        for nft in nfts:
            src = nft.get_property("src")
            name = nft.get_property("alt")
            if name not in nftDic:
                newDic[name]=src
                finish = False

        for key,value in newDic.items():
            r = requests.get(value)
            print("Downloading:"+ key)
            imagePath = ".\\image\\" + key + ".jpg"
            with open(imagePath,'wb') as f:
                f.write(r.content)

        nftDic = nftDic | newDic
        if finish:
            break
        else:
            print("next page")
            cc.send_hotkey("{PGDN}")

if __name__ == "__main__":
    main()

More sample:
10 lines of Python code to upload a video to TikTok, Instagram, Twitter

A Spotify robot