Introduction

This is a little side project I did to try and scrape images out of reddit threads. There's a few different subreddits discussing shows, specifically /r/anime where users add a lot screenshots of the episodes. And I thought it'd be cool to see how much effort it'd be to automatically collate a list of those screenshots from a thread and display them in a simple gallery. The result looked like this

PRAW

PRAW is the Python Reddit API Wrapper, that provides a nice set of bindings to talk to Reddit.

To scrape Reddit you need credentials. The way to generate credentials is hidden away at https://www.reddit.com/prefs/apps where you have to register a new "app" with Reddit. Connecting is as simple as

import praw

reddit = praw.Reddit(client_id='id', \
                     client_secret='secret', \
                     user_agent='useragent', \
                     username='username', \
                     password='DevToIsCool')

Traversing reddit is made simple by the API, for example printing all of the comments in a thread.

submission = reddit.submission(url="https://reddit.com/r/abcde")
for comment in submission.comments.list():
    print(comment)

Finding links

99% of the images I was looking for are posted to imgur so I just matched on those. I used a regular expression to extract the links. I always recommend using a tool like RegEx101 that makes it really easy to debug your regular expressions as they can be pretty brain bending.

    REGEX_TEST = r"((http|https)://i.imgur.com/.+?(jpg|png))"
    p = re.compile(REGEX_TEST, re.IGNORECASE)

Check if an image still exists

One of the problems I found was dead image links, so I created a simple helper that checks the status_code for that link.

# Check if a link still is exists
def checkLinkActive(url):
    request = requests.head(url)
    if request.status_code == 200:
        return True
    else:
        return False

Getting Thumbnails

To save bandwidth and your mobile data I wanted to return a smaller version of the image. In imgur you can append a size character onto a URL to get it at a different size, for example 'l' large and 's' small.

# Add a letter to an imgur url to make a small thumbnail
def getImgurThumbnail(url, size):
    startStr = url[:(len(url)-4)]
    endStr = url[len(url)-4:]
    return startStr + size + endStr

Putting it all together

Putting all of these bits together you get

def getImages(url):
    submission = reddit.submission(url=url)
    # Tell API to return all comment in thread, results are
    # paginated by default
    submission.comments.replace_more(limit=None)

    # Create RegEx object for matching images
    REGEX_TEST = r"((http|https)://i.imgur.com/.+?(jpg|png))"
    p = re.compile(REGEX_TEST, re.IGNORECASE)

    imageMatches = []
    for comment in submission.comments.list():
        matches = p.findall(comment.body)
        for match in matches:
            if checkLinkActive(match[0]):
                imageMatches.append(
                    {"image": match[0], "thumbnail": getImgurThumbnail(match[0], "m")}
                )

    return imageMatches

Trying it out

I decided to stand up a quick demo of this, using an Azure Function to host my new function and a simple web form to allow people to try it out. Just copy and paste a Reddit URL and the function will return any images.

The Demo App uses Bulma for the look and feel, and a little bit of JQuery for the loading of the page.

If you want to give it a go, you can have a play on my site here.

I'll be looking in a future article at providing a show name search instead of having to paste individual episode URLs. Happy Reddit scraping!