Dev Diary Week 2 - Reading a Feed

This post is an entry in a weekly development diary on building a feed-aggregator-based blog on Glitch. Only a small part of building an app is code; the bulk of your time is spent planning, experimenting, making mistakes, getting frustrated, and making progress in baby steps.

The plan

As I discussed in last week's entry, I'm building a web app that will do three things:

1. pull in RSS feeds of my content from various platforms;
2. combine them into a single superfeed; and
3. automatically generate a blog from that firehose feed.

If we look at this project as a black box, here's how it should work:

INPUT          +---------------+          OUTPUTS
               |               |
               |               +----> Aggregated RSS feed
OPML file ---->+ ~rss-firehose |
               |               +----> "Content firehose" blog
               |               |
               +---------------+

In other words, to get your own "content firehose" blog, you would:

1. Remix the ~rss-firehose Glitch app.
2. Drop in an OPML file of your own content feeds.
3. There is no step three!

We're not even close to this yet — right now, ~rss-firehose is just a simple remix of the ~hello-sqlite starter app on Glitch. Let's think about the steps to get us from where we are, to where we want to be! As a side benefit, this will give me a rough outline for the work I want to do each week. I only have a few hours per week to dedicate to this project, so I want to be thoughtful about how I approach that time.

Roughly speaking, here's how I expect things to go. This might change! I'm estimating based on a few things, like how comfortable I am working in Node, SQLite, and how deeply I understand the overall scope of the project. I've built blog templates before, but I've never built a feed reader!

• Week 1: Define the scope of the project ✅
• Week 2: Pull in a single RSS feed
• Week 3: Parse an OPML file into multiple RSS feeds
• Week 4: Save feed entries to a SQLite database
• Week 5: Generate an aggregated "firehose" RSS feed
• Week 6: Create a basic linklog front-end
• Week 7: Improve the front-end design

With our pre-requisite week 1 done, let's move on to week 2.

Reading an RSS Feed

Here's what I plan on doing this week; the next section will discuss how it went, and what I learned.

We want to implement a way to read a single RSS feed in our app. Now, we could figure out how to implement a basic RSS feed reader, but there are a few different formats — I can think of at least three different versions of the RSS spec itself, plus Atom, plus... well, you get the point.

Instead, we can build upon the work of those that came before us. RSS has been around for a very long time, so there are great packages we can use in our project to parse feeds in our app.

The package I see being used most in similar projects is feedparser, so we're going to add that to our app. I've never used feedparser before, so I'm going use my time budget for the week experimenting with reading a single RSS feed and outputting it to the server console.

Just to make it a bit more fun, I'll work with my own Dev.to feed!

How did it go?

I worked on the app over a few days, for a total of about four hours.

I started by remixing the actual app so that it's not affected by this week's work. In my remix, I added the feedparser package to package.json and got down to work.

When I'm working off a remix of a Glitch starter app, I usually start by deleting all the files and code I don't need. I'm skipped that this time, because it's nice to have some boilerplate code that I can examine for the SQLite stuff that will come later.

I then added feedparser.js, which takes a feed, tries to parse it, and returns a feedResponse object. The parsing-related work was mostly taken straight from the feedparser package README. The feedResponse object has two properties that get initialized as the parser does its thing: a meta object containing feed metadata, and an items array for feed entries.

Finally, I added an /api/parse/:feed endpoint to server.js — you can add a feed address to this endpoint in the browser, and it'll return the JSON of the feedResponse object. Try it by opening this link in a new tab, or add an RSS/Atom feed of your choice at the end of this link (but make sure it's url-encoded): https://rss-firehose-readfeed.glitch.me/api/parse/

I usually like to ask myself a few questions at the end of my workweek, and I think they're a good fit here too.

What went well?

Generally, adding the feedparser library went well. The README and sample code is thorough enough that I could follow along and make it do what I needed it to do.

What did I have trouble with?

I'm not very experienced with using the (deprecated) request library, and spent a while trying to get port the feedparser sample code to newer libraries in vain. Not wanting to blow my time budget for the week porting code that otherwise worked fine, I decided instead to just stick with request. I can always change that later.

What did I learn?

I learned how to use the feedparser library! I didn't dive into any of the more complex example code with additional dependencies, because the feeds I tested with worked fine as-is.

What will I work on next week?

Next week, I'm going to work on adding some feeds to an OPML file, parsing that file, and handing them off to the /api/parse/:feed endpoint I created this week!

So, that's week 2 wrapped up! Here's where the app is at now:

Do you use a feed reader?

Personally, I like NetNewsWire, and I sync feeds between my devices using Feedbin.

Which on is your favourite? Let me know in the comments!