This post is part of a series.
This is the second post in a series I'm writing about a new Minimal Viable Product we've released at Voxgig that turns your podcast into a chatbot. Your visitors can now ask your guests questions directly!
The first post is here: Building a Podcast Chatbot for Voxgig
In this series, I'll dive into the code that implements the chatbot. It's all open source so you can cut and paste to have your own one.
Since it is production-grade code, not just an example for the sake of some "content", we'll navigate through the code in baby steps by focusing on specific tasks, rather than trying to go top-to-bottom. Production code has to do a lot of things, so rather than trying to start at the start, we'll start in a reasonable place so you can see useful code right away.
We have a podcast RSS feed URL, and we want to trigger ingestion of the episodes into the chatbot. Where do we begin?
Well, we have to download and parse the RSS feed. There's a great RSS parser package that we can use: rss-parser - thank you to Robert Brennan! To get the RSS feed we need one line of code:
await Parser.parseURL(feed)
That returns the contents of the feed (which is XML), as a JSON document so it's nice and easy to work with. We'll loop through all the episodes, download the audio, get it transcribed with a speech-to-text service (Deepgram in the first version), and then sprinkle some Retrieval Augmented Generation magic pixie dust over everything to make the chatbot work.
But first, let's do some software architecture. This is not meant to be a toy. The system deploys to AWS, and uses Lambda Functions, DynamoDB, S3, SQS, and other fun AWS stuff.
Also, the system is a microservice system when deployed, but a local monolith when developing locally. How that all works is something we'll come back to in later posts.
Since we are downloading RSS feeds, we are effectively an RSS reader, and thus we can have the concept of "subscribing" to a feed in our system. That means we need to have a microservice that can accept an instruction to subscribe to a podcast.
The microservice is called ingest and the message that we send to the microservice to subscribe to a podcast (and optionally trigger ingestion) looks like this:
{
aim: 'ingest', // The "aim" of the message is the `ingest` service.
subscribe: 'podcast', // The instruction to the `ingest` service.
feed: 'https://feeds.resonaterecordings.com/voxgig-fireside-podcast'
}
There are more properties, but we'll come back to those later. Also, I'm a big fan of the Three Virtues: Laziness, Impatience, and Hubris. The string "aim" is three characters and one syllable, whereas "service" is seven characters and two syllables. Also "service" is a term that is going to be horrendously overloaded and over-used in any codebase. Using short non-technical Anglo-Saxon words to stand for project-specific concepts is a great way to reduce overall confusion in a large code base that you will have to maintain for a long time. My favorite programming tool has always been a thesaurus.
I'm completely ignoring, for now, all questions about how this message, which is a JSON document, gets routed to the ingest microservice. But let's look at the two main ways that we can send this message.
In code, you can send this message (perhaps as a result of a new user filling out a form with their RSS feed URL) by using the Seneca Framework microservices library:
const result = await seneca.post({
aim: 'ingest',
subscribe: 'podcast',
feed: 'https://feeds.resonaterecordings.com/voxgig-fireside-podcast'
})
Oh wait, we forgot to be lazy. Let's try that again:
const result = await seneca.post('aim:ingest,subscribe:podcast',
feed: 'https://feeds.resonaterecordings.com/voxgig-fireside-podcast'
})
Notice that there is no indication of how this message is transported to the ingest service. No HTTP calling code, no message topic, nothing. That's important. Because the shortest path to the dreaded distributed monolith is not to use a message abstraction layer.
The other way to send this message, which you'll see quite a bit in this series, is to use a REPL:
$ npm run repl-dev
> @voxgig/podmind-backend@0.3.3 repl-dev
> seneca-repl aws://lambda/pdm01-backend01-dev-monitor?region=us-east-1
Connected to Seneca: {
version: '3.34.1',
id: '6ejf/monitor-pdm01-dev@0.3.3',
when: 1708685110275
}
6ejf/monitor-pdm01-dev@0.3.3> aim:ingest,subscribe:podcast,doIngest:false,feed:'https://feeds.resonaterecordings.com/voxgig-fireside-podcast'
{
ok: true,
why: '',
batch: 'B2024022310460874',
mark: 'Ml032jm',
feed: 'https://feeds.resonaterecordings.com/voxgig-fireside-podcast',
doUpdate: false,
doIngest: false,
doAudio: false,
doTranscribe: false,
episodeStart: 0,
episodeEnd: -1,
chunkEnd: -1,
podcast: Entity {
'entity$': '-/pdm/podcast',
feed: 'https://feeds.resonaterecordings.com/voxgig-fireside-podcast',
t_mh: 2024020722371877,
t_m: 1707345438776,
t_ch: 2024012801013692,
t_c: 1706403696926,
id: 'pdfxc5',
batch: 'B2024020722371839',
title: 'Fireside with Voxgig',
desc: '\n' +
' This DevRel focused podcast allows entrepreneur, author and coder Richard Rodger introduce you to interesting leaders and experienced professionals in the tech community. Richard and his guests chat not just about their current work or latest trend, but also about their experiences, good and bad, throughout their career. DevRel requires so many different skills and you can come to it from so many routes, that this podcast has featured conference creators, entrepreneurs, open source maintainers, developer advocates and community managers. Join us to learn about just how varied DevRel can be and get ideas to expand your work, impact and community.\n' +
' '
}
}
I literally just REPL'd into my live system and sent that message so I could cut and paste that example for you.
That doIngest parameter lets me turn off ingestion. No point ingesting a podcast I'm already up-to-date with, just for the sake of an example. The message action, implemented in the ingest service, is synchronous, so I get a response back with the details of the podcast stored in my database.
I can trigger almost any behavior in my system, at any point in the ingestion pipeline, using the REPL. I can do this live on AWS, or I can do it locally (with hot-reloading), so the debugging experience is just lovely.
In the next post, we'll look at the implementation of the aim:ingest,subscribe:podcast message action in detail. That will set us up to understand the other messages that focus more on the RAG implementation.
But you don't have to wait for me: ingest source code on github.
Top comments (0)