Improving Access to Public Domain Audiobooks with Libridex

The Genesis of Libridex

The Libridex app was born out of a love for reading and an understanding of the challenges that many people face when trying to engage with books in today's fast-paced world. As someone who cherished reading in my youth but now struggles to find the time, I've come to appreciate the convenience of audiobooks, which transform mundane tasks into opportunities for learning and entertainment. However, classic public domain books can often be lengthy and dense, making them less enticing for modern readers. Libridex draws inspiration from music streaming platforms like Pandora and Spotify, which use algorithms to recommend songs based on user preferences. Similarly, Libridex aims to make classic literature more accessible and enjoyable by offering curated samples of public domain audiobooks, guiding users to discover different sections of these timeless works. While there's still much work ahead, the goal of Libridex remains to bridge the gap between classic literature and contemporary audiences, enriching lives through the gift of timeless stories.

The Recommendation Problem

I quickly noticed a glaring gap in the audiobook ecosystem—lackluster recommendation systems. This frustration became the driving force behind Libridex. Most audiobook platforms rely on rudimentary recommendation algorithms that often miss the mark. Users find themselves with audiobooks that don't align with their tastes, resulting in a less-than-optimal reading experience. Libridex was conceived to tackle this issue head-on by harnessing the power of machine learning to deliver tailored recommendations.

Existing Projects

There are some great existing projects focused on improving access to public domain audiobooks, including:

• LibriVox - A non-profit that relies on volunteers to record public domain texts. They offer a large free catalogue of audiobooks.
• Lit2Go - Provides free public domain audiobooks, plus text and resources for teachers. Based out of the University of South Florida.
• Loyal Books - Also offers free public domain audiobooks, focused on literature. Read by volunteers.

Building the Foundation

Every great project needs a strong foundation, and for Libridex, that meant assembling a diverse and extensive dataset. To achieve this, I embarked on an ambitious data-gathering mission, scraping data from LibriVox—a UK-based nonprofit that provides public domain audiobooks. This dataset, comprising approximately 15,000 unique URLs, would serve as the cornerstone of Libridex's recommendation engine, driven by cosine similarity. This intricate algorithm breathes life into the platform, analyzing audiobook descriptions to measure their similarity. It's a solution that conquers the challenge of repetitive terms and phrases in book descriptions, ensuring that the recommendations it serves are genuinely aligned with users' unique preferences.

The Development Process

Building Libridex required addressing both the frontend interface and the backend recommendation engine. For the frontend, I utilized React with Semantic UI for layout and styling. This provided components like the audio player, book info popups, user account management, and more.

I initially built my own user authorization system from scratch using JSON web tokens, hashing, encryption etc. However, this became complex quickly. I learned it's better to rely on a service like Auth0 that handles user auth out-of-the-box.

For the recommendation engine, I scraped over 15,000 public domain audiobook metadata records from LibriVox using Python and Selenium. To analyze these and generate recommendations, I employed a vector space modeling technique called TF-IDF (term frequency–inverse document frequency) to convert text into numeric vectors.

A challenge was that common words like "the", "and", etc skewed results. To account for this, I used cosine similarity to compare word vectors based on the angle between them rather than magnitude. This measures semantic similarity more meaningfully.

Lessons Learned

Here are some valuable lessons from this project:

1. Authentication: Don't reinvent the wheel when it comes to authentication. Instead, leverage a reliable provider like Auth0.
2. Static Analysis: Prioritize static analysis before dynamic deployment. It's essential to establish a strong foundation from the start.
3. Machine Learning Pipeline: Building a robust machine learning pipeline is crucial. It facilitates iterative development, version control, and continuous monitoring. Tools like AWS SageMaker are well-suited for this purpose.
4. Filtering Strategies: Begin with content-based filtering and then progressively refine recommendations based on user behavior over time. Consider transitioning to collaborative filtering for more personalized results.
5. AI Narration: Explore the potential of AI narration services like Anthropic and ElevenLabs to expand content accessibility rapidly and efficiently.

Potential Next Steps

If I were to continue building:

• Establish a Vector Database
• Implement a continuously-running ML pipeline with SageMaker
• Containerize services (Flask API, React frontend, etc)
• Incorporate AI narration and enhance recommendations
• Develop collaborative filtering as user data comes in

The Future is AI Narration

While the human narration provided by the above projects is great, AI voice synthesis holds enormous potential for rapidly expanding access to audiobooks. Companies like Anthropic and ElevenLabs are already creating high-quality AI narration that could be applied to public domain books. Rather than relying solely on volunteers, AI narration could help bring even more classic literature to life in audio form. Libridex may want to explore incorporating these technologies as it develops.

Conclusion

By harnessing the capabilities of modern technology, Libridex aspires to redefine how users discover and immerse themselves in the world of audiobooks. It's not a trivial project, but Libridex holds the promise of making reading more accessible, engaging, and tailored to each individual in today's digital age. Overall, Libridex aims to build on the work of existing public domain audio projects to make classic literature more fun and accessible using an AI-powered recommendation engine. There is clearly both a need and an opportunity here to open up these timeless works to new audiences.