Nevo David

for Gitroom

Posted on May 25, 2023 • Edited on Aug 22, 2023

🎤 Building the biggest list of useful open-source libraries 🎉🥳😎

#webdev #programming #javascript #tutorial

TL;DR

Every day we see "awesome" GitHub lists. They are great.
The main problem is that it sometimes needs to be updated.
Maintainers need more time, or they leave the company.
It also needs to reflect usefulness.

I have decided to build a list that will do two things

Help new libraries get some traction.
Let people vote for their favorite library.

Before we start

I am super happy to tell you how I build it.
But first, if you have a GitHub repository, please add it to our website here.
You can later be able to find it here:
https://github.com/github-20k/awesome-opensource

I wanted to save money 🤑 💰

I have the technicality level of building my own CMS and hosting it on AWS - taking care of the scaling and CI / CD. But it would take me a ton of time and also money.

So I bought a $2 monthly WordPress host from DreamHost.

Don't fall off your chair - yes, it's slow, and yes, it won't be able to survive significant traffic. So I used it + cached almost every possible result on the website (including the search).

The good thing about WordPress is that you can use their API out of the box with GET - /wp-json/wp/v2/posts; also, adding an advanced custom field plugin reflects automatically on the results.

Limitation 🙅🏻‍♂️

Slow; everything must be cached.
Impossible to perform "search."
Long build times (everything is cached).
No real-time updates.
Almost impossible to implement infinite scroll (must be pagination based)

NextJS to the rescue

I picked NextJS. They have a great SSG mechanism for caching all the pages. I didn't use the new app router approach, I don't find it playing nicely, but I did do a tutorial about.

I have cached everything 🤣
We have seven routes

Trending list (/page/1)
New List (/new/1)
Categories list (/productivity/1)
Languages list (/language/javascript/1)
Categories + Languages list (/language/javascript/category/productivity/1)
Site information
General search (I will explain how I implemented it)
And generate tweets (we will talk about it later)

You can probably imagine that the "Categories + Languages" list is the biggest, and it's true; if I added one more filter like that, I would have thousands of pages.

Vercel free plan, play nice when you don't do background processing.

But if you do need background processes, you will experience a problem with the API.

Serverless Function - Free version 10 seconds limit, Pro version 60 seconds.

Why do I need an API if everything is cached

Thank you, Nevo, for asking.

I wanted to let people add their own libraries.

For that, I need to do multiple things:

Use GitHub GraphQL to fetch the repository information - avatar, owner, and home page URL.
Use ChatGPT 😉 on the README.md file to summarize it.
Insert everything into WordPress.

You can probably guess that it takes time to use ChatGPT for long texts.

So I decided to do two things:

Do the first job (GitHub GraphQL) sync with one of the /API paths. (around 10 seconds)
Use Serverlessq to push all the information into a queue running another API route. It will retry it three times in case it fails (around 40 seconds)

The upvote mechanism 👆🏻 ⬆️

I save the number of upvotes over WordPress; it's a simple API request that does +1 on the record.
I know - it's an atomic action, there is no queue, and it can cause a deadlock. When I get to the time, there is massive traffic to the website. I will move it to a queue.
For now, I have a more significant concern.

How do I show the new amount of upvotes without re-rendering the page?

I started to try complicated caching stuff and API. But then it hit me - Supabase.

Supabase

I did genius stuff - still proud of myself.
I created a new table on Supabase called "votes" and added the date the vote was counted.
I also added an additional props in NextJS getStaticProps called date - and set there the current date.

So every time I have a new cache build, the page gets the new date. But I just use the old date if there is no new build.

All that was left to do was to ask Supabase - to give me all votes after this date.

Then I just show Cached votes + Supabase votes.

And when there is a new build, I would probably get from Supabase that there are 0 recent votes.

Get tweeted!

I have lately seen many tech influencers making lists of the ten best libraries; I thought that I could help influencers and library owners by letting influencers build lists on the website and tweet them.