So, first time writing anything public, ever. Hi!
I picked up a side project and thought I'd write a bit about the choices of tech, hardware, services and so on. For the moment I'm gonna keep it a secret exactly what it is I am working on but I can still talk about the broader view.
First things first, my name is Steeve, I currently live in Romania but I'm originally from Sweden, where I spent the first 33 years of my life (currently clocking in at almost 40). I'm co-founder of a startup in Sweden called Matspar where we collect data from online grocery stores (only Sweden at this moment) and help users figure out which store would be cheapest.
Now on to this new project which currently receives most of my free time where I'm not coding on Matspar. In its most basic form it's a webpage where people (with account) can upload a certain type of file and have it parsed, and then later dig around in this data. This might not sound that hard but it's a lot of data and needs to be extremely snappy, which is why I chose to write it in Golang since it's a good combination of fun to write and fast to run.
This project initially started as a small cli application for personal use and someone snowballed into this huge thing that I'm now super stoked about. Also gives me a chance to learn about some pretty cool technologies I've yet to dig my teeth in to.
The tech I've chosen for this project is as follows:
- Next.js - Looking into maybe switching to Remix since I am a bit unhappy with Next.js overall
- GraphQL (Golang) for the public facing API
- gRCP (Golang) for the internal API
- PostgreSQL for storage
- Redis for some storage but mostly for cache
- HashiCorp Consul for service discovery
- HashiCorp Nomad for docker orchestration and deployments
- Traefik for dynamic
The "hardware":
- 1 x 2950X (16-Core ThreadRipper), 128G ram, 2 x 960 NVMe SSD
I did a lot of benchmarking before finally settling on this choice. What I pay for this dedicated server over at Hetzner I get basically nothing over at AWS (which would have been my preferred choice). Just for a reference, I can parse around 35K chunks of uploads per second on my i7 laptop. I maxed out at 20K on AWS and I get a whopping 180K/s on the ThreadRipper. I have in my backlog to benchmark an i9 too since it has higher single core clock speeds and if that pans out I might switch to 3 of those machines instead which would land me around $250/mo.
The cloud services:
- GitHub for code hosting
- GitHub actions for building and deployment
- Vercel for hosting the site
- CloudFlare in front of the API, mostly because I get free SSL without hassle
- Digital Ocean Spaces for storing uploaded files and backups
- Magic for authentication, though I am NOT happy with the latency from Europe to their AWS servers in the US so I might have to find an alternative or implement passwordless login myself.
I guess that's it for now and I am sure I forgot to mention a lot of tech being used but I hope to write something about this project bi-weekly here for whoever might be interested. And hopefully you guys can come back with some feedback on improvements and new stuff I've yet to disover!
Have a great one and stay safe!
Steeve
Top comments (6)
Hey, thats's awesome! Would love to hear more about the project. I got a few questions for you:
What is it about NextJs that you don't like? Did you write the GQL api on your own? If so, why? Why not use something like Hasura Cloud? Is there any particular advantage you see in using gRPC instead of simple REST apis?
Afaik you don't need to use Cloudflare because vercel does SSL for you, you are adding extra latency there.
Here's a link: vercel.com/blog/automatic-ssl-with...
Regarding Next.JS:
I have multiple pain points, my main grief is their routing, I very much prefer the way react-router does it. I realize Remix seems to have adopted this file based routing and even added some (imho) bad conventions when it comes to loaders having to be named the same as the route. I just have the constant feeling that Next is trying working against me :-)
Regarding GQL:
I am building my own because of performance, there's a lot of combining data from redis and postgres (I host my own redis and PostgreSQL on the one server I am running).
Regarding gRPC:
It's just so extremely convenient to use gRPC to talk between microservices, plus I've wanted to dig in to that for a good while now so this was the perfect opportunity! :-)
Regarding CloudFlare:
I only run it in front of the API, which is hosted on my dedicated server (again, for performance reasons). Hosting the API on Vercel (AWS Lambda) would have added a lot of latency, plus, reduced the performance with thousands (maybe hundreds of thousands of percent!). I'm pretty sure my server will have parsed and returned the result before Lambda has even started (even on warm starts). The main site (which I host on Vercel) is set up using their SSL already!
Thanks for your input ❤️
Thanks for the answer! NextJS does file-based routing and quite a few people I've met aren't really fans of it. I wasn't too at first. Remix afaik does the same thing but provides a more granular set of tools to optimize for your use case.
If the issue is having to name your component the same as your route, what you can do is to create the component in a different directory (name it views, screens, etc.) and then import that component, and export it from the pages directory. You could use the
babel-module-resolver-pluginto help ease on the relative import paths (the ../../../ nonsense).I really like the fact that you mentioned that your server would've finished a task before a cloud function would, but then again though you have to worry about scaling, and that's quite tedious. In your particular situation do you think it is worth this potential future work?
Yeah I'm probably giving Remix a go pretty soon, hopefully there'll be some more content around it out soon to really help cement that decision. I really want to support the great work Ryan and Michael is doing for the React ecosystem.
About future scaling, it's not going to be a problem to switch over to cloud solutions in the future to meet the demand, but I also enjoy running my own servers (10+ years of sysadmin background). At my startup we are all-in on AWS so that's probably where I would set stuff up if I needed to. The performance difference per $ for going from bare metal to cloud is pretty insane though and this specific applications need to extremely fast in order to be helpful.
I was on the sysadmin team for one of the most trafficked sites in Sweden for a few years, which at that point ran on bare metal servers in three different DCs in Stockholm. That said, my dream scenario is definitely to host this on AWS at some point, but since performance per $ is extremely important in the beginning, bare metal is the way to go! :-)
Hey @snorkypie for the passwordless auth, I would recommend Did as a good alternative. Super simple to setup and “just works” and based in the EU / U.K. too.
amazing