Brought to you by engine.so - a tool to instantly create a public self-service knowledge base for your customers with Notion.
Google Firestore i...
For further actions, you may consider blocking this person and/or reporting abuse
What I like the most is the reactive database. Something changes in the database and is changed automatically if you are subscribed in the front end.
How can I get this feature without Firestore? Ty
Yep, I liked this feature too. Right now I use GraphQL subscriptions in Hasura to get this same functionality. It's pretty convenient to set them up in Hasura too, which is usually a more painful process if you write your own GraphQL server.
MongoDB has it built-in. They call it "change streams" I think.
thanks , that is what I have been looking at for a while. I was impressed by the live sync feature by metheor js, but hated the rest of that framework.
you can look into pouchdb , couchdb pouchdb.com/
46kB gzipped!! something like pouchdb-'lite' could have been more popular.
Still better than firebase though.
This is one of the main selling points for sure. When you do mobile, one of the most difficult things to handle is bad network. With this feature you don't need to worry about http codes, timeouts, retries, etc.
I think it can be easily implemented with Web Sockets, problem is that need a full time machine, I guess, but talking about SQL you can up a Socket API in same machine that you install a Postgress or some else; Most of noSQL databases, as others says, have it streams native, but is all basically a Web Socket implementation
I have been using Firebase realtime db and firestore for over 2 years now. About 6 months ago I added an amazon MySQL instance to store orders and advertising data for amazon sellers. As you can imagine a seller can have hundreds if not thousands of records a day and firestore just isn’t robust enough for data warehousing. Overall I’m pretty happy with firestore for most scenarios but it just can’t replace a sql db.
As I read your article I did get the feeling part of your frustration was due to your design of your data handling and was not necessarily issues with firestore.
I agree with you Don, I did a pretty large application that was ready for production that hit some snags with Firebase but reverted back to SQL for now. I am now evaluating using Firebase in conjunction with SQL to get the best of both worlds. I think a lot of larger applications need multiple data stores for different needs. They are just merely a tool in your tool belt and not the only tool. I am evaluating using Firebase more as my front end query store with SQL being the version of truth and for large analytics type jobs.
How do you keep everything in sync?
We don't have to sync. The data is distinct between the 3 databases we have. Let me explain...
Our app helps amazon sellers optimize their product listings and analyze their sales. Most of our data is stored in either firestore or the realtime db. Firestore was not in production when we began so a lot of our core data is in the realtime db. We want to move all of it into firestore but it is not a trivial task.
Product orders and advertising performance data is loaded every 6 hours into the sql db. A seller (our users) can have thousands of orders each day. Their advertising data can be even greater. We originally tried to use firestore for this data but it was like trying to put in a nail with a screwdriver instead of a hammer. We have a lot of analytics and this is much easier and runs faster in sql.
In summary, the order data and advertising metrics don't need synchronization to our operational data such as product listings.
Thanks that makes a lot of sense. We went down the road of moving everything to Firebase but hit the snags with needing some sql things. I feel like my pattern would be the opposite than yours, SQL first since we have a lot of legacy there and push to firebase on an api trigger or something.
My suggestion: Parse Server - parseplatform.org
1) OpenSource
2) Node.Js + Express + MongoDb or PostgreSQL
3) REST API, GraphQL or SDK / libraries for various languages / technologies
4) Allows the use of relationships between tables using Pointer or Relation. No duplicate data.
5) Another 30 query operators, including relational queries.
It has practically all the features of Firebase (since Google copied most of them there in 2016.).
"Changing a username 'should' be expensive as it is such a rare event" is a proprietary, domain decision and not for the database to decide. To state that such an event is "expensive" and "rare" imposes an artificial constraint.
NoSQL data modeling is typically driven by application-specific access patterns
I wrote "Should" based on my experience modeling in NoSQL.
With great power comes great responsibility.. with NoSQL you can have a post document containing the post, all comments and all the usernames.
If usernames are changed "too often" you should have references to users instead.
My guess is that the extra speed when loading a post from not reading all commenters user profiles is well worth the rare and expensive username change event
I think Firestore is a great database and I actually think the things you dislike about it are a big plus to it.
As far as the first point of it being propriety does not really matter to me at all. You should structure your code in a way which if you do want to change databases you can do so relatively easy just basically rewriting queries/writes to the new database form. For example, I have a small example node app on my GitHub if I wanted to change from MongoDB to some other database all I would really have to do is change this repository file. this is because I only have one table/repository so depending how many tables you have it can be a bit more work but not hard at all in my eyes.
The point about it being hard to develop/test with is also not really true in my eyes. It is pretty simple to create a development environment and production environment which uses 2 separate firebase applications (which is how it should be they should never rely on the same stuff). Firebase also provides local emulators to run these services locally on your machine to make it quite easy if you set up your application correctly. Here is the introduction to the emulator's post on firebase.
As far as pricing structure paying for only document reads/writes/deletes you do is a big plus in my eyes as you are only paying for what you use and not over-provisioning database capacity in which you are not using. Sure they do not give you any discounts for having faster queries, however, the pricing model would be much more confusing and annoying if you had to factor in query time as well.
I do not personally use Firestore that often (only really in test projects as I am deep into the AWS ecosystem). So I cannot say how annoying it might be to query data. However, I do have a ton of experience in DynamoDB and have helped a company quite recently switch off of Postgres and they saw a huge improvement not only in query speeds but in cost as well because they are no longer paying for over-provisioned databases.
I will admit trying to learn how to properly use NoSQL databases is a bit hard because you have to rethink how you model your data and it is not always the same from database to database in the NoSQL world, however, it brings some huge benefits over SQL especially in performance at scale and clustering data across multiple servers. Though of course, some cases like analytics are better on SQL where you may have an extreme variety of query patterns as you build NoSQL for the query whereas you build SQL for the data. Hence the performance gains on NoSQL, especially at scale.
This is shaping up to look like a pretty great alternative to Firebase powered by Postgres: supabase.io/
thanks for sharing your experiences.
you say that you worked for 9 months with firebase. can you estimate how much would be saved if you had used an other db to begin with? so how much time of that do you feel was wasted and is saved with this article for all of us?
Honestly firestore was probably a time-save for the first month or so of the project. It was running and hooked up to my frontend day 1 which is huge. The issue though is that the first 90% of what you want to do w/ firestore will be easy but that last 10% will be incredibly difficult. I'd say if you're building a personal project and not a business and that project has a concrete "finished" point, then that sounds like a good use-case for firestore.
If you think you'll be working on the project for more than 1-2 months, I think it would be worth your time to look elsewhere. For my project personally I started hitting walls after month 1. I think around month 2-3 is where if I had chosen another solution I would've been developing faster.
firebase is awesome, it is free for MVPs, reads are fast, client caching OMG! Their pricing policy is fair, I would not expect a communist free firestore. It is clearly optimized for mobile apps. Recently they pushed firebase emulator which allows you to emulate their stack locally. The only issue I see here is the lack of knowledge of NoSQL therefore making the wrong pick by being seduced with the pros of NoSQL. SQL has much more pain points in my opinion than firestore. BUT SQL is also great, no doubts here. The ability to discern the right choice for the problem is what makes a great developer (and human in general). All of the technologies are great, only the choice can be wrong here.
Great article pointing out all things Firebase!
My team once talked to a Firebase engineer from Google who said there are really 2 primary groups of users serious users for Firebase:
Odds are if your not in these two groups, Firebase probably isn't the best choice for your project.
If your in the 2nd group, you probably will end up migrating off the platform at some point which is ok, as long as you take that into account. Its better to build fast, get feedback and go from there than spend time and money building for a scale that you never meet and need.
The 1st group are the ones that build around the limitations of the platform, which arguably could be done with other solutions, but is done with Firebase.
One company we ran into, which was in the first group, essentially "tacked on" multiple databases onto the Real Time Database for different use-cases. So they had an SQL DB for managing/reviewing/aggregating data, but used the RTD as their "client-side" DB for example. (Firestore wasn't an option at the time)
The biggest benefit of Firebase that isn't easily replaced with something else is authentication. Sure, you have Auth0 and other auth providers, but Auth0 is super expensive. The database is easy to replace, even MongoDB is a nice easy option if you want something like Firestore.
Every time I go to switch, I go down the problematic rabbit hole of implementing auth and remember just how painful OAuth authentication truly is. The most promising option appears to be Supabase, which is eventually going to support every Firebase feature that you can host yourself.
Just on the 1st point, you can use Terraform and the firebase cli to do the automated setup of a new environment.
You can create a Google project, set up firestorm and create storage buckets via TF. Then use the firebase cli to deploy indexes & cloud functions.
Your speaking out of my heart.
Ive developing a flutter app and I started out using Firebase for everything and did most of the database work with firestore on the client side.
Problem with this approach is that if you have write critical tasks on the client side users which do not update their app will use deprecated formats and apis.
Solution was to use Firebase functions to outsource all writes to the serverside but then I had to notice how hard it is to manage a large set of firebase functions. At times we had 80. Then you have to deal with deployment and management of these functions to make them effeciently bc. functions have cold start if not called frequently. Calling functions over http was super slow even in prod with 10k users.
Happens to be that I deployed to much files but this is another problem. The documentation is so basic and suggest so many bad practices you have to rely on 3rd party information for functions and deployment.
Long story short I am now running AppEngine with NestJS using Firebase to handle all write critical tasks and use Firestore in the frontend to read only.
Functions do background tasks likes counters and copying data.
Thats the reason why I am now thinking of moving away from Firestore because it boasts about how easy it is but in the end its just another database with less query capability then its competitors.
For almost all queries you have to create indices but there are no tools to manage indices and check wich indices you actually need so it is very hard to track for it and it is meant like this because you pay for indices.
Also you will reach the limit of firestory very early requiring you to use functions and other workarounds to generate collections because of limited query capability.
After all this hassle I might have just use SQL.
Never the less firebase still have done its part into quickstarting the project and get a working production version in 3 months so it has its place and firebase auth is still nice.
Your arguments are definitely valid. Having faced similar barriers, i.e. N+1 problem, etc., my solution was to offset these issues by also storing and coordinating data in Firebase realtime db, specifically data with high i/o such as chats, or reference data such as username/profile pic look-ups. Essentially anything that would incur high doc reads/writes.
Not saying splitting into 2 distinct db's did not add complexity or is the right choice for all, and the verdict is still out for me, but it is one solution. Since the realtime db lacks the scaling of Firestore and would eventually require sharding to scale, guess time will tell.
Points 1 and 3 might be true, but point 2 make little sense to me.
"Firestore is built to make processing cheap on the server-side. But guess what? You pay per document so whether a query takes 1ms or 100ms doesn't matter to your wallet. This means that Firestore is optimizing to make their costs cheaper. Not yours. And since you have to overfetch data and manually filter it on the client side you actually end up with a more expensive and slower query overall. This is why I moved away from Firestore. After seeing that this was their business model, it proved to me that there's no way I want to try to scale with this product."
This is a terrible argument. Obviously, making their costs cheaper also makes your costs cheaper. If they allowed queries that took longer, then don't you think they would charge more for queries?
All valid points. I thought the same things.
Have you considered Google's other DB products ? such as Cloud SQL or Cloud bigtable ?
Here they have a few use cases, with cost estimations : firebase.google.com/docs/firestore...
How do these costs compare with the alternatives (MongoDB, dynamoDB, etc...), or even with a SQL DB from another vendor) ? MongoDB tells me I have to start with USD $57/month!! Firestore estimates USD $12/month for 5,000 Daily Active Users.
Also, Google has the following product (Database Migration Service), which may help with your concerns in 1). I have yet to try this product, so I cannot comment much. Have you tried it ?
Thanks a lot for your post. I used to be comfortable with firebase, but you make a very relevant point. Thanks for your suggestions too, specially contentful.com, I've never heard about it and it seems to be a powerful tool. I'll give it a try.
I care about the price and monitoring as well.
Convince me to use SQL
Pacharapol Withayasakpunt ・ Aug 11 ・ 1 min read
It seems that NoSQL is better for denormalized, but RDBMS is better for normalized. Still, many RDBMS can indeed store and query JSON (not sure if Hasura supports that, though.)
But it seems that I can just use MongoDB first, and migrate later, if trouble comes?
BTW, what database in the backend for your GraphQL? Postgres? Also, do you write your resolvers, or use Hasura?
Lastly, is there an economical way to use NewSQL, like Spanner?
Good questions.
MongoDB is a great choice for database. It's open-source, has better support for NoSQL relationships and such than firestore, and I overall have liked working w/ it in the past. If you like it go for it!
I'm now rolling Hasura w/ Postgres as my database. I don't write resolvers I just use the CRUD operations that Hasura exposes. If I need to do something more complex I can use Hasura "Views" (which you might be interested in), so likely I won't need to write resolvers for a while (maybe ever).
Postgres has JSON storage which is a pretty great feature about it.
Never used NewSQL, sorry! I'll check it out though.
Your user/reviews example is more of a problem of poor design. With NoSQL one would store the username and profile pic together in the review document (denormalization), so no N+1 problem here. Now, on username change, you would need to update each comment made by the user. Using queries, you would only access and be charged for the resulting documents and not all of the reviews. We can then expect this upfront write cost to be quickly amortized by the many more subsequent reads: A username may change his username twice a year, but we need to retrieve the reviews on each page visit. Using SQL, we would need costly joins on each page visit.
Very good read! Thanks.
About item number 3. I find these as solved problems in MongoDB + Mongoose.
But your final decision is the best. Good engineering mate! 👍
Yep, I should've mentioned in the article but part of my gripes with firestore is they give you a NoSQL database, but none of the "foreign key" management that you can do with stuff like mongo. So it puts that weight on your shoulders to update manually, which kinda sucks.
This is really insightful, thanks for sharing
I also ran into these troubles with my last side project that used Firestore. I came across faunadb.com which is a serverless database option that I’m really keen to try on my next project
Thank you for sharing your experience. I have always been on the fence when it comes to Firebase and Firestore. It is helpful to read other people's opinions on that topic.
In many corporations there are even 3 environments on the dev company ( dev, stage, prod ). I've also seen 4 (dev , stage, client stage, client prod)
I know it's the same case of propietary DB, but have you checked FaunaDB?
Recommend our project : DB3 Network - Decentralized Firebase Firestore Alternative.
This article confirmed all of my suspicions about using Firebase as my backend. It's cool, but it is definitely for a very specific type of app and an equally specific type of developer.
Insightful read I had no idea Google Firestore was like that.
Hello, you say you now use Hasura + Postgres
But what about offline/sync with the mobile clients? Didn't you use it on Firestore?
Thanks.