In the spirit of sharing failures and learning from them, I'd like to tell you a story about a time a younger me helped create a "clever" system ...
For further actions, you may consider blocking this person and/or reporting abuse
Hey @anna, thanks for sharing your experience. With one sentence, would you recommend using a real GraphDB (e.g. Neo4j) in this case. To be honest, I'm not sure what is your advice by just reading the Good/Bad/Ugly sections :D.
I'm asking as I'm considering endeavouring the GraphDB journey soonish...
Hi Stan, thanks for your comment! I'd recommend using a real graph database over a "fake" one, in any case. Our mistake in this case wasn't really "we should've used Neo4j instead of Postgres" - our mistake was that we didn't hide the implementation details well enough to make it easy to deal with. It also turned out in the end that the business needs didn't require something as robust as a graph organizational structure.
Regardless of which technology you go with, just keep in mind that you want to make it easy to swap out later if need be.
Hi Anna, thanks for the post. It's a useful insight at the process of decision making in a team.
While I agree with the conclusion (going with the tested and tried way) I would argue that in a way you did exactly that by sticking with the techonologies you were using before and if this is not a case of "we should have actually tried plan b", in reality.
Initially you said:
which is perfectly reasonable but in the end you also said:
Couldn't this be a case of a well thought out structure (the graph idea) that had to be bent and implemented on top of something not designed for such?
I'm not saying that going the graph database route would have resulted in the perfect system, I can't know that. What I'm saying is that the team actually chose the boring technology, it might have been too boring for the business logic.
That's a really good point! I'd go one step further than boring technology, and include boring patterns. The biggest problem we had developing with the graph was that we allowed implementation details to leak out of our models into into other parts of the application. We created a codebase that felt unfamiliar and unintuitive to new developers, with pitfalls and traps that hamstrung them wherever they turned.
In our case, we thought we were setting ourselves up to try out something new - if it worked, we could just switch to a "real" graph database later on. Because we didn't do a good job of isolating the graph code in the database, we instead ended up with a hybrid Ruby/Postgres graph implementation that was near impossible to change or get rid of.
Thank you rhymes!
I feel you, you were probably drowning in self joins at the DB level and checks in the Ruby code to dance around AR. At least you now know the limitations of object relational models :-D
That's for sure! 😂💯
Thanks for your story. It was definitely interesting to read.
I do agree with your third lesson learnt that one should use the right tool for the right job.
And I tend to agree that nobody should use a shiny new technology just for the sake of using that shiny new technology. Even though in some cases it might be a good decision.
But, I think you use this idea showing that for you it was a bad idea to use a "shiny new technology". I think it would have been a good thing, but actually, you didn't introduce any new technology. If I understand well, you kept using Postgres and ActiveRecord that were not new at all to you.
On the other hand, you tried to implement an almost 300 years old concept (the one of graphs) with old tools and it didn't work out that well. Reading about the issues you had on your journey, those are the main reasons why one should consider using a native graph database.
Anyway, I don't want to judge old decisions, in hindsight many are clever. And it's always respectable to admit bad decisions, failures.
The main thing I don't agree with that you used anything new and shiny as the concept of graphs is far from new and you didn't use any new technology based on graphs.
“After some research and exploration, it was decided that we'd avoid bringing in a true graph database (like Neo4j, OrientDB, etc.). Cost was potentially an issue, and we wanted to avoid forcing our developers to learn new concepts and a new query language”
=> Sorry but this decision is so stupid in many ways I cannot stay quiet...
Anyway well written and thanks for sharing. Conclusion is gold : right tool for right use cases. Architects are like vampyrs they do not like silver bullets.
Hi Cédrick - in hindsight, yes: it would have been worth it to use an actual graph database for this project. The cost wasn't the technology itself, but the cost of deployment, hosting, and instrumentation - and at the time, the PaaS we were using had less-than-stellar support for actual graph databases. The reasons we chose to implement the graph ourselves were flawed - and the whole reason I wanted to share :) Thanks!
I would rephrase it a bit: Do I even need INSERT SHINY THING HERE? ...do I really need it NOW?
enphasis should be on building evolvable software. Descisions of today should not be implemented in a way that they are set in stone - and all the pain and fear of making THE DESCISION will go away. Implementing graph in sql is a very good idea, as you try out new data structure without overhead of leaning how to run and query niché solution. When data structure was proven to pay off, you should have used all abstraction levels a good developer places here and there to swap it for more mature and flexible tool. This is what IoC is there for, remember all your interviews? “... well with ioc in orm layer we can change database from mysql to ...”
there are ways to achieve reasonable separation of concerns - by watching at abstraction levels you have in single module (as in class) and extract something that doesnt fit )
experimenting is awesome, but we all need to learn how to do it safely )
Good article.
really nice article