While creating a roguelike game in golang I got to the point of needing to save the objects. Naturally my first thought was to use a database, and of course I didn't want to use something heavy and external, but rather light and embeddable, so I chose storm, a toolkit for bbolt. bbolt is a key/value datastore that writes to disk. But when one Screen's worth of objects is a few hundred MB, and you have 240 Screens in the world, well... the query results tend to get mangled after saving just 2-3 Screens. I had very weird results wherein the last 12% to 40% of the results from one Screen query weren't being processed correctly. Combine this with the query taking a few seconds due to the DB easily growing by 300MB for each Screen added, and it becomes apparent that a different method is needed.
Enter the flat-file database. Instead of one giant datastore, I decided that each Screen would be a named folder, and inside there would be a separate file for each object. This yielded much better speeds for the program. For this method I used scribble. This project uses JSON as the file structure, but I don't need the files to be human-readable so I modified it to use encoding/binary. There is a library that was created to be able to use encoding/binary as a drop-in replacement of encoding/json by implementing the Marshaler and Unmarshaler interfaces: github.com/kelindar/binary
By using encoding/binary instead of encoding/json the file sizes are reduced and there's also a bit of a performance increase.
Obviously I need to tune the program to not use so much space, but I like to test things under a high load to see how it performs, and in this case it paid off.
My version of Scribble can be found here: