Motivation
Firestore Native, the document based interface is much easier to work with than the Datastore mode when working with hierarchical data. That is your data is not table/record based, but more graph like. But it does not support custom marshaling like the Datastore mode does. But I have a work around that I implemented.
Solution
I added MarshalMap and UnmarshalMap functions that mimic the contract that MarshalJson and UnmarshalJson interfaces support. They could be interfaces, but that would entail a lot of duplicate code because all the implementations would be exactly the same as these two functions for every type.
Instead of returning and consuming []byte they return and consume map[string]interface{}, or technically map[string]string (document) and map[string]interface{} (nested document or array) only in practice.
Firestore Native has a very novel idea of "Collections" which are much better than nested arrays. But that is another article.
The way it works is I Marshal my custom struct to JSON, then Marshal them to a map[string]interface{}, but since all the fields in the JSON are of string type, it ends up being map[string]string for structs and map[string]interface{} for nested structs.
I looked at using the structs and mapstructure packages and they worked, but after a few iterations I discovered I did not need the power or the complexity of what they provide and ended up not using them.
It should be trivial to support something like this natively in the API.
I know my solution is a naïve solution that is not very efficient in time or space on the machine, but it is time efficient when writing the code.
This gives me very granular control over how the data is represented in Documents and how to convert it back to a type safe struct thru the JSON.
Making every value field a string, even the number and timestamp gives me more control over my data and makes it easier to read in the console when debugging. Boolean types are the only thing I do not think needs to be a string but I usually store them as string anyway for consistency and semantics. yes/no, on/off, enabled/disabled, etc is always more explicit and informative than true/false semantically.
This is only a first version of this, when I get my application launched and I can get real world datasets and metrics I plan on trying to optimize this as needed.
Here is the code
Imports for the following code snippets, I hate when people leave out the imports and leave you guessing what modules they are actually using.
import (
"encoding/json"
"github.com/rs/zerolog/log"
)
This is just a Must wrapper around json.Marshal() because I want to make sure all the data is correct at runtime. Yeah, I love Erlang, hate me if you want, this works!
func MustMarshalJson(o any) []byte {
bytes, err := json.Marshal(o)
if err != nil {
log.Fatal().Err(err).Msg(err.Error())
return []byte{}
}
return bytes
}
I know the use of log.Fatal() to panic the application is controversial, I like using it, I use it consistently through my application. If you do not like it then do not use it. I do not like having to put a return below it every time because the compiler does not realize that line will never be reached.
This is just a Must wrapper around json.Unmarshal() because I want to make sure all the data is correct at runtime. Yeah, I love Erlang, hate me if you want, this works!
func MustUnMarshalJson(bytes []byte, o any) {
err := json.Unmarshal(bytes, o)
if err != nil {
log.Fatal().Err(err).Msg(err.Error())
return
}
return
}
This takes a struct that has the appropriate json:"" tags or custom MarshalJson/UnmarshalJson interface implementations and converts it to a map[string]interface{}
This is implicitly an Must style interface, it works or it fails and stops the world. Thank you Joe Armstrong!
func MarshallMap[T any](o T) map[string]interface{} {
m := make(map[string]interface{})
MustUnMarshalJson(MustMarshalJson(o), &m)
return m
}
This takes a map, converts it to JSON then converts the JSON to a struct.
This is explicitly a Must style interface, it works or it fails and stops the world. There is no way to recover from this, you can either do it or not do it, there is no try.
func UnmarshallMap[T any](m map[string]interface{}, o T) {
MustUnMarshalJson(MustMarshalJson(m), o)
}
And here is how you use this, I included the boilerplate client creation for completeness, I hate when people leave out stuff that makes an example incomplete and leaves you guessing what the setup is actually doing.
Here is what a Write to a Document looks like.
func Create(ctx context.Context, a *Account) error {
client, err := firestore.NewClient(ctx, os.Getenv("GOOGLE_CLOUD_PROJECT"))
if err != nil {
log.Fatal().Err(err).Msg(err.Error())
}
defer func(client *firestore.Client) {
err := client.Close()
if err != nil {
log.Err(err).Msg(err.Error())
}
}(client)
docRef := client.Doc(fmt.Sprintf("account/%s", a.Id))
m := server.MarshallMap[*Account](a)
_, err = docRef.Create(ctx, m)
if err != nil {
return err
}
return nil
}
Here is what a Read from a Document looks like.
func Get(ctx context.Context, id string) (*Account, error) {
client, err := firestore.NewClient(ctx, os.Getenv("GOOGLE_CLOUD_PROJECT"))
if err != nil {
log.Fatal().Err(err).Msg(err.Error())
}
defer func(client *firestore.Client) {
err := client.Close()
if err != nil {
log.Err(err).Msg(err.Error())
}
}(client)
docRef := client.Doc(fmt.Sprintf("account/%s", id))
docSnapshot, err := docRef.Get(ctx)
if err != nil {
return nil, err
}
a := Account{}
server.UnmarshallMap[*Account](docSnapshot.Data(), &a) // << this is the magic part
return &a, nil
}
The great thing about this is you can Marshal/Unmarshal an entire document tree for free.
Here are the struct definitions I am using, this is real code from a real project I am working on.
type Account struct {
Id string `json:"id"`
Name string `json:"name"`
Email string `json:"email"`
Photo string `json:"photo"`
EmailVerified timestamp.Timestamp `json:"email_verified"`
Created timestamp.Timestamp `json:"created"`
LastUpdated timestamp.Timestamp `json:"last_updated"`
}
and the custom Timestamp I like to use for better control over the representation of the data.
type Timestamp struct {
t time.Time
}
func (ts Timestamp) String() string {
bytes, _ := ts.MarshalText()
return string(bytes)
}
func (ts Timestamp) MarshalText() (text []byte, err error) {
return []byte(ts.t.UTC().Format(time.RFC3339Nano)), nil
}
func (ts *Timestamp) UnmarshalText(b []byte) error {
t, err := time.Parse(time.RFC3339Nano, string(b))
if err != nil {
return err
}
ts.t = t.UTC()
return nil
}
func (ts Timestamp) MarshalJSON() ([]byte, error) {
return ts.MarshalText()
}
func (ts *Timestamp) UnmarshallJSON(b []byte) error {
return ts.UnmarshalText(b)
}
func (ts Timestamp) MarshalBinary() (data []byte, err error) {
return ts.MarshalText()
}
func (ts *Timestamp) UnmarshalBinary(b []byte) error {
return ts.UnmarshalText(b)
}
I originally had the time.Time as an anonymous composition, but I decided I did not want my Timestamp to be implicitly convertible to time.Time as it was confusing serialization in other modules. So I gave it a private name t and have functions that convert/adapt to and from time.Time for compatibility with other libraries. Explicit is better than Implicit, I love Python philosophy more than the language.
func From(t time.Time) Timestamp {
return Timestamp{
t: t.UTC(),
}
}
func To(ts Timestamp) time.Time {
return ts.t
}
You should ALWAYS store and transport timestamp data in UTC the only time you should ever be using local time zones is when you are displaying something to the user.
All calculations should be done on UTC values and only converted to local timezones for display.
Top comments (0)