If you're writing backend code, you should always assume that your service is going to crash. 💥

This was something I learned when I was designing reliable code that was performing (effectively) a transaction using Megastore. This service needed to be live, always: our failure cases were that it crashed, or we were upgrading the binary.

This is fairly understood for webservers, where e.g. a Node server can be shut down only by Ctrl-C, or a Go server can only "fail":

func main() {
  http.Handle("/foo", fooHandler)
  log.Fatal(http.ListenAndServe(":8080", nil))
}

But it's less clear what a crash means while performing some operation, like database or file operations. When can your server crash awkwardly?

• when you're writing a file to disk over a previous file: for a time, neither file will exist

  • ... solve with writing a temporary file and replacing the previous one

• updating an aggregate count (e.g., when you insert a row, add one to the 'total'): if the second operation fails

  • ... solve with database transactions

• marking an email as being sent: what if the database fails?

  • ... sending a second email is probably fine ¯\_(ツ)_/¯

In all these cases, it pays to code defensively. This isn't a long post, but just something to think about: at what point could the power 🔌 be unplugged from your provider? What's the risk?

Thanks!

19 👋