NickBlow

Posted on Jun 20, 2019

Familiar day #3 - The Revenge of the JavaScript

For people who are new, I'm blogging along as I'm building my new tabletop roleplaying assistant.

Where are we at?

I've set some goals. I've created some terraform configuration and a Google Cloud account. I deployed a Compute Engine instance that does nothing.

What are we doing today?

We're going to create some storage - we're going to set up a Terraform Backend. These are great for a few reasons, avoiding local configuration, avoiding storing sensitive values on disk (or accidentally committing them to Git 😱) and allowing collaboration with multiple people. It needs a Google Cloud Storage bucket to work, so we're going to create that with the CLI - we could set it up with terraform too, but we don't want to accidentally destroy it when tearing down resources.

Then, we're going to hopefully start some coding, and create a server that actually returns a response. I might get around to deploying it, so let's see.

We have to use the gsutil tool that came with the Google Cloud SDK we installed earlier to create Cloud Storage. You can find instructions to follow along here.

⚠️Warning: you get free REGIONAL storage in US regions only⚠️.

The code we're going to run is:
gsutil mb -p [PROJECT_ID] -c regional -l [REGION] -b on gs://[BUCKET_NAME]/

-b means bucket policy only, which reading the documentation seems to mean that you can't modify individual objects' Access Control Lists, which as our Terraform backend will contain sensitive data seems to be what we're after - we can lock down the bucket.

You can now see the bucket has been created in the console.

Now, I'm going to update my terraform configuration to set this bucket as the backend. I'm going to add these lines to my variables.tf file I set up last time.

variable "backend_bucket_name" {
  type = string
  default = "MY_BUCKET_NAME"
}

I still haven't decided if I want to commit this file, as it contains project and bucket names. I will probably write a quick script that interactively sets it up for users later.

Let's add the backend to our main.tf file

terraform {
  backend "gcs" {
    bucket  = var.backend_bucket_name
    prefix  = "terraform/state"
    credentials = var.service_account_location

  }
}

Note: the syntax highlighting doesn't work because they changed how expressions work in Terraform 0.12. It compiles correctly, I promise!

So all we need to do is run terraform init. Or is it - oops, I get this error:

Initializing the backend...

Error: Variables not allowed

  on main.tf line 10, in terraform:
  10:     bucket  = var.backend_bucket_name

Variables may not be used here.

Aaagh. I want to keep the terraform files generalizable. Perhaps this needs to be another thing that I have a script automatically generate for people who want to deploy by themselves! I'm keeping my project id and bucket name secret and out of version control.

I added it to another file, backend.tf, where I added my configuration, which looks a little like this

terraform {
  backend "gcs" {
    bucket  = "MY_BUCKET"
    prefix  = "terraform/state"
    credentials = "~/.gcp/terraform-sa.json"
  }
}

Terraform init now gives me the promising

Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "gcs" backend. No existing state was found in the newly
  configured "gcs" backend. Do you want to copy this state to the new "gcs"
  backend? Enter "yes" to copy and "no" to start with an empty state.

I said yes, because why not, re-ran terraform plan, and voila

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Perfect! All our state is now stored inside Google Cloud. I removed my local .tfstate and re-ran terraform plan once more to check. All good.

Checking on the Google Cloud Console:

Awesome.

A note on why I started with Terraform

I know it's a little boring, but I'm starting as I mean to go on. I don't want to create any infrastructure I can't reproduce. Even with this blog, I don't want to have to go back and work out what I did to deploy X bit of software. It's also good practice to have infrastructure as code for everything, and allows your infrastructure to be version controlled and changed in an auditable way.

What now?

Well, I could set up some more resources (maybe Cloud Firestore), but I don't really know what I need right now. Let's start writing some code and get things working locally, then work out what we'll need.

I've set up a list of my planned features on GitHub, reproduced below for posterity. They will probably change.

Real time chat
Real time shared virtual tabletop (VTT)
Pluggable rules engine
In game news and campaign journal
Character sheets with programmable macros
Push notifications
Customisable random generators. Automatic integration into other components TBD.
Map building tools (TBD)
Importing stat-blocks with computer vision (TBD)
Developer API (TBD)

Some of these are obviously more aspirational than the others, but at a high level I have a good vision of what I want to build.

If I wanted to do things truly lean, which I am a fan of, I'd focus on building the smallest possible unit of value, get it out to users ASAP, refine my ideas and iterate. I'm still going to do it somewhat lean, but I'm building a lot of this for myself, which means that I'm the user. So to some extent, if I think it's good, it is good. If other people like it, great.

Having a look at the features, there's a few things that jump out. The real time aspect is interesting from an engineering point of view, as I'm currently working on a Go library capable of handling GraphQL subscriptions over a HTTP/2 Server Sent Events transport. I'll probably open source it separately and then use it in this project.

The virtual tabletop is going to be an enormous undertaking, so we'll leave that for later. The same goes for some of the more speculative features (the ones I've marked as TBD).

The other thing that's interesting is the in game news and campaign journal. I see this as a pseudo-blogging platform, but with lasers and dragons. It also feels like the most complete 'product' by itself. There are a million chat apps, the rules engine and character sheets are cool but need the virtual tabletop to be complete.

Some problems

First off, I'm going to use LitElement. All major browsers support shadow DOM, it's fast, it builds off native web functionality (namely WebComponents). The other option I'd take seriously is React, but I'm betting on WebComponents in the long run. Unlike the Polymer Library, which often felt like there was a 'Polymer' way of doing things, LitElement feels much more interoperable with vanilla HTML5 and vanilla Javascript.

However, not all is roses in LitElement (although, the same can be said of React). A lot of this is to do with some buggy implementations of ShadowDOM in various browsers, standards that are not quite ready yet. It also doesn't work in JSDOM, which makes it a little painful to test. The other issue is a lack of server side rendering. There's a library that does server side rendering for LitHTML, which is the rendering library that underpins LitElement.

This can cause issues with CDNing (assuming we can get it cheaply, free was one of the goals after all!), but stuff like rehydrating content is non-trivial, and we can cache all the javascript really well anyway. It's really fast too.

The worst issue

Obviously we need a text editor for our campaign journal feature.

The biggest issue is the getSelectionAPI in shadow DOM. It's poorly supported across browsers, and doesn't work at all in Safari. There's a new api getComposedRange, which at time of writing is still not finalized. Obviously working out selections is a really important feature of text editors, so most of them don't work out of the box. There's a polyfill that doesn't really work exactly the same way, and modern editors who use many features of the Selection object also fail, as it only polyfills part of the Selection object.

I've seen this polyfill work in QuillJS as part of the Vaadin Rich Text Editor component, but the exact details of their implementation is minified and therefore obfuscated, so I don't know how complicated it will be to get it working. The other option is passing it is as a slot into our WebComponents, which means that it's going to be rendered as the element's light DOM. It does however then need to be passed down all the way from the top level. We could also render it in an iframe, but that has its own set of issues.

Additionally, we could mix-and-match technologies, and render the campaign builder pages with just HTML/CSS (potentially via LitHTML), or even React. This means we have to be very careful about our CSS, but we're going to control the entire app, we don't need to necessarily make a fully isolated text editor. To start with, we'll try the slot method, and if that doesn't work then we can revisit this decision.

Let's write some code.

I've rambled enough. I'll create a 'hello world' Lit Element page, get it to render, then call it a day.

I've initialised an npm package, and run npm install lit-element --save. We don't need the web components polyfill as we're not supporting old Edge (if you're on Edge, grab the Chromium based Edge Insider).

We're going to try ParcelJS, but we might have to switch to webpack later. I created a 'frontend:dev' script parcel frontend/index.html in the package.json, and I created index.html in the frontend folder for now (I might have to change that when I decide how to serve it). Eventually I might have multiple package.json files in the project and maybe manage it with LernaJS, but let's see how unwieldy it gets.

The content of index.html

<html>
<body>
  <script src="./js/app.js"></script>
</body>
</html>

And in app.js for now

console.log('hello world')

I'm just going to quickly check parcel works by running npm run frontend:dev

Awesome. That's easy though. Can it run our WebComponent? I'm just going to grab the example from the LitElement home page. It has fancy things like Javascript Decorators and tagged template literals. We'll switch it to TypeScript later.

import { LitElement, html, property, customElement } from 'lit-element';

@customElement('simple-greeting')
export class SimpleGreeting extends LitElement {
  @property() name = 'World';

  render() {
    return html`<p>Hello, ${this.name}!</p>`;
  }
}

This example has a few things going on. It's using LitElement as a base component, which at some point extends the base HTMLElement, making it a WebComponent. It's using decorators to define a property (something you can get and set on the element), and to define the element tag name (simple-greeting). The render function used a tagged template literal, so the string template will be passed to the LitElement html function. The render function itself handles doing the minimal changes to the DOM, and updating when a property (in this example name) changes.

Copy pasting it into VSCode is already giving us an error, saying we need to enable it in our jsconfig or tsconfig. Let's ignore that for one second and see if parcel handles it.

We need to update index.html to include our custom tag to test it.

<html>
<body>
    <simple-greeting><simple-greeting/>
  <script src="./js/app.js"></script>
</body>
</html>

Running frontend:build again doesn't work:


Server running at http://localhost:1234 
🚨  /Users/nickblow/Personal/familiar/frontend/js/app.js:3:0: Support for the experimental syntax 'decorators-legacy' isn't currently enabled (3:1):

Oh dear. Let's add a tsconfig, and see if that fixes it. Worst case we'll move to WebPack.

🕒🕒🕒30 minutes later🕒🕒🕒

Aaaand I'm reminded why I hate the Javascript ecosystem again. I ended up changing so much configuration I may as well have been using WebPack, so that'll be the next task. I ran into this issue, and the only fix was to add last 1 Chrome versions to my browserslistrc (for browserslist). That's awful. I will fix this with WebPack properly later. We want to keep ES6, but we also want to support a few more browsers. Open WC has a whole bunch of good advice, examples and defaults which we'll use.

In addition I added (a lot of this was trial and error and I need to clean it up):

tsconfig.json

{
    "compilerOptions": {
        "module": "es6",
        "noImplicitAny": true,
        "removeComments": true,
        "preserveConstEnums": true,
        "allowJs": true,
        "allowSyntheticDefaultImports": true,
        "sourceMap": true,
        "moduleResolution": "node",
        "target": "es2015",
        "experimentalDecorators": true,
    },
    "include": [
        "frontend/**/*"
    ],
    "exclude": [
        "node_modules",
        "**/*.spec.ts"
    ]
}