Solving life problems with JavaScript

A story of how, how I’m a big dumb dummy, but JavaScript is good.

(originally posted on Medium)

TL;DR: 💯

I didn’t make an important appointment, and there were none available for next month… So I used JavaScript to find one within a few days! I used Chrome DevTools to figure out how the appointment booking website work, and wrote some JavaScript to email me when appointments became available!

Surprise, surprise. I messed up. 🤪

I recently moved from New Zealand 🇳🇿 to Sweden 🇸🇪, and to be honest with you, it’s been pretty straightforward! I was lucky enough to get a role where a relocation agency organised almost my entire relocation. I had to fill out a few forms and pack my stuff. In fact, so much of it was handled for me that I didn’t do my one super important task— book a biometrics appointment at Migrationsverket in Stockholm. This appointment is a critical step in the migration process that would allow me to get a personnummer. That number would allow me to open a Swedish bank account, and start getting paid! People book these appointments weeks and weeks in advance, so it wasn’t like I could walk up and get it sorted.

To make this a bit more embarrassing for me, my girlfriend had been trying to find a time for her own appointment. When she asked me if I needed to do the same, I assured her (wrongly 😅) that the relocation agency would sort out mine. She had been having trouble finding an appointment on the website. We thought the website didn’t work, but in reality, the appointments are very scarce. They get snapped up as soon as they become available!

I wanted to get it sorted out as soon as possible (cause money 💰)! I checked if I could get an appointment in another city, or if I could travel to another country. My relocation advisor reassured me that people often cancel their appointments, so I should keep checking. There are 14 different appointment locations in Sweden, and it takes six clicks to get the list of available times! That means it takes a bit of time to go through and look at all the options. I’m way too lazy for that, and scared that I would miss out cause I was looking at the wrong place at the wrong time.

I can automate that! 🤖

Luckily for me, I write code in the same way that I live my life — badly. I tend to persevere when it comes to figuring out how things work. I’ve had to build up a set of tools for getting myself out of situations like this one. I was pretty confident that I’d be able to reverse engineer the website, and automate the process. I considered using Puppeteer to drive a browser and fill out the forms, but I thought there might be a simpler way. I decided to use HTTP requests, to get the data from the Migrationsverket website, and to send myself an email whenever a new appointment became available.

Figuring this out seemed like it would be quite straightforward, but turned out to be far from it. Let’s look through the steps I took to figure out what was going on.

Step 1) Where does the data come from?

First, I used the Network tab, so that I could watch what requests the browser was making as I used it. I navigated to the website, filled out the form, and clicked the calendar to look at each week. The Network tab let me watch the requests as they happened. Each click triggered a request to “BehaviorListener.1-form-kalendar":

When I looked into each individual response, I saw the JSON data for the list of appointments coming back! Woohoo 🎉!

I thought that I would be able to make a request to that URL and be done 😎… That looked something like this:

Alas, if only it was that easy. When I ran the above code, I got back a chunk of HTML, containing the following 😔:

“The requested page has expired. To countinue booking try to go back using the navigation buttons in your web browser.”

(Yes, “countinue” is in the real response)

Step 2) How do I get the real data?

I knew where the data was coming from, but I couldn’t actually get it. So what was going on? I had a few clues. The response mentioned “expired”. This implied that there is some state involved. That hinted to me that there could be cookies involved, and I knew I could confirm this quite easily. By using the “copy as cURL” tool in DevTools, I could replay the request including all the cookies and see what I got:

When I replayed the whole request including the cookies and other headers, I got back the expected response!

Step 3) How do I get the data for different locations?

The next issue was that the original request URL didn’t contain an actual “query”. It has the start and end dates, but what about the location? I needed more information.

I went through the appointment process several more times, looking for clues. I noticed that no matter what location I selected, I always ended up with a URL in the following format:

As I fiddled around a bit, I discovered that the last query parameter didn’t seem to matter . It was just the current timestamp. Of the remaining params, start and end are important, but they didn’t give any new information. The first parameter is interesting though — it changed with each new page request. Where was it coming from? The server must generate it, but how?

The next thing I tried was to look at the Network tab again, this time looking at the bigger picture. What requests does the site make throughout the whole process? I turned on the “Preserve log” and “Disable cache” options, so that I could see requests across the flow:

The short answer is that there are lots of requests. Unfortunately, I couldn’t find out anything useful here, so I had to try something else…

My next idea 💡 was to disable JavaScript. You can do that by going to the DevTools settings (the three vertical dots for “Customize and control DevTools”, followed by “settings”), and clicking the “Disable JavaScript” checkbox:

Now when I tried to go through the process again, clicking the “Continue” button didn’t work! Google Translate was also broken 🤪! That verified that the form submit was being handled by JavaScript! Now I needed to explore and see exactly what it did. I used the “Inspect Element” tool to get a better look:

Inspecting the element revealed that it had an id, which meant I had another clue! I used the DevTools “Search” to look through the whole site for any references to that id:

The search revealed the there is a click handler attached to this button. That particular code looked like this:

I added a breakpoint to see where the real redirect would go to:

And now I had more clues! A new URL, with a new page, and new query parameters. What happens if we make a request to that URL?

This time I got a real response 🥳:

Du har valt att boka tid för 2 personer för att lämna fingeravtryck och bli fotograferad hos NATIONELLT SERVICECENTER 1 STOCKHOLM

or

You have chosen to book time for 2 people to leave fingerprints and be photographed at NATIONAL SERVICE CENTER 1 STOCKHOLM

The response also contained the URL for “BehaviorListener.1-form-kalendar". It even included the query parameter with the changing number values. It was looking pretty good!

I went through the appointment process with each of the different locations. This allowed me to deduce that the enhet query parameter controlled the location. I built up a little mapping for the different possible values:

At this point I had three crucial bits of information:

1. The format of the URL to request the generated calendar page. Requests to this URL also return the necessary session cookies.

2. The format of the URL to request the JSON data for the appointments.

3. The list of different locations for the enhet parameter.

Step 4) How do I put this all together?

My little node.js script became a bit more complex. It creates a first request based off the location, then looks through the response to find the URL for IBehaviorListener. It then takes the random numbers from that, and then creates a second request based on the random number. The response is a set of appointments!

There’s a few little “tricks” here that make it work:

1. By including { jar: true }, it tells the request module to store cookies. That means that the second request doesn’t give us the “expired” message.

   1. A tiny little Regular Expression to find the right data in the body of the first request. This is particularly brittle, and suspect to breaking if their code changes (😢).

Step 5) How to get it running automatically?

I already had the basic functionality in place. All I needed to do was to make it loop over each of the locations and check if there were new appointments available. This involved some gross code (🤢), but hey, it worked!

Every five seconds, the script would make a request to the next location, and store the list of appointments in the ALL_APPOINTMENTS object. Unfortunately, the appointments endpoint sometimes returns nothing, even though are available appointments. The data eventually shows up, so we have another timeout that waits a few minutes before the script starts sending emails. It’s unfortunate that this is necessary, but this actually explains why my girlfriend was having such a rough time with the website! After two minutes, the information should be stable, and any “new” appointments should actually be new. This means we can send an email notification.

Step 6) How do we send an email?

Thanks to the wonderful node.js ecosystem, this step was actually easy! I used nodemailer, and it worked wonderfully:

This sends emails to me from me, with a little alias (+biometrics-appointment) so I could filter them. The SENDER_EMAIL and SENDER_PASSWORD get set as environment variables. `SENDER_PASSWORD had to be a generated “app password” to get around my two factor.

This meant that each night, I’d receive quite a few emails 😅:

The whole thing:

And putting it all together, the whole script looked like this:

This 123 lines of code meant that instead of waiting almost a month for an appointment, I had one booked in Stockholm within three days. I could also find an appointment for my girlfriend on a date that coincided with her visiting me. All in all, it was a great success!

Wrapping up:

I made it to my appointment! I was a little bit late (of course 🙄), but it went fine, and now I have a biometrics card, and a bank account. JavaScript can’t solve most of my day to day mishaps, but it sure is nice when it can! DevTools is very useful, and the tricks we use to debug our own sites can also be used to figure out how other websites work 🚧.

Anyways, this was kind of dumb, but also kind of fun, and I hope you found it entertaining/useful. Please reach out to me on Twitter (@phenomnominal) and let me know what you think!

🦄