Using Twilio to build the internet

If you’ve ever travelled internationally, you’ve probably asked yourself: “do I buy an expensive roaming data plan, do I jump from one free wifi hotspot to the next, or do I throw caution to the wind and go connection-free in an unfamiliar place?”. Going anywhere without real-time navigation isn’t an option if you’re as directionally-impaired as myself and get lost walking in a straight line. I always have to opt for the data plan which can set me back $80 for a measly 1GB. The lack of competition in the Canadian telecom industry is driving our data prices to be some of the highest in the world, and this large extra travel cost frustrated me to the point where I decided that I was going to do something about it.

As any reasonable person would do, I decided to build a browser for my phone that would transfer all of the content over SMS, while preserving the look and feel of a real browser. Since my phone plan at the time included unlimited SMS, I’d be able to use this app to get unlimited internet anywhere! I figured that this would be slow and a little old-school, and so my new project “Dial-Up” was born.

When I think SMS and code, I think Twilio. A few years back, an integration was released that let you answer surveys over SMS/voice between Twilio and FluidSurveys, the startup that I was working at (later acquired by SurveyMonkey, and I’m now back with the founders and working on Fellow.app). I thought that it was extremely cool, and so I was excited to finally get to use Twilio’s services for my own non-traditional use case!

There are two components to build for this project:

• Phone app: unlimited SMS, will act as the browser.
• Server: unlimited internet, will do all the actual webpage loading on behalf of the phone.

When I was starting this project it was intended to be a helpful tool for just myself, so I built it for Android only, in Java (there were more answers on StackOverflow about SMS for Java than Kotlin!). I built the server side of the project in Node.js, because I thought that it would be hilarious to use JavaScript on the server (where it doesn't belong), to make a JavaScript-less browser (where it's supposed to be).

The image above shows the flow of information between each service. Let’s dive in and follow the lifecycle of a request through the app:

🔗 Requesting a URL

The first thing that we'll want to do in the app is request a URL to load. The image below shows the layout of the app's home page, which provides a text box to enter the URL and a "Go" button. When the "Go" button is pressed, a few things happen:

• If the app hasn't been granted the required permissions, it will request SEND_SMS, READ_SMS, and RECEIVE_SMS.
• The URL will be shortened: https://www. will be removed since it's a given that it should exist, and any query parameters will be taken away since this app won't allow for anything fancy like that.
• The resulting URL will be sent via Android's built-in SMS API to a phone number that we own on Twilio.

☎️ Setting up Twilio

Next up, we'll need to set up the phone number we own on Twilio. I decided to use Twilio's webhooks which let me specify a URL that all SMS sent to my number should be forwarded to. I set up my webhook like this:

After saving this, sending a text message to the number I set up will send a POST request with a json payload to the specified URL containing all sorts of information about the message, such as the sender's phone number, the country it originates from, and when it was sent.

🌎 Getting the webpage and sending it over SMS

At this point we've been able to specify a URL and send it via SMS through Twilio, which will have forwarded it to our server. Let the real fun begin! 🎉

As a developer who tends to work on seemingly small components at a time in frameworks like React, it's easy to forget just how large the HTML that makes up our websites ends up being. If you View page source on your favourite single-box single-button simple-looking search engine, you'll notice that the HTML holding it together is almost a quarter of a million characters long. With SMS having a limit of 160 characters, transmitting that directly over SMS would take more than 1,300 messages!

That's not going to fly.

Even with unlimited message sending capabilities, SMS doesn't have guaranteed delivery. We'd need to be able to figure out which messages weren't received by the phone and resend them, which would add a lot of overhead to the already long time that it would take to receive that many messages at once.

My phone tends to start dropping messages as soon as it gets more than ~10 at a time, so I set a goal to get the 1,300 SMS down to 10, reducing the size by over 99%.

It was an ambitious goal, but those kinds of impossible targets and interesting problems are exactly what drew me to computer science in the first place. Hitting it would mean getting a lot more creative than just using Gzip, so I ditched all ideas around traditional compression and got to work.

Compression step 1: Goodbye JavaScript! 👋

The browser that we're building isn't going to support JavaScript, CSS, images, or anything that you wouldn't find in a website out of the 90s (animated illustrations and visitor counters aside) because of the large overhead it would add for little benefit. The first thing that we'll do after getting the HTML for the requested website is remove everything that doesn't serve an explicit purpose for our browser.

I used sanitize-html for this step, which lets you specify tags and attributes that should be kept or removed from some HTML, as plain lists or as functions of their values. Here's part of the configuration that I used:

const sanitizeHtml = require('sanitize-html');

sanitizeHtml(HTML, {
  allowedTags: ['a', 'input', 'form'],
  allowedAttributes: {
    input: ['value', 'type', 'name'],
    a: ['href']
  },
  exclusiveFilter: (f) => {
    var att = f.attribs;
    return (f.tag == 'input' && att.type == 'hidden') ||
      (f.tag == 'a' && att && (att.href == undefined || 
      att.href.indexOf('policies') > -1));
  },
});

The configuration I set up allows for only text, <a>, <input>, and <form> tags to be kept in the resulting HTML, and only value, type, name, and href attributes to stick around on those tags. I decided on this small list because I felt that in the usage I wanted to get out of this browser, those were the only ones that would provide tangible value and allow for interaction with a site. Since we're cutting out all the CSS by not allowing <style> tags, there's no need to allow class tags (the same goes for JavaScript and other related tags).

sanitize-html also allows removing elements based on a function of their tag and attribute values. Part of the exclusiveFilter that I've defined above removed all hidden elements, links to nowhere, and links to privacy policies and terms & conditions: we're never going to click on them anyway, so why waste the space?

Compression step 2: Shortening common words 📏

Once we've run the HTML through sanitize-html, we're left with a lot of text and links. A lot of languages have some very common words that show up a lot in written text, like "the" or "and" in English. Since we know there's a set of words like this, we can compress them in a deterministic way: by replacing them with single letters (that aren't "a" or "I"). If text is compressed such that the➜t, and➜n, or that➜s, both compression and decompression for these words becomes a simple "find-and-replace-all" for each pair because we know that "s" is not a valid word.
That is the dinosaur and the best thing ➜ S is t dinosaur n t best thing

Compression step 3: Thesaurus-rex 🦖

In the spirit of continuing with the theme of building something totally ridiculous and unnecessary, the second way that I compressed text is by using a thesaurus API. There are a lot of words in English that are overly long and can be shortened while keeping the same approximate meaning, for example penitentiary ➜ jail like in the image below (that's a 12 character to 4 character compression!). By using a thesaurus API, we can find synonyms for long words and do a replacement. This method is absolutely a lossy compression (usually both in actual data and in meaning), but it works, and it's fun!

Compression step 4: A new approach to links 🔗

It wasn't obvious at first because they're hiding when HTML is rendered, but the links in anchor tags were taking up the majority of the remaining space. Behind every 10 character blue word on a page is a 200 character-long URL, and that's a problem. It's a pain to preview links on a phone, so when I'm clicking on them I don't care what the link is as long as it brings me to where it's supposed to. Because of that behaviour, I decided that sending the true href value of an <a> isn't important, and as long as clicking a link can bring me to where I want, I can save a lot of space.

sanitize-html lets you define a function to modify attribute values, which is what I made use of to modify the links. When a link is encountered in the HTML, the phone number the website is for and the real link URL are passed to the function below, which stores key/value pairs of {phone_number}_{shortUrl}/realUrl in Redis, where the shortUrl is a random 3 character string.

const redis = require('redis');
const redisClient = redis.createClient(process.env.REDIS_URL); 

const urlShortener = (phoneNum, url) => {
  if (url) {
    const urlShort = Math.random().toString(36).substr(2, 3);
    redisClient.set(`${phoneNum}_${urlShort}`, url);
    return urlShort;
  }
  return '';
};

The final HTML will have all links replaced with short codes generated from the above code. When a link is clicked on from the app, that short code is sent to the server (over SMS) which knows from its format to look up the full value in Redis, and to retrieve the website from that real URL.

For a website like Wikipedia which is almost entirely links, this adds a lot of value to the compression.

Compression step 5: HTML to gibberish Ω

We've now compressed all of our text and removed as much HTML as we can from the page, so we're ready for the last step before sending the web page to the app!

The SMS charset that we're using is called the GSM-7, and it includes all English letters, numbers, basic symbols... and greek letters! We've already used up all the single English letters in part 2 of the compression, but unless we're looking at websites about math or science, there are probably no greek letters in the HTML.

We can compress the finite set of HTML keywords with these letters, in a similar "find-and-replace-all" method as before. The image below shows the colour mapping between an element and its matching symbols. We can save space by combining characters that we know will show up together, like < with input or value with = and ". Because this mapping is explicit, it's easy to decompress by going in the opposite direction.

Ready for liftoff 🚀

The target that I had set for compression was to get a webpage down from 1,300+ SMS to 10, so how did I do?
I got it down to 3 SMS.
And the best part? None of the code that I wrote was specific to this website, it's generic for any text-based page.

Now that the website is all compressed, we need to send it from the server back to the phone. Twilio provides a great node helper library that does all the heavy lifting. This is all that's required to get the messages sent back to the phone:

const twilioClient = require('twilio')(
    process.env.TWILIO_SID, process.env.TWILIO_AUTH_TOKEN);

// Divide HTML into the max sized SMS - 5
const smss = HTML.match(/.{155}/g);

// Send out all the SMS via Twilio
smss.map((sms, index) => {
    twilioClient.messages.create({
        body: `${index+1}/${smss.length} ${sms}`,
        from: process.env.TWILIO_NUMBER,
        to: req.body.From,
    });
});

📱 Reconstructing the website in the app

On the Android side, a BroadcastReceiver is set up to listen for incoming SMS from our Twilio number. Once all the SMS that make up a website are received, they're chained together and decompressed following the steps of the compression in reverse (skipping over the Thesaurus-Rex 🦖). The resulting HTML is passed to a Webview component (a Chrome browser within an app, which accepts URLs or HTML), and our website is displayed!

The end result for google.ca looks like the image below, which includes the compressed SMS text. This is what the website looked like 15 years ago, not too shabby for a free internet connection!

And that’s how I cheat the system and get unlimited internet! This method works pretty much only for text-based websites and it can be slow (it is named Dial-Up after all), but I know that I’d rather be able to load a search result in 10 seconds using this app for free than have to find a wifi hotspot every few minutes to make sure I’m still walking in the right direction.

If you want to hear more about how I built this project and see it explained with the help of some Downasaurs, check out my talk from JSConf EU 2019, take a look at the code on my website, or send me a message @alexandras_dev!