req, an HTTP scripting language

Programming languages are always something that have fascinated me, how they're
designed, how they're implemented, and how they're used. Whether they're a DSL
(domain specific language) or more of a generic programming language. A
programming language is always something I had wanted to take a stab at
creating, even if it ended up being terrible, or being of no true utility, but
only for the sake of learning. Well, over the Christmas break I decided to
occupy my time on developing a language, one that was small and simple, designed
for a specific use case that I had encountered but hadn't found a solution for.
The language that I ended up developing was req, a language designed
only for HTTP scripting.

• Overview
• Why
• Conclusion

Overview

What do I mean when I say HTTP scripting? Perhaps an example would be best to
demonstrate, followed by an explanation,

Token = env "GH_TOKEN";
Headers = (
    Authorizationn: "Bearer $(Token)",
);

Resp = GET "https://api.github.com/user" $Headers -> send;

if $Resp.StatusCode == 200 {
    User = decode json $Resp.Body;
    writeln _ "Hello $(User["login"])";
}

Above is what req looks like. It looks like your typical language, however it
offers first-class support for making HTTP requests and working with their
responses. The language makes use of builtin commands to handle the sending
of requests, the encoding/decoding of data, and the reading/writing of data.
These commands also return values that can be stored in variables. The output
of one command can be sent as the input of another command via the ->
operator. There is no support for user defined commands.

That's what the language looks like, and this is how it is run,

$ req user.req

the above example makes use of the GH_TOKEN environment variable, so if we
wanted it to actually function we would need to make sure that was set
before invocation,

$ GH_TOKEN=<token> req user.req

So, from a brief overview you can see that there is some familiarity with other
languages out there. I call this a scripting language, as opposed to a
programming language, as it is interpreted, and extremely limited in scope and
capabilities.

req can also be used via the REPL, which is accessed simply by invoking the
binary and passing no arguments to it. This can be used as a scratchpad to plan
out what you want your scripts to do, or as a means of exploring an HTTP service
and its endpoints,

$ req
req devel a5ddbe7 Sat Jan 29 11:34:38 2022 +0000
> Resp = GET "https://httpbin.org/json" -> send
> writeln _ $Resp
HTTP/2.0 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 429
Content-Type: application/json
Date: Sat, 26 Feb 2022 14:14:56 GMT
Server: gunicorn/19.9.0

{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

Why

Now, why did I want to create yet another scripting language. Two reasons.

The first was for the sake of learning. Some time ago I had received copies of
the books Writing An Interpreter In Go and
Writing A Compiler In Go. I had worked through the
first book at the start of 2020, and enjoyed what I had learned, wanted to put
what I had learned to practice. At the time however, I couldn't think of a fun
or interesting language that I would have wanted to develop, so I shelved the
prospect of it for some time, which brings me to my second reason...

I think most developers have to interact with an HTTP service at some point
during their day job, in a way which would require some form of scripting.
Perhaps you're trying to debug an API, so you pull open a terminal and fire off
a few curl requests, and see what response comes back. Or maybe you
want to scrape a site for its information. Either way, you're doing something
that involves some tinkering.

I have been there too. And it is this scenario that made me wonder if there was
a tool out there that allowed for easily working with HTTP requests and their
responses in a programmatic way. Sure, you could use curl and shell scripting,
and wrangle the data through sed, awk, and jq to get the data you need, but this
approach can be fragile. On the other hand you could use a full fledged
programming language. This way, you would have more control, but it can be
perhaps a bit too verbose at times if all you want to do is send off an HTTP
request.

This is was prompted my development on req. A high-level scripting language that
allows you to easily send out HTTP requests, and work with their responses. The
main benefit of the language, in my opinion, is that it tries to make working
with HTTP requests as semantic as possible, take the following,

Resp = GET "https://httpbin.org/json" -> send;

here we want to send a GET request to the https://httpbin.org/json endpoint.
Writing this out either in a script or the REPL can feel more natural than what
you might otherwise write when using curl, for example. Then let's say we want
to decode the response data into JSON,

Data = decode json $Resp.Body;

again, it's like we're describing what we want to do with the response. This is
what I wanted to achieve with this language, keep it limited in scope, and
hopefully offer some utility in the realm of HTTP scripting.

Conclusion

What I've covered in this post is a simple overview of the language, and the
reasons behind it's implementation. I haven't gone into my justifications
as to how/why the language was designed the way it was, but that could perhaps
be another post down the line. If what I've shown so far interests you, then
feel free to take a look at the code for it on GitHub:
https://github.com/andrewpillar/req and feel free to spool through the
documentation there too, to get a sense of the language.