TIL CLI - A tool to write and manage a Today I Learned Repo on Github

tl;dr; til is a wrapper around a few other tools to simplify the management of a TIL repo, such as https://github.com/pjambet/til. I recently published it as a ruby gem.

The code can be found on GitHub.

Intro

At first glance this small CLI might not seem like it does much, and it really doesn't, but while building it I ended up learning quite a few things, three to be exact:

• How to use an external command line tool, such as fzf, and feed it input, similar to using a unix pipe, like echo "a\nb\nc" | fzf, as well as reading the output of the command, but programmatically, in Ruby.
• How to implement a flow similar to what happens when you type git commit without the -m/--message option and it prompts you with an editor, vim by default.
• How to use the Gitub API to create a new commit, without using the git cli.

Before jumping in, here's a summary of what til actually does:

• It first loads the list of all existing categories in your TIL repo, and then uses fzf to prompt you to pick the category for your new TIL. You can also choose to add a new category.
• Once you picked a category, it uses your default editor, as configured through the $VISUAL or $EDITOR environment variables, or vi if none of those are defined. You can then type what you actually learned today.
• After saving and closing the text editor, til will grab the content of the file, and commit it to the configured GitHub repo
• It also takes care of maintaining the README.md file so that it contains a nicely organized index of all your TILs. It keeps a list of all the categories at the top, and includes a link to each TIL below, grouped by category

Using fzf

I don't actually use it that much, but I love how fzf improves CLI interactions. I thought it would be a great addition for the workflow I wanted with til. Most of the time I will be reusing existing categories, and as someone who makes a lot of typos, I always look for ways to avoid having to type anything.

The main way that fzf gets its input in from STDIN. You can test that from the command line: echo "a\nb\nc" | fzf will load fzf with three items a, b & c.

I was able to get the list of folders using the GitHub API ruby library, Octokit. Each folder is a category, and I needed to feed that to fzf, through STDIN, in Ruby.

It's worth mentioning that I could have used the backtick approach, but that was a tiny bit too much magic for me and also, there are few blog posts out there that recommend against using it, and favor the system method. In this
case we're not really dealing with user input, so it doesn't matter that much from a security standpoint.

The Process.spawn method accepts a bunch of options, there are a lot, but these are the ones that are interesting to us here:

:in     : the file descriptor 0 which is the standard input
:out    : the file descriptor 1 which is the standard output

So, we can create a pipe with IO.pipe and pass the reader as the :in argument to spawn, so that we can write with the writer from the main process and the process on the other end, the one created by spawn will receive it.

Note: it is important to close the writer in the initial process, if you don't, fzf still thinks that there might be more to read, and it shows it by displaying a spinner in the bottom left corner of the terminal.

We can use the same approach to get the content that fzf will output. Once a selection is made, fzf writes it to STDOUT. So we create another pipe, and give the writer as the :out option, the process started by spawn will
be able to write to it and we can then use the other end of the pipe to read from it and get the selection from the user in the main process.

This is what til does, as you can see on GitHub.

A quick look at the C code defining the backtick function shows that it uses the pipe syscall, so we're essentially reimplementing something fairly similar to what ruby does for us with `echo 'a' | fzf`.

Note (another one): As I was writing this post, I realized that Ruby has another method related to spawning new processes and dealing with STDIN and STDOUT: IO.popen. I haven't looked too much into it yet, but it looks like
it could simplify my code a little bit. That being said, the overall approach described above is still valid.

Using an external editor

For years I used git from the cli and relied on its commit workflow without really wondering how it actually made that happen. In case you're not familiar with it, from the CLI, when you commit with git commit you essentially have two
options, you either provide the commit message inline, through the -m/--message option, or you leave this option blank
and git opens an editor for you, by default vim.

What is actually really cool with this is that you don't have to use vim, you could use pretty much any other editors,
that being said, you probably want to pick one that is quick to start so you don't have to wait just to type a commit
message. It's a little bit trickier with editors using a dedicated window such as vscode or macvim. GitHub's
documentation has a page explaing how you can use the most common visual editors as git editors.

So, as a git cli enthusiast, I wanted to replicate the same worflow: you picked a category, now let's write the
content, in the editor you like using, so you can format things the way you want.

It turns out that it's apparently "the right approach" to first look at the VISUAL environment variable and then at
EDITOR as explained here and there.

Reimplementing the git commit workflow turned out to be very little work, you first create a file, Ruby makes that easy
with the Tempfile class, we then use either system or spawn with the value in $EDITOR or $VISUAL (we
default to vi, just in case, so we have something).

The main difference between spawn and system here is how they return, system does not return until the process is
over, wheras spawn returns a pid. Since we basically want to wait for the user to close the editor, system is
easier, with spawn we would have had to use waitpid to wait.

Once the child process is done, we can read the content of the file, and VOILA! We have the content, formatted by the
user, in their favorite editor (unless they ended up in vi and couldn't figure out how to exit).

You can see that til does exactly what I just described

Creating a commit using the GitHub API

And now, the final piece of the puzzle, we have a category and a string representing a new TIL, it's time to create a new commit on GitHub. The GitHub API must have an easy way to do this right?

Well?

You can do it! But is it easy? I'll let you answer on your own.

The new commit needs to contain two changes, the new file we want to create, but also, and this is really one of the
reasons why I wanted to create this tool in the first place, the updates to the README file, to keep the table of
content and the links up to the date with the new TIL.

This blog post was really helpful but the fact that it didn't include any code examples means that I spent a few
hours (😭) trying to get things workings, here's a summary of what I'm doing to create a new commit, I hope you're
ready:

• Get the ref of the latest commit on master
• Get the commit object, with the sha obtained in the previous step
• Get the tree object, based on the latest commit obtained in the previous step
• Get the readme content
• Create a blob for the new file - Interesting to note that a blob does not have a path, we only specify the path when adding blobs to a tree
• Get all blobs that are in the current tree, except the README - This is necessary because we don't want the new tree to remove any files, so our new tree needs to contain all the old files, plus the new file, plus the updated readme, and we definitely don't want the old README
• Figure out what the content of the README should be - We add a new category at the top if necessary and then we add a link to the new TIL at the bottom, keeping the categories sorted alphabetically, and the entries sorted, oldest at the top, newest at the bottom. The code is U (pause) GLY, but it works!
• Create a new blob, for the updated readme
• Add the new blob (at the correct path) and the blob for the updated readme to the blobs list
• Create a new tree with all blobs, all the old one, unchanged plus the new file and the updated readme
• ...
• CREATE A NEW COMMIT, FINALLY
• You thought we were done? Nope, we have to update master to point to the new commit, and now, we're done

You can find documentation about the API endpoints I'm using on the following pages:

• The HTTP API Documentation
• The octokit gem Documentation

See it in action

Conclusion

The current version (0.0.4) is very very basic, but it gets the job done, and I've been using it for a few days already. I have a few thoughts about what I would like to do next:

• A "real" cli, probably written in go, so that it's easier to distribute, with brew for instance. There doesn't seem to be any formulae/casks named til, so I should hurry up!
• A chrome/firefox extension, so you can do the same without leaving your browser
• Improve the code (if you've looked at it, it's ... far from great, really far)
• Show a terminal spinner at the end when creating the commit, since it can take up to a few seconds. I recently learned how to use terminal escape sequences to do this! Read more on my TIL repo (see what I did there?!). But there are also at least two gems that do that for you, here and there.

Questions? Comments? Hit me up on Twitter!

Existing gems

There are a few similar gems, both in names and features available on ruby gems, I checked all of them before publishing mine:

• til_cli, there were two issues with this one:

  • It does not seem to work with latest ruby 2.7.1 because of json 1.8.1 not compiling
  • It runs git commands locally, it basically wraps git calls, which is great, but means that the tool is not really a standalone tool, I wanted something I could run from anywhere, and that would work on its own.

• todayilearned: This is an interesting gem, but it works locally with sqlite, and while this is great, this is not what I wanted

• til: This seems to the same code as the one above, they both point to this GH repo: https://github.com/ip2k/todayilearned

• til_: This looks like an empty gem, there's no "real" code in there as of today.