How I added Ballerina support to GitHub

The story begins when I joined the Ballerina team at WSO2 few months ago. I was working on the Ballerina IntelliJ IDEA plugin one day and suddenly I had an idea. That idea was, “What if we can add Ballerina syntax highlighting support to GitHub as well !?”.

As all of you know, GitHub provides syntax highlighting support for hundreds of languages like Java, JavaScripts, GO, etc. When we send a PR, we see the nicely highlighted diff. Also you can see the programming languages used in a repository by clicking on the language detail bar.

But have you ever thought about how that syntax highlighting is done in GitHub? Nope, right? Neither did I. It just works and we took that for granted and never even bothered to think about it twice.

So once I had this thought, I googled about how the syntax highlighting is done in GitHub. I found out that this capability is provided by the Linguist library.

I checked the repository to see whether there is any guidelines for adding a new language to it. Fortunately, CONTRIBUTING.md contained a section on Adding a language. It looked like Ruby is needed for the process. Also I saw this.

We try only to add new extensions once they have some usage on GitHub. In most cases we prefer that extensions be in use in hundreds of repositories before supporting them in Linguist.

At that time, Ballerina was still on its early stages and did not had much usage in GitHub. Also we were very busy at that time with back to back releases, so I wanted to check it when I have some free time and we have more usage in GitHub.

Few months passed by and finally I found some time to work on this again. By that time, more and more Ballerina files were added to GitHub as well. So I decided to pick from where I left. Surprisingly, adding a language to GitHub was easier than I thought even though there were few hiccups along the way.

In the below sections, I will summarize the steps I used to add Ballerina language to the Linguist library.

Grammar File

First of all, you need a grammar file which defines the rules which will be used for syntax highlighting.

There are various ways to define a grammar, but we need the grammar as a tmLanguage file (textmate grammer) since it is the grammar linguist library supports. Fortunately for me, this was already created for Ballerina VS Code plugin. You can find the grammar file I used here. This grammar was bit older, but it was enough to get things started. Grammar file also should have one of these licences. If you are new to textmate grammar, you can refer to this section for more details.

Note: We add the repository which contains the grammar as a sub repository to the linguist repository. So whenever a new version of linguist repository is released, they update these sub modules as a part of the release process. So that means if you update your grammar in your repo, it will be reflected in GitHub after the next release of the linguist library. So you don’t have to send PRs to Linguist repo every time your grammar changes.

Installing Ruby

So the first thing I did was installing Ruby. It was fairly a straightforward process.

Setting Up

1. Fork the repository.
2. Clone the repository to your local machine.
3. Create and checkout a new branch.

4. Install the Bundler using the following command.

   gem install bundler

5. Navigate to the directory and install the dependencies using the following commands.

   bundle install

   One mistake I did was I did not install the dependencies. This caused me few issues along the way. So I found out how to fix those issues the hard way :)

Adding Ballerina language

Updating languages.yml

So the first step is updating the lib/linguist/languages.yml file. In here the tm_scope field should have the scope name defined in the grammar. Make sure not to change already existing entries in the list. Also you can omit the language_id field since it will be generated and added later.

Also it is important to note that you cannot use a color which is similar to a color which is already in the linguist library. This is tested by test/test_color_proximity.rb file. I guess this is done so that each language will have a distinguishable color.

Adding grammar

So the next thing you want to do is adding the grammar using the following command.

script/add-grammar script/add-grammar https://github.com/ballerinalang/plugin-vscode

This will add the the repo as a sub module. The module name is vendor/grammars/plugin-vscode. This was bit problematic since the name was too generic. So I had to rename the module name to more specific one.

Adding samples

Add few samples to samples/Ballerina directory.

Generating Language ID

Now we have to generate a language ID for our language using the following command.

script/set-language-ids --update

This will generate a new ID for our language.

Testing

Now we have to test our changes. For this, run the following commands.

script/bootstrap
bundle exec rake samples
bundle exec rake test

Sometimes getting the tests running can be too much work, especially if you don’t have much Ruby experience. It’s okay: be lazy and let our build bot Travis run the tests for you. Just open a pull request and the bot will start cranking away.

So if you decide to open a PR instead of testing by yourself, following section is for you.

Opening a PR

Now you have done everything to add a new language. Commit the changes and open a PR. When sending the PR, you need to link GitHub search result which shows wide usage of your languages. For example, refer this.

Linguist Community

Developers in that community is very welcoming. My PR was reviewed very quickly. They pointed out the issues in my PR, so I was able to make sure that it is up to their community standard. If you need any help or if you have any questions, don’t hesitate to ask from them.

Ballerina in GitHub

By adding Ballerina language to GitHub, we get following advantages.

Syntax highlighting for Ballerina files

View Ballerina code percentage in repositories

Ballerina code snippet highlighting

Additionally to what others get from this, it was a massive opportunity for me to learn new things. Now every time I see a Ballerina code in GitHub, I feel very happy because I helped to make it happen. It was probably a once in a lifetime opportunity and I'm very glad that I grabbed it.

Hope you enjoyed this article. It was bit long, but I’m sure you learnt something new :)