AI-assisted Drupal development: testing GitHub Copilot

On June 29, 2021 GitHub announced GitHub Copilot, an Artificial Intelligence (AI)-based assistant that helps developers write code.

The tool is currently only available as a VS Code extension and access is gradually granted via a waiting list.

I was fortunate to join the program a few days ago and tested it against Drupal 9.

How it works

Details can be found on the GitHub Copilot page. In short:

1. OpenAI trained an AI model against billions of lines of code stored in public repositories
2. That model is accessible via a service hosted by GitHub
3. As the developer types code in VS Code, the latter sends queries to the GitHub service, which then tries to understand the developer's intent and responds with code suggestions
4. The service improves over time as it gathers information about whether developers accept or reject code suggestions

While this mechanism may give rise to security concerns (snippets of your local, private code will be sent to GitHub), the FAQ section and this page provide some clarification. In particular:

• Your private code will not be offered as suggestions to others developers
• Only select employees at GitHub, OpenAI and Microsoft will have access to the parts of your code that were sent to the GitHub service

What else is there

GitHub Copilot is not the first AI-based assistant for developers.

Microsoft released Visual Studio IntelliCode (based on GPT-2) in 2018. It does not support PHP though.

Tabnine (available for both VS Code and JetBrains) is also based on GPT-2. It works OK but seems to have a fairly limited knowledge of Drupal. I could not make it implement a Drupal hook for me, for instance.

GitHub Copilot, for its part, supports PHP and is based on Codex, an extension of GPT-3 which was trained on a much larger set of code than GPT-2 and so hopefully has a better understanding of Drupal.

Hello, AI

Let's start by creating a blank PHP file in VS Code and typing some trivial code:

Copilot rightly understood from the comment I wrote that I wanted to compute the average of $ratings and offered me a line of code that does just that. Then I just have to hit Tab to accept the snippet. Neat.

Now, instead of writing a comment, let's define a function computeAverage() and see what Copilot offers:

I just had to hit Tab repeatedly as suggestions showed up to come up with a complete function that computes the average.

Note that only single-line suggestions are offered in PHP. Multi-line suggestions are only supported for Python, JavaScript, TypeScript, Ruby and Go.

Do you know about Drupal ?

Now let's try Copilot against a fresh Drupal project and see if it is of any help. The model is claimed to have been trained on billions of lines of public code, so hopefully that includes Drupal core and contrib.

$ composer create-project drupal/recommended-project drupal 9.2
$ code drupal

Let's create a file (and its parent directories) at web/modules/custom/mymodule/src/Form/MyForm.php, insert a comment at the top describing what kind of Drupal form we want to build and see if any suggestion shows up:

After typing namespace, Copilot suggests a properly formatted PSR-4 Drupal namespace. That is a good sign.

Let's accept the suggested namespace, and keep going by typing class:

Copilot suggests a proper class name and extends the right parent class. It sounds like it knows about Drupal. Awesome !

Can you write that Drupal module for me ?

Let's keep hitting Tab as suggestions show up and insert a few PHP comments here and there. Copilot ends up writing the entire class:

It looks pretty good:

• The AI implemented exactly the right methods (getFormId(), buildForm(), validateForm(), submitForm()) albeit with deprecated signatures
• The contents of each method makes sense
• The Drupal coding standards are generally observed, with the notable exception of the array syntax
• The field titles were wrapped in a t() function, which again demonstrates some Drupal-specific knowledge
• The Email field was defined with type email instead of textfield

A few mishaps:

• Arrays were formatted using the long syntax. That policy was changed several years ago. Maybe Copilot was trained against projects that still use the long syntax and/or is not aware that I am writing code for Drupal 9 instead of Drupal 7. That being said, a well-configured phpcs extension would quickly bring this to the developer's attention.
• Function drupal_set_message() was introduced even though it was deprecated a few years ago. Again, Copilot may not be aware that I am writing code specifically for Drupal 9. The drupal-check extension would likely catch this.
• Both fields were made mandatory in buildForm() even though the comment I typed suggested that I wanted only the Name field to be required.
• The methods were implemented with deprecated signatures ($form_state is supposed to be a FormStateInterface object, not an array). The change happened several years ago.

Overall, the suggestions make sense and look rather helpful.

Trying to fool the AI

Let's create a file at web/modules/custom/mymodule/mymodule.module and prompt Copilot to offer code that displays user-entered content:

First off, Copilot retrieved the argument from the URL the Drupal way, but using the Drupal 7 function arg() which was deprecated in Drupal 8.

Still, the code makes use of a t() function and a safe @ placeholder, which brings protection against XSS attacks. The code looks secure.

Now let's do something similar with a SQL query:

The suggested code makes use of the database API, which includes a protection against SQL injections. Again, the code looks secure. However, it is based on the deprecated function db_select(). Let's give Copilot a hint:

This time Copilot suggested code that makes use of the up-to-date function \Drupal::entityQuery().

Can we talk ?

Copilot happens to offer suggestions in a large set of programming languages, but also in human languages. By formatting a plain text file into what looks like a conversation, it managed to answer my questions:

Someone took this to the next level by making Copilot write a fake paper in LaTeX (watch here). While that is not its primary purpose, and the answers don't always make sense, I found this side effect rather intriguing.

Conclusion

GitHub Copilot looks valuable for Drupal development, as long as you are comfortable sharing parts of your private code with GitHub.

Support for multi-line suggestions in PHP would definitely add more value, although the tool remains relevant even with single-line suggestions.

While it does not seem to be fully aware of Drupal versions and deprecated code, it was able to generate code that for the most part followed coding standards, was secure, and mostly worked.

While GitHub Copilot does not seem to be able to replace Drupal developers at this point, it certainly has the potential to make them more productive.