My Week With GitHub Copilot: AI Pair Programming Review

Before text and code editors, developers were tasked with having the correct code passed to a compiler. Now, support tools like syntax highlighting, function syntax assistance, autolinting, and Artificial Intelligence (AI) assistance are available directly in your Integrated Development Environment (IDE) or the editor where you code.

Now, there’s a tool that promises to improve the developer experience even more: GithHub Copilot, an AI pair programmer. After a year of closed beta, GitHub Copilot is now publicly available to developers and is gaining a lot of attention for its ability to enable developers to harness AI while writing code. According to GitHub, developers can write a comment describing the logic they want and GitHub Copilot will automatically generate a code suggestion for their solution. It sounds pretty cool, right? Our team thought so, but we had some questions, for instance:

Does GitHub Copilot really speed up daily work?

• Are the suggestions valid, or just noise for the developer?
• Is it worth a freelance developer paying for it?
• Is it worth companies paying for it for their team members?
• Does it make you less of a developer if you use a tool like this?

OK, that last one might be a bit philosophical, but the others felt testable. So, to get some answers, I put GitHub Copilot to the test. In this article, I’ll share my experience after one week using the tool and dive into key findings from this trial period. I’ll also use my 20+ years of experience as a Senior Full-Stack Developer to compare it with my day-to-day before Copilot. Hopefully, my experience will give other developers a better understanding of how this tool works and if it lives up to the hype.

What Is GitHub Copilot?

To understand how Copilot came to be, it’s first important to understand pair programming.

Pair programming is an extreme programming technique where two developers work together on one computer, one acting as the pilot and the other the copilot. The pilot is on the keyboard while the copilot helps navigate, checking the logic and patterns, while weighing against alternative solutions. While two developers producing one piece of code sounds expensive, the quality of the final work can help avoid added costs of code review and QA.

It was from this idea that AI pair programming came to life. Officially launched in June 2022, GitHub Copilot is an AI pair programming editor that suggests code in real time to help developers work faster and more efficiently. It’s powered by a new AI system, OpenAI Codex, and translates natural language to coding prompts as developers write a comment or a line of code. While it’s free for students, it’s $10/month or $100/year for developers (with a 60-day trial).

Now, with that background covered, let’s dive into some key findings from my week with GitHub Copilot. I’ll run through the installation, testing and how it looks in action.

Initial Tests

To have a consistent testing environment, I set up two directories: one with plain JavaScript (no dependencies) and one with a Next.js application, so I could play around with React and an established React framework briefly.

Plain JavaScript

In an empty .js file, I started typing to create a simple function to convert from hex color to rgb. This is a common feature – most of us know the concept of converting hex numbers to integers – but let’s be honest, most developers don’t have solutions memorized. And unless there’s a library in the system, we’ll search for the simplest solution, validate and move ahead.

Using Copilot, here’s how it can be achieved:

Cool, right? Notice that I ignored the first suggestion as it looked too complex. I was writing a basic test with a sample color and I only had to hit Tab.

React in Next.js

This is a more complex test, but let’s keep it simple for now. Here’s a test page, but nothing specific from Next.js:

There’s a mix between GitHub Copilot and the IDE’s code completion. Do you notice a small delay to display the suggestion? This may be because it’s too vague, but it’s hard to tell. GitHub Copilot’s suggestion is okay, and it also offers an export default, but it isn’t needed in this case.

The code above is enough to test the environment. Out of curiosity, let’s see how it looks if we hit Ctrl + Enter:

You can also use Alt + Tab, of course.

We’ve completed the dummy tests. Next, let’s see GitHub Copilot in action.

Real-Life Action

My current project, where I’ll be using the tool, has Laravel on the back-end and React on the front-end.

Starting with Back-end Code

I only had back-end work on my first day using Copilot, so my expectations were low (no optimization for PHP, remember?)

Note that the tasks were related to the Laravel framework: a modern PHP framework and VSCode with some very established extensions that are able to complete “use” statements and pop up PHP function parameters. Prettier, the code formatting tool I mentioned earlier, alerts me about silly syntax mistakes at saving. In my day-to-day, I purposely avoid indenting code; that way, if Prettierdoesn’t auto-adjust for me at save, I know it’s not working and I have something to fix.

Complete the Line

The first effect is “filling in the blanks” suggestions. This is similar to Google’s auto-suggest function, where typing in Gmail or Google Docs will recommend words or phrases to finish a sentence for you. Copilot shows obvious completions, like end-of-line semicolons or closing brackets/parentheses.

Verdict: GitHub Copilot is helpful in this case, but not impressive. To compare, Prettier does the same at saving, while also automatically indenting your code.

Variable Filling

When you use a function, GitHub Copilot suggests what variable to fill in depending on your context.

This suggestion is 99% of the way there in my context, but the correct answer is:

Verdict: Okay, GitHub Copilot, you won this one.

Message Suggestion

This is a common pattern. You make an early condition to check user access and throw an exception returning a message:

This doesn’t come from external code, but it used my context to suggest an especially useful message. So GitHub Copilot not only knows I need to fill a message, but also an authorization message that should be inferred by the previous code.

Verdict: For those not used to Laravel, it’s a pretty common pattern to get the current user. So this is amazing functionality!

Library Discovery

I was working with the Cloudinary API through their PHP library. The code is supposed to be different depending on the mimetype. Then I get:

The case syntax isn’t impressive as any IDE snippets plugin can do this for any language. But, the mime type goes deeper: it considers the function name to infer the related mime type (video/mp4). The cloudinary_php suggestion looks impressive, but it’s not correct. I could actually write a code like this in some situations, but the solution I need is very different. And so I still have to check the API reference to get my final solution.

Verdict: Developers won this round. Sorry, Copilot.

Model/Database Suggestion

In this example, there’s a switch syntax building a custom filter for the Booking model in Laravel. After pressing Enter, Copilot suggests a real field. But wait, this isn’t on the active file!

When I accept by hitting Tab I get:

This is pretty much the answer, except $value[‘id’] should be $value.

Verdict: It’s not 100% correct, but it saves time because I need to type less and only requires a minor tweak. It’s pretty useful.

Alternatives to GitHub Copilot

Are there any worthy alternatives to GitHub Copilot?

The short answer: at the time of this writing, the only alternatives are Tabnine and Amazon CodeWhisperer (the latter just launched a preview).

The long answer: OpenAI Codex started as an open-source project that would allow, in theory, many tools to use its base to create various alternatives. But this is not a cheap enterprise! They recently moved to a more traditional closed business model and received a strong investment from Microsoft, which owns GitHub.

Some open-source projects started using the remaining open-source training model, but most are stalled. Unless they were to receive funding, it would be pretty hard to achieve the same level of efficiency GitHub Copilot offers now. The latest version of its model is now available for licensing, preferably by Microsoft. As other companies can license OpenAI Codex, it’s possible we will have more alternatives in the future.

Tabnine aims to provide a similar product and already has a customer base (they claim to have more than 1 million developers using their product). They stand for using only open-source code with permissive licenses for their Open Source Trained AI model (MIT, Apache 2.0, BSD-2-Clause, BSD-3-Clause) from GitHub repositories or customer’s repositories/local code. You can also download the model to your machine and get even faster responses. Their Pro subscription comes in at $12 US per month.

In June 2022, Amazon launched a preview of CodeWhisperer, another tool similar to GitHub Copilot. This seems to be a legitimate competitor, but there’s not enough information to compare them so far.

It’s likely that we’ll have more competitives to evaluate in the future. For now, we’ll say it’s “to be continued…”

Addendum: FauxPilot is an open source project that aims to run a model locally, using a Nvidia GPU to process the data, and VSCode GitHub Copilot plugin itself.

Key Takeaways from My Week with GitHub Copilot

GitHub Copilot certainly has its strengths and weaknesses. Here’s a roundup of key takeaways from my trial period with the tool:

Fewer Browser Searches

With GitHub Copilot, I don’t need to switch to the browser to search as often. Common functional solutions are available after a few words + Tab, and established patterns used in the most used frameworks/libraries are there, too. It’s a great time-saver.

Context Matters

My expectation of how Copilot should work was wrong. At first, I thought it was an “ask me something and I return some options” kind of flow. But even if I don’t ask anything, I still get impressive suggestions in the context of my work.

“Context” is key here. The AI considers factors like the frameworks or libraries in use, the variables in the current file, and related files and scope in order to produce better suggestions. Patterns are also considered in part of the answer as well. When you embrace the technology and understand its strengths and weaknesses, you start to get more out of it without changing your flow.

It’s Best for Experienced Developers

Although AI assistance might look like a good learning tool, it’s not. Code suggestions are just that: suggestions. The developer needs to be able to evaluate these suggestions and choose to accept, skip, or ignore them. In order to use the tool properly, it’s necessary to know the context, logic, and expected output to make a smart (and fast) decision. Otherwise, the suggestion can create errors.

GitHub Copilot actually confirms this insight. The GitHub Copilot FAQ entry Fairness and broader impact states that “inexperienced developers may struggle to use GitHub Copilot to effectively generate code, and their lack of experience might inhibit their capability to effectively review and edit suggestions made by GitHub Copilot.”

Verdict: It’s Worth the Money

From my time using the tool, I concluded that yes, GitHub Copilot is worth the cost, and here’s why:

• You switch less to the browser for searching solutions, saving tons of time.
• You type less with smarter code completion than your IDE offers.
• You get tiny boosts with contextualized solutions that sum up at the end of the day.
• You can focus on the bigger picture, architecture, and integration.

Given the rates most software developers are making these days, I think that if this tool saves you even one minute per day, the $10 monthly fee easily pays for itself. In reality, I think you save a lot more time than that, and what you get out of the tool increases with use (you get used to learning how to get better answers). Sounds like a good deal to me.

Final conclusions

Developers aren’t encyclopedias. Even recent solutions we’ve worked on aren’t always readily available in our brains. It takes a lot of searching and research for small solutions, like a for/foreach pattern ten-line function created five years ago for a dead project. The majority of problems that have to be solved every day have already been solved many times over and are available as question-and-answer forums or public code from open source projects.

An AI-powered tool like GitHub Copilot can speed up daily work and bring so much knowledge to your fingertips in your IDE, but it won’t do the job for you. At the end of the day, developers are still responsible for delivering the work even if they use AI assistance. With that said, advanced tools like this won’t make you less of a developer, but they will improve your productivity.

For decades, companies have been trying to automate coding. And, although many common programming activities were automated, abstracted or simplified, real people still need to do the work. Maybe in the future we’ll need another kind of programming, but at least for now, there still needs to be someone’s actual brain in action to solve humanity’s problems.

Final tips for using GitHub Copilot:

• Don’t try to follow GitHub Copilot. Instead, let it follow you.
• Be suspicious of long suggestions. It’s easier to get fooled by them.
• Don’t trust by default. Read the suggestions and be sure they make sense.
• Context matters a lot. Give Copilot as much context as possible by keeping related files open to get wider contextual suggestions.

And remember: GitHub Copilot is AI. Ultimately, you’re in charge and responsible for the end work.

This post was created using parts of an article my colleague, Rafael Goulart, recently wrote. If you would like to learn more, I high recommending reading his full review here: https://www.scalablepath.com/full-stack/ai-pair-programming-github-copilot-review