Introduction
Recently, AWS announced a machine-learning powered service that helps improve developer productivity by generating code recommendations based on developers' comments in natural language and their code in the integrated development environment. The service, called Amazon CodeWhisperer, is still in preview to be used at no cost. This service is similar to GitHub copilot, which Microsoft launched last year.
In the past few months, I had a chance to experiment with this new service in a few use cases. As a machine learning (ML) developer, I had the advantage of utilizing ML to help develop ML solutions. Thus, I am writing about some observations after getting early access to this service. Besides, I am giving specific suggestions on how to make it smarter and more accessible.
The service in action
The service provides real-time code suggestions based on the comments in the code editor and previous codes in the same document. The service may suggest line completion or complete code blocks (e.g., methods).
On Visual Studio, there are some handy shortcuts that make the use of the service more convenient. While the extension is enabled, the service provides online inference similar to the auto-complete feature supported by many IDEs. However, the user could hit the (Alt+C) to see the recommendations without waiting for the response.
Below is an example of writing the famous binary search method
Interestingly, the service may suggest multiple code snippets that could be easily navigated (with left/right arrows) to choose the most suitable recommendation.
Amazon CodeWhisprer is like the companion that tries to whisper in your ears with the right code. Hence, it is a pretty fancy and super descriptive name. Good job on naming the service.
A deep dive, How to get the most out of the service?
AI code companion is a powerful tool that can boost developer productivity. Despite it may be argued that such a tool could replace the developers in the future, it is still too early to jump to this conclusion as the service is like any other service: Garbage in Garbage out. That's, it heavily depends on the input it takes to return good results. Below is an example of how the input quality can totally affect the output quality.
Here, the provided description was so vague without any clear requirements, so the output is chaotic imports after waiting for a relatively long time.
As the input description becomes more clear, the output becomes so much better as shown below for a similar, yet clearer problem.
In addition, the quality of the recommendations significantly improves as the user adds more context i.e., as the developer writes more code. For example, it is expected to get faster and more personalized results while working on one project in comparison to either isolated tasks on the same document or at the early moments of the project when the context is just still not enough.
Nevertheless, the service is not expected to return helpful answers for infamous custom tasks. Below is an example of the same binary search problem but with slight modifications to the input format.
Apparently, the engine couldn't understand the slight modification to the problem (i.e., allowing duplicated elements) and still produce the same code suggested earlier.
As the service is still in preview, it is expected to have areas for improvement. Below is a curated list of actions that can make the service much better.
Inference speed:
As might be noted in the examples above, the service takes non-trivial time to suggest recommendations. I believe there is considerable space for improvement in this aspect.
Consistency and real-time capabilities:
The service is expected to give real-time recommendations while the developer writes the code. However, the real-time suggestions might not give any output at a specific time instant. Surprisingly, hitting the (Alt+C) shortcut returns workable solutions without changing anything (i.e., at the same time instant).
End-user Customization:
The recommendation engine under the hood uses a huge code library from many sources that were written for different purposes. It is justified to enable more customization for the sources that you accept for some projects.
Also, it might be beneficial to predict the codes based on the project theme. For example, Machine Learning development is completely different from developing mobile applications.
As another example, the user might want to work on a project that requires multiple blocks of codes that need to be designed and aggregated. On other projects, it might be needed to prioritize line completion more than block suggestions.
The list of examples for customization is huge and needs careful design.
Solutions Ranking:
It is a great feature to suggest multiple solutions. However, in practice, the ranking of these solutions is not optimal and the user needs to navigate all solutions to find the right suggestion. This is can be tedious and reduce the overall productivity gain.
Problem Customization:
The engine effectively understands common problems found in the training corpus. However, it is a challenge to adapt to the subtle modifications of the same problem.
Conclusion
In summary, AWS CodeWhisprer (and AI code companions in general) is not magic after all that can solve all problems. However, it is a great tool to enhance developer productivity by focusing on the right problems instead of tedious repetitive tasks.
To get the most out of the AWS CodeWhisprer (and AI code companions in general), the following actions might help in achieving the desired goals:
- Concise comments: the clearer and more well-defined the input tasks, the higher the probability of getting quality results.
- Unified Projects: The AI engine collects information from the whole document. Hence, it enriches the context continuously. Thus, it is more beneficial to use it on tasks that have a connection in one way or another.
- Avoid advanced custom problems: the less popular the problem, the higher the probability it will not return any helpful answer.
Top comments (0)