Automating Automated Tests with Ponicode

Last week, I heard about this nifty unit testing tool called Ponicode. It utilizes AI and context information from your code to automatically generate test cases for your functions. At the time of this writing, it is available as a Visual Studio Code extension.

As someone who's written a decent amount of test cases throughout my early career, whether it's been on my co-op work terms or my side projects, I wanted to give this a spin. Setting up automated tests and configuring runners can take up a lot of time and distract me from the main tasks at hand, so I wanted to see how a tool like this can improve my workflow.

The documentation provides a lot of useful information on how the tool works, which languages/syntax it currently supports, and information on how to install the tool, so I recommend checking it out to get started.

Once I installed Ponicode, I checked out the example project they included, which is a simple function that checks whether a provided string is an email, and it seems to work pretty well so far!

Another thing I enjoy so far is having a graphical interface to add and organize test cases for my code. You can add any of the test cases from the generated list to your project by clicking on the "+" button beside it. All added test cases will be written to a file named <file>.test.js, where <file> is the original name of the file you're generating tests for.

The tests will be designed for Jest, which is a downside for me because I prefer to use Mocha for testing my JavaScript-based applications. The Ponicode team explains on their blog that they chose to integrate with Jest first in order to get as many people as possible to try out the tool. [1] Completely understandable, but it still would be nice if they eventually provide support for other runners such as Mocha.

Let's see how Ponicode works outside of their sample project and on a more real-world project. I started opening up some of my open source side projects to see how well Ponicode would work for them. Ponicode only supports JavaScript at the time of this writing, so none of my React, TypeScript, Golang, or Python side projects will work. Also, only globally exported functions are supported at the moment, meaning no classes, static methods, getters/setters, or constructor methods. In addition, class instances, callbacks, and symbols aren't supported either. [2] This makes Ponicode pretty restrictive to just the JavaScript ecosystem at the moment, but hopefully support for other languages and frameworks will come soon.

Lately, I've been making updates to an open-source Atom extension that I created and currently maintain called syntaxdb-atom-plugin. One piece of functionality of the code, for percent encoding search terms sent to the SyntaxDB API, looks like this:

export default class PercentEncoder {
    static percentEncodeChar(char) {
        return '%' + char.charCodeAt(0).toString(16);
    }
    static percentEncodeStr(str) {
        var result = '';

        for (var i = 0; i < str.length; i++) {
            result += this.isReservedChar(str[i])
                ? this.percentEncodeChar(str[i])
                : str[i];
        }

        return result;
    }
    static isReservedChar(char) {
        return reservedMap.has(char);
    }
}

As mentioned above, static methods unfortunately do not work with Ponicode at the time of this writing. But, since this entire static class is better off as a single exported method anyway, I can refactor this and hopefully get a real-world example of Ponicode-generated tests for this article.

After performing the refactor, I was able to get Ponicode to generate some test cases for me:

Very nice! We can see here that some characters within the strings are URL encoded, and after double checking with the Wikipedia article on percent-encoding, I can safely use these as expectations within my tests.

Ponicode isn't perfect though — in one of my old projects back in 2015, hacka-news, I have a function that takes in an array of Hacker News article IDs, and only returns a slice of those ids up to the limit number that the user requested.

var pruneResults = function(ids, limit){
    var prunedIDs = ids.slice(0, limit);
    return prunedIDs;
}

Unfortunately, Ponicode does not seem to pick up on what ids and limit mean.

If I change the parameter names to arrayOfIDs and limitNum respectively, the results look more suitable:

I can also adjust the parameter name in the percent encode function from the earlier example to get Ponicode to generate better test coverage:

Ponicode also provides a feature where it invokes your program, observes the inputs and outputs of your functions, and uses those observations to improve test cases further. I decided not to use this as I don't believe it's very practical for real-world test development, especially if you're working with a library (such as hacka-news) or an extension (such as syntaxdb-atom-plugin), which are not normally invoked directly, and will require writing scaffolding code to get the Ponicode runner to execute code within these projects.

During my research, I also found an insightful article on Ponicode from Sylvain Leroy, who has much more experience than I do in this area. [3] I recommend checking out his article if you want to learn more about Ponicode, as he provides a more critical analysis of the tool and suggests some great ideas that can improve it further.

Overall, this addon has a lot of potential, even as just a nice GUI frontend that allows me to plop in test cases easily to get things rolling. However, where this addon falls short is the results of their AI-generated test cases, which seem to mostly provide irrelevant test cases and require a bit of tweaking of function parameters to improve. Despite this, I believe that the tool can have the potential to produce higher quality test cases in the future. Ultimately, I don't see myself using this tool for day-to-day developer work, but I am interested to see future developments of the tool — hopefully with more language support, more accessible features, and more effective test case generation.

[1] https://blog.ponicode.com/2020/08/14/jest-versus-mocha-which-testing-framework-for-you/

[2] https://ponicode.com/docs/using-ponicode/testable-functions

[3] https://sylvainleroy.com/2020/07/23/ponicode-my-feedback-and-a-mixed-overall-feeling-about-the-tool/

Cover Photo by Digital Buggu from Pexels