In my last post, I raised some concerns with Amazon CodeCatalyst and how it might be received in an enterprise environment, especially the single sign-on capabilities. After that, I spoke with some of the CodeCatalyst team and clarified my perspective for them. I got the sense they were aware of most of the issues I raised, and they seemed very open to my feedback. I can’t share all the details we discussed, but I look forward to what they have coming in the future.
One thing that the initial post left out was my favorite part of CodeCatalyst: the quality reports.
Quality Reports
Amazon CodeCatalyst supports multiple tests, code coverage, software composition analysis (SCA), and static analysis (SA) reports with little to no configuration. CodeCatalyst workflows support attempting to discover the report files automatically. You can also specify where the files are located and provide additional configuration to help fine-tune the generated reports and what causes the workflow to fail.
Test Reports
You can configure CodeCatalyst to display the results of your unit tests or any other automated test run during your builds as a report. The report will show you the pass rate of your tests, how long the tests took to run, and the different test cases and whether they passed or failed. CodeCatalyst supports the Cucumber JSON, JUnit XML, NUnit XML, NUnit3 XML, TestNG XML, and Visual Studio TRX test report formats(1).
Code Coverage Reports
CodeCatalyst’s code coverage report shows the line and branch coverage from an overview and per-file basis. It also details the coverage within different folders in your file tree. It allows you to dig into the details of a specific file and see what portions of the file are covered by the tests. CodeCatalyst supports the JaCoCo XML, SimpleCov JSON, Clover XML v3, Cobertura XML, and LCOV code coverage report formats(2).
Software Composition Analysis Reports & Static Analysis Reports
CodeCatalyst has separate sections in the documentation for software composition analysis (SCA) and static analysis (SA), but there is a lot of cross-over between the two. The documentation lists SARIF as the only supported SCA report file format(3). Three formats are supported for SA reports: PyLint, ESLint, and SARIF(4). In my testing, I had trouble getting CodeCatalyst to flag a report as being a software composition analysis report. It never automatically detected the reports, and when I manually defined them, they showed no findings, even though a review of the files indicated they should show something. I suspect there is a subtle difference in the SARIF formatted output between what CodeCatalyst expects and what the tools I tried generated. Ensure that the SARIF output from your tools matches the standard and that the reports show the output correctly. Otherwise, you could end up with a false result. CodeCatalyst detected ESLint output without any issues.
Reports Access
Accessing the quality reports is easy, and you can view them for a specific workflow run or directly from a pull request. If you have pull requests going through multiple revisions, you can even view the reports for the different revisions. One situation I commonly run into is working with an engineer to improve their unit test coverage and the quality of their unit tests. Being able to jump back to their updates from a week prior quickly will help save time when trying to look at incremental progress.
CodeCatalyst’s Differentiator?
I’ve never seen easy access to code quality reports in GitHub, GitLab, or BitBucket without going through a lot of configuration and integrations with outside tools. This is one area where CodeCatalyst truly shines. When I first saw these reports and how easy they were to set up and access, I immediately shared a project with my boss and some of my peers, and we spent a good amount of time discussing the difference this would have made for us over the years. Other tools provide many of the same features as CodeCatalyst, but when it comes to reports integrated into the solution, CodeCatalyst seems to be in the lead.
Top comments (0)