Automated Testing of Cloud-Based ERP: How to Implement and What to Expect?

A famous quote by Yogi Berra, a renowned American professional baseball catcher, and coach, says: “You can observe a lot by just watching.” When it comes to cloud-based ERP applications, we can paraphrase this phrase: “You can achieve a lot by just testing,” and it will be especially true for automated testing.

Cloud-based applications have a great advantage — flexibility. This allows systems to stay agile and quickly adapt to changes in a rapidly changing business environment. However, this means constant changes with customizations and integrations. At the same time, the cloud software vendor shows up with new releases and another set of changes comes into play. As the implementation of cloud ERP grows, customers have a need to validate their processes after each change. How can you do this, when manual testing approaches are time-consuming and tedious? Here is how we can achieve a lot with automated testing.

In this article, we won’t talk about the advantages of automated testing of cloud-based ERP platforms in theory. Instead of this, we’ll explain how we implemented automated testing in practice and demonstrate how it helped to achieve better results for our client. Let’s get started!

Retail ERP Software: Structure and Challenges

The project we are going to talk about is Comcash, a cloud-based ERP (Enterprise Resource Planning) software with a wide range of features. It includes Point of Sale software and E-commerce software that allows small and medium businesses to sell their products both offline at physical shops and online at e-shops.

The main goal of the platform is to provide retailers with a new way of selling groceries, organizing products and inventory, performing sales, managing customers, managing and tracking warehouse activity, and performing accounting with the help of only one powerful cloud tool. The completely integrated software solution allows retailers to cover all aspects of the process quickly and efficiently.

Before going further, let’s take a detailed look at the project ecosystem that includes web, desktop, and mobile modules. The web module consists of:

• Admin Panel — for managing the whole system
• E-shop — for performing sales online

The desktop module includes:

• POS — a native Windows desktop application where sales are performed
• Desktop application — for tracking and displaying orders

The mobile module has:

• A mobile POS application — used to reduce the load in the stores during the ‘line busting’ hours; with the help of this app customers are able to make a purchase right at the place where the mPOS employee stands in the store
• Mobile application — for managing warehouse activity
• Mobile application — for tracking and displaying orders
• An intelligent display working in tandem with its parent’s POS Terminal that helps customers to track the processing of their order in real-time (what products have been added to the list, products price and quantity, etc.)

As you can see, the project consists of many subsystems, and components, with complex logic, and structure. While the system continues to grow, errors began to appear unexpectedly, and it was necessary to fix them as soon as possible. Moreover, the speed of release of software products to the market is an important part of business development. But a continuously growing project requires much more time and human resources for its development and testing.

At this point, the main tech challenges included:

1. Continuously expanding project meant more and more new features added for testing
2. Extra workload caused a lack of time for the QA team to conduct a complete regression testing
3. Adding new project functionalities resulted in some existing features or even parts of current functionality breaking

All these problems could stop the store’s operations and customer service. The number of bugs in production and requests from the client to fix them could increase as well. Moreover, test cases required the creation of a large amount of test data and combinations, and without prepared data, it was impossible to test as many scenarios as possible. A user without the necessary permissions could have access to reports, sales, or customer information in the Admin panel; there was a risk of entering incorrect data into the system or deleting it altogether. This might result in revenue loss, reputational damage, and operational disruption of the store.

Over time, it became clear that the tech challenges greatly slowed down the testing of new features and the development of the project and might negatively impact the final product.

Implementing Automated Testing: What We Did and What We Got

To solve the tech challenges, we came to the conclusion that it would be more rational to automate the verification of all main scenarios. After analyzing the basic flows of our system, we identified that the most critical is the Discounts functionality from the POS side. This is a large module with complex logic, and mathematical calculations, which could take two weeks of manual time to check. Therefore, we decided to automate this feature. To do this, we needed to explore the possibilities of writing tests for a desktop application, and we found a suitable solution such as Appium and WinAppDriver with results visualization using the Allure report.

This allowed us to pull off most of the tasks from manual QA engineers:

• Test data was created and values were calculated by autotests taking into account all conditions and combinations
• Responses from the server were processed without the participation of manual QA engineers
• A large number of tasks for testing, sales, taxes, and discounts were transferred from manual testing to automation

However, a lot of problems related to the UI and API parts needed to be solved as well:

• A data leak from the Admin panel by requests from an unregistered user
• Stopping the store’s operations and losing customers due to the impossibility of creating a sale or a customer in the Admin panel with the proper permissions
• The accidental deletion of some components in the Admin panel led to the removal of products in the e-shop and POS
• The risk of fraudulent transactions with reports
• The lack of clarity on how the business and its processes are performing due to the discrepancies in financial reports
• Possible income loss and penalties for providing inaccurate information to the tax authorities due to incorrect tax calculations

One More Step to Get Better Results

To make the system more effective and safe, it was necessary to adopt effective solutions that allow for fixing vulnerabilities and covering all possible permission combinations inside the system using API and main user flows for POS and UI modules. As a result, POS, API, and UI autotests were implemented in the project that helped us achieve significant results

The key business benefits for the client included:

• Improvement of customer service by eliminating problems with using the Admin panel options
• Increasing the level of protection and security of system data
• Quick detection of critical bugs that might block the store
• The ability to deploy hotfixes to production frequently and quickly

Thus, we achieved an integrated system of solutions for each direction, which allows us to solve the testing tasks on the project in a comprehensive and quick manner. For example, here are the results of one of the recent releases.

Now we run tests on a regular basis before each release and get detailed metrics which are analyzed for further actions:

1. One of the key metrics to judge the efficacy of the QA process is the bug classification metric. It helps us to identify the most problematic areas of the system and increase test coverage to avoid critical bugs in production.
2. The test coverage metric discovers uncovered areas of the project to make it more robust. Also, test coverage can serve as a basic measure of risk: the more test coverage attained, the lower the chances of production defects.

Once we identify the benefits of autotests within the QA team, we wanted the developers to be able to run tests at any time. For this purpose, integration with CI was arranged. And the stakeholder could catch the clear and accurate ‘big picture’ of the processes on the project at any time.

The Bottom Line: Business Benefits of Automated ERP Testing

By implementing automated testing in the cloud-based ERP software, we managed to successfully solve a bunch of problems with a positive impact on the final product quality and the project itself. Collecting all unique scenarios helped us to make it possible to automate all kinds of combinations and significantly reduce the number of bugs in production. Eliminating the human factor when checking core functionality helped us to avoid inaccuracies in calculations and as a result, to assure our client of receiving correct and real numbers and statistics.

Also, we were able to reduce regression testing time. Having automation testing implemented we managed to save about 30 hours of manual time for the QA team for each release to test new functionality and perform priority business tasks. An average automation testing time is about 33 minutes, and the difference in time savings is more and more noticeable with every release. Optimizing processes in the QA team helped manual QA engineers to be more focused on testing critical functionality. What is even more important, adopting automated testing gave an opportunity to the client to make sure that the key functions of the system are working correctly quickly and at any time.