Test Automation For Blockchain - How to improve development speed and code quality, Part 1

Many blockchain projects don't survive long after hitting the initial production state. For most, the lack of proper software testing is one of the main reasons for their demise. It's estimated that over half a billion dollars worth of cryptocurrency has been lost due to bad code in the last year alone. You probably heard about The DAO's code loophole, which allowed attackers to drain out 3.6 million ETH (worth $70 million at the time) from the Ethereum-based smart contract. Another notorious case was the Parity bug which resulted in over $150 million permanently frozen. Even Bitcoin itself is not immune to hacks. Late last year, a bug discovered in the code allowed malicious individuals to artificially inflate Bitcoin's supply via double input. If the bug wasn't quickly identified and addressed, it could have had catastrophic effects on the network. This is just the tip of the iceberg — there are plenty of smaller incidents caused by inexperienced or inattentive developers that don't make the headlines.

What does this tell us? In development, things can go wrong fast and the outcome can be ugly. This is why software testing is so important for any project utilizing blockchain technology, such as blockchain platforms, blockchain applications or blockchain-based services.

In this article, we will discuss our experience and best practices with software testing while developing Lisk, a blockchain application platform. We will also show you how implementing automation testing improved our internal workflows and code reliability. Lastly, we will show you how you can get involved in testing our open source software.

This is a long blog post, but we've broken it down into bite-sized pieces for you.

Part 1:

1. Introduction to Blockchain and Lisk
2. What is Software Testing?
3. Testing blockchain applications adds new metrics to traditional software testing
4. Why blockchain developers need to pay much more attention to details
5. Automation testing can significantly cut down the release process
6. Different types of automated tests
7. Continuous Integration (CI) is best practice when it comes to automation testing
8. Which CI platform to pick? Travis CI vs CircleCI vs Jenkins

Part 2:

1. Software testing is not enough - introducing Quality Assurance
2. How manual testing slowed down our software development process
3. How we implemented Quality Assurance at Lightcurve
4. The results of having a QA team in place
5. Our blockchain network's QA testing process
6. Get involved in our open source automation testing
7. How to start contributing to our QA
8. Which QA tools can we offer you?

Introduction to Blockchain and Lisk

You've probably heard about blockchain in the context of cryptocurrencies like Bitcoin, but what makes this new technology so special? The blockchain, which is a type of distributed ledger technology (DLT), is an open, distributed database that is able to record transactions among parties permanently and in an efficient, verifiable way. Those transactions are packed into blocks, cryptographically-signed and form the actual chain. Data stored in blockchain can't be altered or tampered with, as all the records are immutable. Once data is saved into the ledger, it stays there forever. The blockchain is also a decentralized network, meaning there is no central authority with control over it.

It's easy to get excited about blockchain, but very difficult to actually develop a working product on top of it. Blockchain developers still seem to be a rare breed of programmers for various reasons, including the steep learning curve required. Lisk plans to tackle these problems head-on by making it easy for every developer to build and deploy blockchain applications in JavaScript. Lightcurve is a blockchain product development studio and consultancy based in Berlin, Germany. We're currently focused on developing all aspects of Lisk, including the product, marketing, community, and design.

What is Software Testing?

Basically, software testing is defined as a set of activities that can be done to guarantee that software is behaving as expected and it's bug-free. A proper software testing process will help you to identify and prevent mistakes, ensure that the actual implementation of particular features matches the requirements and increase overall confidence in the code.

Testing Blockchain Applications Adds New Metrics to Traditional Software Testing

Testing blockchain applications is not much different from testing non-blockchain ones. However, with blockchain, additional test metrics are involved, for example:

• Chain size. The longer the blockchain is, the more data it contains. This results in more space taken on the datastore. The chain can grow really fast and there is no limit on its actual size, as new blocks are constantly being added. We need to know the maximum possible amount of data stored on the chain over a certain amount of time. This way we can estimate how much space a blockchain can take, for example after a year from now.
• Throughput in blockchain is measured by calculating the number of transactions per second (TPS). Higher TPS is always better, but it comes with a cost of increasing overall network load and not every node is able to catch up. TPS as a factor of scalability is a challenge and a hot topic of discussion in the blockchain industry. Many projects are chasing TPS blindly without focusing on other, more important indicators of performance. This often results with projects becoming less decentralized, which in turns negates the ideology of blockchain.
• Security and cryptography. The code needs to be constantly reviewed and audited to ensure that there are no flaws regarding the creation of new tokens, maintaining account balances, verifying blocks, or transactions signatures.
• Data integrity ensures that all the data stored on the blockchain is consistent between network nodes. This is guaranteed by cryptography for blocks and transactions. However, some blockchain applications calculate and maintain states (for example actual account balances) in the memory or helper databases in order to make sure that data consistency is preserved. There is no other way than just comparing data directly between nodes.
• Data propagation is the distribution of data from one node to another. A decentralized network can consist of thousands of nodes and all of them need to maintain a reasonable amount of connections with each other. When some portion of data hits one node, it needs to be forwarded to the entire network without disruption. The time this propagation takes is an important metric.

Why Blockchain Developers Need to Pay Much More Attention to Details

If a bug was introduced in a centralized system, providing a fix would be relatively straightforward. Even if it corrupted some data, it would be easier to correct. This is because in most cases, the company maintaining the application has complete control over the data. Given that blockchains are immutable ledgers, corrupted data is incredibly hard if not impossible to correct. Complicating the process even further, delivering a fix needs to be coordinated with all participants of the decentralized network. With Lisk, we must coordinate every release with hundreds of node operators, as well as block producers called delegates in our DPoS consensus algorithm. Taking all these scenarios into account, the consequences of bugs in blockchain applications can be much more dangerous than in centralized software.

In development, things can go wrong fast and the outcome can be ugly.

Automation Testing Can Significantly Cut Down the Release Process

Now that we have discussed the importance of testing in a blockchain project, we can divide the actual tests according to the way we execute them:

• Manual testing is executed by QA/test engineers and is useful when some of the test scenarios are not yet migrated or ready to be executed in an automated way, such as new features. However, not everything can be done in this way for various reasons, including difficulty, time and budget constraints. Manual testing is overall the most time-consuming method.
• Automation testing is the basis of continuous delivery. It is a development methodology that allows development teams to safely deploy changes to production. Continuous delivery enables developers to find bugs quickly and helps teams deliver new releases with confidence. For complex applications, automation testing can reduce the time required for the release process from months or years to days or even hours. Investing time to implement a high-quality test suite can dramatically enhance developers' productivity. However, it requires some effort to implement and maintain both the test scenarios and the infrastructure for executing them.

Different Types of Automated Tests

We can distinguish a few types of automated tests, such as unit tests, integration tests, and functional tests. However, in some of the existing test suites, those types are confused with each other and there is absolutely no distinction between them. This makes a test suite not well-suited for anything in particular. It's very important for a developer to understand different types of tests as each one has a unique role to play.

• Unit testing is used as a fast feedback mechanism for a developer and "first line of defense" during the development process. In unit testing, particular units of code (functions) are tested independently from each other with simulated input and the main focus is to test the logic in the unit. This means I/O operations (like file access, Ajax/RPC calls, DOM interaction), timers, and expensive functions (encryption) are usually faked.
• Integration testing is a defense mechanism against protocol changes (e.g. argument order) in mocked dependencies in the unit. Stubbing is often avoided in integration tests and actual interaction between units is being tested. Therefore, integration tests are more expensive than unit tests.
• Functional testing, to paraphrase Eric Elliott, is usually considered as a subset of integration tests, as they test all pieces of the application together (in the context of the running application).

For complex applications, automation testing can reduce the time required for the release process from months or years to days or even hours.

Continuous Integration Is A Best Practice When It Comes to Automation Testing

Continuous Integration (CI) is a software development practice which depends on frequent integration of the code into a shared repository. Every time a team member commits some code changes to version control (for example, Git) the automated process of building and testing the code can be triggered. Developers are encouraged to submit their code together with tests (unit, functional, integration) after the completion of every small task (fixing an issue, implementing a feature). Automated build system fetches the latest version of the code from the shared repository and applies changes if needed. It then executes all scheduled actions against it, like executing tests. CI is considered best practice because software developers need to integrate all the changes done by them with the rest of the changes made by other members of the development team. It helps to avoid merge conflicts, difficult bugs, or duplicated efforts. This is because CI forces the code to be merged into a shared version control branch continuously, allowing them to identify possible problems earlier and more easily. It also minimizes both the time spent on debugging and the time required for code reviews, allowing developers to focus more on adding features.

Which Platform to Pick?: Travis CI vs. CircleCI vs. Jenkins

The few popular CI platforms currently available vary by features and flexibility. Some of them are free. For others, you need to pay.

• CircleCI is very easy to start with. There are some free plans available, but with some limitations, like 1 concurrent job with 1 container and no parallelism. It's a cloud-based tool, so you don't need to host the infrastructure on your own. It has various integrations such as GitHub, Slack, Jira, etc. The initial setup is very easy and they have great customer support.
• Travis CI is very similar to CircleCI, however, Travis CI is more flexible. It allows you to run builds and tests on Linux and Mac OS X at the same time, as well as support more languages (they even provide tutorials for them). It's free for open source projects.
• Jenkins is the leading open source automation server and definitely the most flexible one. You need to host it yourself, so it requires some more effort to initially set up and later to maintain. Jenkins allows you to have full control over any aspect of your builds. It can also be extended using plugins and there are hundreds available already, so you can integrate it with basically every tool you want to. While using Jenkins for small projects can be a bit of an overkill, it's great for large ones.

If you want to compare CI platforms in more detail, there is a very nice comparison available on Stackshare.io.

That’s it for Part 1! Check out Part 2 of the walkthrough to read about the importance of quality assurance, our experience in implementing automated testing and links to get involved in the QA testing of our blockchain.