The Internet of Things is one of those concepts that seemed like science fiction only a few decades ago but are totally real these days, just like self-driving cars or delivery robots. IoT devices surround us everywhere, and their presence is only becoming more ubiquitous. By 2025, there are projected to be over 75 billion connected IoT devices, and the global IoT revenue is expected to hit $96 billion the same year.

The IoT market is a fascinating example of the most advanced use of technology that benefits businesses, regular users, and even whole cities. However, behind every successful Internet of Things product, there are many rounds of testing and quality assurance. In this article, we will talk in detail about IoT testing: what it entails, how to do it the right way, which challenges you can face along the way, and why, despite all challenges, IoT QA is still a critical aspect of creating Internet of Things products.

What is an IoT device?

IoT, or the Internet of Things, refers to physical devices that are equipped with various sensors and can connect to other devices via a communication network. The IoT technology combines all the available components into a system that helps make fast and effective decisions based on real-time data.

IoT is the most common term used to address this group of devices. Still, there has been a lot of discussion in the tech community about whether it’s the right term to use because the internet is only one way for an IoT device to connect to other devices and transfer information.

Where are IoT devices used?

The IoT technology has hundreds of potential applications, and this number is only growing by the year. We can divide all the possible uses of IoT devices into two categories: business and personal.

Business, or professional uses of the IoT technology, can include a variety of applications in enterprise management, agriculture, and the creation of the smart city environment. Personal uses, on the other hand, typically involve home security, smart home technology, and healthcare, although the latter stands at the intersection of professional and personal uses.

What is IoT testing?

The Internet of Things testing is an accumulation of efforts to ensure the quality of an IoT solution. This can include a variety of testing types, methods, and tools. Typically, IoT QA consists of checking the functionality, performance, UX, security, and network connectivity of an IoT device and the infrastructure surrounding it.

“Testing IoT products is more challenging than testing a regular software product simply because most solutions need to be tested completely from scratch. While there are plenty of approaches, scenarios, and toolsets for testing a mobile or a web application, testing each IoT product requires a completely different strategy that usually cannot be used for another solution.”

Andrii Nikitenko, Head of Operations at TestFort

What needs to be tested in an IoT device

IoT devices are a unique combination of hardware and software, and the hardware and software in question are usually absolutely unique. This is why two IoT QA strategies are never the same. At the same time, an IoT device can be broken down into several different layers that need to be tested both individually and as a system. Here are those layers, or, as it’s commonly referred to, the IoT testing framework.

Device layer

This is the layer of an IoT product that is mostly physical and therefore requires IoT testing experts with both hardware and software testing expertise. This layer typically includes various sensors, actuators, and control elements, as well as the firmware that is responsible for its correct operation. It’s worth noting that this part of an IoT solution often performs in challenging environments that need to be accounted for and simulated in the testing process.

Connectivity layer

This layer is responsible for transferring the data between the device layer and the remaining layers. Depending on the specifics of the product and its architecture, IoT testing on the connectivity layer may deal with gateways, connectivity protocols, and network communications. This is one of the most challenging layers to test because of the endless variability of settings and protocols.

https://j6v5t8r7.stackpathcdn.com/wp-content/uploads/2022/12/2-IoT-Testing.png

Data processing layer

At the data processing layer, the IoT QA team will investigate how the solution handles the data obtained from sensors and transferred through the network. Specifically, it’s important to look at how data is stored and processed, as well as how it’s analyzed by the system. For example, the team will check how the solution handles data storage to avoid memory shortage, or whether there’s anything standing in the way of efficient data processing.

Application layer

The application layer of an IoT solution mainly has to do with the software side of the operations — specifically, the front-end and the back-end components of the product. This can include the user application, the control application, and possibly the BI application, provided that the nature of the product calls for its use. This layer is the closest one to traditional software testing but still faces unique challenges.

“For us, the first step in creating any IoT testing strategy is analyzing the current state of the product and the client’s requirements. We look at not only whether the product adheres to the requirements, but also whether the requirements themselves adhere to the needs of the market. This is where relevant IoT QA experience comes into play. For example, if the product is missing the feature of Over-The-Air updates, we will always suggest the idea of adding this feature to the client, so that the product gets one more competitive advantage and complies with the industry standards.”

Andrii Nikitenko, Head of Operations at TestFort

Types of testing commonly used for IoT QA

Although each Internet of Things testing project is different from the rest and involves its own sequence of steps, there are certain types of IoT testing that can be found in pretty much any IoT QA project. Here are the IoT testing types that every Internet of Things testing endeavor needs.

“Testing a web or mobile application or a website more or less follows the standard procedure, and there are elements that need to be tested in every regular software product. The situation, however, is different with the Internet of Things testing. There are many devices that don’t have a user interface, which means UI testing is irrelevant there. And then there are devices that have a connected web, mobile, and desktop application, so the scope of testing is significantly bigger. Overall, there cannot be two identical IoT testing projects.”

Andrii Dumin, Hardware QA Engineer at TestFort

Functional

IoT functional testing is responsible for checking whether everything in the device and the application is working properly and conforms to the functional requirements. This can include testing the operation of the product, as well as GUI testing, which serves to confirm that every interface element is in the right place and corresponds to the right action.

https://j6v5t8r7.stackpathcdn.com/wp-content/uploads/2022/12/3-IoT-Testing.png

Usability

The absolute majority of IoT devices are designed to be interacted with in some way by real human beings, which is why usability testing is a must. With this type of testing, the IoT QA team will make sure that the device and the application it’s connected to create a positive user experience regardless of the user’s level of technical proficiency.

Smoke

Smoke testing is one of the IoT testing types that are present in nearly every software product released to the public. Smoke testing is one of the first steps in the QA process and helps the team make sure that the current build is stable enough and ready to be tested further. Skipping this step, or insufficient smoke testing, can lead to disrupting the whole testing process and to a significant portion of the testing budget going to waste.

Performance

When it comes to IoT device testing, it’s important to remember that many of them operate under unusual conditions and handle large amounts of data. This makes Performance testing one of the types of IoT testing that can be successfully automated, provided that the IoT QA testing team has the necessary expertise and possesses the proper selection of tools.

Compatibility

Some IoT solutions are designed to work on just one platform, while others are intended for use on a myriad of devices, platforms, and operating systems. This is why compatibility testing is important for all IoT projects but absolutely crucial for many of them. For this type of IoT testing, it’s very important to run the tests on as many devices as possible, and this is one of the instances where TestFort’s collection of over 250 physical devices proves particularly helpful.

Integration

Due to the fact that a typical IoT solution consists of multiple modules, it’s important to test these modules not just independently, but also as part of the integration process. In other words, the QA team will look at how successfully the new additions to the existing setup — whether it’s a new piece of software or a new hardware module — are integrated into the system.

Scalability

One of the ultimate goals of an IoT product is to increase the number of customers and ensure presence in as many territories as possible. This is why it’s essential to include scalability in the IoT testing process. It will help the stakeholders avoid any unpredictable performance issues the solution can face as the customer base starts growing.

Connectivity

The connection between the different layers of the IoT solution is what actually allows it to be called an Internet of Things solution. Without a stable connection, an IoT device is just a piece of plastic that cannot do much. This stage of IoT device testing should involve checking the mobile and wireless connectivity, as well as any related types of connectivity that are specific to the product.

Security

By some estimates, around 70% of all IoT devices globally are currently vulnerable to security attacks. And with the majority of IoT solutions dealing with sensitive data in one way or another, whether it’s the smart home technology or a healthcare device, security is not something to be taken lightly.

Localization

With some exceptions made for IoT products designed for a limited local market, IoT devices are typically marketed and sold all over the world. Therefore, English being the only available interface language won’t cut it. Localization testing not only tests the correctness of the translation, but also makes sure the product complies with local standards and requirements.

Regulatory

IoT products are a unique type of software and hardware that has unique characteristics and often deals with the most sensitive aspects of human existence. From home security and smart cities to medical devices — the IoT market is intensely regulated. This makes regulatory testing one of the key IoT testing types for any solution.

“QA engineers do not function as robots, working only within the specified requirements. Instead, we use a comprehensive approach to QA. For example, even when the client only requests us to do performance testing or localization testing, we will still be able to spot bugs outside of the given area and report them to the client to ensure a higher quality of the product”

Andrii Nikitenko, Head of Operations at TestFort

IoT QA team: Ideal composition and training

We’ve already talked about the IoT testing process, types, and other aspects of quality assurance for the Internet of Things products. However, behind every successful testing project, there is a team of IoT testing experts. So who can test an IoT device, how many people do you need to test a product successfully, and what kind of training is required for the job?

To answer this question, we spoke to the most prominent IoT testing experts in our company. Here is what they think:

• As it’s often the case in software testing, the size and seniority level of the team is determined only by the project specifications. It’s entirely possible to test a small IoT device with a team consisting of just one QA. But there are also projects that require hundreds of testers to comprehensively test the product and obtain reliable results.

• With the exception of tiny QA projects that can get away with just one experienced engineer, the minimal size of a QA team is 2 to 4 engineers plus 1 QA lead. Naturally, if the project requires automation, an automation QA is also necessary.

• Although prior experience with IoT devices is a big advantage for a QA engineer, it is not always a deal breaker. In other words, a QA engineer with a background exclusively in software testing can successfully test IoT devices, provided they are ready to learn.

• On the other hand, when a team consists exclusively of engineers with no IoT testing experience, it can slow down the project and potentially cause some bugs to go unnoticed. This is why the team must include at least one engineer with relevant experience.

• To be good at IoT QA, the engineer must possess some important skills and qualities — most importantly, an ability to think outside the box and find effective solutions in a dynamically changing environment. For example, an IoT testing engineer needs to be able to quickly think of a way to test a parking meter that is designed for use in Canada in sub-zero temperatures to make sure it’s able to withstand drastic weather conditions.

“I firmly believe that a person with just a software testing background can successfully test IoT solutions — of course, provided that there is at least another team member with relevant testing experience. After all, all of us regularly test IoT products in real life when using countless devices that surround us. So while the Internet of Things devices are definitely unique, IoT testing is not out of bounds for beginners.”

Andrii Dumin, Hardware QA Engineer at TestFort

How much does testing IoT products cost?

Given how much unique hardware and software is needed for a typical IoT product, developing an Internet of Things solution typically costs a lot. Still, it’s important to take into account the expenses linked to testing the product. Similarly to testing regular software products, there are dozens of factors that can influence the cost of the project. Here are just some of them:

1) The size and seniority level of the team
2) The presence of strict deadlines
3) The amount of functional and non-functional requirements that need to be tested
4) The complexity of the requirements
5) The need to create specific test conditions
6) The need to implement testing automation
7) The use of unusual testing techniques

The factors listed above can be more or less applied to testing most software and hardware products. However, when testing IoT solutions, the team operates on a different scale. For example, while regular software products need to be tested on a specific number of hardware/software/OS combinations, many Internet of Things products require the use of specially designed physical environments, which understandably leads to extra costs.

At the same time, the biggest expense of an IoT testing project is the cost of labor. The salary of an IoT QA engineer depends on their level of expertise and, more importantly, location. For instance, a US-based IoT testing engineer makes between $90,000 and $110,000, based on Indeed.com, whereas an IoT & Embedded QA engineer in Ukraine makes between $20,000 and $30,000 annually. This is why many IoT product owners now prefer to outsource their testing needs abroad — it’s an easy way to ensure an impeccable quality of testing while keeping the costs relatively low.

IoT automation testing: Why we need it and how it works

For most software products, testing automation is a handy tool that helps increase the efficiency of testing and cover more aspects of the application. However, it’s rarely the most crucial part of the process. This is not what the situation is like with testing IoT devices. The use of automation is one of the pillars of a well-tested Internet of Things product, and there are several reasons for that:

• An IoT product consists of multiple components that work on different layers, and automation testing can ensure the widest test coverage in a limited time.
• Testing a typical IoT solution runs on a large scale, as there are countless features, combinations of conditions, and performance aspects to explore.
• Automation is one of the few ways to perform in-depth connectivity and security testing of an IoT device, which directly influence its popularity, reputation, and more.

Therefore, the key goal of IoT automation testing is to provide an effective framework for speeding up and enhancing the testing process, resulting in the software of higher quality that goes through faster release cycles. However, the volatile nature of IoT QA and the unique IoT testing challenges teams face require the team to design the automation framework from scratch, which certainly takes time and effort.

“IoT automation testing helps eliminate human error and increase the number of tests, which directly influences the quality of the product. However, automation is not completely replacing manual IoT QA any time soon. No matter how advanced testing automation tools are getting, they still cannot imitate a pair of human eyes and a pair of human hands with 100% efficiency.”

Andrii Dumin, Hardware QA Engineer at TestFort

Possible challenges of testing IoT devices

The benefits and importance of IoT device testing and IoT application testing are not up for debate. And Internet of Things testing as a whole has come a long way in the past few years with new tools, approaches, and best practices helping IoT testing experts be even better at what they do.

At the same time, like any software testing niche, IoT QA is not without its challenges. Here are the top 6 IoT testing challenges most teams encounter sooner or later.

Lack of standardization

Many IoT device types have been around for years and are regulated by certain industry standards. However, there are lots of completely unique IoT solutions that are hard to categorize, and the standardization industry clearly struggles to keep up with the IoT industry. The lack of strict standardization means the QA team has nothing to base the testing strategy upon and no blueprint that can serve as a canvas for testing.

User behavior is hard to predict

There are IoT products that are barely interacted with by human beings other than the setup and maintenance process. These can include different agricultural equipment, factory machinery, and logistics devices. And then there are IoT products that are interacted with every single day, such as medical devices and wearables. This means the team has to account for most of the possible ways the user can interact with the product, which takes time and experience.

https://j6v5t8r7.stackpathcdn.com/wp-content/uploads/2022/12/4-IoT-Testing-1.png

Too many variables

An average IoT product is a combination of hardware, software, cloud technologies, and design features that is not used in any other product, even if their functionality is similar. Moreover, IoT solutions can use different connectivity protocols, so much so that some IoT testing experts believe that it’s not even possible to know every connection present in the device, let alone test it. Too many variables in a product mean a longer, more labor-intensive testing process.

Hardware-related issues

The actual IoT device — the piece of hardware that the user interacts with — is usually autonomous to a degree but still needs to be regularly maintained and have its battery charged. Unfortunately, both things often prove challenging. As a result, IoT QA engineers may choose to go lightly on testing and updates, which, in turn, can make the solution more vulnerable to security issues and jeopardize the whole related network.

Fast time to market

IoT product owners face a fierce competition in the market these days. With some exceptions of absolutely one-of-a-kind products, there are usually several alternatives available for every IoT product you can think of. This is why product owners feel pressured to release their solutions faster than their competitors. This creates an additional challenge for the QA team, who now have to try and complete the same amount of testing in less time.

Increased cost of testing

There are many reasons why testing IoT devices typically costs more than testing regular software solutions. In addition to the possible seniority and prior experience requirements for the QA engineers, it’s also often necessary to set up a testing lab where the team can create the required conditions for thoroughly testing the product. It’s very expensive to do in-house and also triggers additional costs when working with an outside vendor.

Why the Internet of Things testing is so important

The fact that software and hardware testing is essential for ensuring the quality of the product, and therefore the company’s reputation and revenue, is not up for debate. However, testing IoT solutions takes all of that to the next level, as the cost of error for an IoT product is significantly higher.

What’s the worst thing that could happen when an eCommerce site displays poor performance during Black Friday sales? The customers cannot finish the purchase and the company loses revenue as a result. But we probably don’t need to explain the potential outcome of a home security system being infected with malware due to insufficient security testing, the whole smart city infrastructure coming to a halt due to one malfunctioning component, or an insulin pump administering the wrong dosage.

This is why the key QA principle that states that exhaustive testing is hardly ever possible is more true than ever for Internet of Things testing. In other words, the more testing is done and the more aspects of the solution are covered, the better for everyone, including the owners and end users of an IoT product.