By Brainbox Consulting Software Testing April 17, 2020

What is IoT?

The Internet of Things, known as IoT, refers to the collection of physical devices that are connected to the internet, gathering and sharing data.

These devices can sometimes talk to other related devices and act on the information shared, a process called machine-to-machine (M2M) communication. Humans can also interact with these devices to set them up, to give them instructions, to gather information or access data.

The IoT involves extending Internet connectivity to beyond standard devices; like desktops, laptops, smartphones or tablets. This includes everything from cellphones, coffee makers, washing machines, headphones, lamps, wearable devices and almost anything else you can think of.

Source: iso

Building Blocks of IoT

Physical devices/Sensors
Gateway
Cloud
Analytics (optional)
Use Interface (Mobile/Web Applications

How IoT works?

An IoT system consists of smart devices that are web-enabled and use embedded systems, such as sensors, processors, hardware for communication and software applications, to collect, send and act on the data they gather from their environment. IoT sensors share the data they gathered by connecting to an IoT gateway from which the data is sent to the cloud to store or to be analyzed or to take certain action depending on the situation.

IoT devices can communicate with other connected/related devices, share the information among them and react to it as necessary. Users can also interact with these smart devices to access data or give them a command to perform certain actions.

Artificial Intelligence (AI) and Machine Learning are widely being used in IoT today to make the data available to users according to their personal preferences, thus improving the efficiency of IoT devices and enhancing user experience.

What is IOT Testing?

IoT Testing means checking/testing of IoT devices. There is an increasing need to deliver better and faster services and a huge demand to access, create, use and share data from any device. The thrust is to provide greater insight and control, over various interconnected IoT devices.

IoT brings many new things to the table in terms of testing requirement, including quite a few unusual device types like home appliances, watch sensors etc.

With such an eclectic mix of products in the IoT ecosystem, it becomes imperative for the QA team to come with an effective test strategy to meet the quality of these diverse products. Hence, the IoT testing framework is important.

Why testing is important in IoT?

IoT is a rapidly developing market. Count of devices and software in IoT is increasing day-by-day. According to Gartner, approximately 20.4 billion devices get connected by the end of 2020. Delivering robust, high-quality IoT solutions quickly to the market is a key requirement for businesses.

IoT system involves different devices, applications and data systems which makes it very complex and each of these components has different requirements for testing. The lack of interoperability between these components can cause issues. The technology companies develop solutions independent of each other, using different platforms and this may result in issues in the integration of components with one another.

End-users benefit from increasing connectivity of devices, but this, in turn, adds to the complexity in development and testing of IoT solutions.

Hence it is essential to implement a robust testing process that covers not only the device but also the interaction of the device with the cloud and mobile applications (including all the integrations), to ensure the product is functioning as intended.

Iot Testing Challenges:

Platform Diversity

IoT devices use different protocols such as as Message Queuing Telemetry Transport (MQTT), Extensible Messaging and Presence Protocol (XMPP), Constrained Application Protocol (CoAP) among others to communicate with controllers and each other. Hence it is required to test them across various protocols to ensure the devices work seamlessly.

Cloud Diversity

IoT ecosystem can use different cloud platforms to send and collect data and to connect different components of the system. Some of them are Azure IoT, IBM Watson and AWS. So it is imperative to test IoT devices across these cloud platforms to guarantee their usability.

Security and Privacy

The amount of data collected and shared by the connected devices in an IoT system is huge. The concern of data leak and unauthorized access rise due to this high volume of data sharing. So it becomes critical and of high priority to identify the vulnaribilities and address them.

Real time data velocity

Connected IoT devices need fast communication channel and this is ensured by good network status. The devices can face problems due to slow internet connection, bad network infrastructure or problems with network hardware, affecting their performance. So IoT devices and applications must be tested under these different conditions to avoid any loss of data if in case any of the above issue appear.

Interoperability

Interoperability between components of an IoT system like physical device, gateway, cloud and application is a key challenge. Each IoT device has its own hardware and application software. Th software interacts with hardware to issue commands or analyze the data collected by the device. Since there are so many variants of hardware and software, as well as different versions of firmware and operation systems, it is a real challenge to test all the combinations and interoperability.

Scalability

IoT solution deployment entails thousands of interconnected devices, which connect to servers over network. Server infrastructre is built from applications from different vendors and on multiple interconnected services. Testing such complex environment and simulating real time situations is a challenge.

User experience

A consistent and hassle free user experience is most important for any IoT solution. Hence to test the user friendliness of the IoT devices and applications across all platforms of mobile, desktop or any other smart device is very critical and also challenging.

IoT testing strategy

Device-level testing

Device Communication:

Testing should be done to verify that the data transmitted between the device and gateway is linear and properly acknowledged. If the communication between them doesn’t work properly, there may be a chance of data loss.

Connectivity:

Connectivity test validates both wired and wireless network configuration of the smart device and it can switch between different networks and connect to different security settings.

OTA/Upgrade:

Tests should be done to ensure the device performs as expected without any data loss when a new software or hardware upgrade occurs. Also, it should be tested to make sure that they can be downgraded to previous versions if necessary.

Cloud level testing

Most of the test at cloud level (Unit/Integration/System and Performance ) is API based and are easily automated. These test should be part of continues integration system to provide early feedback.

Performance/Scalability/Reliability: The two most important aspect of the cloud are:

How quickly it responds to the user actions
How well it scales to accommodate user demand

So testing should include these aspects. Example of load test would be to simulate a million users accessing the service at once and observing how the service responds.

o What’s the maximum level of latency that occurs if you scale from a thousand to a million units?

o If the servers are resilient enough, the customer should experience at most a “glitch,” but should still be able to continue using the connected device.

Security

The purpose of the security test is to make sure the data flowing through the cloud to all its endpoints is protected. This is particularly important for the devices that collect and store sensitive information. One way to test cloud security is through penetration testing. This involves locating vulnerabilities at data paths to ensure that the firewalls, and other security measures, are set up correctly and doing their job.

Data Integrity:

IoT is all about data sharing, hence test should be designed to assure that the data remains consistent through data creation to data consumption/aggregation.

Application-level testing

Performance Testing:

Performance testing is necessary to determine the scalability, reliability, stability and responsiveness of the application. Without this test, the software may suffer from issues such as slowness, crashing or inconsistent behaviour across different platforms or when under load.

Compatibility Testing

Due to diversity in mobile devices, browsers, operating systems and platforms, a compatible test should be designed to check if the application behaves as expected in all the conditions.

Usability

The Usability testing is done to measure the user’s ease of handling and experience of the application.

Connectivity

Since the application will be using the internet, it is necessary to test its behaviour when it changes between the network, when there is a loss of connectivity or when there is a slow/intermittent network. Also, it should be tested on different network speeds/types like 2G, 3G, LTE or WiFi etc.

False tolerance tests:

These are negative tests that check abnormality of the application when subjected to undesirable conditions.

Some of the common false tolerance tests are when the application is left idle for a prolonged period, kill and relaunch the application etc.

Value-added testing

Exploratory Test

Exploratory testing is an unscripted testing technique used to discover unknown issues. The learning from exploratory tests provides good feedback in designing the future regression test suite.

Field testing:

Generally, the final step is field testing. When you have a product you believe is fully functioning and ready for customers, test it among a small group of 10-20 users in each region you plan to deploy a product.

Regulatory and conformance:

A well tested IoT product may also require the right certifications to enter the market. IoT devices generally have to meet various certification requirements for the network, protocol compliance, device drivers, app store submissions, etc.

Test tools and Frameworks

To test complex IoT solutions effectively and execute a wide range of tests mentioned above, the right set of automation, simulation, virtualization, monitoring tools/frameworks are vital.

Some of the supported tools/frameworks are listed below:

Device/Protocol Simulators:

IoT simulators allow you to design, create and test IoT applications and devices without actually using real IoT boards.

Mobile/Web application Testing tools:

They provide automated functional mobile testing that replicates end-user experience and ensures that the application works as expected.

Commonly used tools are Appium, Selenium, SeeTest, Test Complete, Kobition, Appuim Studio.

Security testing tools:

They can be classified into threat modelling, static code analysis and runtime threat inducing tools.

Tools such as Micro Focus® Fortify on Demand, OWASP ZAP, VGC and Microsoft®. Threat Modelling Tool identifies threats, prioritize them and also provide recommendations on how to fix them.

API testing tools:

Most cloud solutions are now built using REST APIs and Web services. Tools such as Postman, SoapUI, Progress® Telerik® Fiddler™, etc. test their connectivity, response, and performance.

Performance testing tools:

Some of the commonly used tools for application performance testing are Jmeter, Blazemeter, Neoload, Loadrunner, etc.

Conclusion

IoT involves the interconnection of multiple components. This makes the interaction and compatibility between the components highly complex. So a good testing strategy with

emphasis on Test-As-A-User (TAAS) approach and use of appropriate tools, can ensure quality and robust IoT solution to be delivered.

The value of Testing in Internet of Things (IoT)