Testing of Mobile Applications

Introduction

Mobile Application Testing is a process by which application software developed for handheld devices is tested for its functionality, usability, and consistency. Apart from functional testing, which – most importantly – ensures that the application is working per the requirements, performance (behavior & performance under certain conditions; bad connections or low battery), security or compliance testing are also very important. Usability is one of the keys to commercial success and therefore testing is crucial to verify if the application is achieving its goals and if it is getting a favorable response from users.

Mobile Application Testing Types

Mobile application testing can be done in two ways, Testing by Engineer and Automation. We have discussed in detail about test automation of Mobile applications in the article here.

• Testing by Engineer

Testing by Engineer can be described as the part of mobile testing that requires human input, analysis, or evaluation. Consequently, this kind of testing is a user-centric approach, mainly focusing on explorative ways of monitoring whether a mobile application meets user requirements and expectations. Especially when it comes to the testing of the application’s look and feel, the usability or the reproduction of bugs, testing by Engineer on real devices is vital.

How to perform mobile application testing by testing by Engineer?

To perform testing by Engineer, you must organise and standardize the tests. By taking these steps, you can repeat them in any device without any changes in the flow. And you will be able to reproduce an error again if it is necessary.

Without organizing, you are just going around in circles and hoping to find something wrong. Organizing tests will save time, and it will be easier to reproduce errors. There are a few steps to follow to create testing by Engineer.

• • Plan the test

Most things start with a good plan, and testing is no exception. Your first step when preparing a test is to plan out the steps that will comprise the test, and your route for executing them.

• • Test

Upload the app to be tested on the platform. With the test planned, run it. Test everything you have planned, note the route you used and how each point was addressed to avoid confusion when the test is run again. When an error appears, try to reproduce all the steps the same way you did the first time. Determine what caused the error.

• • Note the errors

Registering errors is necessary to make them easier to reproduce again. Show your test to the developers to help find out what is causing the errors. After solving each of the problems, perform all tests to verify that nothing was broken. Use an appropriate tool to share your test results with QA and the development team and keep a history of your tests and issues.

• • Repeat

We must repeat all the steps again after performing any kind of code change, reproducing the tests exactly as from the start, and remembering to note any changes, thus reducing the test error rate of testing by Engineer.

• • Select devices

Select different devices to perform tests. Try distinct device brands, platforms, and operational systems. Differentiate whenever possible the hardware and software, check the version of auxiliary software, and take note if there are differences between updates. And because there are often differences between screens and resolution, checking if the app is responsive is mandatory.

Advantages and Disadvantages of testing by Engineer

Advantages

• • It is more cost effective in the short-termed period.
  • Testing by Engineer is more flexible.
  • Better simulation of user actions.

Disadvantages

• • Testing by Engineer test cases are hard to be reused.
  • Less effective of execution certain and constant task.
  • Test running process is slow.
  • Some kinds of test cases couldn’t be executed testing by Engineer (load testing).

• Automation testing

Test automation can be described as the process of automating different test cases or scenarios by using a specific tool and language, instead of testing manually with a group of human testers and physical devices.

How to perform mobile application automation testing?

Selecting the right mobile testing tools for script generation is the first step toward automated testing. Having chosen the perfect tool, you should start thinking about setting up the testing environments.

Benefits and drawbacks of automated testing

Benefits:

• • The biggest benefit of automated testing is time savings, especially achieved through regression testing and continuous integration.
  • Another important advantage of automated testing is cost reduction. Since it requires fewer resources, test automation also reduces the overall costs involved in testing.
  • Furthermore, testers benefit from the repeatability of tests while using an automated approach. This helps to avoid the risk of human errors.
  • A powerful automated testing test suite helps to ensure that test scripts are kept up to date, and testers can cover every feature (increased coverage) within the application.
  • Finally, automated tests are re-usable and can be applied on different versions of the software – even if the interface changes.

Drawbacks:

• • The initial effort of automated mobile testing seems often rather high due to initial direct costs.
  • Furthermore, automated testing requires expert knowledge especially when it comes to writing automated test scripts.
  • Not every test can be automated and not every project is suitable for automated testing.

Mobile Application Testing Techniques

There are different types of testing which can be performed on a mobile device. Some of them are,

• Functional Testing
• Compatibility Testing
• Localization Testing
• Performance and Stress Testing
• Security Testing
• Memory Leakage and Power Consumption Testing
• Interrupt Testing
• Usability testing
• Installation and un-installation  testing
• Certification Testing

Consider testing of a mobile application for online flight ticket booking system for example test scenarios in the mobile test techniques below.

Functional Testing 

Functional testing is the most basic test for any application to ensure that it is working as per the defined requirements. Similar to other user-interface based applications, mobile applications require several human interactions in user scenarios.

Example test scenario:

• Verify that flight availability is displayed for a chosen source destination on a selected date only.

Compatibility Testing 

Compatibility testing has got the highest stack when it comes to mobile application testing. The purpose of a mobile app compatibility test is to ensure an app’s key functions behave as expected on a specific device. It is not possible test the compatibility on all available devices. So, depending on the customer requirement and market study, select top 10 devices of most used phone models and OS. This can either be physical devices or cloud farm of devices accessed through tools like Browser Stack.

Example test scenario:

• Verify that flight search is performed successfully with Android device.

Localization Testing 

Nowadays, most of the apps are designed for global use and it is very important to care about regional trails like languages, time zones, etc.  It is important to validate the app’s functionality when time zone changes.

Example test scenario:

• Verify that there is no UI or data truncation issue when we use the mobile app with different languages.

Performance and Stress Testing 

Mobile performance test covers client application performance, server performance, and network performance. With the help of performance testing tools, we can identify the existing networks, servers, and server-side application bottlenecks, given the predefined loads and transaction mix.

Stress testing is a must to find exceptions, hangs, and deadlocks that may go unnoticed during functional and user interface testing. Some of the criteria for stress testing are as follows:

• Load your application with as much data as possible to try to reach its breaking point.
• Perform the same operations over and over again.
• Perform the repeated operations at varying speeds – very quickly or very slowly.

Example test scenarios:

• Verify that flight available check takes only a reasonable amount of time.
• Check 1000 users are accessing the mobile app to search for domestic flights.

Security Testing 

Vulnerabilities to hacking, authentication, and authorization policies, data security, session management and other security standards should be verified as a part of mobile app security testing. Applications should encrypt username and passwords when authenticating the user over a network.

Example test scenario:

• Verify that the application does not get operated with same user credentials on two different mobile devices.

Memory Leakage and Power Consumption Testing 

Memory testing is exceptionally important for mobile applications to ensure that each application maintains optimised memory usage throughout the user journey. One more important test is checking the battery usage while the application is running. It is important to make sure the battery usage is minimum and does not affect the device.

Example test scenarios:

• Do flight availability checks for ten times and note down increasing memory usage for each check.
• Use the mobile app to search for flight availability and check the power consumption remains minimal.

Interrupt Testing 

An application, while functioning, may face several interruptions like incoming calls or network coverage outage and recovery. So, testing should be done to verify app handles these interruptions well. The state of the application should not change because of interruptions.

Example test scenario:

• Verify that flight availability check gets paused and resumed back after receiving an incoming call.

Usability Testing 

Usability testing evaluates the application based on the following three criteria for the target audience: Efficiency, Effectiveness and Satisfaction.

It is very important to get usability testing in place from the early stage of the application design, and it should not be done only when the application is completed. Usability testing requires heavy involvement of users, and the output might affect the application design, which is very difficult to change in the later stages of the project.

Example test scenario:

• Flight availability check should be on the home page.  Sponsored advertisement should not be displayed in the middle of the content.

Installation, Un-installation and Updates Testing 

This testing verifies that the installation and un-installation process of the application goes smoothly without the user having to face any difficulty. It is also important to test the application works as expected when an update is done on it, without breaking any functionality.

Example test scenarios:

• Verify that the installation process is smooth and does not take long.
• Verify that all the application related files are removed successfully after un-installation.
• Verify that the application works successfully after automatic update.

Certification Testing 

To get a certificate of compliance, each mobile device needs to be tested against the guidelines set by different mobile platforms.

Example test scenario:

• Verify that the application adheres to the policy of iOS phones when installed on iPhone.

Testing different types of mobile applications

There are 3 types of mobile applications: Web, Native and Hybrid.

• Web mobile application testing:

Web mobile apps are websites optimised for mobile browsers. These can be tested similar to web browser applications. These uses user agent ad-ons that help render the content of mobile we browser on desktop browser.

What to test in Web mobile applications:

• • UI testing
  • App performance
  • Connectivity issues
  • Battery usage

• Native and Hybrid applications testing:

Native mobile apps are smartphone apps specifically designed for a particular operating system—iOS or Android. Whereas Hybrid apps can be well described as a mixture of native app and mobile web app. They are a way to expose content from existing websites in App format.

What to test in Native and Hybrid mobile applications:

• • Functional testing
  • Compatibility testing
  • Connectivity testing
  • Performance testing

Some special scenarios to which importance should be given while testing mobile applications

• When an application switches to a different application while performing an action

Take an example of an online shopping mobile application. In this case, when you order a product from the app, it will redirect you to a banking website to complete your payment. This is a crucial testing point for an application. It is important to make sure the transition from the shopping application to banking application happens without any issues and also comes back to the shopping app once the payment is complete. It is important to make sure no data is lost in the process.

The way to go for testing in such situation is to do API testing. API testing will make sure that the interface of the application successfully transits to a different application and vice versa.

• When there is an interruption while the mobile app is running

When some interruptions occur while the application is running, like receiving a phone call or notification, it is important for the app to behave as expected when the interruption is over. The state of the application should not be affected by the interruption that occurred. The app should run in background during the handling of interruption by the user and then resume its state as before.

So, Interrupt Testing is a subset of Functional Testing for a Mobile Application. And, to conduct Interrupt Testing, you would follow the same Mobile Application Test Frameworks and Tools. It is the skill of the testers to conceive these scenarios. Once done, you would design the test cases and execute in the exact same way as any other test.

Case Study: Mobile Application Testing and Automation

At Brainbox, we provided mobile application testing service for a client who were launching their mobile application. Their application was the smart and free app to personalise your photos and videos by simply clicking your logo from the keyboard, add a message and instantly share it on your social media platform. A very simple way to generate brand awareness which makes online marketing easier for the companies.

​For the best possible solution to get application to market on time and within budget, a comprehensive testing strategy is required. This mobile testing strategy should include a device and a network infrastructure, an optimised selection of target devices, and an effective combination of testing by Engineer and automated testing tools to cover both functional and non-functional testing. We provided both testing by Engineer and automated testing (using our Brainbox Framework) services for our customer. We will see the approaches that we followed.

Customer’s main requirement was,

• To verify the application is working as expected on the selected mobile devices (Device Compatibility) ​
• Feedback on the “Ease of use” ​

Challenges:

• Client is commercially launching their app in a few weeks.
• Usability, functionality, compatibility, and security areas needs through testing to avoid surprises/risks (users may uninstall app as soon as they install) ​
• No detailed requirement specifications.
• It was a challenge to perform testing on handheld devices, and then also develop a automation test suit.

Test Strategy:

• Understand context, technology, architecture
  • Review system requirements, review software design and architecture
  • Understand Context of the business
• Define test strategy
  • Test Data and Test Design
  • Test Plan outline
  • Requirement coverage
• Develop system test specification
  • Detailed test plan
  • System test environment
  • Test cases and Test data preparation
  • Requirement traceability matrix
• Test Execution
  • Defects
  • Test Reports
  • Assess Application Maturity and fill gaps

Functional Testing:

• 100 % System Requirement Specification (SRS) coverage of all testable functional requirements​.
• Automation of critical and most used functionalities.

Non-functional Testing:

• Usability​
  • Screen Orientation (Landscape/Portrait) ​
  • Accessibility of menu items​
  • Error Messages​​
  • Gestures tests​
  • Switching between application​
• Performance​
  • Application size​
  • Startup time/Launch time​
  • Responsiveness of the application​
  • Memory, CPU utilization and Battery Consumption​
• Compatibility​
  • Testing the application on Multiple devices/Different Operating Systems/Screen Resolutions
• Network Specific Checks​
• Installation/Upgrade tests​

There was a weekly status report shared with the stakeholders on progress and deliverable. Defects were logged and defect management board meeting was held twice a week.

Client Benefits:

The product was delivered on time and within budget, and the client was fully satisfied with our proactive communication, international standard of delivery, support, and effective documentation process. The following are the key benefits we added to our quality service during the test management.

• Brainbox test team worked like a product partner with a strong belief in shared growth and exceeded quality expectations of the client.
• Our extensive experience in providing independent testing services helped the client take care of their product at different stages smoothly without any botheration and remain focused on sales and strategic planning.
• On-time delivery and on-schedule reporting provided the client an edge in product’s market introduction and saved their time.
• Our expertise in mobile test automation and testing capacity provided the client guaranteed quality assurance.
• Accelerated QA process helped our customer to take fast and confident decisions regarding release cycle and market launch.

Summary

Mobile application testing has become an integral part of application development process, with increase in development of critical and complex mobile applications. At Brainbox, we provide solutions of testing by Engineer and automation of mobile applications, using our in-house test automation framework.

Contact us at sales@brainbox.consulting for more information and we would be happy to assist you with testing related services.