Test Plan for Mobile Application Testing

1. Overview:
This document explains the testing methodology for a mobile application, and is to be used as a guide for the testing activity.

2. Introduction
2.1. What is Mobile Application
A mobile application is software that runs on a hand-held device (phone, tablet, e-reader, iPod, etc.) than can connect to Wi-Fi or wireless carrier networks, and has an operating system that supports standalone software.

2.2. Type of Mobile Application
Mobile Applications are mainly divided into three types.

• Native Application: A native app can only run on one mobile operating system such as iPhone, Android, Window Phone and BlackBerry. For using it user have to install it from market.
• Web App (wap): Web application can run within a mobile device’s web browser. No need to install it, this kind of application delivered on fly as needed via internet.
• Hybrid app: A hybrid app is built using web technologies, and then wrapped in a platform-specific shell that allows it to be installed just like a native app. Facebook is the good example of the hybrid application.

Note: Mobile handset testing is different from mobile application testing.

2.3. What is Mobile Application Testing
Mobile application testing is a process by which application software (Native/Wap) developed for hand-held mobile devices is tested for its functionality, usability and consistency.

2.4. Mobile applications categories
Mobile application is categorized in 6 categories.
• Communications
Email Client, IM Client, Social Networking client etc.
• Games
• Multimedia
Image Viewers, Video Players, Audio Players etc.
• Productivity
Calendars, Notepad/Memo/Word Processors, Banking/Finance etc.
• Travel
City Guides, Currency Converters, GPS/Maps etc.
• Utilities
Profile Manager, Screen Savers, Address Book etc.

3.  Scope
The scope of testing is to test the operating characteristics of an application that runs on mobile devices supporting iPhone, Android, BlackBerry and Window Phone. The tests are organized by requirement category such as UI, usability, functionality, security, performance, memory leakage, interrupt testing, installation testing, certification testing etc.

The procedure for carrying out testing in terms of preparation of test cases, test environment setup, defects logging and reporting are explained.

4. Test Plan and Strategy
This is describing the “Testing Approach” to achieve testing objectives.

4.1. Testing Levels

4.1.1. Unit Testing
The objective of Unit testing is to verify that a particular module of source code is working properly. Unit tests are designed to test a single class or component or module in isolation.

Developers run unit tests in their own environment and only for the components they are working on.

4.1.2.  Integration Testing
The objective of Integration testing is to ensure that aggregates of units perform accurately together. An independent test team can perform the integration testing.
100% integration testing is not possible on real devices so we can use emulator or dev environment for same.
Here we can perform Functional testing, Interface testing, Usability testing, performance and security testing.

4.1.3.  System Testing
The objective of System Testing is to identify defects that will only surface when a complete system is assembled. Here we check the application behavior with different kind of environment and hardware combination.

We can perform system testing on real devices or the third party tools like perfecto Mobile or Device anywhere. It is intended to validate the application as a whole.

Here we can perform functionality testing, usability testing, security testing, installation testing etc.

4.1.4. Regression Testing
The objective of regression testing is to check that old functionality has not been broken by new functionality or changes made in application.

Like system testing we can perform regression testing on real devices and third party tools like perfecto Mobile or Device anywhere.

Here we can perform all testing types Installation, UI, Usability, Performance, Security, interruption etc.

We can also automate the regression testing in mobile domain. Given below are the lists of tools which can help automate our testing.

Robotium – Robotium is a test framework created to make it easy to write powerful and robust automatic black-box test cases for Android applications.

Robotium has full support for Activities, Dialogs, Menus and Context Menus.

FoneMonkey – Record and playback functionality testing of iOS apps on iPhone and iPad.

Perfecto Mobile - MobileCloud-Automation allows users to execute automated testing on real handsets and tablets located anywhere in the world. Users can automate their testing using the intuitive web-based user interface, which allows the creation of keyword-based ScriptOnce™ scripts. Once the scripts are done, users anywhere in the world can access the devices and run the scripts.

4.1.5. Acceptance Testing
Acceptance testing refers to the execution of a set of test cases on the target device itself. It is intended to verify that a particular product or application meets the requirement specifications of the mobile application under test.

Testing Mobile Business Applications

4.2. Type of Testing

4.2.1. UI Testing
All mobile platforms have certain submission guidelines to follow before the application can be available commercially. Apple AppStore is very strict on guidelines. Most of the applications submitted are rejected due to small errors on AppStore. We need to verify (UI test) that application is meeting guidelines:

Key Points for UI testing

For performing the UI testing of Native and WAP applications, we should keep the mobile OS UI guideline in mind.
Guideline for Android: Click here to see details.
Guideline for iPhone: Click here to see details.
Guideline for different screen size: Click here to see details.
 
4.2.2. Usability Testing
Usability testing is a technique of software testing, through which we validate the simplicity of the software product. Under this technique we check that a new user how easily understand the functionality and behavior of the application and in any complication, how application communicate it to the user.

The main objective of usability reviews and functional testing is to help a business succeed.

Key Points for Usability testing

Given below are the techniques of usability testing

• Walkthrough
• Mock up review
• Business requirement review
• Assist in preparing the guidelines

4.2.3. Functionality Testing
Functional testing ensures that the application is working as per the requirements. Most of the test conducted for this is driven by the user interface.

Key Points for Functional testing
For performing functional testing, we should keep application requirement in mind.

4.2.4. Security Testing
Smart phones are fast replacing traditional computers. As the user base is rapidly shifting to mobiles, hackers are also shifting their attention to mobiles. Due to this trend, conducting security tests on these applications has become a necessity.

Key Points for Usability testing
For validating the security of application we should check the given below points

• Sensitive data leakage (inadvertent or side channel)
• Unsafe sensitive data storage
• Unsafe sensitive data transmission
• Activity monitoring and data retrieval.
• Unauthorized dialing, SMS, and payments.

4.2.5. Performance Testing
This testing process is undertaken to check the performance and behavior of the application under certain conditions such as low battery, bad network coverage, low available memory, simultaneous access to application’s server by several users and other conditions.

Performance of an application can be affected from two sides: Application’s server side and Client’s side.

Key Points for Performance Testing
For performing the performance testing we should check the given below points.

• Check memory, CPU and battery usage, when application is in active state.
• Check memory, CPU and battery usage, when application is running in background.
• Check response time of the application in different conditions.

Performance Testing can be performed using various third party tools available in the market. Eg. WatchDog, System Panel Light etc.

4.2.6. Interruption Testing
An application while functioning may face several interruptions like incoming calls or network coverage outage and recovery.

Key Points for Interruption testing
The different types of interruptions are
• Incoming and Outgoing SMS and MMS
• Incoming and Outgoing calls
• Incoming Notifications
• Battery Removal
• Cable Insertion and Removal for data transfer
• Network outage and recovery
• Media Player on/off
• Device Power cycle

4.2.7. Installation Testing
Certain mobile applications come pre-installed on the device whereas others have to be installed from the store. Installation testing verifies that the installation process goes smoothly without the user having to face any difficulty. This testing process covers installation, updating and uninstalling of an application.

Key Points for Interruption testing
For performing installation/upgrade testing, we should choose the handset with different type of hardware, configuration and API support.

4.2.8. Certification testing
Before going to release, every application should be tested against the guidelines set by different mobile platforms. This is very important to publish an application in app store.

Key Points for Certification testing
For performing certification testing we should validate application’s UI, navigation and business flow against the app market.

4.2.9. Compatibility Testing
Mobile application's Compatibility on devices of different screen sizes, resolutions and hardware. Need to ensure the application work exactly as you want them to across all devices.

In cases of open source mobile OS (Android), there are number of manufacturer who customizes mobile phone architecture, so for insuring the application support on all devices we will perform compatibility testing.

Key Points for Compatibility Testing
For performing compatibility testing, we should choose the handsets with different screen size, OS version, and configuration.

4.2.10. Network Testing
There are number of career available in mobile domain. To insure that the application is working properly with different careers and their different network bandwidth, we do network testing.

Key Points for Network testing
Testing on different network speed:
1. Low
2. Medium
3. High

Testing during change of network speed
1. Low to high
2. High to low

Testing in different network types
1. 2G (GPRS, CDMA, EDGE)
2. 3G
3. Wi-Fi
4. Different types of plan based on the service provider.

4.2.11. Localization Testing
Localization testing is the part of the software testing process focused on the language and internationalization of the application. Localization includes translating of the application user interface and adapting graphics for a specific culture/locale.

4.2.12. Synchronization Testing
Synchronization is a feature that enables exchanges, transforms and synchronizes data between two different applications or data stores. Synchronization could be either cradle-based or wireless. This category lists the different failure that could be encountered while synchronizing data between two applications or between two devices like PC and Mobile.

5. Testing Flowchart

6. Test Deliverables 

6.1. Test Plan/Strategy
It shows testing approach to achieve testing objectives.
6.2. Requirement Traceability matrix
Documents the links between the user requirements for the system you're building and the work products developed to implement and verify those requirements
6.3. Test Cases Document
Document with a description and expected result for each test case.
6.4. Test Progress Report
The Pass/Fail status of each test cases and the list of issues.
6.5. Test Summary and Detailed Report
 
7. Challenges in Mobile Application testing
Mobile app marketing is growing now very fast, but in testing field number of challenges arrives due to variation in handsets, phone carriers, networks supported and application written in different languages.

7.1. Mobile OS
There are different Mobile Operating Systems in the market. The major ones are Android, IOS, Windows Phone, and Blackberry. Each operating system has its own limitations. Testing a single application across multiple devices running on the same platform and every platform poses a unique challenge for testers.

Critical areas for Browser based application:

• Browser based application always expect the connectivity.
• Speed and coverage are the critical aspect.
• Cache related issues.

      Critical areas for Installable application:

• Can be installed and uninstalled in the device
• They can be downloaded from

1. App Stores
2. OAT (over the air)
3. transferred via USB
4. wireless media like Bluetooth, infra etc

7.2. Mobile Browser
Mobile browser compatibility is a big challenge for the wep application as there are a list of in-build and third party browser available in App Markets.

7.3. Mobile Device
Mobile devices differ in screen sizes, input methods (QWERTY, touch, normal) with different hardware capabilities.

7.4. Network Carrier
There are over 400 mobile network operators in the world; out of which some are CDMA, some GSM, whereas others use less common network standards like FOMA, and TD-SCDMA. Each network operator uses a different kind network infrastructure and this limits the flow of information.

7.5. Mobile User Location
A user, who belongs from any graphical location and cutler can install and use the application.

8. Mitigating Challenges

8.1. Mobile OS
Training required for resources on various OS and their standards.

8.2. Mobile Browser
Market research, use of Simulators, real devices and third party tools.

8.3. Mobile Device 
Testing with various internet bands and network speeds.

8.4. Mobile Carrier
Testing with various internet bands and network speeds.

8.5. Mobile User Location
We should study the different cutler and implement it (localize) for that.

9. Schedules for Testing
This will be decided in consultation with the project manager.

10.  Risks and Assumptions
Risks

• Device availability
• Any new feature addition/modification to the application which is not communicated in advance.
• Any delay in the software delivery schedule including defect fixes. Any changes in the functional requirements since the requirements were signed-off/formulated.
• Lack of Resources and Expertise if the technology is relatively new.

Assumptions 

• Every release to QA will accompany a release note specifying details of the features implemented and its impact on the module under test.
• All "Show-Stopper" bugs receive immediate attention from the development team.
• All bugs found in a version of the software will be fixed and unit tested by the development team before the next version is released
• All documentation will be up-to-date and delivered to the system test team.
• Devices, Emulators and other support tools will be fully functional prior to project commencement.
• In case of lack of required equipment or changes in the feature requirements, the test schedules may need to be reviewed.

11.  Entry and Exit Criteria
Entry Criteria 

• Development of the application is complete.
• Successful completion of unit testing for the applications.
• Release of software to the test environment.
• Dedicated resources are allocated.
• Approved test bed to carry out system testing.
• Test environment is up and working.

Exit Criteria 

• All test cases have been executed and at least 95% have passed successfully. The remaining 5% do not impact critical functionality
• All test results have been evaluated and accepted.
• There are no showstoppers or high criticality defects unresolved or outstanding

12.  Test Metrics
Following metrics will be captured and reported as part of the Test
• Summary Report
• Test Design effort
• Test execution effort
• Number of Test Cases executed
• Number of Defects and their classification
• Test Coverage (Number of test cases executed/Number planned)

13.  Logging Tests and Reporting
Some third party applications will be used for reporting bugs found during test execution. The QA team will log defects in the tool as testing progresses.

14.  Roles and Responsibilities
The roles and responsibilities of the testing team for the project are as follows:

14.1. Project Manager / Test Manager Responsibilities
• Overall responsibility for managing the testing process
• Approval of test documents
• Approval of inspections/reviews done as per the test plan
• Providing resources for the project
• Arrange relevant training if required.

14.2. Test Lead
• Requirement gathering
• Planning and estimating for testing
• Tracking and monitoring the testing as per the test plan
• Reporting the project status

14.3. Test Engineer
• Creating the test cases as per the test plan
• Executing the test cases
• Documenting the results and logging errors.
• Test Plan
• Test Cases Document – Document with a description and expected result for each test case.
• Test Results – The Pass/Fail status of each test cases and the list of issues.
• Maintain Reports and Metrics.

15.  Best Practices to Improve QA skills and QA Process
• Know your team and team members and be aware of the available skillset and specializations.
• Meet discuss and talk.
• Identify and bring to notice all training needs
• Self-assessment at regular intervals is important
• Know and Manage Automation expectations
• Report outcomes
• Discuss failures and reasons
• Think ahead of time in future and be future ready
• Share knowledge within team and across team.

16. Best Practices for success of project
• Identify Automation needs (Answer, Where, When, Why, Who etc.)
• Identify Tool, perform tool validations, create Comparison Matrix
• Layout a Structured Process
• Decide on team roles
• Consensus on current activities for execution
• Layout Plan, Strategy etc.