Updated: May 10, 2021
Before being published to app stores, each mobile application is tested. However, the testing process is not so simple and clear-cut. We have outlined the key steps of the mobile application testing process that will help you obtain high-quality results for the entire project.
Preparation for the testing process
When conducting a technical audit, we often face the fact that QA specialists join the project later than they should. If you start the testing process only after you get the first build of the application, there is a risk that mistakes were made at the stage of determining technical requirements and specifications. Reworking the final product is always more challenging. Below we describe the important first steps that will save you time and money.
1. Device models and OS versions
Once you have your project idea, your business logic, and your target audience, you should determine the devices and the OS versions your mobile application will be compatible with. These will be the devices you will run your tests on as well.
To figure out the best devices for your app, you need to answer the following questions:
What devices are the most popular in the application launch region?
If the project already has a website, which mobile devices are used more often to access it?
What devices are the most popular with your target audience?
What OS versions are required for the business logic of the project?
What OS versions are the most widespread in the world at the moment?
As a rule, we use the following resources: Apple and Google distribution dashboard, Device Atlas, and Statcounter (a very convenient resource with the information about platforms, devices, and data organized by country).
Your choice of the devices and OS will affect the functional requirements, application design, and the devices and OS used for testing.
Mobile vendor market share worldwide Mar 2020-Mar 2021. Source: Statista
Usually, we recommend choosing at least 3-5 reference devices for each platform with a different screen aspect ratio, different DPI, and different versions of OS.
2. Specification and design prototype testing
Once the technical specification is completed, a QA specialist should be involved. If the design is ready as well, having a QA specialist at this stage is even more important. The specification and design prototype testing allow you to identify defects before they become an integral part of the product.
A QA specialist will identify the incompleteness of descriptions, ambiguous wording, contradictions, and imprecisions. Design prototype testing eliminates the discrepancy in the technical specification and can help to identify problems with user experience.
This approach will save you from challenges later down the line, and it will save time for the entire team and keep you on budget.
Test planning
At this point, the project has the final approved specification and design. The project management methodology, which usually determines the appropriate testing method, has also been chosen. Everything is ready for the development stage. However, before you start testing, you need to take a few more key steps.
3. Test plan
Well planned is half done.
The fundamental document for the QA process is the test plan. This is a regulation document that describes everything related to testing a project. It includes:
Project overview
This includes the introductory information that will allow any specialist to easily understand the idea and purpose of the project, its target audience, description of all parts of the project, and the parts that are subject to testing. Most of the time, the project has a back-end, an administration system, as well as at least one mobile application. Sometimes the project may contain several administration systems or a website in addition to mobile applications.
Links to the main project documents
This includes everything that may be useful for QA specialists, including the links to project space, technical specification, project stack information, design source files, and a list of test logins and passwords.
Links to test checklists and test cases should be added in this block as soon as they are prepared. At the start of the description in this section, links to document templates can be placed.
Necessary equipment
In the first step, we identified the device models and OS versions that the application should support. Now, we need to determine which device models and OS versions will be tested. It’s extremely rare to test 100% of OS versions and device combinations.
Using emulators is another issue to consider. When you perform mobile application testing using an emulator, some difficulties may arise. For example, issues may come up with Face / Touch ID, fingerprint sensor, camera, voice control, device’s memory testing, voicing interface elements, shaking the device, multi-tap, and other control gestures. Plus, there is a significant difference between the application's behavior in the emulator and on a real device. Thus, some bugs are reproduced only on real devices and vice versa.
Interaction with other applications or devices
If the project involves integrations with other applications or devices, the QA should know the details. It is important to determine how integration testing will be performed and how to access the devices. For example, this is relevant for testing IoT mobile applications, where you need to have specific knowledge about the device function or the object which the application is integrated with.
Testing team
The test plan should specify a team of testers and the role of each member. However, it’s important to not only fix the team members on paper but also to hold an intro-meeting with the team to discuss the roles, plans, tasks, and expectations of all project participants. All QA specialists must keep all testing documentation up-to-date throughout the entire development process.
Project space, bug tracking tools
The whole team should know the exact list of systems and tools used on the project. For example, here are a few of our favorites: ADB, Crashlytics, Postman, Instabug.
We recommend devoting time to learning new tools regularly.
Top 10 bug tracking software according to G2
Testing start and finish
Formulate the criteria and the deadlines for testing if the release date is fixed. If you plan to arrange private demonstrations during the development process, these dates are also important to fix in the QA team work plan.
4. Test design
At this stage, it is necessary to analyze the project’s requirements, assess the risks, and consider options for reducing them with minimal costs. Usually, during this analysis, we try to answer the following questions:
What types of mobile app testing should be used based on the project’s technical requirements?
What problems may arise during the testing process? How to avoid them or minimize their impact on the project?
What product risks can be identified? How can we reduce them, and what are the possible consequences?
What factors outside of the team’s control could influence the project?
Are there any known release dates for new OS versions? What actions could be taken for them not to affect the planned release date of the application?
Are there any risks that may arise during the process of publishing applications on the AppStore and Google Play?
How will a possible user load increase on the application be handled? What can be done to test the load?
How fast should the application load, and how fast can users surf the application menu and its functions?
What test documentation is needed for the project?
5. Types and forms of testing
Let's take a closer look at all the possible forms and types of mobile application testing. Usually, you would decide on the type of testing based on the project requirements, the budget for testing, the planned release date, and the team’s expertise.
Testing forms
manual,
manual + farm,
manual + automatic,
manual + farm + automatic.
Testing forms
As a rule, the most winning scenario is a combination of testing forms and types. For example, autotests give the most accurate results and are available for reuse, but writing them can take an unreasonably long time. You can also perform a manual test of any application, sometimes even without any preparation (so-called “hallway testing”). The farm allows you to run testing simultaneously and even remotely on multiple devices.
The correct combination of testing forms and types allows you to optimize the testing budget and get a high-quality product.
Types of testing
Functional Testing
This is the type of testing that ensures that the application performs its functions correctly and meets the expectations described in the technical specification of the project. Depending on the features of mobile devices, this type of testing can include:
Installation and version update testing, which checks that the application can be installed, launched, and updated to the new version without any problems.
Interruption testing that is aimed at preventing users from losing data in forms they fill out or contents they create in case they get interrupted and leave the application to perform another task on the device.
Testing the app’s performance with different networks and during the network switch, which makes sure that when the wi-fi signal disappears, the app will connect to an available mobile network. If there is no access to any network, the application should display a correspondent error. The application should resume where the user left off without losing user input once the network is available again.
Testing the interaction with other devices and applications, which runs such tasks as taking a photo using the device's camera app or sharing content in social network apps.
Testing the effect of the application on battery drainage and the device’s memory.
In-app payments testing, including Apple Pay, Google Pay, and purchase restore.
Testing the behavior of the application with different screen orientations.
Testing the tap on push notifications.
Testing the error handling not only on the client’s side but also on the server.
Automation testing
Automation testing does not replace the manual one since the creation of autotests and their maintenance are time-consuming, especially when the requirements change quickly enough. However, autotests are a good option for performance testing and regression testing of important application features. Regression tests must be carried out every time after a bug has been fixed or new functionality has been added to make sure that those actions haven’t affected the rest of the functions. Due to the frequent repetition of the same actions, the human factor comes into play, and QA engineers could miss critical errors. Autotests avoid the human factor and free up time for more creative manual checks and new features testing.
User interface testing
This concerns the assessment of the interface elements impact on performance. The app's user interface is checked to be compliant with the approved design prototype and the current guidelines from Apple and Google. Also, this type of testing includes checking the syntax (texts and formulations) for compliance with the style of the application, its requirements, and logic.
Usability and A/B testing
This is an evaluation of the application’s clarity and ease of use. It is difficult for team members to see the application with the eyes of the unprepared and unfamiliar with the specification and design users. For this type of testing, analytics is the first thing to set up. Analytics allows testers to see how much time users stay on the screen, how much time they take to fill in the forms, how often they go back to the app, how many times they activate a certain button. Once this has been accomplished, the users are divided into two or three groups that are given access to the application builds that contain small differences in one element of the interface or any particular part of the functions. Based on the collected statistics, we can decide which changes are perceived by users better. This type of testing should be taken into account whenever you estimate the time and budget for testing.
Localization testing
If the mobile application is designed for users from different regions, text translation and its display should be accurate. Cultural specificity and differences in the format of data output, as well as different units of measure should be equally taken into account. For example, the decimal separator can be a period or a comma, and the date display can differ, depending on the region. Our team loves to use the Localise tool to organize localization in mobile applications.
Accessibility testing
This is the assessment of how convenient the work with the application is for people with various health conditions. For example, it tests whether the interface elements are correctly voiced. We recommend reading about the basic rules and tips for checking the inclusiveness of an application on the Global Infrastructure Hub website or in a series of articles from Usable Net on Medium.
Security and compliance testing
This type of testing ensures the security of user data. For example, you cannot proceed with the registration in the application with a password that is too simple like “qwerty” or “1234567.” The application should not request unnecessary permissions and should work properly, even if the user hasn’t given permission to use the requested data; sensitive data like logins and passwords should not be transmitted in an unencrypted form in requests to the server.
Performance testing
With the help of special tools, this type of testing checks the back-end performance with normal predicted load (the number of simultaneously active users and the amount of transmitted data), with higher than planned load, and during prolonged work with a normal load.
Read also: Different Types of Mobile Application Testing for Boosting App Quality
6. Testing documentation
The test plan is the first document on the list. Based on the test plan, forms, and types of testing, the following documents are drawn up:
Checklists
It is important to describe the acceptance criteria, as these criteria will help evaluate whether the application performance is acceptable.
Test cases
The test case includes a sufficient number of tests that cover all the functionality. As we previously mentioned, it is unfeasible to test all the possible combinations of input parameters. Choosing the right data is key to identifying potential bugs.
Once test cases are written, they get divided into sets that will run in different situations, for example:
Tests for separate application modules.
Smoke test, which is a set that covers only the main functionality and is performed on each new build to decide whether the build is ready for more precise testing.
Acceptance test, which is a set of tests covering all the functionality. This set runs until the product’s quality level required for publication is reached.
As a result of this stage, you will get a list of test cases containing a description of the idea validation, the necessary steps, the expected result, and test-suites. Test suites are sets of sequentially executed test cases, for example, for end-to-end testing.
7. Bug report
When working with checklists and test cases, QA specialists look for the possible inconsistencies between the expected and the actual results in the application. However, simply finding them is not enough. It is important to describe all the bugs' details in a way that will allow the development team to easily reproduce and fix them.
For this reason, we use our bug report template for each project. No doubt, bug reports may look different for different types of testing and at different project stages. The type of project space and the project management methodology can influence the reports’ contents as well. However, there is a standard set of data/fields in every bug report:
Bug # in case the system does not assign numbers automatically.
Title. A short description of the defect, which gives an understanding of what goes wrong, where, and under what conditions.
Severity.
Priority from low to block level depending on how the business logic of the application is affected.
Steps to reproduce the bug.
The environment. This always includes the device information, such as version and build number, as well as the description of specific conditions under which the defect is reproduced, for example, screen resolution for GUI bugs.
Expected and actual results. This describes the contradiction between the actual application behavior and the expected behavior described in the requirements. This discrepancy proves that the issue is not trivial, but is a defect that needs to be fixed.
Attachments (screenshots, videos, sketches, logs).
Read also: How to Write a Bug Report That Will Make Your Engineers Love You [+Free Bug Report Template]
8. Testing
The testing process lasts until all the planned functionality is implemented and the required level of quality is achieved. For this reason, keep in mind the following:
Each new part of the functionality obtained at the new iteration is covered by the tests developed at the previous stage;
The interaction of new features with other implemented modules has to be tested;
Ensuring that bugs that were previously discovered and fixed by the developers are no longer reproduced is achieved through confirmation testing;
Regression testing is performed to make sure that the new functionality and bug fixes have not affected previously stable functionality;
Written test cases have to be updated and supplemented. Missing this step will render future tests ineffective over time (the so-called “pesticide effect”).
App testing cycle
Based on the results of a significant intermediate or final testing, we draw up an additional report - a test report. Essentially, this is an extended version of the bug report, which gives the Project Manager and the Product Owner an overview of the product’s quality. This kind of test report may contain a link to the executed test-suites indicating the devices and the OS covered, functional quality assessment, information about the severity status of the remaining bugs, and compliance with the acceptance criteria, as well as recommendations for improvement.
9. Pre-release testing
Acceptance test includes precise and complete testing, which may still be required to stabilize the application performance and identify minor bugs. At this stage of testing, it's significant to compare the planned functionality and the one accepted to be sent to the stores for publication. You may need to eliminate some redundant functions or add new ones.
Final report on pre-release testing
The whole process of mobile application testing should be well-documented and be as up-to-date as possible. Double-check that the necessary entries have been made, then make your test summary report.
Test summary report should describe:
Important information identified as a result of tests;
Information about the quality of testing;
Summary information about the quality of the tested mobile application;
Statistics obtained from reports of various incidents;
Information about the types of testing and the time spent on each of them;
Indication that this mobile application is suitable for use in the current state of quality and meets all the criteria of accepted functionality and quality of work.
The decision to publish the application in app stores should be made after the test summary report.
10. QA responsibilities after release
The work of the QA team does not end after the application release. If the team continues to work on the second version of the project, the whole process may start from the very beginning. However, even when further work is not required and the project is functioning successfully, it is best to set time aside for testing. In this case, the testing process will be a periodic check, which may include one or all of the following events:
Testing after the new OS versions release. This is necessary because changes not compatible with a previously developed application may occur.
Testing after SDKs updates. If the project exploits third-party SDKs or libraries, their updates could also lead to errors.
Testing following user comments. Users can report errors by email or in comments in the AppStore and Google Play. It is important to try to reproduce the reported bugs and deliver them to the development team or the designers for fixing.
Testing according to the minimum checklist. For post-release projects, it may be sufficient to take the previously used checklist and execute a sample check for the most significant functions.
Mobile application testing is a complex process, and setting it up requires effort from the entire team. You can adjust the steps of the testing process described above for each project. However, do not skip the above 10 steps during the testing process. This way, you are guaranteed to get a reliable result in the form of high-quality mobile applications.