Why is mobile testing difficult?
Currently estimates suggest that there are over six billion smartphone users in the world. With that level of prevalence, it should come as no surprise that since 2017, over half of all web traffic has been from a mobile device. But the increase in mobile use has brought new challenges for web and software developers.
There are a lot of different mobile devices with which users might interact with your application. To make matters worse, even seemingly identical devices can behave in different ways when it comes to real-world app performance.
The huge amounts of variation in actual mobile device use can make testing difficult. If you wanted to manually test for all the possible end-use cases imaginable you would need to have a stockpile of thousands of devices and a whole engineering team of testers working around the clock. While crowdtesting can get you more eyes on your product than ever before, it still won’t be able to cover every single configuration.
To complicate matters further, most phone and tablet users expect their apps to keep pace with updates to their mobile devices. With both Android and iOS rolling out continuous patches and updates to their operating systems, you’ll need to do the same.
For instance, if you’re targeting a market that’s very different from your existing ones, the best course of action might be to rebuild your GTM strategy from scratch.
Alternatively, if you’re targeting a market that’s quite similar, you may only need to translate your content and make minor alterations to your sales strategy.
Sometimes it can be even easier than that - for instance, if you’re a US company targeting the UK market, you might only need to find a local banking partner and start accepting orders
What can make mobile testing difficult
With device fragmentation and seemingly limitless real-world variables to account for, difficulties with mobile testing can stem from several complexities that are intrinsic to working with mobile devices:- Wide variety of mobile devices
- Developing for both Android and iOS operating systems (OS)
- Different OS versions are in use at the same time
- Regular OS updates
- Integrating with a complex app ecosystem
- Unpredictable interactions with third-party code
- What constitutes a localization MVP?
Most common mobile testing challenges
Testing a mobile application comes with a whole host of unique challenges. Many of these challenges come down to what gets referred to as fragmentation—the sheer diversity and variety of actual mobile devices and configurations in use today.
While every mobile development lifecycle will inevitably encounter its own difficulties, most of them can be classified as belonging to one of the following mobile testing challenges.
OS fragmentation
When it comes to mobile operating systems, Apple and Google split the lion’s share of the market. As of September 2021, devices running either Apple’s iOS or Google’s Android system accounted for over 99% of all smart mobile devices, with Android alone making up over 70% of the market for mobile operating systems.
As a result of the Android-iOS duopoly, many app developers have chosen to abandon support for other platforms and focus on the two systems that cover the vast majority of mobile usage.
Interoperability
While for some apps it might be enough to simply build two versions—one for Android and one for iOS—if your app has a social element to it, or your users expect it to integrate across multiple devices, you will need to consider how the two versions will communicate and interact.
If you plan on building an app that can function outside of Android and iOS, interoperability becomes even more of a challenge.
Screen sizes
Smartphone manufacturers design their devices with varying screen specifications in order to appeal to a broader range of consumers. These days, even a single model of mobile device can come in multiple variants, all with different resolutions and screen sizes.
To maintain consistency across a varied landscape of screen size and resolution, app and web developers need to ensure their products resize correctly for all these many variations.
OS versions
As well as designing for both Android and iOS operating systems, developers have to take into account fragmentation within these systems.
The chart below shows the changing popularity of the most common iOS versions between November 2020 and November 2021. As you can see, at any one time over the last year, no single version of the OS can be said to have dominated, and the distribution of different versions of OS across devices is in constant flux.
As newer versions of each OS introduce new features, you may want to take advantage of these in your app design. But when it comes to testing you will also want to ensure that new features don’t inhibit your app’s backward compatibility with older versions.
Ecosystem integration
Whether you are building a mobile web app, a native app to be downloaded from one of the popular app stores, or some kind of hybrid application, modern mobile users expect their apps to integrate seamlessly into the wider application ecosystem.
From simple integrations like building a “log-in using Facebook/Google” option into your mobile website, to more complex ones that require communication between multiple apps from different developers, ensuring your integrations work properly requires a rigorous testing process.
Different browsers
If OS fragmentation is the primary concern of native app developers, different browsers are more likely to be a challenge for mobile web developers.
While there have been attempts to standardize the way browsers render web pages, the success of attempts to implement standards has been limited. This is partly because the popular browsers all use different rendering engines. Safari and Opera are built on WebKit, but Chrome uses Blink, Firefox uses Gecko, and so on. These different approaches to graphical rendering affect what your web pages look like on the client side.
As well as the Android and iOS inbuilt browsers (the mobile versions of Chrome and Safari), there are several other browsers you will need to test for. As the chart below shows us, after Chrome and Safari, you’ll need to account for Samsung browser, UC browser, Opera, Firefox, and more.
Device settings
Because mobile device settings vary according to OS version and device hardware, as well as user preference, it can present a huge challenge for testers.
As you implement the final stages of your testing cycle, you will want to keep in mind that any or all of a wide variety of device settings can be changed at any time when your application is in use. Font size, night mode, and accessibility setting can all impact the way your app works.
Language and cultural specifics
For developers marketing their application to an international audience, making sure that different language versions all function smoothly without improper translation can be hard work, especially when it comes to languages not spoken by your core development team.
Mistranslations can occur in cultural contexts as well as linguistic ones. Besides different alphabets and characters to think about, there’s also different numerical systems, currencies, date, time, and measurement standards to think about, to name just a few variables.
To compete in local markets, mobile developers need to implement what is known as localization testing. Localization testing checks how a given mobile application is received on the ground in different geographic locales
Location awareness
Another reason for the importance of localization testing is that a growing number of apps are dependent on geo-location data to function properly. For example, they might use location-dependent algorithms to provide real-time information about the user's surroundings or to share information between app users in the same area.
The challenge of location-dependence means app developers need to negotiate both infrastructural differences between locations (like divergent network coverage) and legal ones (like different privacy laws).
Usability
In the field of mobile app testing, the end-user experience of a product is the ultimate measure of whether a testing and quality assurance program has worked.
The challenge of ensuring usability on real devices is really a kind of meta-challenge that encompasses many of the other difficulties that are specific to mobile application testing. While usability testing may be a discipline in its own right, the various stages of exploratory testing and functional testing are also ultimately in place to create an enjoyable experience for the end-user.
Common bugs found during the mobile testing process
In mobile development, a “bug” refers to any error, flaw, or fault in an application that causes it to produce an undesired outcome, or to behave in unintended ways. The process of finding and fixing bugs is known as "debugging" and is a central pillar of all software testing.
While this broad definition of a bug means that the list of things that could count as a bug is impossible to ever fully express, there are certain bugs that test engineers from across the spectrum of mobile development tend to encounter.
While some bugs can make a product unusable, others are just annoying. However, a frustrating user experience is one of the leading reasons people uninstall an app - and a high uninstall rate can end up costing your business money.
High battery usage
When using a mobile app, few things are more frustrating than battery drain. High battery usage can be caused by any one of the critical or non-critical paths in an application. The most efficient apps optimize battery usage for each function and ensure functions don’t run concurrently if they don’t need to.
High memory usage
Another instance of poor resource allocation, mobile testing can indicate when an application is using excessive amounts of ROM or RAM. One common cause of high memory usage is an excessive memory churn rate. This happens when the number of allocations of temporary objects an application makes in a given amount of time is too high.
Picking up on excessive memory usage is important because it’s one of the biggest contributing factors to people uninstalling an app.
Button and click frustration
In both native and web apps, unresponsive or poorly designed buttons can be frustrating to users. There are many things that can cause buttons not to work properly. Sometimes the issue is related to different screen sizes, other causes include unexpected interactions between graphical components and having the wrong hierarchy of design elements.
The click-crash
One major bug that can invalidate an otherwise great application is when a specific button causes the app to crash when clicked. This problem can often go overlooked if the button that’s causing the issue is one that rarely gets used, is hidden deep within the app, or only causes the crash when a particular sequence of variables has occurred.
Slow transitions
Abnormally slow transitions between different app pages and tabs are some of the most common bugs that mobile testers need to look out for. Even a delay of a few seconds can introduce friction in the user experience and cause frustration.
Display problems
Mobile users expect their apps to smoothly rotate between portrait and landscape orientations. They also expect the ability to toggle between portrait, landscape, and auto-rotate options.
Other display issues include distorted dimensions, unexplained whitespace, improper rendering, and resizing issues.
While software automation tools exist that mean you don’t have to manually design a separate user interface for every conceivable screen size, you will need to test properly to ensure that your resizing is working smoothly on different devices.
Security flaws
More than ever before, information security is a primary concern for mobile developers.
Consider the recent “log4shell” exploit, where a zero-day vulnerability in the popular log4j Java library put the data security of countless organizations in jeopardy and forced many developers to completely reavaluate their Java applications.
As the log4shell example demonstrates, even widely-used libraries have the potential to introduce security flaws into your mobile product, proving the importance of thorough security testing.
Scrolling
As an integral part of the user experience of a mobile device, being able to scroll up and down effortlessly is one of the things that everyone expects from an application. Despite this, it frequently goes wrong!
Because scrolling functionality is designed from deep within mobile OS architecture, it can occasionally behave in surprising ways. As such, sometimes emulators are not sufficient and only usability testing on real devices can pick up on scrolling issues.
Repeated results on a list
Android ListView is a view which groups several items and displays them in a vertically scrollable list. It’s an important feature of Android app design that affects interface usability.
A common occurrence in Android applications, this bug is caused by problems with the way ListView pulls content from an array or database.
How to mitigate mobile testing challenges
With the seemingly endless array of mobile devices and OS configurations out there, the challenge of software testing for mobile web applications or native mobile apps can be daunting.
The best way to manage mobile testing challenges is to ensure you have a testing strategy in place from project inception. That way you will know right from the beginning which types of tests you will be carrying out, who is to be responsible for the various test stages, and which testing tools you will be using.
Event storming is a lightweight and agile way of mapping a current project that can help you to model your test requirements. It is useful for establishing what the biggest testing challenges are in a project and conceptualizing how the test cycle fits within the evolution of the project as a whole.
Strategy to maximize customer delight
These days, many developers have a great idea they want to pursue but don’t have the necessary mobile testing experts in their team. Thankfully, there are options for outsourcing many mobile testing challenges.
Global App Testing is the test management platform of choice for a wide variety of mobile developers. For example, we tested 5,000 of the largest apps that use Facebook Login across various devices and operating systems—all in just four weeks. By outsourcing elements of the testing process, even a giant company like Facebook estimated that it was able to complete ten times the number of tests in the required timeframe than its in-house team would have been able to alone.
Mobile testing strategies
When designing a testing strategy to maximise customer adoption and retention, you need to consider the different techniques available to you and the skill sets of your in-house and outsourced testing teams. Ask yourself at which point in the software development lifecycle you should be implementing which type of test, and how you can prepare your backlog accordingly.
Most approaches to mobile testing will consist of one or more of the following strategies.
Manual Testing
Manual testing refers to a strategy in which mobile apps are manually tested for defects. It requires a tester to use most of the application's features to ensure correct behavior.
The tester often follows a written test plan that leads them through a set of important test cases.
Test Automation
Also known as automation testing, test automation is the yin to the yang of manual testing. Where manual tests are performed by human testers, automation testing is done via scripts and bots. It is ideal for routine and repetitive test cases.
Global App Testing has built a suite of dedicated automatic testing tools for mobile developers that allow teams to efficiently handle their test case management by automatically performing different types of test runs.
Hybrid Strategy
These days, most mobile development projects employ a hybrid strategic model that includes elements of both manual and automatic testing.
The reason for this is that while test automation lends itself to the execution of routine test cases, early-stage functional testing, and metrical performance testing, it can still be valuable to enlist human testers for usability testing. After all, it’s humans and not bots who will ultimately be using an application, so it makes sense that app testers are still needed during the final stages of quality assurance.
Exploratory Testing
Exploratory testing is an informal testing philosophy that is preferred by agile development teams. In this approach, the tester does not follow any rigorous testing procedure but rather explores the user interface and APIs of the application to intuitively uncover bugs and design additional tests. 
Ad Hoc Testing
Like exploratory testing, ad hoc testing is an unstructured approach to testing that doesn’t describe a specific type of test, but rather a strategic perspective.
The difference between exploratory testing and ad hoc testing is that, whereas exploratory testers use their intuition and programming experience to find defects, an ad hoc approach relies on an element of randomness.
Ad hoc testing is unplanned, can be implemented at any stage of development and there are no criteria for bug reporting or documentation.
Crowd Testing
Crowd testing or crowdtesting is a strategy that employs the help of a distributed network of testers to ensure functionality and usability
Crowd testing can be a great solution for all kinds of development teams. It allows you to quickly scale your test capacity safely in the knowledge that only highly qualified testers with relevant expertise will be assigned the task of testing your product.
With a vetted network of 50,000 skilled testers across 189+ countries, GAT can enhance the efforts of your testing team and deliver key insights into how your app works from a user perspective in specific environments.
Essential mobile testing types
Once you’ve got your strategies in place, you need to know what types of tests to run. Common tests include:Functional Testing
The purpose of functional testing is to verify that every function of an app is working as required. It focuses on testing user flows within the app, as well as each feature individually.
Regression Testing
Regression testing is the process of re-running functional and non-functional tests to ensure that previously developed and tested software still performs as required after a change.
In mobile app development, regression testing should be incorporated into your test strategy both prior to and after release so as to ensure continuous delivery of a functional product following patches and updates.
Interruption Testing
This type of mobile testing checks how an application responds when faced with an unexpected interruption, as so often happens when apps are in use.
When interrupted, an application needs to respond accordingly, either by pausing and returning to the previous state or by performing a certain predefined action. Interruption testing ensures that an app handles interruptions without crashing or otherwise malfunctioning.
Some common interruptions that should be considered while testing are:
- Incoming phone calls
- Incoming messages
- Low battery notifications
- The device being plugged in or out of charging
- Device shutting down
- Loss and restoration of network or WiFi connection
Localization Testing
Because different countries have different network standards, speak different languages, etc, you need a global testing framework if you plan to release internationally. The purpose of localization testing is to make sure that an app performs its key functions under different circumstances, in different locations.
GAT has a global network of testers to help you improve your apps based on real user data and feedback. Localization testing ensures end-users feel as though the products you deliver were built for them and allows you to discover targeted, localized issues that impact your ability to compete in local markets.
Speed Testing
Speed testing tests how fast a mobile app is able to perform its functions.
Because how fast an app performs is so often affected by variables like hardware specifications and internet connectivity, speed testing needs to be carried out under a range of conditions if it is to give a reliable indicator of how fast an app will perform upon release.
Memory Leak Testing
In simple language, a memory leak is a loss of available memory when an application fails to return memory that it obtained for temporary use. When a mobile app is opened frequently or runs continuously, even a very small memory leak can eventually cause it to terminate.
To identify memory leaks, a form of dynamic program analysis known as memory profiling is applied.
Performance Testing
Performance testing defines a type of test that takes place in a controlled environment in order to measure specific performance indicators in isolation.
For example, you might test battery consumption, start-up time, or RAM usage individually for each module before deploying a given app in situ. This way, if any specific element of the app is underperforming you already know where the bug is rather than having to locate the source of the problem based only on general performance metrics.
Security Testing
When we call a mobile application secure, we mean it is trustworthy, free from obvious vulnerabilities, and able to protect the personal information of users from theft or exploitation by malicious actors.
As more and more transactions are carried out on mobile devices, mobile applications are increasingly targeted by hackers looking to exploit information systems for financial gain. As a result, the field of mobile security testing is now one of the most technically advanced subdisciplines of mobile testing.
Remember that security testing isn’t just about financial data either. Information Security Management is concerned with securing all kinds of personal data, helping to uphold your customers’ privacy.
Security testing is also now mandated by various laws and standards that govern the digital sector. For example, here at GAT, we have introduced a range of security measures to ensure full GDPR and ISO27001 compliance.
Testing with Simulators and Emulators
Emulators and simulators allow you to run mobile tests inside flexible, software-defined environments. Employing this strategy can be a cost-effective way to perform many tests quickly.A simulator is designed to create an environment that contains all of the software variables and configurations that will exist in an application’s actual use environment. Simulators don’t attempt to emulate the hardware environment in which an application will be run. As such, they can be implemented using high-level programming languages.
An emulator, on the other hand, attempts to mimic the hardware on which an application is intended to be run as well as software features. To achieve this, they are typically coded using assembly language.
Testing on Real Devices
Real Device testing tends to follow on from simulator and emulator testing. It is an essential component of any mobile testing framework as even the best emulators can never exactly replicate real-world use conditions.
Sometimes, bugs and issues can go unnoticed during prior testing phases and it is only when it comes to using an application on physical handsets that they are spotted. This is why no amount of simulator and emulator testing can replace this crucial final stage of testing before release - and why crowdtesting can make such a difference.
Usability Testing
Usability testing is typically the last stage of the testing cycle. After functional and performance tests have been carried out to ensure an app is behaving as it was designed to, the intention of usability testing is to discover flaws in the interface design and friction in the user experience.
For usability testing, testers use a range of devices and configurations to explore how an app performs under conditions it is likely to encounter on release.
Conclusion
As is the case with other forms of software development, testing is an unavoidable part of the mobile development lifecycle. But mobile developers face unique challenges when it comes to designing their test frameworks, running their test cases, and analyzing their test data.
You should now have a better understanding of the difficulties that device and OS fragmentation present for the mobile development process, and an idea of some of the solutions available to test engineers and quality assurance professionals working in the field of mobile application development.
Real-life isn’t a controlled test environment and no test framework will ever be able to fully replicate the real-world conditions a mobile product will encounter upon release. Thankfully, with crowd testing, automation testing, and multi-device usability testing services from GAT, you are well equipped to run your mobile app through a rigorous test cycle.
At GAT, we believe in facing challenges head-on, equipped with the most appropriate tools for the job and a can-do attitude. That’s why our test management platform integrates with all the most popular bug and issue trackers and enables a hybrid testing strategy that makes the most of the two pillars of mobile testing: test automation and manual testing.
So whether you’re building a native Android or iOS app, a containerized web app, or a hybrid application, choose a testing solution from GAT for your mobile development project and enjoy the benefits of crowdtesting, test automation, and a test management platform with mobile testing at its core.
