QAOps Blog

Conducting Mobile App Testing at Scale

Mobile App Testing at Scale?

You’ve built a cracking mobile app that you can’t wait to show off to the world. Everything is ready from the development side, and your team is pretty darn excited about the features that they’ve work hard to complete. Anticipation in the office is running high as the downloads begin to pour in from the major app stores around the world. Yet, anticipation soon turns to trepidation as a few bug reports come in from a few users. Your immediate reaction is to head over to the QA and development teams and ask them about it.

Well we didn’t have that much set aside for budget and time for testing. We only ran it on a couple of the top devices” explains Stephan your lead developer and tester. “We caught all the big bugs so it shouldn't be that big of a deal”.

By the end of the 3rd day being live on the various stores, you realise there are major issues and reviews have not been kind.

What could have gone wrong? How could you have avoided this and the barrage of meetings needed to address the situation? These are problems that product managers, QA managers and CTOs have to deal with when releasing apps. In fact, while QA might have been the first stop in our fictitious example, QA needs to be scaled and at the heart of your development efforts right from the start.

The ultimate question ends up being: how do you conduct mobile app testing at scale?

Mobile App Testing and Device Fragmentation

If there is one thing that has undoubtedly caused difficulties for those of us in QA, it is the fragmented nature of the devices, operating systems and ecosystems of mobile devices. Testing on different devices becomes an absolute quagmire of ever increasing quicksand. Variations across different models, operating system updates and other download applications all influence the behaviour of a device. While much of this talk is often thrown at Android, Apple and Windows devices also vary enough to have their own fragmentation issues.

The only way to get around this, unfortunately, is by having a system available to you which can accommodate your testing needs based on your audience. Many companies opt to test on the top 10-15 devices for instance and the last few years of operating system versions. But, even this small amount of testing can result in hundreds of variations which means your QA team gets overwhelmed. Some companies opt for the use of virtual devices at this stage to attempt to address the sheer volume required. Unfortunately, manpower and bandwidth are still required to setup the testing, devices and options.

Additionally, virtualised testing does not address contextual factors found when real people use a device in the wild. So, unless you are testing an application which has been designed for use under laboratory conditions, you’ll end up missing out on the exact kind of bugs that got you into that mess to begin with.

Scaling efforts and Continuous Delivery

Besides dealing with the device fragmentation issues mentioned above, scaling your efforts of testing becomes a bigger problem as you move towards faster delivery times and mechanisms. Most software development firms have embraced the move to faster development initiatives moving them towards a state of continuous delivery. Nowhere else is this more clear than in the mobile space. With companies such as Amazon releasing hourly, Facebook multiple times a day and Google running hot releases, you need to ensure that testing the mobile apps that you develop does not become the bottleneck.

Mobile App Testing & Continuous Delivery

You can do this by ensuring that the entire QA process is a feedback loop that delivers in sync and in tandem with your development processes. Make sure that you have enough resources dedicated to your team or that you use a crowdsourced testing platform to fill in the gaps. The more eyes and ears you can get on your mobile app during the testing and development phase, the quicker you can resolve issues and prevent them from turning into app store review nightmares.

Localising Mobile App Testing

Of course, none of the advice already provided will mean much if a bug pops up when the app releases globally. The problem is, you cannot account for the extra issues that pop up in a global app launch.  If you develop a mobile application and assume that your usability testing conducted in the US or the UK will be enough for users in another country, then you are in for a world of hurt. Localised testing provides, as the name implies, localised insights. This includes things such as how that device or mobile application works within the parameters of that location. Factors such as cellular network strength, data access limitations and device accessibility only scratch the surface of what is possible.

If your plan is to release globally, you need some kind of mobile app usability testing where you release. The most important initial phase of testing will be functional exploratory testing to allow testers to explore the app and find bugs through creative means. Later on down the path, you can switch the focus a little into creating and executing test cases. Whatever process you elect to use, the only way you are going to get a successful global app is via testing with the actual people in the markets you hope to capture.

None of the advice and suggestions provided here is going to provide a bullet proof mobile app release without any bugs. What the advice and suggestions will do, is at least lower the acceptable level of risk for you and your organisation when you release or update your apps. In fact, while the suggestions presented here focus on the initial release of your mobile app, the testing can and should be part of your ongoing release plans and obviously we can help you with your  mobile app testing.

Every time you build a new feature, launch into a new market or develop for a different OS, exploratory testing in the hands of the professional testers who match your target market should be the bedrock of your plan.

Nick Roberts

Nick Roberts

Nick is a senior global technology leader having worked in an array of SaaS companies.

Schedule a Demo