In 1984 Cem Kaner coined the phrase Exploratory Testing and used an interesting analogy in a 2008 talk to compare the differences between it and automated/scripted testing. Before diving into exactly what exploratory testing is, imagine for a moment that you are a police crime scene investigator who has been assigned a murder to investigate. But for this investigation you are specifically restricted to a predefined set of scripted actions you can take to get to the bottom of the crime. You cannot deviate from this script and thus are severely hamstrung in solving the murder unless the outcomes of the scripted actions and questions lineup. This is exactly what scripted/automated testing does. It hamstrings the test process only looking for a predefined set of designed questions and thus, the need for exploratory testing.
Table of contents
- Exploratory testing definition
- Exploratory Testing Example
- When does exploratory testing fall short?
Exploratory testing definition
Exploratory testing “emphasises personal freedom and responsibility” in testing because the individual tester tests, learns and iterates throughout not looking for a pre-designed answer that cannot deviate. A lack of test scripts doesn’t mean a lack of preparation. Instead, it leads to less constraints on the tester. Exploratory tests are all about testers exploring an application to identify and document potential bugs. Testers embark on a process of investigation and discovery in order to effectively test a product. That's what exploratory testing is all about.
The three stages below are not independent of one another and exist as a single, cohesive function or test cycle.
One of the most important functions that a tester needs is an understanding of the app or website that they are testing. This understanding provides context and includes information such as competitive benchmark data, industry knowledge and company details. For certain types of applications, a tutorial may be necessary. Much of this learning will happen in real-time, as they begin to explore and test the software.
An understanding like this ensures that the tester is able to take in all manner of inputs relating to the app or website when he or she is performing the actual test. Contextual knowledge allows testers to provide details surrounding results that they may find and whether or not they are applicable.
They also need to understand deeply the requirements of the specific test that was requested. A deep and comprehensive understanding of the testing requirements arms the tester with the right amount of knowledge to dive deep into the app or website. When the tester knows the requirements, including the scope of test coverage and the metrics they might need to consider, they can make sure to keep their testing and related bugs within those functional realms.
It is also important that the tester knows what the desired outcomes of the testing might be. It is not enough to merely look for bugs but to know what the end goal is for the testing cycle. For early iterations, they might be gathering test data on flaws. For later iterations, the goal may be to ensure that previous bugs have been resolved in such a way that new functional bugs have not been introduced (regression) somewhere else. Sometimes the software testing is to ensure that particular functions work as expected and no bugs found are the desired outcome per se.
A major difference between exploratory and scripted testing is in the test design. Designing differs from scripting in that while the test has specific parameters or rules, it is not done in a preset path or prescribed manner. Exploratory testers are able to conduct the test in a way which they deem fit and do not have a desired or expected outcome. Oftentimes, the exploratory testing technique leads to developing more rigorous, scripted test scenarios over time.
Designing also affords teams the option to map out the various techniques a tester might use. Test management can help decide the device, circumstances, or conditions if that has not been established yet by the test requester. Traditionally however, testers and test teams map out a time box of the test, the number of testers needed and other important pieces of the cycle. The cycle is still not a formal test case base nor is the tester writing out test cases during the testing itself. Testers can use notes, mindmaps, flowcharts, decision tables or any manner of organisational tools at their disposal.
Finally, the tester is given the freedom to complete the test as they feel free to. As soon as the test idea is written or requested, it can be conducted. This freedom means that nobody is waiting for scripted requirements and creative work can begin. The tester can start observing and learning about the application or website. They’ll probe and explore how functions work, how they interact with one another and how those pieces work together so that further exploration can take place. Results are then compiled and reported back through the appropriate methods.
This method of test execution encourages and rewards both the product designed and tester by allowing an unconstricted approach to finding bugs. There is a stark difference between exploratory testing and adhoc testing although frequently the two are construed to mean the same thing. Adhoc approaches are distinguished by their lack of defined process and approaches.
When you think about this exploratory approach, it can be easy to think that the test methods rely on intuition or gut. However, this is not the case. Intuition and gut play a role during when a tester goes to test the application or software, but in the test plan there will be specific goals, functional areas and areas in which functionality is assessed.
It is best to imagine the functional exploratory tester as a detective who is investigating a crime. The crime has been committed by the software engineering team (expected functionality) and the detective is working his or her way through the clues to determine whether or not everything makes sense. Providing this level of freedom to assess your products and services ensures you deliver a level of quality your consumers absolutely demand.
Exploratory Testing Example
A client of Global App Testing uses exploratory testing in a very popular iOS and Android mobile gaming app. When this client is ready to release a new version of their application, usually on a bi-monthly basis, they call upon exploratory testers from the global crowd to functionally explore the app. Testers from around the world will be given general scenarios from which to test expected behaviour of the new version.
For instance, the end user might expect to be able to play against another human or computer player. In doing so, they might need to use tokens or in-game inventory items before battle. The exploratory testing may require a tester to ensure that this functionality works and the player can complete the battle as expected.
Test automation simply doesn’t work for these scenarios. While a script could ensure that tokens or inventory items are purchasable, it can’t emulate a human doing battle! This is the inherent human aspect of exploratory testing.
Exploratory testing in Agile development
Exploratory testing is commonly used by teams following Agile methodology. Agile development places emphasis on collaboration and responding to change, and this testing approach relies heavily on these aspects.
Software testers are encouraged to freestyle in their approach, playing around with the product as part of the testing process. This combination of simultaneous learning and testing is incredibly well-suited to an Agile-based team, but might not fit a traditional software development team.
When does exploratory testing fall short?
Exploratory testing doesn’t always provide the most efficient use of time, however. There may be situations where a more traditional workflow makes more sense. In the same example as mentioned above, if the company releases a new functionality that is unrelated to the in-game inventory or currency, it makes little sense to use testers in a functional way. Simple regression tests can be conducted to ensure that the other features have not been inadvertently affected by the new features released elsewhere.
It is important that a good test plan is put in place at your organisation so that you can be efficient and use time wisely as release deadlines approach. We often run into companies who cut short their testing time in cycles because they have simply run out of time before release. However, with a good plan in place (and using the right scalable solutions) you can avoid these kinds of issues.
What type of testing does your organisation need? Are you developing a mobile application for consumer use? Or, does your company focus on B2B applications and thus is focused on delivering API functions? Whatever the products or services you offer, different types of testing and testing tools are required at different stages. When parts of your products and services are stable and only exhibit small changes, you might only need to consider regression testing. However, new functionality and changes might demand you utilise exploratory testing to ensure all the bugs are found before release.
As part of your organisation's testing, exploratory testing and test case creation/execution can be executed through the use of a professional crowdsourced QA team. Leveraging the experience of experts, your organisation can access decades of testing experience designed to provide you with continuous testing strategies that speed up your development process. Contact us to learn more.