Developing software is hard work; that much is no secret. The process is time-consuming and often complicated. There’s rarely ever a straight road leading from the formulation of an idea for a new type of software to the moment that new software rolls out.
One of the reasons why software development is such a complex process is that there are many, many layers of testing that each piece of software has to go through.
Whether you’re adding new functionalities to an existing platform, developing a brand-new app from scratch, or implementing fixes to improve your software, you’re going to wind up running a lot of tests before the final version is made public.
There are multiple distinct rounds, or layers, involved when it comes to software testing. This guide is going to focus on smoke testing in particular. More specifically, these are the smoke testing-related topics we will be covering:
What is smoke testing?
Smoke testing is the means by which a software build’s stability is tested. It’s used to determine whether a new build is ready for the next phase of testing, or whether that new build needs to be made stable first.
It’s helpful to think of smoke testing as a step in a flow chart that creates a diverging branch for stable and unstable builds, like this:
Smoke tests have to be conducted on every single new build. This is because they let software developers determine whether the features they’ve included in the new builds are working, or whether they need to fix them before conducting further testing.
Smoke testing is a type of acceptance testing, and it’s similar to smaller-scale, quicker versions of the kind of regression test you’d use to assess a major functionality. Much like regression testing, smoke testing helps developers figure out whether the software is performing as it should, and whether it’s ready to move into the next phase of testing.
In other words, smoke tests let software developers verify the validity of the builds they’ve created. By implementing a smoke testing stage before other formal tests are done, they can ensure that no time or resources are wasted during the later testing stages.
There are three types of smoke testing, and we’ll dive into those later and explore each one in turn. The short of it is that smoke testing is highly adaptable, and can be adjusted to suit the needs of just about any business that would consider using it.
Why is smoke testing important?
Without smoke tests, it’s a waste of time to run other tests. In short, smoke testing is important because it’s an indicator of whether a build is even ready for formal testing. Without it, major issues slip through the cracks and have the chance to stay within a build for longer—that’s why smoke testing lets developers squash bugs early on.
Aside from bugs, smoke testing is important because it paves the way for other types of testing, and performing more tests on builds improves the overall quality of those builds. Without smoke testing first, you can’t move on to functional testing, which means the build can’t progress.
It’s also worth considering that smoke testing is integral to multiple other stages in the testing process. Smoke testing is used during system testing, for example, and makes build validation easier and faster across all testing types.
Why is it called “smoke testing”?
The term “smoke testing” didn’t come from the software development industry at all, far from it, in fact. It originated with the construction industry. More specifically, it comes from the way construction workers test water pipes.
Water pipes can’t have leaks, or floods could come about quickly and damage infrastructure. That’s why workers sometimes pass smoke through water pipelines before exposing them to water. If the smoke creeps out of the pipes anywhere, except at the ends, then that’s where the leaks are.
The same idea of validating the functionality of builds is why smoke testing in software development carries that name. You can’t run a network of water pipes if even one join of one pipe leaks; in the same way, you can only use software builds when all of their components work as intended.
What is smoke testing for?
Smoke testing is used for determining whether a software build is stable, and whether that build is ready to go through the next set of tests.
Its purpose is strikingly similar to its namesake, as described in the previous section. Just as water pipes can’t afford to have leaks where the water could get out, software builds cannot be tested, let alone used, if they’re unstable. The software has to be able to run correctly, without crashing or showing other signs of instability, to have any chance of performing its intended function.
We’ve established that smoke testing checks for stability. What this means is that it ensures that all the critical functionalities of the software in question can run seamlessly, or at least, without crashing. Whether it looks beautiful in the process (and can handle a broad range of, if not all, intended situations) is for later testing stages to determine.
When to use smoke testing
Since smoke testing is a type of build verification testing, it normally sees the most use at the start of the testing cycle for any new builds. That includes both new code that will become the foundation of brand new software, and new builds that are developed to be integrated with existing software.
In short, if a developer wants to know whether their software is working as it should, smoke testing is used.
For example, let’s say you want to build a new mobile app from scratch. That means you’ll be joining a rapidly-growing market with the potential for lots of revenue:
Of course, with such rapid revenue growth, this market is highly competitive. You cannot afford to send your new app out into the world without testing it extensively first and guaranteeing that as many bugs as possible have been ironed out in the first version you’ll publish.
To get your app to that stage, you’d have to know for certain that the builds on which it’s based are seamless, error-free, and completely stable. That’s where smoke testing comes in. You would want to start smoke testing the app’s components in the early stages of development, and then continuously as the app’s code is improved and edited.
Types of smoke testing
Considering how crucial smoke testing is to successful software build development, it’s no surprise that there are multiple different varieties.
The most common three are hybrid testing, manual testing, and automated testing. We’re going to take a look at what sets these three apart from one another, as well as what makes them uniquely useful.
Opting for the manual method means employing human testers to conduct your smoke tests. For example, let’s consider the mobile app development scenario again.
If you’re developing a brand-new mobile app that needs testing, you can get your manual testers to try and run it across a variety of devices (depending on which one(s) they own). The results might look something like this:
Despite being more time-consuming than complete automation, manual testing comes with the built-in benefit of exposing your app to the kind of use it would be seeing after being put on the market. That means your test cases are going to be more indicative of your app’s practical functionality.
Instead of relying on human testers, you can choose to use tools to automate the process. Aside from boosting your company efficiency drastically, automating your smoke testing means you get to save money and resources that would otherwise be used on manual testers.
The automated testing process is also necessarily quicker than any others. That’s because automation tools can perform their designated tasks much more rapidly than a human counterpart could.
The hybrid testing method combines the best parts of the other two types (manual and automated tests). It essentially involves employing some automation in the smoke testing process, while also relying on a degree of manual human labor to test the software.
This type of testing has an advantage over the other two in that it mixes the benefits of both of them. A development team that uses hybrid smoke testing can ensure that both the human testers and the automated system can vouch for the viability of their builds.
Smoke testing vs. sanity testing
We’ve already established that smoke testing is a way of testing the stability and validity of new builds, so to compare it with sanity testing, we’re going to define the latter.
A sanity test is performed on software builds that have just gone through small code edits and improvements. These edits aimed to remove bugs without introducing any new problems in the process. Sanity tests confirm that this is the case; that the improvements to the build squashed bugs without creating new ones, while also retaining functionality.
Sanity tests are a subset of regression tests. Their aim is distinct from smoke tests, since smoke testing aims to confirm broader functionality, while sanity testing checks whether smaller fixes are working as intended. This graphic shows the difference between the two:
As you can see, both tests need to be passed for any given build to move into later testing stages. However, smoke testing focuses on initial stability, whereas sanity testing emphasizes smooth continuous integration of bug fixes and software updates.
Sanity testing can be compared to unit testing. Both sanity and unit tests focus on smaller parts of the bigger whole to ensure functionality. However, unit testing hones in more closely on just one unit, whereas sanity testing’s emphasis lies in testing the rationality (or, as it happens, sanity) of the choices developers made in fixing bugs.
Benefits of smoke testing
So far, we’ve seen that smoke testing is a necessary thing that all software builds must go through to guarantee stability. That might make smoke tests look like just another item to add to the long checklist your software developers doubtlessly have to maintain already. And if it’s just another item, why not skip it? What makes it so valuable?
The answer to both of those questions lies in the long list of benefits that come with smoke testing. Below are the most significant advantages of performing smoke tests.
It’s incredibly useful to be able to verify, reliably and early on, that your builds are stable. This makes them more usable at later stages, and reduces the amount of work developers have to put into manually searching for and reporting bugs late into the development cycle.
Guaranteeing that things are functional moving forward also provides a certain degree of peace of mind. Everyone involved in the later stages of the testing process will be able to work knowing that they don’t need to worry about huge bugs within the code causing endless glitches.
In short, nobody’s time gets wasted.
It also helps that smoke testing often enough to guarantee system stability, makes for a solid foundation to later improve upon. It’s easier, after all, to add to something that’s already good than it would be to try and overhaul a broken system later down the line.
Having stable software builds across the board makes it significantly simpler to increase revenue and overall performance. That means your company gets to join the others expecting to grow their revenue in the near future:
One of the biggest advantages of smoke tests is that they’re very easy to conduct. In the next section, we’re going to go through all the necessary steps to perform a smoke test—and there are only four steps altogether.
That means you can run smoke tests often without using up a lot of resources in the process.
Due to their simplicity, smoke tests are compatible with just about every testing methodology you could think to combine them with. Whether you want to insert smoke testing after non-functional testing, add it as a step more than once in the process, and/or put in a smoke testing stage right before your hardware testing, the simplicity of the process ensures you can do it.
Identifies bugs easily
The fewer bugs that make it to users, the better. That’s why it’s important to catch bugs as early on as possible; it gives developers more time to fix them, and ensures that the most prominent ones never make it to end-users.
By smoke testing often, you can ensure that your software is relatively bug-free when it rolls out. At the very least, you can promise your users that you’ve already caught and fixed any potentially harmful bugs.
Improves end-product quality
It’s simple: Fewer bugs mean a better product, and smoke testing often leads to fewer bugs.
Providing a high quality product means your customers will be more satisfied, which in turn creates a better reputation for your company.
The value of quality assurance—and the role it plays in customer satisfaction—can’t be overstated. When your software runs seamlessly, with no bugs hindering customers’ experiences of it, they’re going to have a better time using it.
In other words, they’ll have those priceless customer experiences that your company needs to focus on:
Smoke testing can help make that happen. All you need to do is use it often, and well, to ensure you’re catching big bugs before you even begin testing for other types of flaws.
Makes QA teams’ work easier
Your QA teams have a lot that they need to do, and cannot afford to waste time exclusively running large test suites to catch issues smoke tests could be catching instead.
If you want to maximize the efficiency of your QA testing, you should run more smoke tests, and with greater frequency. That’s because your smoke tests can catch bugs that your QA teams then do not have to spend time sifting through code for.
Since the process of smoke testing is both simple and efficient, you can use it to ensure your QA teams have enough time and resources left to really search for bugs with a fine-toothed comb. That way, you get to roll out smooth and error-free software, all while saving money.
There’s little point in wasting time working on a build that’s riddled with errors and barely functional. That’s why you’ll want to rely on smoke tests to point out all errors affecting stability. That way, your developers get to remove them early, and then each subsequent build that adds to the original gets to be founded on a solid and bug-free piece of software.
In other words, smoke tests increase the reliability of your foundational code. They also help your new builds integrate more easily, since these will also be relatively clean of major issues.
Furthermore, smoke testing improves the efficiency of your other testing stages. That’s because those other stages won’t be affected by the problems your smoke tests catch, and which can then be fixed.
Lessens integration risks
In this specific instance, we’re not referring to the kinds of integrations that let you stack your tech into one unified app. Those look like this:
The integrations that smoke testing helps with are the kind that involve taking an existing piece of software and adding new functionalities to it. Those functionalities are added by means of fresh builds, each of which would need to be tested first.
When that testing includes smoke testing, you can guarantee that each component of the resultant integrated software has been validated already. That makes for smoother, more stable integration overall.
Also, you get to lower the chances of your original build rejecting the new content when you’ve got proof that the new content is stable and free of software-breaking bugs.
In other words, thanks to smoke testing, you can make it easier to incorporate new content into your existing apps and software. That means you can continuously improve the software you present to your customers.
Saves time and resources
Being able to find the sources of issues quickly and efficiently by smoke testing means your teams have to spend less time combing the code for the potential culprits. That saves time, as well as resources—especially if your team doesn’t use much automation.
Further to the point on automation, the smoke testing process can be fully automated if you choose to opt for that route. The next item in this list will elaborate on that point.
Can be automated
Automation saves a lot of resources, as well as time. Bots can always complete processes and workflows faster than their human counterparts. While you cannot automate the processes that rely on human ingenuity and ideas, you can certainly save yourself a lot of time by having AI programs handle the smoke testing process.
Opting for automation comes with a wide array of benefits.
One particular benefit is that when you automate a process, you can guarantee that every time that process is completed, it’s happening to the same standard. That means you can eliminate the unpredictability associated with human error.
You can also run tests more often. That’s because automation helps your teams spend less time on repetitive tasks.
All you need to do is assign an automated AI bot to run smoke tests, and you can have as many as you like done very quickly.
At the same time, you don’t need to use automation at all to make the most of smoke testing.
That’s because another benefit of the process itself is highly flexible and agile. With three types of smoke test, there’s always going to be at least one that suits your needs.
As great as automation can be, it isn’t for everyone. Still, if you don’t want to use automation for any reason, you can still smoke test by choosing to do it manually. This method comes with its own advantages, as we’ve seen in the section discussing it more extensively.
The flexibility of smoke testing isn’t limited to automated and manual testing, either. Smoke testing works on all kinds of builds, with minimal adjustments to the actual process itself.
Practically every business that deals with software in any capacity can use smoke testing in some way or form. The agility that’s necessarily built into the process ensures that it’s usable by all of those businesses as well.
Delivers feedback quickly
Getting feedback regularly is vital to all kinds of areas of business. The fact that collecting it from customers is considered by over half of the respondents to a 2019 survey to be the most effective tactic for improving the customer experience goes a long way to indicate as much:
Of course, software developers need to secure the quality of the product they’re crafting before putting it out for customers to try out. That means they can’t rely on the power of customer feedback to push them to keep improving.
That’s where testing feedback comes in.
With smoke tests, your developers can get feedback quickly, and regularly. In other words, they’ll constantly get to see what they did well, and which areas need improvement.
When you’re using an automated method of smoke testing, you get more feedback. You also get it faster. Both of these things mean developers get more opportunities to smooth out errors, and that they get to see how their progress is looking all the time.
By choosing a hybrid or manual approach, your developers get to receive feedback from human testers. That means they can see how their software is performing with users in real-time, which boosts the chances of catching the kinds of bugs that eventual customers will be likely to encounter.
It also helps that with this constant feedback being provided, developers won’t need to wait on the submission of bug reports before getting an indication of where they should focus their efforts. Smoke tests can point out the biggest bugs, while later subsequent tests will highlight the smaller and more elusive issues.
Ensures your API testing process is smooth
API (application program interface) testing is a crucial part of the software development cycle, and it’s another one that can benefit massively from the incorporation of smoke testing into that cycle.
This type of testing is geared towards analyzing an API to determine whether it’s able to perform its expected functionalities. API testing also verifies whether the software build in question can consistently handle the necessary security protection your app or software needs, as well as assessing its overall reliability.
Since the entire API testing process is all about assessing the validity of your software builds, it becomes a lot easier when it’s preceded by extensive smoke testing.
There’s no need for your API testers to waste their time looking for bugs that aren’t there, and the bugs won’t be there if smoke testing catches them first. It therefore follows that API testing relies on effective smoke testing.
How to perform smoke testing
We’ve covered the benefits of smoke testing, and outlined why it’s a good idea to spend time conducting smoke tests. Now it’s time to learn how.
As mentioned earlier, there are three different ways to perform a smoke test: You can go the manual, hybrid, or automated route. Regardless of the specific type of smoke test you’re conducting, you’ll need to follow the steps listed in the tutorial below to conduct it successfully every time.
Step 1: Decide on the number of smoke test cases you’ll need
Before you can start a smoke test, you need to know how many test cases you’re going to need. If you don’t know the minimum number of test cases you’ll need to guarantee that everything is working as intended, you’ll find yourself having to slow down and recalculate during later stages.
To avoid this, make sure you take the time to consider how many times each build you’re smoke testing has to actually be run through the test. That means adjusting the settings on your automated software and/or determining the number of manual testers before beginning testing.
It might be tempting to say you’ll run a test just once, and then accept that result as the truth. However, the more often you test, the more powerful and useful the results are:
That’s why you’ll want to choose your number of test cases (in the graphic, number of observations) carefully.
As an added bonus, selecting the number of test cases in advance guarantees that you’ll move through the smoke testing phase more quickly. This is because you’ll clearly know when you’re done.
Step 2: Create smoke tests
The next part of the smoke testing process involves crafting the test suites that will be used in Step 3. That means preparing the test cases and the test script, as well as adjusting your methods to the type of tests you’ve chosen (manual, hybrid, or automated).
In this step, it’s also important to make sure that your tests are well-suited to the type of business you’re running.
For example, a SaaS company might have to run some of the most extensive smoke tests, since software is their entire lifeblood. An ecommerce business, on the other hand, would be more likely to seek basic functionality first, then add tweaks and improvements later.
Step 3: Run smoke tests
Of course, you can’t perform smoke tests without this crucial step: Actually running the tests.
The most important thing to keep in mind here is to follow the guidelines you set out in previous steps. If a number of your manual testers suddenly cancel, for instance, you’ll want to make sure you have replacements at the ready.
By the time you reach this stage, you should already have the necessary plans prepared to deal with those kinds of eventualities. Of course, there can always be surprising developments. That’s why it’s important to be actively involved in monitoring the process and watching for hitches.
As you’re running your smoke tests, ensure that you (or your testers, whether manual or automated) record the results of each one. This will make the next step easier.
Step 4: Analyze smoke tests
Analyzing the results of your smoke tests will help you to determine whether the overall result is a pass or a fail.
The parameters for passing and failing depend on a lot of things, most notably your own company’s needs and standards.
As before, if you’re a SaaS company, your standards for the quality of your software (or web application) would necessarily be quite high. That means that if 10% of your tests indicate an unstable build, you’d probably want to send it back for improvement. The same percentage could qualify a build as stable for other developers, though.
The thing to keep in mind during the analysis stage is, in other words, that you’ve got to adjust your expectations to your own requirements. For other things to bear in mind, consider the tips in the next section.
Whether you’re running a smoke test for the first time or you’ve been doing it for years, there are tips every smoke tester can benefit from bearing in mind. Here’s a quick list of the top tips and best practices for effective smoke testing.
- Run smoke tests often, and early. The sooner you can catch errors that need fixing, the sooner you’ll have a stable build on your hands. That’s why it’s important to run your smoke tests as early—and often—as you can.
- Never skip the smoke testing stage. It might be tempting to assume that a build is stable and ready for the next round of testing, but that’s a dangerous way to think. You can never be sure of the validity of a build without testing it first, so make sure you always run your smoke tests as a matter of course.
- Test everything. This tip is connected to the previous one. Regardless of whether the build in question will be moving into integration testing, performance testing, or sanity testing next, you’ve got to run smoke tests on each and every build. No exceptions. Every kind of software build benefits from being smoke tested.
- Ensure you’ve got a checklist. A testing checklist is especially important for anyone opting for manual testing, though anyone can benefit from organizing their process using one.
- Track your smoke test results. If a build keeps failing your smoke tests, it’s important to know. Likewise, if a particular developer’s builds always pass their smoke tests, you’ll want to be aware of this. Incorporate results tracking into your smoke testing workflow to keep on top of these types of things.
- Test more than once. One test might not catch every potential error. Running your builds through the smoke testing process more than once increases the chances of stopping any major bugs in their tracks, as well as providing you with some extra peace of mind.
- Choose the right kind of smoke test for your software. If you’re running a small business with a limited budget, it might not be feasible for you to try to hire enough testers to go with the manual approach. In that instance, the hybrid or automated method would likely suit you best. Make sure you make your choice according to your needs.
If there’s one thing you take away from this guide, let it be this: Smoke testing is a highly efficient, hugely useful way to validate the stability of your software builds.
Testing often is important for a number of reasons, including securing build stability, proving the validity of new builds, and ensuring smooth integration. In a word, it’s efficient. Conducting more tests comes with a wide array of benefits, and carries almost no drawbacks.
Smoke testing, in particular, is a handy way to guarantee that your key features work as intended, which can only make your software builds look better to their users. It helps your software deliver on the expectations your customers have for the standard of quality they want to see, which, in turn, creates a more positive image of your brand.
It also doesn’t hurt that smoke testing often is a sure way to speed up your release cycle, since it ensures that your builds work properly, early on in the development cycle.
Benefits aside, conducting a smoke test is quite easy. All you need to do is follow the steps set out in this guide, and you’ll be smoke testing effectively (and reaping all the associated benefits) in no time.