The Ultimate Guide to Smoke Testing

Introduction

This is the ultimate guide to smoke testing, our guide to everything you need to know about what smoke testing is, when to test, when to use it, and mistakes we’ve seen clients make in their smoke testing process. 

Global App Testing can offer tests turned around in as quickly as 6 hours, designed to support you by taking edge cases and execution of tests in your automation test queue out of the hands of your quality engineers, to give you the bandwidth to automate more tests. 

Which is handy! Because as we’re about to see, smoke testing has the highest ROI of any kind of test when it comes to automating your test. If you want to get started with Global App Testing, click the button below.

Recap | What is smoke testing?

“Smoke testing” refers to broad-but-shallow functional testing for the main functionality of a product.  It’s so-called because of the joke application of an electrical engineering concept: in electrical engineering, your product passes a “smoke test” if your device doesn’t catch fire when it turns on.

The goal of a smoke test is to prevent the needless expense and time waste of deeper tests on a broken product. A smoke test asks “is the titanic floating?” If we run the test and find that the titanic is indeed floating, we’ll still have to conduct other tests to verify that the titanic is seaworthy. If the titanic is not floating, we know that there is no point in running those additional tests yet.

A smoke test can be used in multiple kinds of build environments, but is strictly functional. It should try to touch every component of your product (“broad”) but must be fast to achieve its goals (“shallow”). You might test that you can set up a bank account and transfer money (for a banking app), or buy an item (ecommerce), or make a social post (social software). 

(Fun fact – the term smoke testing has multiple distinct origins. In electrical engineering, smoke testing was a real test, by attaching the circuit of an electrical device to a power source to see whether it caught on fire. In plumbing, smoke can be applied to the testing pipes by sending smoke down them, top check for leaks.) 

What is a successful smoke test?

Because a smoke test is designed to save your organization time and money, this guide will argue your smoke tests should reflect those goals from a best practices standpoint. One common failure category of smoke tests is that they are too thorough – the purpose of a smoke test is not to catch all bugs, but to make your testing workflow more efficient by validating whether your product works in its core functions, quickly.

Definitions | How is smoke testing different to… sanity testing, or regression testing?

Regression testing is quite specific – it refers to the complete functional test of a whole software system when a new feature is introduced or launched (check out our regression testing page). 

“Sanity testing”  is less common in use and has a less precise meaning than either regression or smoke testing. However, we’ve seen sanity testing defined as a subset of regression testing which bears some similarities to smoke testing. Like smoke testing, “sanity testing” is a big picture test of core functionality to verify that further tests should go ahead. Whereas smoke tests apply to unstable and initial builds, sanity tests apply to stable build undergoing a regression test series for a new feature. 

Smoke testing as part of your test anatomy

Like all testing, your smoke testing quality process is highly specific and can be traced back to the commercial and operational incentives of your organization. Global App Testing generally advocates for higher quality processes and products in our book Leading Quality. (But then, we would say that, wouldn’t we?)  

In a narrow sense, the moment to run a smoke test is, “whenever you want to check the application is working”. In a more prescriptive sense, there are key times this crops up during which smoke tests are a sensible investment: 

Common moments to run smoke tests

• Before you commit code to a repository – if you don’t run your full test suite in a local environment, you should at least make sure you haven’t broken anything so severe that it shows up in a smoke test. Testing in the initial pre-commit stage can be done by devs working on local devices by running a Git test script coupled with a client-side hook.
  
• Before a large test series including regression and acceptance testing. Although an automated smoke test would theoretically save time against any manual test, how much time you’ll save is proportional to the scale of the test series you’re about to undertake. We undertake manual smoke tests on clients’ behalf before a major test series, too – but the saving is more marginal. 
• After deployment – immediately after deployment is a sensible time to undertake a smoke test to ensure that everything is still working properly. 
• Any time – as long as your smoke test is rapid (and probably, automated), we would generally advocate for a liberal smoke testing policy. Any time that you need to test your system is working in a big picture sense demands a smoke test.

Smoke testing within CI/CD, DevOps, agile

Because of the purpose of smoke tests in generating faster failures and shorter feedback loops, they have become closely associated with modern programming methodologies such as testing for agile and frameworks which focus intensely on speed of deployment. With DevOps in particular, while it’s not a technical requirement to have automated smoke tests, it is probably a de facto one. 

How do I execute a smoke test?

Automated, Manual, Hybrid

You can execute any manual test including smoke tests by logging into your Global App Testing dashboard, entering or uploading the appropriate test cases and pressing “launch” to receive your results back in as little as 2 hours. Express tests with Global App Testing are made to offer manual tests with the ease, convenience and speed of running automated tests. We have also integrated with Jira, TestRail, GitHub, Slack and Zephyr Squad so you can launch and receive test results where your teams like to work. 

For more complex and bespoke testing for our clients, we will often run a quick smoke test to verify that the test is worth doing as a matter of policy. 

To write the appropriate test cases, identify all of the areas of your product that you think a smoke test should touch and write out their core functionality step-by-step. Executing the tests properly (i.e. writing them out and writing “pass/fail” next to each step) rather than ad hoc play testing as some developers prefer is likely to ensure that your test is working as intended and that you don’t cut corners.

Writing an automated smoke test 

This is outside of the scope of this article, because this depends on the language in which your application is written and other factors which relate to your smoke testing approach specifically. However, here is a useful resource to help you write it.

Smoke testing best practices and failure categories

Smoke testing best practices

Here’s our advice on what to do when you’re smoke testing: 

1/ This is the highest ROI of “automating” any test – you should probably automate it

Because of the frequency with which you’ll undertake smoke testing, the savings of automating your smoke tests are the greatest of all the savings we’d associate with the automation of any test. We’re a manual testing solution so we often have to make judgments about when our clients would be better off with an automated test (or even give our clients the bandwidth to automate more of their testing suite) – but a smoke test is usually better off done by a program.  

2/ Run smoke tests frequently

Once your smoke test has been automated, particularly if parts of your testing suite are manual or time-intensive, there are only advantages to testing more frequently. We’d recommend you run the smoke test at every stage of the production environment, from before you commit right through to after deployment. If your smoke test is manual, consider running smoke tests 

3/ Ensure the whole system is touched

One failure mode of building a smoke test is that only part of the system gets tested by the test. If your smoke test is not sufficiently broad, it will fail to find major fault in some modules. Ensure that your test cases touch every function of the system, without looking into the complexer instances of each function. If you have a modular system, you might need to ensure that the relationship between modules is tested in addition to the modules themselves.

Smoke testing failure categories 

1/ Lack of focus 

The most likely way that your smoke testing will be executed poorly is that you lose sight of the point of the test. The test is to prevent wasted time, not to find every bug thoroughly. Excessive depth is therefore the most common failure category of smoke tests, because by conflating a thorough test with a quick check, a test suite is likely to fail in both categories. Similarly, failure to touch every software module can mean that the product is not broadly tested and that tests begin which are not economical from the perspective of time or money.

2/ Failure to identify whether it will save time 

Failure to (correctly) identify when and whether a smoke test is going to save your organization time is the second most common category of failure. Probably, you are smoke testing less frequently than optimal, and you could save your organization more time by automating smoke tests and testing more often. But in some cases, you may be smoke testing pointlessly where a smoke test will not save your organization any time. 

Get more advice, straight to your inbox

If you’ve enjoyed this article, we write loads more like it. You can sign up to our content essentials below and get it straight to your inbox – all of our tips about how and when to test, what to test, and how to test to drive quality in your organization.