A flaky manual test produces different results on repeated runs with no code change. Unlike automated flakiness, which usually traces to timing or brittle selectors, manual flakiness comes from ambiguous test cases, environment drift, dirty test data, and tester fatigue. All four are fixable without hiring.
For a small QA team, a handful of flaky results erodes confidence in every cycle. Engineers start questioning whether failures are real bugs or noise, and that doubt slows releases. This guide covers the root causes, how to write test cases that resist flakiness, how to standardise environments and data, and how to quarantine unreliable tests before they contaminate your suite.
A flaky result occurs when the same manual test case produces different outcomes on repeated runs, even though the application code has not changed. One session marks a feature as passing; the next flags it as failing.
In automated testing, flakiness usually traces back to timing issues or brittle selectors. In manual app testing, the causes are different. Human factors, environment drift, and vague instructions all play a role.
The downstream cost is significant. Developers spend time investigating non-issues, product managers delay launches, and the QA team's credibility takes a hit. Addressing flakiness early protects both your release cadence and your team's standing.
Small teams wear many hats. A QA engineer might write test cases in the morning, execute them after lunch, and triage bugs before the end of the day. Context switching increases the chance of missed steps and inconsistent execution.
Limited device inventories add risk. If your team owns only a few phones and laptops, you cannot reproduce every user environment. A test that passes on one device may fail on another because of OS version, screen resolution, or network speed.
Budget constraints often mean shared test environments. When two testers run sessions against the same staging server, one session alters the data state for the other, producing an unreliable outcome that has nothing to do with application quality.
Documentation gaps compound it. On a larger team, a test architect maintains a central case library and environment runbook. On a five-person squad, that documentation lives in scattered spreadsheets and personal notes.
| Cause | What it looks like | Fix |
|---|---|---|
| Ambiguous test cases | Two testers take different paths through the same case | Atomic steps naming exact inputs and expected responses |
| Environment drift | Cached data and stale config accumulate between sessions | Reset to a documented baseline before each session |
| Device and network variability | A dropped packet reads as an application timeout | Record conditions alongside every result |
| Tester fatigue | Steps skipped or outcomes misread late in a long session | Rotate assignments; cap sessions at 90 minutes |
| Test dependency | One early failure cascades through every downstream case | Each case sets up its own preconditions |
A case that says "verify the checkout flow works" leaves too much room for interpretation. One tester completes checkout with a credit card; another uses a digital wallet. Different paths expose different behaviour, and neither is wrong on its own.
The fix is specificity. Each step should name the exact input, the expected on-screen response, and the device or browser to use.
Test environments degrade. Cached data, leftover accounts, and stale configuration accumulate between sessions. A test relying on a fresh database behaves unpredictably when old records persist.
This is especially common with payment and KYC flows, where test credentials expire or get rate-limited. Resetting to a known baseline before each session eliminates the category.
Manual testers interact with real hardware, and real hardware introduces variability. Battery state, available memory, background processes, and signal strength all influence how an app responds.
A Wi-Fi connection dropping packets intermittently causes a timeout that looks like an application bug. Documenting network conditions alongside each result separates environmental noise from genuine defects.
Running the same regression suite for the third time in a week is mentally taxing. Fatigued testers skip steps, misread expected outcomes, or rush validations, producing a mix of genuine failures and human error that is difficult to separate.
Rotating assignments and scheduling shorter blocks reduces this. Keeping sessions under 90 minutes preserves focus.
When one case depends on the output of a previous one, an early failure cascades. If "create account" fails for an environment reason, every downstream case assuming that account exists also fails, none of them for a real bug.
Designing each case to set up its own preconditions makes the suite resilient. Independent tests run in any order, and a single failure stays contained.
Good test cases are the first line of defence. A well-written case removes judgement calls from execution and anchors every action to a specific, observable outcome. Our guide to writing functional test cases covers the fundamentals.
Break every interaction into a single action and a single expected result. Instead of "fill in the registration form," list each field. Atomic steps make it obvious where a failure occurred.
State the starting environment at the top of each case. Which OS version? Should the account be logged out? Is the cart empty? Preconditions eliminate the uncertainty that introduces variation.
A pass criterion is not "the page loads." Specify what the page shows: a confirmation banner with the order number, a redirect to the dashboard, an email arriving. Define what counts as failure so the tester does not guess.
Attach annotated screenshots or short recordings. A visual reference removes remaining ambiguity about what the expected outcome looks like, particularly useful for UI and localisation tests, where subtle differences matter.
Document every detail: OS version, browser version, installed extensions, database seed. Reset to this baseline before each session. Teams using snapshots or containerised setups can automate the reset.
Running smoke tests and regression tests on the same server invites data collisions. Dedicated environments keep sessions independent.
If your app serves users on mobile networks, simulate those conditions consistently. Network throttling profiles in browser developer tools replicate 3G or 4G speeds. Running every session at the same setting removes network variability from results.
Pin browser versions, OS images, and third-party sandbox endpoints. An unexpected browser update changes rendering behaviour overnight, turning a stable test flaky.
Dirty data is a silent source of flakiness. When test accounts accumulate history, outcomes become unpredictable.
Small teams rarely have access to the device and OS combinations their users rely on. Limited coverage creates blind spots where flaky results hide.
Global App Testing gives you access to a vetted network of professional testers across 190+ countries. You can target specific devices, operating systems, languages, and payment instruments without purchasing hardware or expanding headcount.
Tests run on real devices under realistic conditions. Bug reports include video evidence and reproduction steps, making it straightforward to verify whether a reported issue is a genuine defect or an environment-specific anomaly.
Not every flaky test can be fixed immediately. Quarantine isolates unreliable tests without letting them pollute your main results.
You cannot fix what you do not measure. Four metrics give visibility into process health. The thresholds below are starting points to calibrate against your own baseline rather than industry standards.
| Metric | What it measures | Suggested starting threshold |
|---|---|---|
| Pass-rate consistency | Share of cases returning the same result across consecutive runs | Investigate below 95% |
| Defect rejection rate | Bugs closed as not reproducible or working as expected | Investigate above 10% |
| Mean time to diagnose | How long to determine real bug versus environment issue | Investigate above one hour |
| Flakiness score | Inconsistent results as a ratio of total runs, per case | Sort by score to prioritise rewrites |
A high defect rejection rate often points to flaky results rather than poor development quality, which makes it the most useful early warning of the four.
Manual testing and continuous testing are not mutually exclusive. Triggering manual cycles at specific pipeline stages catches what automated scripts miss without slowing your cadence.
Deploy to staging and run automated smoke tests. Once those pass, launch a manual cycle covering flows needing human judgement: payment checkouts, localisation accuracy, complex multi-step workflows.
Feed results into the tools your developers already use. Our platform integrations put bug reports alongside automated test output, so there is a single source of truth and less context switching during triage.
Set a fixed window, such as four hours. Crowdtesting platforms run hundreds of cases in parallel across time zones, returning results well inside that window.
Use this at the start of every test cycle.
Flaky manual results are not inevitable. They are symptoms of gaps in case design, environment management, data hygiene, and coverage. Addressing each systematically lets even a small team produce dependable results every cycle.
Start with your test cases: atomic, explicit, anchored to observable outcomes. Standardise environments and isolate data. Track metrics and quarantine unreliable tests before they infect the main suite. For how this fits a wider process, see QA testing best practices.
When you need to scale coverage across devices, geographies, and real-world conditions, talk to our team about filling the gaps your own devices can't reach.
Manual testing best practices
15 tips to write functional test cases
Automated testing vs manual testing
QA testing: process and best practices