Flaky Tests: How to Find, Quarantine and Fix Them

A flaky test is one that passes and fails on the same code without any change. One flaky test is an annoyance. Twenty of them teach your team to ignore red builds, and at that point your automated suite has stopped protecting you.

These are the practices our test automation engineers use to bring unreliable suites back under control.

Measure flakiness before you fix it

You cannot fix what you cannot see. Record the result of every test on every run, then calculate a simple stability score: the percentage of runs on unchanged code where the test passed. Anything below 98 percent deserves attention. Most CI tools can export this data, and a basic spreadsheet is enough to start.

Quarantine, do not delete

When a test is confirmed as flaky, move it into a quarantine group that still runs but does not block merges. This keeps the main pipeline trustworthy while you investigate. Set a rule that quarantined tests must be fixed or removed within two weeks, so the quarantine does not become a graveyard.

The five most common causes

• Timing assumptions. Fixed sleeps and tests that assume an element appears instantly. Replace them with waits for a specific condition, such as a network response or a visible element.
• Shared test data. Two tests editing the same account at the same time. Give each test its own data, created at setup and cleaned up after.
• Order dependence. A test that only passes when another test runs first. Run your suite in random order occasionally to expose these.
• Unstable environments. Shared staging servers, third party sandboxes and slow CI runners. Mock external services where you can and give CI enough resources.
• Brittle selectors. Locators tied to styling or layout. Use dedicated test attributes or accessible roles and names instead.

Be careful with automatic retries

Retries make a pipeline look green, but they hide real problems, including genuine race conditions in your product. If you use retries, log every retry and review them weekly. A test that needs a retry is still a flaky test.

Make stability part of code review

New tests should pass reliably before they merge. A simple rule works well: run any new or changed test 20 times in CI, and only merge it if it passes all 20. It adds a few minutes to review and saves hours of debugging later.

Keep people in the loop

Automation is at its best when paired with human testing. Exploratory sessions from manual QA testing often reveal the product behavior that makes a test flaky in the first place. For help deciding what belongs in the suite at all, read manual vs automated testing: when your app needs each. If your suite has lost the team’s trust, talk to a Charlie Testing engineer about a stability audit. Once a flaky test is fixed, keep it earning its place in your regression testing suite.