Does a retry make a test flaky?
No. A retry inside a single run doesn’t mark a test as flaky.
Flakiness is worked out statistically across runs, not from retries. Testomat.io looks at a test’s pass rate over its last 100 runs, and flags it as flaky when that rate falls between your configured Minimum Success Rate and Maximum Success Rate.
Example. With the thresholds set to 40% and 60%: a test that ran 14 times and passed 7 times has a 50% pass rate. That’s inside the range, so it’s reported as flaky.
You can adjust both thresholds in your analytics settings to make detection stricter or looser.
Finding tests that retry
Section titled “Finding tests that retry”Retries are still recorded, they just don’t affect flaky detection. To find runs containing retried tests, use TQL:
finished and has_retries > 2You can also include testrun.retries in your report templates to show retry counts in reports.