Why is there a test count mismatch between Playwright and Testomat?
Context
Section titled “Context”- Playwright version: 1.62.1
- Reporter version: v2.9.0
- Retry strategy:
retryStrategy: 'isolated' - Playwright reports 1662 tests while Testomat shows 1646 tests.
- 16 missing results due to parameterized tests being merged by test ID.
Answer
Section titled “Answer”The mismatch in test counts between Playwright and Testomat is due to the way parameterized tests are handled. In Testomat, parameterized tests are merged by test ID, which results in missing results and incorrect reporting of flaky tests.
To resolve this issue:
- Avoid using parameters as tags in your test titles. Parameters should be included in the test title to ensure they are distinguished properly.
- Manually update test titles to ensure they differ not only by their tags. This can be done by removing the tag before the parameter from the title in both Testomat and the source code after the import.
The Testomat team is aware of this issue and is working on a fix to address the behavior where parameters are reported as tags, which causes the current logic to break.
If this doesn’t work
Section titled “If this doesn’t work”If you continue to experience issues, consider creating a public Run report in Testomat and share it with the support team for further investigation.