Flake detection before browser tests can silently green
This walkthrough documents the Playwright retry report gate added for
RFD-BDD-001 D-5 in pramaan-app.
What changed
The e2e CI job now runs the WebKit and WebKit BDD suite with
Playwright retries enabled and writes a JSON report to
playwright-report/flake-detection.json.
pnpm run check:flake-budget counts Playwright tests
whose aggregate JSON status is flaky. CI sets the
budget to 0.
A test that fails first and passes only on retry is not treated as green evidence. The gate prints the flaky test title and exits non-zero when the budget is exceeded.
Bad-input proof
pnpm exec playwright test -c scripts/bdd/fixtures/playwright.flake-fixture.config.ts pnpm run check:flake-budget -- --report playwright-report/seeded-flaky.json
The seeded fixture lives outside the normal e2e suite. It deliberately fails on the first attempt and passes on retry, proving the gate trips on a flaky pass without poisoning the required browser suite.
Expected operator flow
- Run Playwright e2e with JSON reporter output.
- Audit the report with
pnpm run check:flake-budget. - Fix or quarantine intentionally before raising the budget above zero.
Files changed
5 file(s), +224 / -2 lines
| Path | Role | + | - |
|---|---|---|---|
| .github/workflows/ci.yml | Workflow | +7 | -2 |
| package.json | Config | +1 | -0 |
| scripts/bdd/check-flake-budget.ts | Script | +198 | -0 |
| scripts/bdd/fixtures/playwright.flake-fixture.config.ts | Script | +13 | -0 |
| scripts/bdd/fixtures/seeded-flaky.spec.ts | Script | +5 | -0 |