Measure the iframe reveal budgets from inside the page (#2189)

The three reveal tests timed their budgets from before `page.Goto`, so a
slow navigation was spent against a window that belongs to the iframe. In
`TestIframe_StaysHiddenUntilTheDocumentReportsInited` that made the test
vacuous rather than flaky: on a navigation between 2.5 and 5 seconds the
loop bounding the visibility assertion had no budget left, ran zero times,
and the test passed having asserted nothing. Reproduced by delaying the
demo document by three seconds, where the assertion ran 0 times before and
runs 23 after. The timeout test had the mirror of it, with navigation
counting toward the lower bound that exists to catch a shortened fallback.

An init script now records, in the page, when the widget's iframe element
enters the document and when its visibility first flips. `create-iframe.ts`
arms its fallback a moment earlier, on the detached element, so these read
a shade short and every bound is conservative in the same direction.

Both bounds were also wider than the thing they guard. The hidden window
now runs almost to the fallback rather than half of it, and the lower bound
sits just under it rather than at three quarters, which a fallback
shortened to four seconds used to clear.

CI reruns a failing test once rather than failing the build on the first
flake. A browser suite has a floor no amount of care removes, and one flake
failing the build is what stops people trusting the suite. Once rather than
twice, because a rerun stops at the first pass and each further attempt
only widens the window where a real intermittent regression is absorbed.
What needed a rerun is written to a report and uploaded with the traces,
which are kept whether or not the job went green: a run that recovered on
the rerun is exactly the one whose evidence used to be discarded.
This commit is contained in:
Dmitry Verkhoturov
2026-08-21 22:05:51 -05:00
committed by GitHub
parent 7c312da199
commit a0879b2336
4 changed files with 189 additions and 31 deletions
+32 -5
View File
@@ -59,7 +59,9 @@ jobs:
tests:
name: Tests
needs: vet
timeout-minutes: 30
# generous against the docker build plus two go test invocations: a job cancelled on
# timeout skips its own failure steps, so the run would end with neither logs nor traces
timeout-minutes: 45
runs-on: ubuntu-latest
permissions:
contents: read
@@ -89,18 +91,43 @@ jobs:
- name: Build & start the stack
run: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -f compose-e2e-test.yml up -d --build --quiet-pull --wait
- name: Install gotestsum
run: go install gotest.tools/gotestsum@v1.13.0
# each rerun is its own `go test` process carrying the same timeout, so the ceiling is
# 8m for the first attempt plus 4 x 8m of reruns, inside the job's 45. a job cancelled on
# timeout skips its own upload steps, which is the one outcome worth designing against.
# four failures covers one test failing across all three engines, counted as its three
# subtests plus their parent; more than that is a regression and should not be rerun.
# a browser suite has a flake floor no amount of care removes, and one flake failing the
# build is what makes people stop trusting it. one rerun, not two: a rerun stops at the
# first pass, so each further attempt only widens the window in which a real intermittent
# regression is absorbed. what needed a rerun is written to a report and uploaded, since
# a green job is a job nobody reads the log of
- name: Run e2e
run: cd e2e && go test -tags=e2e -count 1 -timeout 20m -v ./...
# shared across the rerun so it exercises the same threads as the attempt that failed
env:
E2E_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
run: |
cd e2e && gotestsum \
--rerun-fails=1 \
--rerun-fails-max-failures=4 \
--rerun-fails-report=rerun-report.txt \
--packages=./... \
--format standard-verbose \
-- -tags=e2e -count 1 -timeout 8m
- name: Server logs on failure
if: failure()
run: docker compose -f compose-e2e-test.yml logs --tail=200
- name: Upload browser traces
if: failure()
- name: Upload browser traces and any rerun report
if: always()
uses: actions/upload-artifact@v7
with:
name: playwright-traces
path: e2e/traces/
path: |
e2e/traces/
e2e/rerun-report.txt
retention-days: 30
if-no-files-found: ignore