The three reveal tests timed their budgets from before `page.Goto`, so a
slow navigation was spent against a window that belongs to the iframe. In
`TestIframe_StaysHiddenUntilTheDocumentReportsInited` that made the test
vacuous rather than flaky: on a navigation between 2.5 and 5 seconds the
loop bounding the visibility assertion had no budget left, ran zero times,
and the test passed having asserted nothing. Reproduced by delaying the
demo document by three seconds, where the assertion ran 0 times before and
runs 23 after. The timeout test had the mirror of it, with navigation
counting toward the lower bound that exists to catch a shortened fallback.
An init script now records, in the page, when the widget's iframe element
enters the document and when its visibility first flips. `create-iframe.ts`
arms its fallback a moment earlier, on the detached element, so these read
a shade short and every bound is conservative in the same direction.
Both bounds were also wider than the thing they guard. The hidden window
now runs almost to the fallback rather than half of it, and the lower bound
sits just under it rather than at three quarters, which a fallback
shortened to four seconds used to clear.
CI reruns a failing test once rather than failing the build on the first
flake. A browser suite has a floor no amount of care removes, and one flake
failing the build is what stops people trusting the suite. Once rather than
twice, because a rerun stops at the first pass and each further attempt
only widens the window where a real intermittent regression is absorbed.
What needed a rerun is written to a report and uploaded with the traces,
which are kept whether or not the job went green: a run that recovered on
the rerun is exactly the one whose evidence used to be discarded.
Dmitry VerkhoturovandGitHub
a0879b2336