Dmitry VerkhoturovandGitHub
4d5dae20e2
* Pin the published /web surface in the e2e suite
#2178 renamed the widget bundles from .js to .mjs and the URLs earlier
releases served under those names stopped resolving. Three were noticed
from the demo site; the rest, including every locale chunk, were found
only by requesting the whole surface of both images over HTTP. #2192
restored them with a server-side alias, and nothing in the suite would
have caught the break or would notice it returning.
Two cases with deliberately different criteria. The documented names are
written out, because the documentation decides that list and not the
build: an operator pastes privacy.html into an OAuth application, the
nginx manual proxies index.html by name, and the integration guides start
from the embed script. Everything else is taken from the build itself, so
whatever the bundler emitted has to serve identical bytes under its
legacy .js name and parse as a classic script, which is the premise
serving one under the other rests on. A third case requests a name that
does not exist, without which a fallback serving one page for everything
would keep the whole table green.
All of them check the content type as well as the bytes: nosniff is set
on every response, so a bundle served as text/plain is as broken as one
that 404s while comparing equal.
On e3d1d0e2, the commit before the alias, this fails with 32 red subtests.
* Cover the widget behavior the e2e suite never drove
Removing npm from the widget takes the jest tests with it, and everything
here was protected by jest or by nothing at all.
Signing out was untested at every level, and the panel repainting is the
half that always works: the assertion after the reload is the one that
catches a session the server never ended. The edit form has to hand back
the source that was posted and not the rendered comment, which #2040
shipped the other way round, taking every entity and tag the author had
written with it. A draft has to survive a reload and be gone once the
comment is posted. A refused comment has to stay in the form with
something said about it, so that case drives the backend's own
restricted-words code and then retries with the route removed.
Uploads had no browser coverage in either direction. The posted case
asserts naturalWidth instead of visibility, since a broken src still
renders as an empty box, and the failing case holds the intercepted
request long enough for the in-flight state to be observed: without that,
"the text is unchanged afterwards" would hold for an upload that never
started.
Moderation and reader-side hiding are asserted from a page other than the
one that made the change. Hiding seeds a second author, so it proves one
person is hidden and the thread not emptied, and both the blocked
author and the moderated one carry the run id in their names: a block and
a verification are properties of the user and outlive the run in the
stack's database, so a fixed name works exactly once.
Locales were the largest hole. Each catalog is a chunk fetched at
runtime, and loadLocale falls back to english on any failure instead of
throwing, exactly as an unrecognized name does, so every case compares
the rendered string against the file on disk. One case per catalog
covers that they are all served and render; another fetches one through
the widget document and asserts it parsed, which is the half a chunk that
serves but fails to parse would slip through. The delete page and the
last-comments stylesheet had no coverage of any kind.
Two things the suite itself needed. The widget's own aria-label is
translated, so commentFormSel only ever finds an english widget and a
localized case cannot use widget(). And the auth probe is capped at two
requests a second for the whole suite, hard-coded in rest.go: the added
cases pushed past it and the suite began manufacturing its own 429s,
which render as a signed-out widget and fail whichever test happens to be
signing in, so the pacing gap is wider and the locale cases stub the
probe they never needed.
* Harden the e2e harness against silent failures and stale stacks
Three things the suite could not tell you about itself.
A browser failure nothing asserts on now fails the test that caused it.
Uncaught exceptions are the ones worth the machinery: a widget throwing
while it renders leaves most of these cases green, since they assert on
elements the browser lays out either way. Rate-limit responses are
recorded as well as logged for the same reason. A test driving an error
path declares what it expects by substring, so a case that means to
break something says which thing.
The stack the suite adopts is now checked against the sources under
test. Every checkout builds the image tag the compose file names, so a
stack from another worktree, or from this one before an edit, answers on
these ports and passes every readiness probe while serving code nobody
is looking at. stamp.sh digests what goes into the image, compose passes
it as the revision label, and a mismatch is refused with what to do
about it. Checked after our own build too, or a stamp that never reaches
the image would be a guard that silently passes everything.
assertSignedIn no longer waits out the whole timeout on a refused status
read. /auth/ is capped at two requests a second for the entire suite, a
bare literal at rest.go:242, and a case signing in on two pages spends
that twice; when the read that repaints the panel is the one the limiter
turns down, the widget shows signed out over a session that exists and
no later request will ask again. The short first wait now ends in a
focus handoff, which the widget answers by re-probing, and only then
does the real wait run. A sign-in that genuinely failed still fails,
since the second read finds no state either. That is what
TestComment_AdminPinsAndVerifies and TestComment_BlockedAuthorCannotPost
were failing on in CI while passing locally.
signInAnon takes the page for that reason, and its callers pass it.
* Cover iframe geometry, the embed contract and five deployment modes
Seventeen cases for the parts of the widget that broke repeatedly and
that nothing here could see, plus the areas jest was the only check on.
Geometry is the biggest of them. The widget measures its own document
and posts the number for the parent to apply, and every way that has
gone wrong is invisible to assertions about elements, which read the
same whether the frame is right, twice too tall or a strip. So: the
first height the parent is given describes rendered content and not the
preloader, the frame matches the document it holds and does not stand
24px taller, no_footer leaves the last comment inside the frame, and the
frame follows the sign-in dropdown and the growing textarea and comes
back down again. Each was verified by reintroducing the defect it covers
and watching it fail: a 63px report before the real one, six pixels of
body padding, and a frame sized under the content.
The embed surface is the other half the widget cannot see. An element
placeholder gives way to exactly one iframe carrying the embed's own
marker, a second createInstance reuses it, destroy takes it away with
its handles, and a theme change after load reaches both the element and
the document. A page that posts a message of its own, which is all
embed.ts's own title observer does, no longer empties an open login
form.
Five configurations that cannot share an instance get one each, since
each changes the widget for every reader: an admin's unlimited edit
window, a session carried in a header and not a cookie, an instance with
no auth provider to offer, anonymous voting, and the notify module,
without which email_notifications is false and the subscribe control
never renders at all. The subscription round trip is the one place a
token from a real message is exchanged for state the server keeps.
Two things surfaced there and are left alone, both said so in place. The
panel confirming an unsubscribe cannot be observed, because the click
changes the step and the dropdown closes on an element no longer in the
rerendered view, which the component notes as its own awkwardness; the
case asserts the request and the answer, which is what decides whether
the reader still gets mail. And the widget takes the subscribed state
from user.email_subscription, absent from what it hydrates the user with
on the next load, so after a reload it offers to subscribe somebody who
already is.
simple_view needs no instance, being a query parameter, so both branches
run against the main one. A transient failure of the status probe has a
case too: the session belongs to the server, and one refused answer must
not end it.
The suite runs about four and a half minutes now, so the workflow's own
budget goes to 20m to match the Makefile, well inside the job timeout,
and the workflow stamps the stack it starts the way the Makefile does.
The locale case that loads the widget document directly is renamed for
what it protects, the origin and CSP its chunks are fetched under.
Telegram gets no test: the base URL is formatted inline inside
go-pkgz/auth, so nothing here can point it elsewhere, and the fix
belongs upstream in v1 and v2 both. Reported as #2208.
Three things CI found that a laptop cannot. The anonymous sign-in form
validates its input against pattern="[\p{L}\d\s_]+", and E2E_RUN_ID is
"<run id>-<attempt>" on a runner, so every username built from it
carried a hyphen the browser refused to submit: no request was made and
the case waited out its timeout on a panel that was never going to
change. Names are built by anonName now, which drops what the pattern
does not allow and adds the pid, so a second run against a surviving
stack does not meet its own blocked and verified users. signInAnon waits
for the request the submit makes, so the next such refusal fails as
itself.
The admin instance takes its admin from an email address, not a name.
remark42 hashes an anonymous id from the name and the client address
together, to tell apart two people picking the same name, so the id
written into ADMIN_SHARED_ID belonged to nobody on a runner and the
instance had no admin at all: the countdown stayed, and the backend
refused the edit. An email id is sha1 of the address, which is the same
everywhere.
The subscription case clears its own precondition and confirms through
the page's session. The dev user is shared and a subscription outlives
the run, so the panel opened on the subscribed step; and the panel moves
to that step while its token textarea is still on screen, leaving no
moment at which the control to submit it exists.
Three settings of remark_config get cases of their own, none having had
any: __colors__, which is the one setting that travels through
window.name and not the query string, so nothing else in the suite would
notice the path going; the url override, which is how a canonical
address keeps one conversation across pages that differ; and the
subscription controls an integrator turns off, with the both-shown case
as the control.
Writing the url case turned up a backend defect, reported as #2204 and
not fixed here: a thread url containing "&" cannot be commented on at
all. Sanitize runs the locator
through SanitizeAsURL, which round-trips it through bluemonday, so the
url is stored html-escaped; the bucket is created under the escaped key,
the read-back uses the real one and answers 500, and every later find,
count and feed asks for the real url and is told the thread is empty.
Any page addressed with two query parameters is affected. The case uses
a single-parameter url for that reason.
Five more from the same audit, none needing a service. Collapsing a
thread shrinks the frame, which every other geometry case would miss:
they all assert growth, and a widget that only grew would satisfy them
while leaving a hole under each collapsed thread. Voting gains the
direction nothing covered, downvoting and its survival of a reload, and
the rule the other vote cases work around, that your own comment offers
no buttons and the backend refuses the vote anyway.
A vote with the X-XSRF-TOKEN header stripped has to be refused. That
check is why a document navigation, an iframe src among them, is always
anonymous and why the widget hydrates its user over XHR, so anything
designed around that wants it pinned.
An unrecognized locale has to render English, which is loadLocale's only
observable guarantee: it falls back the same way for a name it does not
know and for a chunk it cannot fetch. And a comment's timestamp has to
be the reader's own, which is the one part of rendering that cannot move
to the server, asserted from a context in Kiritimati against the same
Intl the widget uses.
A host page on an origin the widget is not served from, which is the
separate-domain setup the manuals describe and the configuration readers
actually hit problems with. Every other host page here is served by
remark42 itself, so the cross-site path was never taken: an nginx on its
own name and port serves e2e/hostsite, and the case asserts the frame is
revealed, which means its document loaded and reported itself inited
across the origin boundary, and that the thread it renders is the one
the page's address names. Signing in is left out on purpose, an embedded
cookie needing SameSite=None, which browsers take only as Secure, and
this stack speaks http; that is #1139 and not something a case here can
settle.
The other half is ALLOWED_HOSTS. The no-provider instance names only
itself, so a page elsewhere embedding it is refused by the browser, the
document never runs, and the reveal comes from the widget's own fallback
five seconds later. Both directions are worth holding: without the
fallback a mistyped host leaves a permanently invisible widget with
nothing to say why, and without the refusal the setting does nothing.
The host page's title reaching the stored comment gets a case, the path
running the other way from everything else here: the page posts its
title into the widget, the widget sends it with the comment, and it is
what a feed and the admin listing show. Set after the widget is up, so
it covers the observer embed.ts installs and not the value read at boot.
max_shown_comments has no case. The setting reaches the widget, appears
in the iframe's query string and changes nothing: four comments render
with it set to two, which is #812, still open, where the reproduction is
now recorded. A case for it would be red on master.
The README gains what the suite cannot reach. Every service here speaks
http, so anything the browser gates on the page protocol is invisible: a
Secure cookie, anything keyed on window.location.protocol, and the
SameSite=None with Secure and Partitioned form that is the only one an
embedded frame can still use. That is not hypothetical, setAuthCookie
having decorated its cookies with __Host- on any https page and survived
precisely because nothing here runs on one. The cross-origin case names
the assertion to add if the stack ever gets TLS, which is the reload:
the widget holds its token in memory for the life of a page, so signing
in and posting without reloading passes while persistence is broken.
And the trap waiting for whoever acts on that: playwright's own default
--disable-features argument carries ThirdPartyStoragePartitioning, and it
beats both --test-third-party-cookie-phaseout and
--block-third-party-cookies passed through Args, so a run meaning to
prove the third-party case keeps an ordinary third-party cookie exactly
as it would with no flags at all. IgnoreDefaultArgs is the lever, and a
blocking run has to assert a control before anything it reports can be
believed. None of it reaches the widget's own storage fallback either:
IS_STORAGE_AVAILABLE stays true with partitioning enforced, chromium
partitioning localStorage instead of denying it, so comments.html needs
webkit and not a flag.
The downvote case now actually corrects. It claimed the score ends where
the second vote leaves it and never cast one, so #728, a reader taking a
vote back, could break with it green. It also turns out the opposite
vote takes the first one back instead of flipping it, so the score
returns to zero and never reaches +1, which is what the case asserts,
before and after a reload. Renamed for what it covers.
12 KiB
End-to-end tests
Drives the widget in a real browser through playwright-go against a remark42 built from this checkout.
The import path is github.com/mxschmitt/playwright-go, which is what the module declares even though its repository is playwright-community/playwright-go. Do not rewrite it to match the repository URL: the versions that carry the matching path cannot install their driver.
Prerequisites
- Docker with compose, which the suite shells out to
- A Go toolchain matching
e2e/go.mod - Network access on the first run: the Playwright driver and the browsers are downloaded into the user cache directory,
~/.cache/…on Linux and~/Library/Caches/…on macOS, and that download is the slowest part of a cold run
Running
make e2e
The suite brings the compose stack up itself when it does not find one already answering, and tears it down again afterwards. To keep the containers between runs, start them first:
make e2e-up
make e2e
make e2e-down
Run from e2e/; the compose path is relative to it. A single test:
cd e2e && go test -tags=e2e -run TestComment_ReplyNestsUnderItsParent -v ./...
make e2e-ui runs with a visible browser and leaves the stack up. The env vars behind it:
E2E_HEADLESS=falseshows the browser and slows it to 50ms a stepE2E_KEEP=1leaves the containers running afterwards, which only matters when the suite brought them up itselfE2E_DEBUG=1logs every HTTP response of status 400 or aboveE2E_BROWSERS=chromiumnarrows the engines the rendering tests use, which is the quickest way to shorten a local run
Rate-limit responses are logged whether or not E2E_DEBUG is set, because they surface otherwise as unexplained locator timeouts.
The build tag keeps these out of go test ./...; nothing runs without -tags=e2e.
When something fails
A failed test writes a Playwright trace to e2e/traces/, which CI uploads as an artifact. Nothing else writes one, so a run carrying the artifact is a run with a failure to look at. Open one with npx playwright show-trace e2e/traces/<name>.zip.
CI does not retry a failing test. The suite is young enough that a failure is evidence about the suite itself, and a retry is what would hide an intermittent regression. E2E_RUN_ID stamps the threads a run uses with the CI run id, so a thread url in a trace names the run it came from. It does not carry the data across: the stack is disposable, and a local run against the same id gets those urls on an empty database.
Beyond that: docker compose -f compose-e2e-test.yml logs for the server side, and mailpit's web UI on http://127.0.0.1:8025 for anything email.
A failing test also logs whatever the browser wrote to its console, which is where a failed request or a widget-side error shows up. Those are context only. What does fail a test on its own is an uncaught exception in the page and a rate-limit response, both of which otherwise corrupt a run silently: a widget that throws while rendering still satisfies most assertions here, and a refused /auth/status renders as a signed-out reader.
The stack the suite runs against
The suite refuses a running stack that was not brought up from the sources under test. Every checkout builds the image tag the compose file names, so a stack from another worktree, or from this one before an edit, answers on these ports and passes every readiness probe while serving code nobody is looking at.
e2e/stamp.sh digests the content of backend, frontend, Dockerfile and docker-init.sh; compose puts it in the container's environment and the suite reads it back. It digests content and not HEAD, so a commit touching only the suite does not invalidate a stack. make e2e-up stamps the same way, so a stack started by hand is accepted. On a mismatch the failure says to run make e2e-down.
Before pushing, cd e2e && go vet -tags=e2e ./... and golangci-lint run --build-tags=e2e --config ../backend/.golangci.yml. CI runs both, and neither is covered by a plain go vet ./... because of the build tag.
The stack
compose-e2e-test.yml at the repository root runs six services, each bound to the loopback interface since it holds a known secret and an admin shared id:
- remark42 on
:8080, with the dev oauth2 provider on:8084, anonymous and email sign-in - remark42-shortedit on
:8081, withEDIT_TIME=15sand anonymous sign-in only, since the dev oauth2 provider's port is fixed at 8084 and cannot be published twice. It exists so the expired-edit path is observable without holding a test open for the default five minutes - remark42-adminedit on
:8082, withADMIN_EDIT=trueand the same short window, for the unlimited window an admin is supposed to get - remark42-jwtheader on
:8083, withAUTH_SEND_JWT_HEADER=true, where the token arrives in a header instead of a cookie and the frontend has to keep it itself - remark42-noauth on
:8085, with no auth provider at all, which the widget has to say something about, and withALLOWED_HOSTSset to its own address so it doubles as the instance that refuses to be framed elsewhere - remark42-anonvote on
:8086, withANON_VOTEand theVOTES_IPit depends on, since the default configuration turns an anonymous vote down - host-site on
:8090, an nginx servinge2e/hostsite/, which is a page on an origin the widget is not served from. Every other host page here is served by remark42 itself, so without it the separate-domain setup the manuals describe is never exercised.post.htmlembeds the main instance;restricted.htmlembeds the one whoseALLOWED_HOSTSnames only itself, which is the refusal case - mailpit on
:8025, which catches the email-auth verification message and the subscription token for the suite to read back
The main instance enables the notify module (NOTIFY_USERS=email). Without it email_notifications is false in the config, the widget never renders the subscribe control, and the whole subscribe, confirm and unsubscribe flow is unreachable from a browser.
The four remark42 instances beyond the first offer anonymous sign-in only, for the reason remark42-shortedit does: the dev oauth2 provider binds a port fixed at 8084 and cannot be published twice. remark42-adminedit gets its admin from ADMIN_SHARED_ID, since the anonymous provider derives the user id from the name and the id for a chosen name can be written into the compose file ahead of time.
Three settings exist for the tests and not for realism, and each is there for a reason:
REMARK_URLuses a hostname, not127.0.0.1. The dev oauth2 server binds whatever host it reads out ofREMARK_URL(localBindAddrin go-pkgz/auth), and a loopback bind inside a container cannot be published. The browser maps the names back with--host-resolver-rules.UPDATE_LIMIT=100, because the default of 0.5 updates a second rejects any test that posts twice in a row.- The suite paces its own calls to
/auth/, which is limited to two requests a second by a bare literal atbackend/app/rest/api/rest.go:242and not by a setting. SeepauseForAuthLimit.
What this suite cannot reach
Every service here speaks http, and nothing in it holds a certificate. Any behaviour the browser gates on the page protocol is therefore invisible: a cookie the widget writes with Secure, anything keyed on window.location.protocol, and the whole third-party cookie form of SameSite=None; Secure; Partitioned, which is the only one browsers still accept from an embedded frame.
That is not hypothetical. setAuthCookie prefixed its cookies with __Host- on any https page, so a real deployment stored __Host-JWT while the backend looked for JWT; it survived because the prefix comes from the page protocol and every test and the dev server run on http. Fixed in #2197, under a second suite pinned to an https page, because this one cannot show it.
There is a second trap waiting for whoever gives the stack TLS and then tries to prove the third-party case. Playwright's own default --disable-features argument carries ThirdPartyStoragePartitioning, and it beats both --test-third-party-cookie-phaseout and --block-third-party-cookies passed through Args. A run configured that way keeps an ordinary third-party cookie exactly as it would with no flags at all, so it proves nothing while looking like it proved something. The lever is IgnoreDefaultArgs on the launch options: drop that default entry and re-supply --disable-features without that one feature. Measured on a cross-site https embed:
| ordinary third-party cookie | Partitioned cookie |
|
|---|---|---|
| Playwright defaults | kept | kept |
| partitioning left enabled | dropped | stored, with its partition key |
So a blocking run has to assert a control before anything it reports can be believed: set an ordinary SameSite=None cookie from inside the widget frame and require the browser to drop it. If it survives, the run is not blocking anything.
None of that reaches the widget's own storage fallback, which the auth panel offers as comments.html when IS_THIRD_PARTY && !IS_STORAGE_AVAILABLE. IS_STORAGE_AVAILABLE stays true even with partitioning properly enforced, because Chromium partitions localStorage instead of denying it, so the probe behind that constant never throws and the condition cannot fire. That case needs WebKit, not a Chromium flag.
The practical consequence is for the cross-origin case in crossorigin_test.go, which asserts rendering and deliberately not signing in. Give the stack TLS and signing in there becomes testable, and the assertion that matters is the reload: the widget holds its token in memory for the life of a page, so a case that signs in and posts without reloading passes while persistence is entirely broken.
Isolation
Each test gets its own comment thread from a query string on the demo page, since the demo page passes window.location.href as remark_config.url and remark42 keys comments by it. A per-run id keeps threads apart from those an earlier run left behind.
Thread URLs deliberately keep the underscores a test name carries, since collapse persistence keys off the page url and a url containing an underscore is the case worth covering.
Browsers
The iframe_test.go group runs in Chromium, Firefox and WebKit. It is about rendering and not logic: the widget holds the frame hidden until its document reports itself inited, and the opaque canvas that guards against is a WebKit behaviour, so Chromium alone would not exercise it.
Those tests address the server as 127.0.0.1 and not by name, since --host-resolver-rules is a Chromium flag and they need no dev oauth2, which is the only reason the hostname exists.
Everything else runs in Chromium alone, for the same reason inverted: those tests sign in, sign-in needs the dev oauth2 provider, and reaching it by name from the host is Chromium-only. Running them in the other engines would mean putting the suite back inside the compose network.
Selectors
The production bundle strips data-testid, so tests use what ships: the stable class hooks the widget keeps outside CSS modules (.auth-button, .auth-submit, .comment-actions, .sort-picker, .preloader), title attributes on icon-only controls, and visible text. Three shapes are worth knowing:
.authonly exists while signed out, so waiting on it hangs after sign-in.widget()waits on the comment form, which is present either way.- The production build hashes every css-module class name to a short opaque id, so a component's own class is not something a test can hold.
roleis: the footer is[role="contentinfo"]and the edit countdown is[role="timer"]. - Comments render through an IntersectionObserver, so one below the fold is an empty
articlewith no text in it. That makes any absence assertion written as a text filter pass whether the comment is gone or merely off screen; count articles instead, which is whatarticleCountis for. - Collapsing a thread hides the comment text, so a locator filtered by that text stops matching the element under test.
TestThread_CollapsePersistsAcrossReloadanchors on the comment's id instead.