* Pin the published /web surface in the e2e suite
#2178 renamed the widget bundles from .js to .mjs and the URLs earlier
releases served under those names stopped resolving. Three were noticed
from the demo site; the rest, including every locale chunk, were found
only by requesting the whole surface of both images over HTTP. #2192
restored them with a server-side alias, and nothing in the suite would
have caught the break or would notice it returning.
Two cases with deliberately different criteria. The documented names are
written out, because the documentation decides that list and not the
build: an operator pastes privacy.html into an OAuth application, the
nginx manual proxies index.html by name, and the integration guides start
from the embed script. Everything else is taken from the build itself, so
whatever the bundler emitted has to serve identical bytes under its
legacy .js name and parse as a classic script, which is the premise
serving one under the other rests on. A third case requests a name that
does not exist, without which a fallback serving one page for everything
would keep the whole table green.
All of them check the content type as well as the bytes: nosniff is set
on every response, so a bundle served as text/plain is as broken as one
that 404s while comparing equal.
On e3d1d0e2, the commit before the alias, this fails with 32 red subtests.
* Cover the widget behavior the e2e suite never drove
Removing npm from the widget takes the jest tests with it, and everything
here was protected by jest or by nothing at all.
Signing out was untested at every level, and the panel repainting is the
half that always works: the assertion after the reload is the one that
catches a session the server never ended. The edit form has to hand back
the source that was posted and not the rendered comment, which #2040
shipped the other way round, taking every entity and tag the author had
written with it. A draft has to survive a reload and be gone once the
comment is posted. A refused comment has to stay in the form with
something said about it, so that case drives the backend's own
restricted-words code and then retries with the route removed.
Uploads had no browser coverage in either direction. The posted case
asserts naturalWidth instead of visibility, since a broken src still
renders as an empty box, and the failing case holds the intercepted
request long enough for the in-flight state to be observed: without that,
"the text is unchanged afterwards" would hold for an upload that never
started.
Moderation and reader-side hiding are asserted from a page other than the
one that made the change. Hiding seeds a second author, so it proves one
person is hidden and the thread not emptied, and both the blocked
author and the moderated one carry the run id in their names: a block and
a verification are properties of the user and outlive the run in the
stack's database, so a fixed name works exactly once.
Locales were the largest hole. Each catalog is a chunk fetched at
runtime, and loadLocale falls back to english on any failure instead of
throwing, exactly as an unrecognized name does, so every case compares
the rendered string against the file on disk. One case per catalog
covers that they are all served and render; another fetches one through
the widget document and asserts it parsed, which is the half a chunk that
serves but fails to parse would slip through. The delete page and the
last-comments stylesheet had no coverage of any kind.
Two things the suite itself needed. The widget's own aria-label is
translated, so commentFormSel only ever finds an english widget and a
localized case cannot use widget(). And the auth probe is capped at two
requests a second for the whole suite, hard-coded in rest.go: the added
cases pushed past it and the suite began manufacturing its own 429s,
which render as a signed-out widget and fail whichever test happens to be
signing in, so the pacing gap is wider and the locale cases stub the
probe they never needed.
* Harden the e2e harness against silent failures and stale stacks
Three things the suite could not tell you about itself.
A browser failure nothing asserts on now fails the test that caused it.
Uncaught exceptions are the ones worth the machinery: a widget throwing
while it renders leaves most of these cases green, since they assert on
elements the browser lays out either way. Rate-limit responses are
recorded as well as logged for the same reason. A test driving an error
path declares what it expects by substring, so a case that means to
break something says which thing.
The stack the suite adopts is now checked against the sources under
test. Every checkout builds the image tag the compose file names, so a
stack from another worktree, or from this one before an edit, answers on
these ports and passes every readiness probe while serving code nobody
is looking at. stamp.sh digests what goes into the image, compose passes
it as the revision label, and a mismatch is refused with what to do
about it. Checked after our own build too, or a stamp that never reaches
the image would be a guard that silently passes everything.
assertSignedIn no longer waits out the whole timeout on a refused status
read. /auth/ is capped at two requests a second for the entire suite, a
bare literal at rest.go:242, and a case signing in on two pages spends
that twice; when the read that repaints the panel is the one the limiter
turns down, the widget shows signed out over a session that exists and
no later request will ask again. The short first wait now ends in a
focus handoff, which the widget answers by re-probing, and only then
does the real wait run. A sign-in that genuinely failed still fails,
since the second read finds no state either. That is what
TestComment_AdminPinsAndVerifies and TestComment_BlockedAuthorCannotPost
were failing on in CI while passing locally.
signInAnon takes the page for that reason, and its callers pass it.
* Cover iframe geometry, the embed contract and five deployment modes
Seventeen cases for the parts of the widget that broke repeatedly and
that nothing here could see, plus the areas jest was the only check on.
Geometry is the biggest of them. The widget measures its own document
and posts the number for the parent to apply, and every way that has
gone wrong is invisible to assertions about elements, which read the
same whether the frame is right, twice too tall or a strip. So: the
first height the parent is given describes rendered content and not the
preloader, the frame matches the document it holds and does not stand
24px taller, no_footer leaves the last comment inside the frame, and the
frame follows the sign-in dropdown and the growing textarea and comes
back down again. Each was verified by reintroducing the defect it covers
and watching it fail: a 63px report before the real one, six pixels of
body padding, and a frame sized under the content.
The embed surface is the other half the widget cannot see. An element
placeholder gives way to exactly one iframe carrying the embed's own
marker, a second createInstance reuses it, destroy takes it away with
its handles, and a theme change after load reaches both the element and
the document. A page that posts a message of its own, which is all
embed.ts's own title observer does, no longer empties an open login
form.
Five configurations that cannot share an instance get one each, since
each changes the widget for every reader: an admin's unlimited edit
window, a session carried in a header and not a cookie, an instance with
no auth provider to offer, anonymous voting, and the notify module,
without which email_notifications is false and the subscribe control
never renders at all. The subscription round trip is the one place a
token from a real message is exchanged for state the server keeps.
Two things surfaced there and are left alone, both said so in place. The
panel confirming an unsubscribe cannot be observed, because the click
changes the step and the dropdown closes on an element no longer in the
rerendered view, which the component notes as its own awkwardness; the
case asserts the request and the answer, which is what decides whether
the reader still gets mail. And the widget takes the subscribed state
from user.email_subscription, absent from what it hydrates the user with
on the next load, so after a reload it offers to subscribe somebody who
already is.
simple_view needs no instance, being a query parameter, so both branches
run against the main one. A transient failure of the status probe has a
case too: the session belongs to the server, and one refused answer must
not end it.
The suite runs about four and a half minutes now, so the workflow's own
budget goes to 20m to match the Makefile, well inside the job timeout,
and the workflow stamps the stack it starts the way the Makefile does.
The locale case that loads the widget document directly is renamed for
what it protects, the origin and CSP its chunks are fetched under.
Telegram gets no test: the base URL is formatted inline inside
go-pkgz/auth, so nothing here can point it elsewhere, and the fix
belongs upstream in v1 and v2 both. Reported as #2208.
Three things CI found that a laptop cannot. The anonymous sign-in form
validates its input against pattern="[\p{L}\d\s_]+", and E2E_RUN_ID is
"<run id>-<attempt>" on a runner, so every username built from it
carried a hyphen the browser refused to submit: no request was made and
the case waited out its timeout on a panel that was never going to
change. Names are built by anonName now, which drops what the pattern
does not allow and adds the pid, so a second run against a surviving
stack does not meet its own blocked and verified users. signInAnon waits
for the request the submit makes, so the next such refusal fails as
itself.
The admin instance takes its admin from an email address, not a name.
remark42 hashes an anonymous id from the name and the client address
together, to tell apart two people picking the same name, so the id
written into ADMIN_SHARED_ID belonged to nobody on a runner and the
instance had no admin at all: the countdown stayed, and the backend
refused the edit. An email id is sha1 of the address, which is the same
everywhere.
The subscription case clears its own precondition and confirms through
the page's session. The dev user is shared and a subscription outlives
the run, so the panel opened on the subscribed step; and the panel moves
to that step while its token textarea is still on screen, leaving no
moment at which the control to submit it exists.
Three settings of remark_config get cases of their own, none having had
any: __colors__, which is the one setting that travels through
window.name and not the query string, so nothing else in the suite would
notice the path going; the url override, which is how a canonical
address keeps one conversation across pages that differ; and the
subscription controls an integrator turns off, with the both-shown case
as the control.
Writing the url case turned up a backend defect, reported as #2204 and
not fixed here: a thread url containing "&" cannot be commented on at
all. Sanitize runs the locator
through SanitizeAsURL, which round-trips it through bluemonday, so the
url is stored html-escaped; the bucket is created under the escaped key,
the read-back uses the real one and answers 500, and every later find,
count and feed asks for the real url and is told the thread is empty.
Any page addressed with two query parameters is affected. The case uses
a single-parameter url for that reason.
Five more from the same audit, none needing a service. Collapsing a
thread shrinks the frame, which every other geometry case would miss:
they all assert growth, and a widget that only grew would satisfy them
while leaving a hole under each collapsed thread. Voting gains the
direction nothing covered, downvoting and its survival of a reload, and
the rule the other vote cases work around, that your own comment offers
no buttons and the backend refuses the vote anyway.
A vote with the X-XSRF-TOKEN header stripped has to be refused. That
check is why a document navigation, an iframe src among them, is always
anonymous and why the widget hydrates its user over XHR, so anything
designed around that wants it pinned.
An unrecognized locale has to render English, which is loadLocale's only
observable guarantee: it falls back the same way for a name it does not
know and for a chunk it cannot fetch. And a comment's timestamp has to
be the reader's own, which is the one part of rendering that cannot move
to the server, asserted from a context in Kiritimati against the same
Intl the widget uses.
A host page on an origin the widget is not served from, which is the
separate-domain setup the manuals describe and the configuration readers
actually hit problems with. Every other host page here is served by
remark42 itself, so the cross-site path was never taken: an nginx on its
own name and port serves e2e/hostsite, and the case asserts the frame is
revealed, which means its document loaded and reported itself inited
across the origin boundary, and that the thread it renders is the one
the page's address names. Signing in is left out on purpose, an embedded
cookie needing SameSite=None, which browsers take only as Secure, and
this stack speaks http; that is #1139 and not something a case here can
settle.
The other half is ALLOWED_HOSTS. The no-provider instance names only
itself, so a page elsewhere embedding it is refused by the browser, the
document never runs, and the reveal comes from the widget's own fallback
five seconds later. Both directions are worth holding: without the
fallback a mistyped host leaves a permanently invisible widget with
nothing to say why, and without the refusal the setting does nothing.
The host page's title reaching the stored comment gets a case, the path
running the other way from everything else here: the page posts its
title into the widget, the widget sends it with the comment, and it is
what a feed and the admin listing show. Set after the widget is up, so
it covers the observer embed.ts installs and not the value read at boot.
max_shown_comments has no case. The setting reaches the widget, appears
in the iframe's query string and changes nothing: four comments render
with it set to two, which is #812, still open, where the reproduction is
now recorded. A case for it would be red on master.
The README gains what the suite cannot reach. Every service here speaks
http, so anything the browser gates on the page protocol is invisible: a
Secure cookie, anything keyed on window.location.protocol, and the
SameSite=None with Secure and Partitioned form that is the only one an
embedded frame can still use. That is not hypothetical, setAuthCookie
having decorated its cookies with __Host- on any https page and survived
precisely because nothing here runs on one. The cross-origin case names
the assertion to add if the stack ever gets TLS, which is the reload:
the widget holds its token in memory for the life of a page, so signing
in and posting without reloading passes while persistence is broken.
And the trap waiting for whoever acts on that: playwright's own default
--disable-features argument carries ThirdPartyStoragePartitioning, and it
beats both --test-third-party-cookie-phaseout and
--block-third-party-cookies passed through Args, so a run meaning to
prove the third-party case keeps an ordinary third-party cookie exactly
as it would with no flags at all. IgnoreDefaultArgs is the lever, and a
blocking run has to assert a control before anything it reports can be
believed. None of it reaches the widget's own storage fallback either:
IS_STORAGE_AVAILABLE stays true with partitioning enforced, chromium
partitioning localStorage instead of denying it, so comments.html needs
webkit and not a flag.
The downvote case now actually corrects. It claimed the score ends where
the second vote leaves it and never cast one, so #728, a reader taking a
vote back, could break with it green. It also turns out the opposite
vote takes the first one back instead of flipping it, so the score
returns to zero and never reaches +1, which is what the case asserts,
before and after a reload. Renamed for what it covers.
779 lines
29 KiB
Go
779 lines
29 KiB
Go
//go:build e2e
|
|
|
|
// Package e2e drives the remark42 widget in a real browser through playwright.
|
|
//
|
|
// The suite talks to the stack in compose-e2e-test.yml at the repository root and brings it
|
|
// up itself when nothing is listening. Files:
|
|
//
|
|
// - e2e_test.go: TestMain, shared helpers, constants
|
|
// - harness_test.go: the stale-stack guard and the browser failures no assertion covers
|
|
// - auth_test.go: dev, anonymous and email sign-in
|
|
// - comment_test.go: post, reply, edit, delete
|
|
// - vote_test.go: voting and its failure path
|
|
// - thread_test.go: sorting and collapse persistence
|
|
// - iframe_test.go: the iframe's color scheme and reveal handshake
|
|
// - geometry_test.go: the height the widget reports and the parent applies
|
|
// - embed_test.go: the surface the host page holds, placeholder to destroy
|
|
// - config_test.go: the remark_config surface an integrator sets
|
|
// - crossorigin_test.go: a host page on an origin the widget is not served from
|
|
// - deployment_test.go: the instances whose configuration is the thing under test
|
|
// - subscribe_test.go: the email subscription round trip
|
|
// - webfiles_test.go: the published /web surface
|
|
// - widgets_test.go: last-comments, counter and the profile iframe
|
|
package e2e
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"log"
|
|
"net/http"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
"unicode"
|
|
|
|
"github.com/mxschmitt/playwright-go"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
const (
|
|
// what the browser asks for. these have to be names, not 127.0.0.1: the dev oauth2
|
|
// server binds whatever host it reads out of REMARK_URL, and a loopback bind inside the
|
|
// container cannot be published, so the browser resolves the names back with
|
|
// --host-resolver-rules and reaches the published ports
|
|
baseURL = "http://remark42:8080"
|
|
shortEditURL = "http://remark42-shortedit:8081"
|
|
// the deployment modes that cannot be a setting on an instance shared with anything else,
|
|
// each covering a regression the default configuration cannot reach. see compose-e2e-test.yml
|
|
adminEditURL = "http://remark42-adminedit:8082"
|
|
jwtHeaderURL = "http://remark42-jwtheader:8083"
|
|
noAuthURL = "http://remark42-noauth:8085"
|
|
anonVoteURL = "http://remark42-anonvote:8086"
|
|
|
|
// a page on an origin the widget is not served from, see compose-e2e-test.yml
|
|
hostSiteURL = "http://host-site:8090"
|
|
|
|
// what this process asks for, since it is not the browser and has no resolver rules
|
|
probeURL = "http://127.0.0.1:8080"
|
|
shortEditProbeURL = "http://127.0.0.1:8081"
|
|
adminEditProbeURL = "http://127.0.0.1:8082"
|
|
jwtHeaderProbeURL = "http://127.0.0.1:8083"
|
|
noAuthProbeURL = "http://127.0.0.1:8085"
|
|
anonVoteProbeURL = "http://127.0.0.1:8086"
|
|
hostSiteProbeURL = "http://127.0.0.1:8090"
|
|
mailpitURL = "http://127.0.0.1:8025"
|
|
|
|
composeFile = "../compose-e2e-test.yml"
|
|
|
|
// every comment form carries this label, whatever mode it is in
|
|
commentFormSel = `form[aria-label="New comment"]`
|
|
|
|
// traces of failed tests land here; CI uploads the directory
|
|
traceDir = "traces"
|
|
|
|
// generous because CI runners are slower and less predictable than a laptop
|
|
waitTimeout = 15 * time.Second
|
|
|
|
// how long assertSignedIn waits before nudging the widget into re-reading auth state. short
|
|
// enough that a refused status read costs a fraction of a test instead of a whole timeout,
|
|
// long enough that the ordinary path never takes the nudge
|
|
authRepaintWait = 3 * time.Second
|
|
|
|
// what a single locator call inside a poll body gets. playwright's own default is 30s,
|
|
// twice the budget of the loops here, so without this a missing element would block one
|
|
// attempt for longer than the whole loop and then report the wrong thing
|
|
pollTimeout = time.Second
|
|
)
|
|
|
|
var (
|
|
pw *playwright.Playwright
|
|
browser playwright.Browser
|
|
startedStack bool
|
|
|
|
// engines beyond chromium, launched on demand for the rendering tests
|
|
extraBrowsers = map[string]playwright.Browser{}
|
|
extraBrowsersMu sync.Mutex
|
|
|
|
// distinguishes this run's threads from those a previous run left in the database.
|
|
// E2E_RUN_ID pins it, so the urls a CI run works on name that run and not the moment
|
|
// the process started
|
|
runID = firstNonEmpty(os.Getenv("E2E_RUN_ID"), fmt.Sprintf("%d", time.Now().UnixNano()))
|
|
|
|
authGate sync.Mutex
|
|
lastAuthCall time.Time
|
|
|
|
contextSeq atomic.Int64
|
|
|
|
// the default client has no timeout, so a port that accepts and then stalls would block
|
|
// a probe well past its own deadline and leave TestMain looking hung
|
|
probeClient = &http.Client{Timeout: 5 * time.Second}
|
|
)
|
|
|
|
// everything under /auth/ is rate limited to 2 requests a second, and that figure is a bare
|
|
// literal at backend/app/rest/api/rest.go:242 and not a setting, so the suite has to pace
|
|
// itself: the widget calls /auth/status on every load, and again on visibilitychange or
|
|
// window focus while an oauth popup sign-in is pending. without this the limiter starts
|
|
// answering 429 and the widget renders as signed out
|
|
func pauseForAuthLimit() {
|
|
// 1200ms, up from the 700 this started at: the widget fires an unpaced /auth/status on
|
|
// every load and again on visibilitychange, so the paced side has to stay well under the
|
|
// 2/s cap to leave room for them. at 700 the suite manufactured its own 429s, and a lost
|
|
// probe renders as signed out, which fails whichever test happens to be signing in
|
|
const spacing = 1200 * time.Millisecond
|
|
|
|
authGate.Lock()
|
|
defer authGate.Unlock()
|
|
if wait := spacing - time.Since(lastAuthCall); wait > 0 {
|
|
time.Sleep(wait)
|
|
}
|
|
lastAuthCall = time.Now()
|
|
}
|
|
|
|
func TestMain(m *testing.M) {
|
|
if err := ensureStack(); err != nil {
|
|
log.Printf("[ERROR] stack not ready: %v", err)
|
|
teardown(1)
|
|
}
|
|
|
|
if err := playwright.Install(installOpts("chromium")); err != nil {
|
|
log.Printf("[ERROR] failed to install playwright: %v", err)
|
|
teardown(1)
|
|
}
|
|
|
|
var err error
|
|
if pw, err = playwright.Run(); err != nil {
|
|
log.Printf("[ERROR] failed to start playwright: %v", err)
|
|
teardown(1)
|
|
}
|
|
|
|
headless := os.Getenv("E2E_HEADLESS") != "false"
|
|
var slowMo float64
|
|
if !headless {
|
|
slowMo = 50 // slow the visible browser down enough to follow
|
|
}
|
|
browser, err = pw.Chromium.Launch(playwright.BrowserTypeLaunchOptions{
|
|
Headless: playwright.Bool(headless),
|
|
SlowMo: playwright.Float(slowMo),
|
|
Args: []string{
|
|
"--host-resolver-rules=MAP remark42 127.0.0.1, MAP remark42-shortedit 127.0.0.1, " +
|
|
"MAP remark42-adminedit 127.0.0.1, MAP remark42-jwtheader 127.0.0.1, " +
|
|
"MAP remark42-noauth 127.0.0.1, MAP remark42-anonvote 127.0.0.1, " +
|
|
"MAP host-site 127.0.0.1",
|
|
},
|
|
})
|
|
if err != nil {
|
|
_ = pw.Stop()
|
|
log.Printf("[ERROR] failed to launch browser: %v", err)
|
|
teardown(1)
|
|
}
|
|
|
|
code := m.Run()
|
|
|
|
for _, b := range extraBrowsers {
|
|
_ = b.Close()
|
|
}
|
|
_ = browser.Close()
|
|
_ = pw.Stop()
|
|
teardown(code)
|
|
}
|
|
|
|
// ensureStack waits for a running stack and starts one with compose when there is none
|
|
func ensureStack() error {
|
|
// the digest of the sources the image is built from, exported so compose stamps the build
|
|
// with it. read back off the running stack below, since answering on the right ports says
|
|
// nothing about which checkout the code came from
|
|
stamp, err := sourceStamp()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if err := os.Setenv(stampEnv, stamp); err != nil {
|
|
return fmt.Errorf("exporting %s: %w", stampEnv, err)
|
|
}
|
|
|
|
// every service, not just the first: compose-dev-backend.yml and `make rundev` also
|
|
// publish 8080, and adopting one of those would run the suite against a developer's own
|
|
// database and fail later as unexplained locator timeouts
|
|
if stackReady(2 * time.Second) {
|
|
return assertStackMatches(stamp, true)
|
|
}
|
|
|
|
log.Printf("[INFO] no complete stack on 127.0.0.1, bringing one up from %s", composeFile)
|
|
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
|
|
defer cancel()
|
|
|
|
// set before running, not after: a failed `up` leaves containers behind, and teardown
|
|
// has to know to remove them
|
|
startedStack = true
|
|
|
|
// --build matters: the image tag this compose file uses is the same one the dev compose
|
|
// files produce, so without it the suite can quietly test an image from another checkout
|
|
cmd := exec.CommandContext(ctx, "docker", "compose", "-f", composeFile,
|
|
"up", "-d", "--build", "--quiet-pull", "--wait")
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
return fmt.Errorf("docker compose up: %w\n%s", err, out)
|
|
}
|
|
if !stackReady(waitTimeout) {
|
|
return fmt.Errorf("compose reported the stack healthy but it does not answer")
|
|
}
|
|
// checked after our own build too, so a stamp that never reaches the image is a failure
|
|
// here, and not a guard that silently passes everything for the rest of its life
|
|
return assertStackMatches(stamp, false)
|
|
}
|
|
|
|
func firstNonEmpty(values ...string) string {
|
|
for _, v := range values {
|
|
if v != "" {
|
|
return v
|
|
}
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// stackReady reports whether every service the suite needs answers
|
|
func stackReady(timeout time.Duration) bool {
|
|
for _, url := range []string{
|
|
probeURL + "/ping",
|
|
shortEditProbeURL + "/ping",
|
|
adminEditProbeURL + "/ping",
|
|
jwtHeaderProbeURL + "/ping",
|
|
noAuthProbeURL + "/ping",
|
|
anonVoteProbeURL + "/ping",
|
|
hostSiteProbeURL + "/post.html",
|
|
mailpitURL + "/api/v1/messages",
|
|
} {
|
|
if err := serverReady(url, timeout); err != nil {
|
|
return false
|
|
}
|
|
}
|
|
return true
|
|
}
|
|
|
|
func teardown(code int) {
|
|
if startedStack && os.Getenv("E2E_KEEP") == "" {
|
|
composeDown()
|
|
}
|
|
os.Exit(code)
|
|
}
|
|
|
|
// composeDown is separate so its context is closed before teardown calls os.Exit. it is
|
|
// bounded because it runs after m.Run, where `go test -timeout` can no longer rescue a
|
|
// wedged daemon and the binary would otherwise hang until the CI job times out
|
|
func composeDown() {
|
|
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
|
defer cancel()
|
|
|
|
cmd := exec.CommandContext(ctx, "docker", "compose", "-f", composeFile, "down", "-v")
|
|
if out, err := cmd.CombinedOutput(); err != nil {
|
|
log.Printf("[WARN] docker compose down: %v\n%s", err, out)
|
|
}
|
|
}
|
|
|
|
func serverReady(url string, timeout time.Duration) error {
|
|
deadline := time.Now().Add(timeout)
|
|
for time.Now().Before(deadline) {
|
|
resp, err := probeClient.Get(url)
|
|
if err == nil {
|
|
_ = resp.Body.Close()
|
|
if resp.StatusCode == http.StatusOK {
|
|
return nil
|
|
}
|
|
}
|
|
time.Sleep(100 * time.Millisecond)
|
|
}
|
|
return fmt.Errorf("%s not ready after %v", url, timeout)
|
|
}
|
|
|
|
// newPage opens an isolated context, so cookies and storage never leak between tests. the
|
|
// context records a trace, kept only when the test fails, since a CI-only failure otherwise
|
|
// leaves nothing to look at from the browser side
|
|
func newPage(t *testing.T) playwright.Page {
|
|
t.Helper()
|
|
return newPageOn(t, browser)
|
|
}
|
|
|
|
func newPageOn(t *testing.T, b playwright.Browser) playwright.Page {
|
|
t.Helper()
|
|
return newPageInContext(t, b, playwright.BrowserNewContextOptions{})
|
|
}
|
|
|
|
// newPageInContext is newPageOn with the context configured, for the cases where what the browser
|
|
// says about itself is the thing under test
|
|
func newPageInContext(t *testing.T, b playwright.Browser, opts playwright.BrowserNewContextOptions) playwright.Page {
|
|
t.Helper()
|
|
ctx, err := b.NewContext(opts)
|
|
require.NoError(t, err)
|
|
|
|
// the reveal timers start when the iframe element is created, so the tests that bound
|
|
// them have to measure from there and not from anything this process can time
|
|
require.NoError(t, ctx.AddInitScript(playwright.Script{Content: playwright.String(iframeMarkScript)}))
|
|
|
|
tracing := ctx.Tracing().Start(playwright.TracingStartOptions{
|
|
Screenshots: playwright.Bool(true),
|
|
Snapshots: playwright.Bool(true),
|
|
}) == nil
|
|
// a test that opens two contexts would otherwise have them write the same file, and
|
|
// cleanup runs last-in-first-out, so the surviving trace would be of the page that was
|
|
// only setting the scenario up
|
|
seq := contextSeq.Add(1)
|
|
t.Cleanup(func() {
|
|
if tracing {
|
|
if !t.Failed() {
|
|
_ = ctx.Tracing().Stop()
|
|
_ = ctx.Close()
|
|
return
|
|
}
|
|
// say so instead of swallowing it: this runs only on a test that already failed,
|
|
// and a silently missing trace is what the reader goes looking for
|
|
// the pid keeps two processes sharing a run id, and so a trace directory, from
|
|
// overwriting each other: the counter restarts with every process
|
|
name := strings.ReplaceAll(t.Name(), "/", "-")
|
|
path := filepath.Join(traceDir, fmt.Sprintf("%s-%d-%d.zip", name, os.Getpid(), seq))
|
|
// the driver creates the parent of the trace path too, but nothing here states or
|
|
// tests that, so the suite makes the directory itself
|
|
if derr := os.MkdirAll(traceDir, 0o750); derr != nil {
|
|
t.Logf("could not create %s: %v", traceDir, derr)
|
|
}
|
|
if serr := ctx.Tracing().Stop(path); serr != nil {
|
|
t.Logf("could not write the trace to %s: %v", path, serr)
|
|
}
|
|
}
|
|
_ = ctx.Close()
|
|
})
|
|
|
|
page, err := ctx.NewPage()
|
|
require.NoError(t, err)
|
|
// before the first navigation, so nothing the page reports on its way up is missed
|
|
watchPage(t, page)
|
|
debug := os.Getenv("E2E_DEBUG") != ""
|
|
page.OnResponse(func(r playwright.Response) {
|
|
switch {
|
|
case r.Status() == http.StatusTooManyRequests:
|
|
// the rate limiter answers the widget, not the test, so without this the failure
|
|
// arrives as an unexplained locator timeout. recorded as well as logged: a lost
|
|
// /auth/status renders as signed out, and every sign-in assertion then fails
|
|
// somewhere that does not name the cause
|
|
recordPageIssue(page, "rate limited", r.URL())
|
|
log.Printf("[WARN] rate limited: %s", r.URL())
|
|
case debug && r.Status() >= 400:
|
|
body, _ := r.Text()
|
|
log.Printf("[DEBUG] HTTP %d %s %s", r.Status(), r.URL(), body)
|
|
}
|
|
})
|
|
return page
|
|
}
|
|
|
|
// installOpts asks for the browser system libraries on CI only: install-deps shells out to
|
|
// apt with sudo, which is right for a runner and wrong for someone's laptop
|
|
func installOpts(browsers ...string) *playwright.RunOptions {
|
|
return &playwright.RunOptions{
|
|
Browsers: browsers,
|
|
WithDeps: os.Getenv("CI") != "",
|
|
}
|
|
}
|
|
|
|
// engines lists the browsers the rendering tests run in. hiding the frame until its
|
|
// document reports itself inited guards against a flash that was a WebKit one, so chromium
|
|
// alone would test the engine that never had the bug. E2E_BROWSERS narrows the set
|
|
func engines() []string {
|
|
if v := os.Getenv("E2E_BROWSERS"); v != "" {
|
|
return strings.Split(v, ",")
|
|
}
|
|
return []string{"chromium", "firefox", "webkit"}
|
|
}
|
|
|
|
// browserFor launches an engine once and keeps it for the rest of the run
|
|
func browserFor(t *testing.T, name string) playwright.Browser {
|
|
t.Helper()
|
|
if name == "chromium" {
|
|
return browser
|
|
}
|
|
|
|
extraBrowsersMu.Lock()
|
|
defer extraBrowsersMu.Unlock()
|
|
if b, ok := extraBrowsers[name]; ok {
|
|
return b
|
|
}
|
|
|
|
require.NoError(t, playwright.Install(installOpts(name)))
|
|
|
|
var bt playwright.BrowserType
|
|
switch name {
|
|
case "firefox":
|
|
bt = pw.Firefox
|
|
case "webkit":
|
|
bt = pw.WebKit
|
|
default:
|
|
t.Fatalf("unknown browser %q", name)
|
|
}
|
|
|
|
b, err := bt.Launch(playwright.BrowserTypeLaunchOptions{
|
|
Headless: playwright.Bool(os.Getenv("E2E_HEADLESS") != "false"),
|
|
})
|
|
require.NoError(t, err)
|
|
extraBrowsers[name] = b
|
|
return b
|
|
}
|
|
|
|
// renderURL is the thread url for the rendering tests. it uses the address directly rather
|
|
// than the mapped hostname, because --host-resolver-rules is a chromium flag and these tests
|
|
// need no dev oauth2, which is the only reason the hostname exists
|
|
func renderURL(t *testing.T) string {
|
|
t.Helper()
|
|
return threadURLOn(t, probeURL)
|
|
}
|
|
|
|
// threadURL gives each test its own comment thread. the demo page passes
|
|
// window.location.href as remark_config.url, and remark42 keys comments by it, so a unique
|
|
// query string isolates a test without resetting the database between runs
|
|
func threadURL(t *testing.T) string {
|
|
t.Helper()
|
|
return threadURLOn(t, baseURL)
|
|
}
|
|
|
|
// threadURLOn is threadURL against a given instance, for the tests that do not use the
|
|
// default one. every thread url goes through here so the rewriting below cannot be missed
|
|
func threadURLOn(t *testing.T, base string) string {
|
|
t.Helper()
|
|
// underscores are left in on purpose: collapse persistence keys off the page url, and a
|
|
// url carrying the separator its storage used to join on is the case that broke it. only
|
|
// the "/" of a subtest name is dropped, having no business in a query value
|
|
name := strings.ReplaceAll(strings.ToLower(t.Name()), "/", "-")
|
|
return fmt.Sprintf("%s/web/?e2e=%s-%s", base, name, runID)
|
|
}
|
|
|
|
// anonName builds a username no earlier run has used. Blocking a user and verifying one are
|
|
// properties of the user, and the stack's database outlives the run, so a name that comes back
|
|
// arrives already blocked or already verified: the case then toggles the state off and waits for
|
|
// something that is being taken away.
|
|
//
|
|
// The run id alone is not enough, since E2E_RUN_ID pins it and a second run against a surviving
|
|
// stack repeats it, which is exactly what `make e2e-up` invites. The pid distinguishes those
|
|
// while leaving CI, where every job is one process, with the run id it wants in the name.
|
|
//
|
|
// Everything outside [\p{L}\d\s_] is dropped, because the form validates its input against that
|
|
// pattern (auth.tsx:332) and the browser refuses to submit anything else: no request is made, and
|
|
// the test waits out its timeout on a panel that was never going to change. E2E_RUN_ID is
|
|
// "<run id>-<attempt>" on CI, whose hyphen is exactly such a character
|
|
func anonName(prefix string) string {
|
|
var b strings.Builder
|
|
b.WriteString(prefix)
|
|
for _, r := range runID {
|
|
if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '_' {
|
|
b.WriteRune(r)
|
|
}
|
|
}
|
|
fmt.Fprintf(&b, "_%d", os.Getpid())
|
|
return b.String()
|
|
}
|
|
|
|
// openThread loads the demo page on this test's own thread and waits for the widget
|
|
func openThread(t *testing.T, page playwright.Page) playwright.FrameLocator {
|
|
t.Helper()
|
|
return openURL(t, page, threadURL(t))
|
|
}
|
|
|
|
func openURL(t *testing.T, page playwright.Page, url string) playwright.FrameLocator {
|
|
t.Helper()
|
|
pauseForAuthLimit()
|
|
_, err := page.Goto(url, playwright.PageGotoOptions{
|
|
WaitUntil: playwright.WaitUntilStateDomcontentloaded,
|
|
})
|
|
require.NoError(t, err)
|
|
return widget(t, page)
|
|
}
|
|
|
|
// embedConfig loads a page carrying a remark_config of the test's choosing and runs the embed
|
|
// script against it.
|
|
//
|
|
// The demo page cannot be used for this: it writes its own remark_config, so anything a test
|
|
// needs to vary has to go into a page that carries no widget of its own. privacy.html is served
|
|
// by the instance and has none, which also keeps the widget on its own origin, where its CSP
|
|
// allows the chunks it loads. host, site_id and url are filled in when the caller leaves them out
|
|
// placeholder, when given, is markup left inside the root before the widget runs, which the
|
|
// documentation promises is cleared once the iframe reports itself inited
|
|
func embedConfig(t *testing.T, page playwright.Page, config map[string]any, placeholder ...string) {
|
|
t.Helper()
|
|
embedConfigOn(t, page, "/web/privacy.html", config, placeholder...)
|
|
}
|
|
|
|
// embedConfigOn is embedConfig on a chosen host page, for the cases where the page's own address
|
|
// is part of what is under test
|
|
func embedConfigOn(t *testing.T, page playwright.Page, hostPage string, config map[string]any, placeholder ...string) {
|
|
t.Helper()
|
|
|
|
for key, value := range map[string]any{"host": baseURL, "site_id": "remark", "url": threadURL(t)} {
|
|
if _, ok := config[key]; !ok {
|
|
config[key] = value
|
|
}
|
|
}
|
|
|
|
pauseForAuthLimit()
|
|
_, err := page.Goto(baseURL + hostPage)
|
|
require.NoError(t, err)
|
|
|
|
_, err = page.Evaluate(`([config, placeholder]) => {
|
|
window.remark_config = config;
|
|
const node = document.createElement('div');
|
|
node.id = 'remark42';
|
|
node.innerHTML = placeholder;
|
|
document.body.appendChild(node);
|
|
}`, []any{config, strings.Join(placeholder, "")})
|
|
require.NoError(t, err)
|
|
|
|
_, err = page.AddScriptTag(playwright.PageAddScriptTagOptions{URL: playwright.String(baseURL + "/web/embed.mjs")})
|
|
require.NoError(t, err)
|
|
}
|
|
|
|
// stubSignedOut answers the widget's auth probe from the browser, for a page that never signs
|
|
// in. /auth/ is capped at two requests a second for the whole suite and the widget probes on
|
|
// every load, so a case that only needs a signed-out widget should not spend that budget: the
|
|
// tests that do sign in are the ones that cannot fake it
|
|
func stubSignedOut(t *testing.T, page playwright.Page) {
|
|
t.Helper()
|
|
require.NoError(t, page.Route("**/auth/status**", func(route playwright.Route) {
|
|
require.NoError(t, route.Fulfill(playwright.RouteFulfillOptions{
|
|
Status: playwright.Int(http.StatusOK),
|
|
ContentType: playwright.String("application/json"),
|
|
Body: playwright.String(`{"status":"not logged in"}`),
|
|
}))
|
|
}))
|
|
}
|
|
|
|
// reload re-navigates and returns the widget once the thread itself has loaded.
|
|
//
|
|
// waiting for the comment form is not enough here: root.tsx renders it as soon as the user
|
|
// fetch resolves, while the thread keeps its own preloader until /find answers. an assertion
|
|
// that a comment is absent would otherwise pass against a thread that has not rendered yet
|
|
func reload(t *testing.T, page playwright.Page) playwright.FrameLocator {
|
|
t.Helper()
|
|
pauseForAuthLimit()
|
|
_, err := page.ExpectResponse("**/api/v1/find**", func() error {
|
|
_, rerr := page.Reload()
|
|
return rerr
|
|
}, playwright.PageExpectResponseOptions{Timeout: playwright.Float(float64(waitTimeout.Milliseconds()))})
|
|
require.NoError(t, err)
|
|
|
|
frame := widget(t, page)
|
|
// the response has arrived; give preact the frame it needs to swap the preloader out
|
|
waitHidden(t, frame.Locator(`[role="list"] .preloader`))
|
|
return frame
|
|
}
|
|
|
|
// postComment types into the form the widget is currently showing and waits for the comment
|
|
// to appear in the thread
|
|
func postComment(t *testing.T, frame playwright.FrameLocator, text string) {
|
|
t.Helper()
|
|
postCommentMatching(t, frame, text, text)
|
|
}
|
|
|
|
// postCommentMatching posts source and waits for a comment carrying marker. the two differ
|
|
// whenever the source has markdown in it, since the thread shows the rendered result
|
|
func postCommentMatching(t *testing.T, frame playwright.FrameLocator, source, marker string) playwright.Locator {
|
|
t.Helper()
|
|
submitForm(t, frame.Locator(commentFormSel).First(), source)
|
|
|
|
posted := comment(frame, marker)
|
|
waitVisible(t, posted)
|
|
return posted
|
|
}
|
|
|
|
// submitForm fills a comment form and submits it. the submit label is Send, Reply or Save
|
|
// depending on the form's mode, so match on the type and not the text
|
|
func submitForm(t *testing.T, form playwright.Locator, text string) {
|
|
t.Helper()
|
|
require.NoError(t, form.Locator("textarea").Fill(text))
|
|
require.NoError(t, form.Locator(`button[type="submit"]`).Click())
|
|
}
|
|
|
|
// comment locates the comment carrying the text.
|
|
//
|
|
// only sound for asserting that something IS there. comments render through an
|
|
// IntersectionObserver (components/root/in-view), so one below the fold is an empty
|
|
// placeholder article and no text filter matches it: an absence assertion written this way
|
|
// passes whether the comment is gone or merely off screen. use articleCount for absence
|
|
func comment(frame playwright.FrameLocator, text string) playwright.Locator {
|
|
return frame.Locator("article", playwright.FrameLocatorLocatorOptions{HasText: text})
|
|
}
|
|
|
|
// articleCount counts the comments in the thread, including any the viewport has not reached
|
|
// yet, which still occupy an empty article element.
|
|
//
|
|
// it is an absence oracle only after a reload. before one the widget keeps a deleted
|
|
// comment's node and only swaps its text, and the backend prunes a deleted comment from the
|
|
// tree only when it has no replies (store/service/tree.go)
|
|
func articleCount(t *testing.T, frame playwright.FrameLocator) int {
|
|
t.Helper()
|
|
n, err := frame.Locator("article").Count()
|
|
require.NoError(t, err)
|
|
return n
|
|
}
|
|
|
|
// replyForm returns the form the widget opened last, which is the reply or edit form.
|
|
//
|
|
// it waits for a second form to exist first: playwright auto-waits for the element a locator
|
|
// resolves to, not for a better one to appear, so filling .Last() too early would type into
|
|
// the top-level form and post a root comment instead of a reply
|
|
func replyForm(t *testing.T, frame playwright.FrameLocator) playwright.Locator {
|
|
t.Helper()
|
|
eventually(t, waitTimeout, "the reply or edit form did not open", func() bool {
|
|
n, err := frame.Locator(commentFormSel).Count()
|
|
return err == nil && n > 1
|
|
})
|
|
return frame.Locator(commentFormSel).Last()
|
|
}
|
|
|
|
// actions returns the action bar of the comment carrying the text
|
|
func actions(frame playwright.FrameLocator, text string) playwright.Locator {
|
|
return comment(frame, text).Locator(".comment-actions").First()
|
|
}
|
|
|
|
// widget returns the widget's iframe once its own content has rendered. it waits on the
|
|
// comment form and not the auth panel, because .auth is only present while signed out
|
|
func widget(t *testing.T, page playwright.Page) playwright.FrameLocator {
|
|
t.Helper()
|
|
frame := page.FrameLocator("#remark42 iframe")
|
|
waitVisible(t, frame.Locator(commentFormSel).First())
|
|
return frame
|
|
}
|
|
|
|
func waitVisible(t *testing.T, loc playwright.Locator) {
|
|
t.Helper()
|
|
require.NoError(t, loc.WaitFor(playwright.LocatorWaitForOptions{
|
|
State: playwright.WaitForSelectorStateVisible,
|
|
Timeout: playwright.Float(float64(waitTimeout.Milliseconds())),
|
|
}))
|
|
}
|
|
|
|
// waitHidden waits for the element to go away. msgAndArgs says what its staying would mean,
|
|
// since the failure otherwise names a selector and not the behavior that broke
|
|
func waitHidden(t *testing.T, loc playwright.Locator, msgAndArgs ...any) {
|
|
t.Helper()
|
|
err := loc.WaitFor(playwright.LocatorWaitForOptions{
|
|
State: playwright.WaitForSelectorStateHidden,
|
|
Timeout: playwright.Float(float64(waitTimeout.Milliseconds())),
|
|
})
|
|
require.NoError(t, err, msgAndArgs...)
|
|
}
|
|
|
|
// pollText reads an element's text with a timeout short enough to be used inside eventually
|
|
func pollText(loc playwright.Locator) (string, error) {
|
|
return loc.InnerText(playwright.LocatorInnerTextOptions{
|
|
Timeout: playwright.Float(float64(pollTimeout.Milliseconds())),
|
|
})
|
|
}
|
|
|
|
// pollAttr reads an attribute with the same short timeout
|
|
func pollAttr(loc playwright.Locator, name string) (string, error) {
|
|
return loc.GetAttribute(name, playwright.LocatorGetAttributeOptions{
|
|
Timeout: playwright.Float(float64(pollTimeout.Milliseconds())),
|
|
})
|
|
}
|
|
|
|
// eventually polls fn until it returns true, for assertions playwright's own auto-waiting
|
|
// does not cover, such as a value read out of the page with Evaluate
|
|
func eventually(t *testing.T, timeout time.Duration, msg string, fn func() bool) {
|
|
t.Helper()
|
|
deadline := time.Now().Add(timeout)
|
|
for time.Now().Before(deadline) {
|
|
if fn() {
|
|
return
|
|
}
|
|
time.Sleep(50 * time.Millisecond)
|
|
}
|
|
t.Fatalf("condition not met within %v: %s", timeout, msg)
|
|
}
|
|
|
|
// pageFetch issues a request from the page's own session, so it carries the browser's cookies
|
|
// and the XSRF header the widget's own calls carry. body may be nil for a bodyless request
|
|
func pageFetch(t *testing.T, page playwright.Page, method, url string, body any) (status int, respBody string) {
|
|
t.Helper()
|
|
|
|
payload := ""
|
|
if body != nil {
|
|
raw, err := json.Marshal(body)
|
|
require.NoError(t, err)
|
|
payload = string(raw)
|
|
}
|
|
|
|
res, err := page.Evaluate(`async ({method, url, payload}) => {
|
|
const xsrf = document.cookie.split('; ').find((c) => c.startsWith('XSRF-TOKEN='));
|
|
const resp = await fetch(url, {
|
|
method,
|
|
credentials: 'include',
|
|
headers: {
|
|
'Content-Type': 'application/json',
|
|
'X-XSRF-TOKEN': xsrf ? xsrf.slice(xsrf.indexOf('=') + 1) : '',
|
|
},
|
|
body: payload === '' ? undefined : payload,
|
|
});
|
|
return {status: resp.status, body: await resp.text()};
|
|
}`, map[string]any{"method": method, "url": url, "payload": payload})
|
|
require.NoError(t, err)
|
|
|
|
out, ok := res.(map[string]any)
|
|
require.True(t, ok, "unexpected shape from the page: %#v", res)
|
|
switch v := out["status"].(type) {
|
|
case int:
|
|
status = v
|
|
case float64:
|
|
status = int(v)
|
|
default:
|
|
t.Fatalf("unexpected status type %T in %#v", v, out)
|
|
}
|
|
respBody, _ = out["body"].(string)
|
|
return status, respBody
|
|
}
|
|
|
|
// mailpitMessage returns the newest message sent to the address, with its body
|
|
func mailpitMessage(t *testing.T, to string) string {
|
|
t.Helper()
|
|
|
|
type item struct {
|
|
ID string `json:"ID"`
|
|
To []struct {
|
|
Address string `json:"Address"`
|
|
} `json:"To"`
|
|
}
|
|
var found string
|
|
eventually(t, waitTimeout, "no message for "+to, func() bool {
|
|
resp, err := probeClient.Get(mailpitURL + "/api/v1/messages?limit=50")
|
|
if err != nil {
|
|
return false
|
|
}
|
|
defer func() { _ = resp.Body.Close() }()
|
|
|
|
var list struct {
|
|
Messages []item `json:"messages"`
|
|
}
|
|
if err := json.NewDecoder(resp.Body).Decode(&list); err != nil {
|
|
return false
|
|
}
|
|
for _, msg := range list.Messages {
|
|
for _, addr := range msg.To {
|
|
if strings.EqualFold(addr.Address, to) {
|
|
found = msg.ID
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
return false
|
|
})
|
|
|
|
resp, err := probeClient.Get(fmt.Sprintf("%s/api/v1/message/%s", mailpitURL, found))
|
|
require.NoError(t, err)
|
|
defer func() { _ = resp.Body.Close() }()
|
|
|
|
var body struct {
|
|
Text string `json:"Text"`
|
|
HTML string `json:"HTML"`
|
|
}
|
|
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
|
|
return body.Text + body.HTML
|
|
}
|