Dmitry VerkhoturovandGitHub
4d5dae20e2
* Pin the published /web surface in the e2e suite
#2178 renamed the widget bundles from .js to .mjs and the URLs earlier
releases served under those names stopped resolving. Three were noticed
from the demo site; the rest, including every locale chunk, were found
only by requesting the whole surface of both images over HTTP. #2192
restored them with a server-side alias, and nothing in the suite would
have caught the break or would notice it returning.
Two cases with deliberately different criteria. The documented names are
written out, because the documentation decides that list and not the
build: an operator pastes privacy.html into an OAuth application, the
nginx manual proxies index.html by name, and the integration guides start
from the embed script. Everything else is taken from the build itself, so
whatever the bundler emitted has to serve identical bytes under its
legacy .js name and parse as a classic script, which is the premise
serving one under the other rests on. A third case requests a name that
does not exist, without which a fallback serving one page for everything
would keep the whole table green.
All of them check the content type as well as the bytes: nosniff is set
on every response, so a bundle served as text/plain is as broken as one
that 404s while comparing equal.
On e3d1d0e2, the commit before the alias, this fails with 32 red subtests.
* Cover the widget behavior the e2e suite never drove
Removing npm from the widget takes the jest tests with it, and everything
here was protected by jest or by nothing at all.
Signing out was untested at every level, and the panel repainting is the
half that always works: the assertion after the reload is the one that
catches a session the server never ended. The edit form has to hand back
the source that was posted and not the rendered comment, which #2040
shipped the other way round, taking every entity and tag the author had
written with it. A draft has to survive a reload and be gone once the
comment is posted. A refused comment has to stay in the form with
something said about it, so that case drives the backend's own
restricted-words code and then retries with the route removed.
Uploads had no browser coverage in either direction. The posted case
asserts naturalWidth instead of visibility, since a broken src still
renders as an empty box, and the failing case holds the intercepted
request long enough for the in-flight state to be observed: without that,
"the text is unchanged afterwards" would hold for an upload that never
started.
Moderation and reader-side hiding are asserted from a page other than the
one that made the change. Hiding seeds a second author, so it proves one
person is hidden and the thread not emptied, and both the blocked
author and the moderated one carry the run id in their names: a block and
a verification are properties of the user and outlive the run in the
stack's database, so a fixed name works exactly once.
Locales were the largest hole. Each catalog is a chunk fetched at
runtime, and loadLocale falls back to english on any failure instead of
throwing, exactly as an unrecognized name does, so every case compares
the rendered string against the file on disk. One case per catalog
covers that they are all served and render; another fetches one through
the widget document and asserts it parsed, which is the half a chunk that
serves but fails to parse would slip through. The delete page and the
last-comments stylesheet had no coverage of any kind.
Two things the suite itself needed. The widget's own aria-label is
translated, so commentFormSel only ever finds an english widget and a
localized case cannot use widget(). And the auth probe is capped at two
requests a second for the whole suite, hard-coded in rest.go: the added
cases pushed past it and the suite began manufacturing its own 429s,
which render as a signed-out widget and fail whichever test happens to be
signing in, so the pacing gap is wider and the locale cases stub the
probe they never needed.
* Harden the e2e harness against silent failures and stale stacks
Three things the suite could not tell you about itself.
A browser failure nothing asserts on now fails the test that caused it.
Uncaught exceptions are the ones worth the machinery: a widget throwing
while it renders leaves most of these cases green, since they assert on
elements the browser lays out either way. Rate-limit responses are
recorded as well as logged for the same reason. A test driving an error
path declares what it expects by substring, so a case that means to
break something says which thing.
The stack the suite adopts is now checked against the sources under
test. Every checkout builds the image tag the compose file names, so a
stack from another worktree, or from this one before an edit, answers on
these ports and passes every readiness probe while serving code nobody
is looking at. stamp.sh digests what goes into the image, compose passes
it as the revision label, and a mismatch is refused with what to do
about it. Checked after our own build too, or a stamp that never reaches
the image would be a guard that silently passes everything.
assertSignedIn no longer waits out the whole timeout on a refused status
read. /auth/ is capped at two requests a second for the entire suite, a
bare literal at rest.go:242, and a case signing in on two pages spends
that twice; when the read that repaints the panel is the one the limiter
turns down, the widget shows signed out over a session that exists and
no later request will ask again. The short first wait now ends in a
focus handoff, which the widget answers by re-probing, and only then
does the real wait run. A sign-in that genuinely failed still fails,
since the second read finds no state either. That is what
TestComment_AdminPinsAndVerifies and TestComment_BlockedAuthorCannotPost
were failing on in CI while passing locally.
signInAnon takes the page for that reason, and its callers pass it.
* Cover iframe geometry, the embed contract and five deployment modes
Seventeen cases for the parts of the widget that broke repeatedly and
that nothing here could see, plus the areas jest was the only check on.
Geometry is the biggest of them. The widget measures its own document
and posts the number for the parent to apply, and every way that has
gone wrong is invisible to assertions about elements, which read the
same whether the frame is right, twice too tall or a strip. So: the
first height the parent is given describes rendered content and not the
preloader, the frame matches the document it holds and does not stand
24px taller, no_footer leaves the last comment inside the frame, and the
frame follows the sign-in dropdown and the growing textarea and comes
back down again. Each was verified by reintroducing the defect it covers
and watching it fail: a 63px report before the real one, six pixels of
body padding, and a frame sized under the content.
The embed surface is the other half the widget cannot see. An element
placeholder gives way to exactly one iframe carrying the embed's own
marker, a second createInstance reuses it, destroy takes it away with
its handles, and a theme change after load reaches both the element and
the document. A page that posts a message of its own, which is all
embed.ts's own title observer does, no longer empties an open login
form.
Five configurations that cannot share an instance get one each, since
each changes the widget for every reader: an admin's unlimited edit
window, a session carried in a header and not a cookie, an instance with
no auth provider to offer, anonymous voting, and the notify module,
without which email_notifications is false and the subscribe control
never renders at all. The subscription round trip is the one place a
token from a real message is exchanged for state the server keeps.
Two things surfaced there and are left alone, both said so in place. The
panel confirming an unsubscribe cannot be observed, because the click
changes the step and the dropdown closes on an element no longer in the
rerendered view, which the component notes as its own awkwardness; the
case asserts the request and the answer, which is what decides whether
the reader still gets mail. And the widget takes the subscribed state
from user.email_subscription, absent from what it hydrates the user with
on the next load, so after a reload it offers to subscribe somebody who
already is.
simple_view needs no instance, being a query parameter, so both branches
run against the main one. A transient failure of the status probe has a
case too: the session belongs to the server, and one refused answer must
not end it.
The suite runs about four and a half minutes now, so the workflow's own
budget goes to 20m to match the Makefile, well inside the job timeout,
and the workflow stamps the stack it starts the way the Makefile does.
The locale case that loads the widget document directly is renamed for
what it protects, the origin and CSP its chunks are fetched under.
Telegram gets no test: the base URL is formatted inline inside
go-pkgz/auth, so nothing here can point it elsewhere, and the fix
belongs upstream in v1 and v2 both. Reported as #2208.
Three things CI found that a laptop cannot. The anonymous sign-in form
validates its input against pattern="[\p{L}\d\s_]+", and E2E_RUN_ID is
"<run id>-<attempt>" on a runner, so every username built from it
carried a hyphen the browser refused to submit: no request was made and
the case waited out its timeout on a panel that was never going to
change. Names are built by anonName now, which drops what the pattern
does not allow and adds the pid, so a second run against a surviving
stack does not meet its own blocked and verified users. signInAnon waits
for the request the submit makes, so the next such refusal fails as
itself.
The admin instance takes its admin from an email address, not a name.
remark42 hashes an anonymous id from the name and the client address
together, to tell apart two people picking the same name, so the id
written into ADMIN_SHARED_ID belonged to nobody on a runner and the
instance had no admin at all: the countdown stayed, and the backend
refused the edit. An email id is sha1 of the address, which is the same
everywhere.
The subscription case clears its own precondition and confirms through
the page's session. The dev user is shared and a subscription outlives
the run, so the panel opened on the subscribed step; and the panel moves
to that step while its token textarea is still on screen, leaving no
moment at which the control to submit it exists.
Three settings of remark_config get cases of their own, none having had
any: __colors__, which is the one setting that travels through
window.name and not the query string, so nothing else in the suite would
notice the path going; the url override, which is how a canonical
address keeps one conversation across pages that differ; and the
subscription controls an integrator turns off, with the both-shown case
as the control.
Writing the url case turned up a backend defect, reported as #2204 and
not fixed here: a thread url containing "&" cannot be commented on at
all. Sanitize runs the locator
through SanitizeAsURL, which round-trips it through bluemonday, so the
url is stored html-escaped; the bucket is created under the escaped key,
the read-back uses the real one and answers 500, and every later find,
count and feed asks for the real url and is told the thread is empty.
Any page addressed with two query parameters is affected. The case uses
a single-parameter url for that reason.
Five more from the same audit, none needing a service. Collapsing a
thread shrinks the frame, which every other geometry case would miss:
they all assert growth, and a widget that only grew would satisfy them
while leaving a hole under each collapsed thread. Voting gains the
direction nothing covered, downvoting and its survival of a reload, and
the rule the other vote cases work around, that your own comment offers
no buttons and the backend refuses the vote anyway.
A vote with the X-XSRF-TOKEN header stripped has to be refused. That
check is why a document navigation, an iframe src among them, is always
anonymous and why the widget hydrates its user over XHR, so anything
designed around that wants it pinned.
An unrecognized locale has to render English, which is loadLocale's only
observable guarantee: it falls back the same way for a name it does not
know and for a chunk it cannot fetch. And a comment's timestamp has to
be the reader's own, which is the one part of rendering that cannot move
to the server, asserted from a context in Kiritimati against the same
Intl the widget uses.
A host page on an origin the widget is not served from, which is the
separate-domain setup the manuals describe and the configuration readers
actually hit problems with. Every other host page here is served by
remark42 itself, so the cross-site path was never taken: an nginx on its
own name and port serves e2e/hostsite, and the case asserts the frame is
revealed, which means its document loaded and reported itself inited
across the origin boundary, and that the thread it renders is the one
the page's address names. Signing in is left out on purpose, an embedded
cookie needing SameSite=None, which browsers take only as Secure, and
this stack speaks http; that is #1139 and not something a case here can
settle.
The other half is ALLOWED_HOSTS. The no-provider instance names only
itself, so a page elsewhere embedding it is refused by the browser, the
document never runs, and the reveal comes from the widget's own fallback
five seconds later. Both directions are worth holding: without the
fallback a mistyped host leaves a permanently invisible widget with
nothing to say why, and without the refusal the setting does nothing.
The host page's title reaching the stored comment gets a case, the path
running the other way from everything else here: the page posts its
title into the widget, the widget sends it with the comment, and it is
what a feed and the admin listing show. Set after the widget is up, so
it covers the observer embed.ts installs and not the value read at boot.
max_shown_comments has no case. The setting reaches the widget, appears
in the iframe's query string and changes nothing: four comments render
with it set to two, which is #812, still open, where the reproduction is
now recorded. A case for it would be red on master.
The README gains what the suite cannot reach. Every service here speaks
http, so anything the browser gates on the page protocol is invisible: a
Secure cookie, anything keyed on window.location.protocol, and the
SameSite=None with Secure and Partitioned form that is the only one an
embedded frame can still use. That is not hypothetical, setAuthCookie
having decorated its cookies with __Host- on any https page and survived
precisely because nothing here runs on one. The cross-origin case names
the assertion to add if the stack ever gets TLS, which is the reload:
the widget holds its token in memory for the life of a page, so signing
in and posting without reloading passes while persistence is broken.
And the trap waiting for whoever acts on that: playwright's own default
--disable-features argument carries ThirdPartyStoragePartitioning, and it
beats both --test-third-party-cookie-phaseout and
--block-third-party-cookies passed through Args, so a run meaning to
prove the third-party case keeps an ordinary third-party cookie exactly
as it would with no flags at all. IgnoreDefaultArgs is the lever, and a
blocking run has to assert a control before anything it reports can be
believed. None of it reaches the widget's own storage fallback either:
IS_STORAGE_AVAILABLE stays true with partitioning enforced, chromium
partitioning localStorage instead of denying it, so comments.html needs
webkit and not a flag.
The downvote case now actually corrects. It claimed the score ends where
the second vote leaves it and never cast one, so #728, a reader taking a
vote back, could break with it green. It also turns out the opposite
vote takes the first one back instead of flipping it, so the score
returns to zero and never reaches +1, which is what the case asserts,
before and after a reload. Renamed for what it covers.
267 lines
11 KiB
Go
267 lines
11 KiB
Go
//go:build e2e
|
|
|
|
package e2e
|
|
|
|
import (
|
|
"context"
|
|
"crypto/sha256"
|
|
"encoding/hex"
|
|
"errors"
|
|
"io"
|
|
"net/http"
|
|
"os"
|
|
"os/exec"
|
|
"slices"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/mxschmitt/playwright-go"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
const (
|
|
// the e2e stack's container, named in compose-e2e-test.yml. addressed directly and not
|
|
// through `docker compose exec`, which resolves the service through a project name taken from
|
|
// the directory the compose file sits in: run the suite from a git worktree and it looks for a
|
|
// project named after the worktree instead of the stack that is answering
|
|
stackContainer = "remark42-e2e"
|
|
|
|
// what the served web root is inside that container
|
|
stackWebRoot = "/srv/web"
|
|
|
|
// enough for a healthy docker CLI and short enough to fail as itself. an unbounded call to a
|
|
// wedged daemon hangs until the package timeout panics the binary, which skips TestMain's
|
|
// teardown and leaves the stack running
|
|
dockerTimeout = 30 * time.Second
|
|
|
|
localesDir = "../frontend/apps/remark42/app/locales"
|
|
)
|
|
|
|
// the /web paths the documentation publishes, plus the legacy names of the same files. each is
|
|
// held by somebody outside this repository: an operator pastes privacy.html into an OAuth
|
|
// application, an nginx config proxies index.html, the integration guides start from the embed
|
|
// script, and the comment form links markdown-help.html. written out and not derived, because
|
|
// the documentation decides the list, so a name joining or leaving is an edit made on purpose
|
|
var documentedWebPaths = []struct {
|
|
path string
|
|
// what publishes it, so a failure names who is holding the URL
|
|
where string
|
|
// the content type it has to keep. nosniff is set on every response, so a bundle served as
|
|
// text/plain is as broken as one that 404s while its bytes still compare equal
|
|
mime string
|
|
// something only the right file contains. without it a directory listing, or an error page
|
|
// carrying a 200, passes for the real thing
|
|
marker string
|
|
}{
|
|
{"/web", "getting-started/installation, the first URL an operator opens", "text/html", `id="remark42"`},
|
|
{"/web/", "linked from most documentation pages", "text/html", `id="remark42"`},
|
|
{"/web/index.html", "manuals/nginx proxies this exact URL", "text/html", `id="remark42"`},
|
|
{"/web/embed.mjs", "configuration/frontend/spa.md, manuals/subdomain, contributing/frontend", "text/javascript", "remark_config"},
|
|
{"/web/embed.js", "the legacy name integrations still hold", "text/javascript", "remark_config"},
|
|
{"/web/counter.mjs", "built from remark_config.components by the loader in configuration/frontend", "text/javascript", "remark42__counter"},
|
|
{"/web/counter.js", "the legacy name the same loader built", "text/javascript", "remark42__counter"},
|
|
{"/web/last-comments.mjs", "built from remark_config.components by the same loader", "text/javascript", "remark_config"},
|
|
{"/web/last-comments.js", "the legacy name the same loader built", "text/javascript", "remark_config"},
|
|
{"/web/privacy.html", "configuration/authorization, pasted into an OAuth application", "text/html", "Privacy Policy"},
|
|
{"/web/markdown-help.html", "linked by the comment form", "text/html", "Markdown"},
|
|
{"/web/400x400.jpeg", "embedded by markdown-help.html", "image/jpeg", ""},
|
|
}
|
|
|
|
// TestWeb_DocumentedURLsResolve pins the published surface as a compatibility contract. A page or
|
|
// an OAuth application set up years ago holds these URLs for good, and the build changing shape is
|
|
// not their problem
|
|
func TestWeb_DocumentedURLsResolve(t *testing.T) {
|
|
for _, tc := range documentedWebPaths {
|
|
t.Run(strings.TrimPrefix(tc.path, "/"), func(t *testing.T) {
|
|
resp := getWeb(t, tc.path)
|
|
|
|
assert.Equal(t, http.StatusOK, resp.status, "%s has to keep resolving: %s", tc.path, tc.where)
|
|
assert.NotEmpty(t, resp.body, "%s resolved but served nothing", tc.path)
|
|
assert.Contains(t, resp.mime, tc.mime, "%s serves the wrong type, and nosniff means the "+
|
|
"browser will refuse it whatever the bytes are", tc.path)
|
|
if tc.marker != "" {
|
|
assert.Contains(t, resp.body, tc.marker, "%s resolved but is not the file it should be", tc.path)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestWeb_UnknownNameIs404 is the negative control for the case above: without it a fallback
|
|
// serving one page for everything under /web would keep every assertion there green
|
|
func TestWeb_UnknownNameIs404(t *testing.T) {
|
|
resp := getWeb(t, "/web/no-such-file.html")
|
|
assert.Equal(t, http.StatusNotFound, resp.status, "an unknown name has to 404, or the cases "+
|
|
"above cannot tell a served file from a fallback")
|
|
}
|
|
|
|
// TestWeb_EveryBundleServesUnderBothSuffixes covers the names the list above does not, and takes
|
|
// its input from the build and not from a list somebody maintains, so a locale joining or
|
|
// leaving needs no edit here. Whatever the bundler emitted has to answer under its legacy .js name
|
|
// with the same bytes and the same type, and has to parse as a classic script, which is the premise
|
|
// serving one under the other rests on
|
|
func TestWeb_EveryBundleServesUnderBothSuffixes(t *testing.T) {
|
|
names := emittedBundles(t)
|
|
// a listing from the wrong place, or one that lost most of its entries to a chunk directory,
|
|
// would leave a handful of cases running and report green. every locale is a chunk of its own,
|
|
// so the catalog count on disk is a floor the build cannot drop below
|
|
require.Contains(t, names, "remark.mjs", "listing is not the served web root: %v", names)
|
|
require.GreaterOrEqual(t, len(names), localeCount(t),
|
|
"%d bundles is fewer than there are locales, so the listing is missing chunks: %v", len(names), names)
|
|
t.Logf("checking %d bundles", len(names))
|
|
|
|
page := parsePage(t)
|
|
|
|
for _, name := range names {
|
|
t.Run(name, func(t *testing.T) {
|
|
legacy := strings.TrimSuffix(name, ".mjs") + ".js"
|
|
|
|
modern, alias := getWeb(t, "/web/"+name), getWeb(t, "/web/"+legacy)
|
|
require.NotEmpty(t, modern.body, "%s serves nothing, so everything below compares emptiness", name)
|
|
|
|
assert.Equal(t, digest(modern.body), digest(alias.body),
|
|
"%s and %s serve different content (%d and %d bytes), so the legacy name is not the same file",
|
|
name, legacy, len(modern.body), len(alias.body))
|
|
assert.Contains(t, alias.mime, "javascript",
|
|
"%s serves the wrong type, and nosniff means the browser refuses it however right the bytes are", legacy)
|
|
|
|
require.Empty(t, classicScriptError(t, page, modern.body),
|
|
"%s does not parse as a classic script, so serving it as %s hands a syntax error to the "+
|
|
"oldest integrations", name, legacy)
|
|
})
|
|
}
|
|
}
|
|
|
|
// webResponse is what the cases assert on: everything read from one request, since the body has to
|
|
// be consumed and closed before the next one anyway
|
|
type webResponse struct {
|
|
status int
|
|
mime string
|
|
body string
|
|
}
|
|
|
|
// getWeb requests a path from the stack. Redirects are followed, because whoever holds a
|
|
// documented URL cares whether the page arrives, not how many hops it took: /web and
|
|
// /web/index.html both answer 301 towards the directory
|
|
func getWeb(t *testing.T, path string) webResponse {
|
|
t.Helper()
|
|
|
|
pauseForWebLimit()
|
|
resp, err := probeClient.Get(probeURL + path)
|
|
require.NoError(t, err)
|
|
defer func() { assert.NoError(t, resp.Body.Close()) }()
|
|
|
|
body, err := io.ReadAll(resp.Body)
|
|
require.NoError(t, err)
|
|
require.NotEqual(t, http.StatusTooManyRequests, resp.StatusCode,
|
|
"%s was rate limited, which is this file pacing itself wrong and not a broken URL", path)
|
|
|
|
return webResponse{status: resp.StatusCode, mime: resp.Header.Get("Content-Type"), body: string(body)}
|
|
}
|
|
|
|
func digest(body string) string {
|
|
sum := sha256.Sum256([]byte(body))
|
|
return hex.EncodeToString(sum[:])
|
|
}
|
|
|
|
// emittedBundles lists the .mjs files the running stack serves, recursively, so chunks landing in
|
|
// a subdirectory stay in the set. Read from the container because the bundler runs inside the
|
|
// image build: the host has no build output, and a list from anywhere else describes another build
|
|
func emittedBundles(t *testing.T) []string {
|
|
t.Helper()
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), dockerTimeout)
|
|
defer cancel()
|
|
|
|
out, err := exec.CommandContext(ctx, "docker", "exec", stackContainer,
|
|
"find", stackWebRoot, "-name", "*.mjs").Output()
|
|
require.NoError(t, err, "listing %s in %s: %s", stackWebRoot, stackContainer, exitStderr(err))
|
|
|
|
var names []string
|
|
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
|
if name := strings.TrimPrefix(strings.TrimSpace(line), stackWebRoot+"/"); name != "" {
|
|
names = append(names, name)
|
|
}
|
|
}
|
|
slices.Sort(names)
|
|
return names
|
|
}
|
|
|
|
// localeCount is how many message catalogs the app carries, each of which the bundler emits as
|
|
// its own chunk
|
|
func localeCount(t *testing.T) int {
|
|
t.Helper()
|
|
|
|
entries, err := os.ReadDir(localesDir)
|
|
require.NoError(t, err, "reading %s", localesDir)
|
|
|
|
count := 0
|
|
for _, e := range entries {
|
|
if strings.HasSuffix(e.Name(), ".json") {
|
|
count++
|
|
}
|
|
}
|
|
require.NotZero(t, count, "no catalogs in %s, so the floor below would be meaningless", localesDir)
|
|
return count
|
|
}
|
|
|
|
// exitStderr is what the command wrote to stderr, which Output keeps out of the parsed result
|
|
func exitStderr(err error) string {
|
|
var exit *exec.ExitError
|
|
if errors.As(err, &exit) {
|
|
return string(exit.Stderr)
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// parsePage is a browser page for the parse probe alone. Not newPage: that one records a trace on
|
|
// failure, and a trace of a page which never navigates sends the reader to an empty recording
|
|
func parsePage(t *testing.T) playwright.Page {
|
|
t.Helper()
|
|
|
|
page, err := browser.NewPage()
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { assert.NoError(t, page.Close()) })
|
|
return page
|
|
}
|
|
|
|
// classicScriptError returns the parse error a classic script parser gives for src, and an empty
|
|
// string when there is none. new Function parses its argument as a function body, which rejects a
|
|
// top level import, export or import.meta exactly as a classic script does, and runs nothing. The
|
|
// grammars are not identical, a function body also accepting return and new.target, but only in
|
|
// the direction that lets more through, so the probe never fails a bundle a browser would take
|
|
func classicScriptError(t *testing.T, page playwright.Page, src string) string {
|
|
t.Helper()
|
|
|
|
v, err := page.Evaluate(`src => {
|
|
try { new Function(src); return ""; }
|
|
catch (e) { return e.name === "SyntaxError" ? String(e) : ""; }
|
|
}`, src)
|
|
require.NoError(t, err)
|
|
|
|
msg, ok := v.(string)
|
|
require.True(t, ok, "the parse probe returned %T instead of a string", v)
|
|
return msg
|
|
}
|
|
|
|
var (
|
|
webGate sync.Mutex
|
|
lastWebGet time.Time
|
|
)
|
|
|
|
// everything under /web/ is rate limited to 20 requests a second, hard coded in rest.go rather
|
|
// than settable, and the bundle case asks for two files per bundle. Without pacing this file
|
|
// manufactures the 429s it would then have to tell apart from a missing URL
|
|
func pauseForWebLimit() {
|
|
const spacing = 55 * time.Millisecond
|
|
|
|
webGate.Lock()
|
|
defer webGate.Unlock()
|
|
if wait := spacing - time.Since(lastWebGet); wait > 0 {
|
|
time.Sleep(wait)
|
|
}
|
|
lastWebGet = time.Now()
|
|
}
|