Compare commits

..
Author SHA1 Message Date
Dmitry Verkhoturov 345eee8ba4 Fix two sentences the Firefox row made false
Both predate #2222 and both are on master now, so they contradict the table
that PR added. One said the header flag keeps a reader signed in in any browser
configured to block third-party cookies; the other said the recommended
arrangement survives a reload whatever the browser's third-party cookie policy.
Firefox's block-all setting is the exception to both, because it discards
partitioned cookies as well, and the same table says so three lines away.

Folding them in here rather than leaving them to a separate pass, since this
branch is already editing the file.
2026-08-24 01:05:03 +01:00
Dmitry Verkhoturov 7ee867e7bf Measure the Safari column on Safari itself
The table's Safari column was WebKit through Playwright, which is the engine
but not the browser: ITP is a Safari layer above it and could have been
stricter. Driven through Safari 27's own WebDriver, all three configurations
match the WebKit result exactly, so the column now says Safari and means it,
and the e2e suite's WebKit coverage is a faithful proxy for this behaviour.

Safari blocks third-party cookies out of the box and still honours Partitioned:
with the header flag the widget's own cookie is readable inside the frame and
the reload keeps the reader signed in, while the control cookie written beside
it is dropped.
2026-08-24 00:06:36 +01:00
Dmitry VerkhoturovandGitHub 3286f028e3 Document what each browser actually does with cross-domain auth (#2222)
Measured on real domains over real certificates, Remark42 on one registrable
domain and the host page on another, with a control cookie behind every
blocked column so a run that blocks nothing cannot report a pass.

Three results the manual did not carry. Safari blocks third-party cookies out
of the box, so AUTH_SAME_SITE=none on its own has already stopped working
there, which makes the old recipe broken today and not deprecated later.
Firefox reaches a working session by a weaker route than Chrome and Safari do:
it accepts the server's attribute-less cookie, and because that cookie is
HttpOnly the browser then forbids the widget's script from replacing it, so the
session rides on an ordinary unpartitioned third-party cookie even with the
header flag on. And Firefox's block-all setting discards partitioned cookies
too, so no configuration survives it.

Two parameter descriptions were wrong in ways that matter here. AUTH_SAME_SITE
default emits no SameSite attribute rather than Lax, which is precisely what
lets the widget's own cookie land on Chrome and Safari. And AUTH_TTL_COOKIE
does not govern the cookie that carries the session under the header flag,
since the frontend hardcodes 200h to mirror the default.
2026-08-23 18:03:49 -05:00
UmputunandGitHub c947a06d48 Release the response before tearing the test server down (#2212)
TestRest_securityHeaders and TestRest_frameAncestors both start a server, read
one response, and then call teardown() partway through the test to start a
second server with different options. The first response body is only closed by
a defer, which does not run until the test returns.

httptest.Server.Close waits on connections still in use, so it blocks on a body
that will not be closed until after it returns. The tests deadlock and the whole
rest/api package dies on the timeout rather than on an assertion.

CI pins go 1.25, where the responses are small enough that the connection goes
back to the pool on its own and nothing hangs. On go 1.27 both tests hang, which
is how this surfaced.

Close the body and the client's idle connections before teardown() in both.
2026-08-23 17:51:52 -05:00
UmputunandGitHub 5f439cf1d5 Qualify what works off-domain, and fix two typos beside it (#2221)
The opening summary said Telegram, Email and anonymous auth "would work
everywhere". That holds only with AUTH_SEND_JWT_HEADER set, and the widget's
own cookie is Secure, so the path is HTTPS-only and bounded by ALLOWED_HOSTS
besides. The sentence now states those conditions. What happens without the
flag is two separate things, whether sign-in succeeds in the frame and whether
it survives a reload, and the body below already separates them.

The other two are older: a stray backtick after "work on any domain", and
"expect" for "except" in a bullet whose neighbour already says except.

Related to #2218
2026-08-23 17:51:47 -05:00
Dmitry VerkhoturovandGitHub 7de51ad2ef Document what actually keeps a cross-domain reader signed in (#2218)
* Document what actually keeps a cross-domain reader signed in

The separate-domain manual tells operators to set ALLOWED_HOSTS and
AUTH_SAME_SITE and says authorisation then works anywhere. That stopped being
true as browsers began blocking third-party cookies: the server-set auth
cookies carry no Partitioned attribute, so a browser enforcing the block drops
them whatever their SameSite value. What survives is AUTH_SEND_JWT_HEADER,
where the token returns in a header and the widget writes its own partitioned
cookie from inside the frame, and the manual never mentioned it. It now does,
with the XSS trade-off and a pointer to the parameter page, and it says plainly
that this rescues Email, Telegram and anonymous but not oAuth.

The parameter page's own mitigation list was left wrong by #2197. It promised
SameSite=Strict cookies and a __Host- prefix on HTTPS; authCookieOptions drops
the prefix entirely and uses SameSite=None; Secure; Partitioned whenever the
widget is embedded on another domain, which is the case the flag exists for.

* Say that the JWT header is sent in addition to the cookies, not instead

Both the flag's own help and the parameter table said the header replaces the
server-set cookie. Service.Set does neither: it writes the header and then
falls through to set both cookies, with a comment saying the cookies are needed
because headers do not survive the OAuth redirect. An operator reading either
description would expect the server to stop setting cookies once the flag is
on, and would misjudge what the flag changes about their exposure.

* Correct three details in the cross-domain documentation

The link to the parameter page used Zola's @/ syntax, which Hugo emits
literally as a relative href since there is no render-link hook. It was the
only such link under site/content; the other manuals use the relative form and
this now does too.

The CHIPS description claimed Partitioned makes the cookie unreadable from any
other page the browser visits. The partition key is the top-level site, so a
different site gets a separate cookie while pages and subdomains under the same
site share it. Overstating isolation on the page an operator reads to weigh
risk is the wrong direction to be wrong in.

And Chrome does not block third-party cookies by default: Google's April 2025
position keeps ordinary Chrome on user choice and names Incognito as the mode
that blocks. Naming Safari, Chrome Incognito and browsers configured to block
them says the same thing and stays true.

* Drop AUTH_SAME_SITE from the recommended cross-domain recipe

Measured rather than reasoned, because it reverses guidance this page has
carried for years. With only the remark42-https service taken back to the
default, both reload cases pass for anonymous and email, under a permissive
browser and under one enforcing partitioning.

The cookie jar after an anonymous sign-in says why. With the setting there are
four cookies: the server's unpartitioned JWT and XSRF-TOKEN, and the widget's
own partitioned pair. Without it there are two, the widget's pair alone, and
the session behaves identically. So the setting is doing something real, which
is what makes the passing run meaningful, and what it does is add an
unpartitioned HttpOnly JWT delivered as a third-party cookie to every listed
domain wherever the browser still permits that. Nothing needs it.

It stays documented for the configuration that does need it, which is one
without AUTH_SEND_JWT_HEADER, where the server's cookies are the only ones
there are.

One prediction the experiment falsified: the attribute case was expected to
fail on the default server-set pair. It passes, because a cross-site Set-Cookie
lacking SameSite=None is refused outright, so that pair is absent from the jar
instead of present with the wrong attribute. The manual now says so.
2026-08-23 15:49:06 -05:00
Dmitry VerkhoturovandGitHub 389189afcf Give each import request in TestMigrator_ImportDouble its own reader (#2220)
The test passed one strings.Reader as the body of both POSTs. client.Do
returns once the response headers arrive, and the import answers 202 before
the transport has finished copying the body, so the second http.NewRequest
reads the reader's Len to set ContentLength while the first request's
writeLoop is still advancing it. The race detector caught it on CI as a write
in strings.(*Reader).WriteTo against a read in NewRequestWithContext, failing
a test nothing had touched.

Reproduced in isolation to confirm the mechanism rather than infer it from the
trace: a handler that answers 202 without draining an 8 MiB body, two requests
sharing one reader, and -race reports strings.(*Reader).Len in
NewRequestWithContext against strings.(*Reader).Read on every run. It does not
reproduce in this package locally, which is why it reads as a flake.

Both requests now build their own reader over the same content. The second one
carries a full body where before it inherited a consumed one, which is closer
to what the case is about: a second import arriving while the first is running
still has to be refused.
2026-08-23 14:58:31 -05:00
Dmitry VerkhoturovandGitHub 6f40926241 Drop the origin-anchored public path from the delete-me bundle (#2219)
deleteme.ts set __webpack_public_path__ to window.location.origin plus /web/,
which discards any path prefix the instance is served under. It is inert today
because that bundle references no asset and loads no chunk, so the value is
assigned and never read, but it is wrong by construction and would resolve at
the domain root the moment anyone adds an image to that page. Removing it
leaves webpack's own publicPath: 'auto', which derives the base from the
script's URL and is right in both arrangements.
2026-08-23 14:58:27 -05:00
Dmitry VerkhoturovandGitHub 2640aaee9e Reach what http cannot: the widget over TLS, embedded cross-origin (#2214)
Every service in the suite spoke http, and the browser gates a whole class
of behaviour on the page protocol: Secure cookies, SameSite=None,
Partitioned, and any code reading location.protocol. None of it was
executed, which is how setAuthCookie came to decorate its cookies with
__Host- on https pages and survive for years.

A TLS pair joins the stack: remark42 with SSL_TYPE=static on 8443, and an
nginx serving a host page on its own name on 8444, both on a self-signed
certificate that e2e/tls/generate.sh makes and .gitignore keeps out. Every
context accepts it, and so does the readiness client, since those are the
only servers either talks to. The instance also runs with
AUTH_SEND_JWT_HEADER, which is what makes the widget write cookies of its
own: without it the client-side writer never runs on any https page here
and every assertion about the attributes it chooses is vacuous.

Three cases. Signing in across origins and then reloading, which is the one
the http cross-origin case cannot make: the widget holds its token in
memory for the life of a page, so signing in and posting says nothing about
persistence and only the reload asks whether the cookie was delivered,
stored under a name the backend reads and sent back from a third-party
frame. The cookies themselves, read out of the browser store while the
widget is embedded elsewhere, since a cookie the browser refused is absent
from that list entirely and one it kept but will not send is worse than
useless: every copy of both names has to be Secure and SameSite=None, at
least one has to be partitioned, and none may carry a __Host- prefix
nothing on either side reads. And the same reload under a browser that
blocks third-party cookies, which the widget's own partitioned pair is the
only reason to survive.

That last one needs a browser playwright does not offer: its default
arguments disable ThirdPartyStoragePartitioning outright, so a run
configured wrongly keeps every third-party cookie and the case would pass
while asserting nothing. IgnoreDefaultArgs drops that list and re-supplies
it without the one feature, and a control cookie set from inside the frame
has to be refused before anything else is read, so a playwright release
that changes the list fails as itself instead of going quietly vacuous.

All three pass against master. What TLS still cannot reach, the OAuth popup
above all, is written down in the README.
2026-08-23 14:58:23 -05:00
Dmitry VerkhoturovandGitHub 250e8ad925 Report the widget height when the sign-in panel closes (#2213)
The sign-in panel is positioned absolutely, so it grows the iframe without
growing the document: `useDropdown` measures the panel itself and posts the
sum, and a ResizeObserver on the panel keeps that number current while it is
open. Closing it resizes no box anything watches. The panel observer goes with
the element, and the document observer in `Root` sees nothing, because the
document height never changed in the first place. Nothing then tells the parent
to come back down, so the iframe keeps the open panel's height and the
embedding page carries a hole under the widget for as long as the reader stays
on it.

The effect's cleanup now reports the height, with no element, so the number is
the document's own. That is the one place both close paths reach: the click
inside the widget, and the clickOutside message the host page posts when the
reader clicks anywhere else.

TestGeometry_HeightFollowsTheAuthPanelAndTheTextarea covers this and has been
intermittently green: whether the frame comes back down without the fix depends
on timing, and it fails on every run here while CI has been passing. The unit
test fails with the cleanup reverted.
2026-08-23 00:26:31 -05:00
Dmitry VerkhoturovandGitHub 4793c1cd2c Fill the instance URL into the embedded frontend at serve time, and stop pinning compressor output in tests (#2198)
* Assert what the image endpoints promise rather than the compressor's output

Three tests pinned the exact bytes or the exact length of an encoded
image, so they fail on any toolchain whose deflate or png encoder emits
something different. CI pins go 1.25 and passes; go 1.27 fails all three,
while the images themselves are perfectly valid.

TestRest_QR now decodes both the golden file and the response and
compares the pixels, which is the same assertion about the qr code and
none about the encoder. The two resize cases assert the decoded image
fits the box resize was given and touches one of its sides, which is what
fitting to a box means and what the function actually promises.

Resolves #2200.

* Fill the instance URL into the embedded frontend at serve time

The widget falls back to a compiled-in URL whenever a page omits
`remark_config.host`. The bundler cannot know that URL, so it emits
`{% REMARK_URL %}` and each distribution substitutes it: the docker image
rewrites the files under its web root at container start, and the release
binary, which serves the build embedded in itself, had nothing doing it.
`prepare-release-assets.sh` filled the marker with `http://127.0.0.1:8080`
before the embed instead, so every copy of the binary shipped pointing at
the visitor's own loopback address, and on an https site the request is
blocked as mixed content besides.

It has been that way since v1.11.0, the first release to embed the
frontend, and the earlier binaries embedded none, so the tarball has never
served a correctly addressed widget.

The placeholder now survives into the embedded copy and the file server
fills it with the configured `REMARK_URL` as it serves, which is what the
docker image already does to its own copy. The image no longer bakes the
loopback address into its embedded copy either, so the fallback it keeps
for a missing web root is correct rather than misleading.

Substituted in html, js and mjs, the same set `docker-init.sh` rewrites,
and the served size is the substituted one so a response is neither
truncated nor left hanging.

Nothing exercised the marker the frontend build emits wherever the instance
url belongs. Every page in the suite sets `remark_config.host` from its own
origin, so the compiled-in fallback is never read, and a distribution that
stopped substituting would keep the suite green.

Two tests. The first reads the served bundles and pages back and asserts the
marker is gone from each and that what replaced it is this instance. The
second covers what the substitution is for: the widget document carries no
host of its own, since `iframe.html` builds its config from a query string the
parent never puts one in, so everything it requests is addressed with the
compiled-in url. It asserts the widget renders and that the config request
went to this instance.

The demo pages cannot show the second. Their loader builds the bundle's own
script url from `remark_config.host`, so a page without one never gets as far
as loading the widget.

Verified by disabling both substitution paths, the serve-time one and the
docker image's, and rebuilding: both tests fail. Editing the files on disk is
not enough, since the file server substitutes as it serves.

The served body now depends on remarkURL, but cacheControl builds its
etag from version and path only. An operator who notices the widget is
addressed to the wrong host, corrects REMARK_URL and restarts the same
binary gets 304 on revalidation, so the client keeps a bundle pointing
at the old host. Cache-Control is no-cache, so it revalidates every time
and never ages out of that state either.

That is the exact situation this substitution exists to fix, so the
validator has to carry the url.
2026-08-22 13:15:27 -05:00
Dmitry VerkhoturovandGitHub 23be25d84a Fix seven widget defects, including the cookies the separate-domain setup needs (#2197)
* Drop the frontend workspace root and re-resolve the lockfile

`frontend/` carried a `package.json`, a `pnpm-workspace.yaml` and the lockfile
for a workspace of exactly one package. Two manifests meant two places to
declare a version, and the app pin was the one that did not win: `preact` and
`@babel/core` were each written twice, and a bump to the app manifest alone
would have been a silent no-op, since `pnpm.overrides` decides and it lived at
the root.

Everything pnpm reads now lives in `frontend/apps/remark42`: dependencies,
`packageManager`, `engines` and the overrides. `frontend/` keeps `.nvmrc`,
`.husky` and `CLAUDE.md`, none of which pnpm reads. The directory nesting
stays: every path in the repository points at `frontend/apps/remark42`,
including the published contributing docs, so moving the package up would have
rewritten 14 files to no benefit.

Moving the manifest kept the old resolutions verbatim, which left optional peer
subtrees the tree no longer reaches: `ts-node` under jest, `@swc/core` under
webpack, `vitest` under `@testing-library/jest-dom`, `tslib` under
`webpack-dev-server`. None is referenced by any config or source file here.
Re-resolving drops 137 packages and moves 59 to versions already permitted by
the ranges in the manifest, 1446 to 1308, with no direct dependency changing
version: the five that look changed differ only in their peer suffix. Every
file `pnpm build` produces is identical in size before and after.

The frontend-deps stage of the Dockerfile sets `CI=true` so the `prepare`
script skips husky, which has no git repository to install hooks into there.

* Stop markdown-only changes triggering heavy workflows, and check the documented versions

`ci-backend.yml`, `ci-build.yml` and `ci-frontend.yml` all end their path
filters with `!**.md`. The e2e workflow did not, so a change to any markdown
file under `frontend/` or `backend/` matched its `frontend/**` and `backend/**`
entries and started a docker build and the whole browser suite. The release
filter had the same hole and two of its own: it names `README.md` and `LICENSE`
on purpose, since `.goreleaser.yml` packages both, so it now excludes markdown
under `backend/` and `frontend/` only. `CLAUDE.md` and the installation page
were listed as well, and neither is packaged.

`ci-site.yml` goes on matching markdown, which is right, since the site is
built from it. It excludes `CLAUDE.md`, so a future `site/CLAUDE.md` cannot
start a site build, and `site/README.md`, which documents how to build the site
rather than being part of it.

The installation page tells a reader that a source build needs Go 1.25, Node
24+ and PNPM 10. Nothing kept those in step with `backend/go.mod`,
`engines.node`, `packageManager` and `.nvmrc`, and the drift is silent: a wrong
version in the docs builds and tests exactly as well as a right one. `.nvmrc`
is the pin with form here, having sat at 16 through the whole node 20 migration
because nothing red ever pointed at it. The check compares each stated version
against its source and holds `.nvmrc` to `engines.node`, and it fails when the
page states no version at all, so removing the claims cannot turn it into a
check that passes by comparing nothing.

Its own workflow rather than a step in an existing one, since the inputs span
the backend module, the frontend manifest and the site.

* Fix the cookie fallback page, asset path, message senders, auth teardown and cookies

Two defects with the same origin: 5825a55b, the January 2021 frontend
rewrite, first released in v1.7.0.

It removed the build entry for comments.html while leaving both the
template and the link to it in place, so the page the auth panel offers
when third-party cookies are blocked has been a 404 ever since, for
exactly the reader who has no other way in. The template needed no
changes; it is built again, and an e2e case now opens it on a thread
carrying a comment and waits for that comment, so the page being served,
its inline script running and it asking for the thread named in its own
query string are all covered. Against an image built without the plugin
entry that case fails on the 404, which is the regression it exists for.

It also fixed the public path to the domain root, so an instance mounted
under a prefix, which manuals/separate-domain documents, asked for
/web/google.svg when its own icons live under that prefix. Fifteen
provider icons in remark.mjs and one in last-comments.mjs. The path is
now derived from the url the bundle was loaded from, which is correct for
both arrangements, and the file loader no longer overrides it.

The host page also accepted postMessage from any window: every frame on a
page can reach window.parent, and the handler resizes the widget, scrolls
the page and opens the profile overlay. It now ignores anything that did
not come from a frame this module created.

A fourth, in the same family: the OAuth flow never tore its polling down.
`subscribed` was declared, checked and cleared but never set, so the guard
against a second subscription was dead code and every provider click
attached another listener pair. The five minute deadline then rejected
without unsubscribing, leaving those listeners and a retry that
reschedules itself for as long as getUser returns null. Cross-domain is
where getUser never stops returning null, so a reader on the arrangement
manuals/separate-domain documents was left polling /auth/user once a
minute for the life of the page, against a route capped at 2 req/s. It
also rejected with no argument, and the caller stores that as the error
state, so the interface had undefined to render. The deadline now tears
the subscription down and rejects with an error.

The message check had a second half. Hardening the parent left the widget
document trusting any sender, and it acts on signout and theme, so
anything holding a reference to the frame could sign a reader out.
`auth.hooks` already checked `event.source !== window.parent`; that check
is now a shared `isFromParent` and the three listeners that lacked it use
it too. The origin cannot stand in for it, since the host page is
whatever site embeds the widget and `ALLOWED_HOSTS` is enforced server
side through `frame-ancestors`.

And createInstance stacked its listeners. It reuses the marked iframe
instead of building one, but installed three listeners plus a title
observer on every call, while destroy could only reach the newest
closure, so a second call without a destroy stranded a set for good. The
listeners of the current instance are now detached before the next set
goes on. Reuse and the ignored config are unchanged: that contract is
open in the backlog note and not settled here.

The auth cookies the embedded case needs were not being delivered, in
both halves of the client's own writer. The name was decorated:
setAuthCookie prefixed with __Host- whenever the page was https, so a
real deployment wrote __Host-JWT and __Host-XSRF-TOKEN while the backend
looks for JWT and the fetcher reads XSRF-TOKEN, and nothing anywhere
reads a prefixed name. Nothing caught it because the prefix is applied
from the page protocol and every test and the dev server run on http;
there is now a second suite pinned to an https page, which is the only
condition that shows it. And the attributes could not be delivered: both
were SameSite=Strict, judged against the top-level site and not the
request's own origin, so a Strict cookie is never sent from a
third-party frame, which is the entire configuration this code exists
for. They now follow the embedding, Strict while the widget shares its
page origin and None with Secure and Partitioned once it does not, since
that is the only third-party form browsers still accept. Over http in a
third-party frame no combination works, and the strict form is written
instead of one the browser would reject outright.

That leaves the client half of #1877 working, whose reporter wanted
AUTH_SEND_JWT_HEADER for exactly this arrangement, and whose first half
merged as #1929. The server's own cookies still carry no Partitioned;
that is upstream work in go-pkgz/auth.

Two plan changes. A review pass corrected its central Path B premise,
which said the first document render is anonymous permanently, in every
configuration: it is anonymous in the configuration remark42 ships,
go-pkgz/auth exposing XSRFIgnoreMethods and remark42 leaving it unset.
The door is not shut, it is closed by a setting, and opening it is
scoped security work and not a flag flip, because GET /deleteme
deletes every comment a user has written and is a GET so the emailed
link works. And the separate-domain arrangement is promoted from a
constraint bullet to a named requirement with acceptance criteria, since
a test that signs in and posts without reloading passes while
persistence is entirely broken.

Review found a seventh, and it was reachable only because of the first:
comments.ejs built its title with innerHTML from the url query
parameter, so restoring the build entry made a reflected XSS live on the
instance origin, where the page is a top-level document, frame-ancestors
does not apply and the /web CSP allows unsafe-inline. The anchor is now
built through the DOM with textContent, and only http and https reach
href, since escaping alone leaves a javascript: url working. Two e2e
subtests pin both halves, and mutation testing separates them: restoring
innerHTML fails four assertions, while keeping the escaping and dropping
only the scheme guard fails the href one alone.

Review also found the poll teardown test did not exercise the poll.
handleWindowVisibilityChange is reachable only from the two listeners
and from the retry it schedules itself, and the test dispatched neither,
so no request was ever made and the assertion compared zero to zero; it
passed with the teardown reverted. It now dispatches focus, asserts
requests are being made and keep coming, and only then that they stop.
And the teardown could not cancel an in-flight getUser: a null resolving
after the deadline ran the code past the await and scheduled a fresh
retry with nothing left to clear it. A closure-local flag checked after
the await stops that, chosen over a second guard at the top of the
handler because only one of the two is detectable by mutation and this
is the one that prevents the stray timer rather than neutering it.

The inline handler in the iframe template accepted messages from any
window while acting on them through location.replace and document.title.
It now takes only the parent, the same check the host page side makes.
2026-08-22 12:34:24 -05:00
Dmitry VerkhoturovandGitHub 4d5dae20e2 Broaden the e2e suite from 21 cases to 63, and harden its harness (#2196)
* Pin the published /web surface in the e2e suite

#2178 renamed the widget bundles from .js to .mjs and the URLs earlier
releases served under those names stopped resolving. Three were noticed
from the demo site; the rest, including every locale chunk, were found
only by requesting the whole surface of both images over HTTP. #2192
restored them with a server-side alias, and nothing in the suite would
have caught the break or would notice it returning.

Two cases with deliberately different criteria. The documented names are
written out, because the documentation decides that list and not the
build: an operator pastes privacy.html into an OAuth application, the
nginx manual proxies index.html by name, and the integration guides start
from the embed script. Everything else is taken from the build itself, so
whatever the bundler emitted has to serve identical bytes under its
legacy .js name and parse as a classic script, which is the premise
serving one under the other rests on. A third case requests a name that
does not exist, without which a fallback serving one page for everything
would keep the whole table green.

All of them check the content type as well as the bytes: nosniff is set
on every response, so a bundle served as text/plain is as broken as one
that 404s while comparing equal.

On e3d1d0e2, the commit before the alias, this fails with 32 red subtests.

* Cover the widget behavior the e2e suite never drove

Removing npm from the widget takes the jest tests with it, and everything
here was protected by jest or by nothing at all.

Signing out was untested at every level, and the panel repainting is the
half that always works: the assertion after the reload is the one that
catches a session the server never ended. The edit form has to hand back
the source that was posted and not the rendered comment, which #2040
shipped the other way round, taking every entity and tag the author had
written with it. A draft has to survive a reload and be gone once the
comment is posted. A refused comment has to stay in the form with
something said about it, so that case drives the backend's own
restricted-words code and then retries with the route removed.

Uploads had no browser coverage in either direction. The posted case
asserts naturalWidth instead of visibility, since a broken src still
renders as an empty box, and the failing case holds the intercepted
request long enough for the in-flight state to be observed: without that,
"the text is unchanged afterwards" would hold for an upload that never
started.

Moderation and reader-side hiding are asserted from a page other than the
one that made the change. Hiding seeds a second author, so it proves one
person is hidden and the thread not emptied, and both the blocked
author and the moderated one carry the run id in their names: a block and
a verification are properties of the user and outlive the run in the
stack's database, so a fixed name works exactly once.

Locales were the largest hole. Each catalog is a chunk fetched at
runtime, and loadLocale falls back to english on any failure instead of
throwing, exactly as an unrecognized name does, so every case compares
the rendered string against the file on disk. One case per catalog
covers that they are all served and render; another fetches one through
the widget document and asserts it parsed, which is the half a chunk that
serves but fails to parse would slip through. The delete page and the
last-comments stylesheet had no coverage of any kind.

Two things the suite itself needed. The widget's own aria-label is
translated, so commentFormSel only ever finds an english widget and a
localized case cannot use widget(). And the auth probe is capped at two
requests a second for the whole suite, hard-coded in rest.go: the added
cases pushed past it and the suite began manufacturing its own 429s,
which render as a signed-out widget and fail whichever test happens to be
signing in, so the pacing gap is wider and the locale cases stub the
probe they never needed.

* Harden the e2e harness against silent failures and stale stacks

Three things the suite could not tell you about itself.

A browser failure nothing asserts on now fails the test that caused it.
Uncaught exceptions are the ones worth the machinery: a widget throwing
while it renders leaves most of these cases green, since they assert on
elements the browser lays out either way. Rate-limit responses are
recorded as well as logged for the same reason. A test driving an error
path declares what it expects by substring, so a case that means to
break something says which thing.

The stack the suite adopts is now checked against the sources under
test. Every checkout builds the image tag the compose file names, so a
stack from another worktree, or from this one before an edit, answers on
these ports and passes every readiness probe while serving code nobody
is looking at. stamp.sh digests what goes into the image, compose passes
it as the revision label, and a mismatch is refused with what to do
about it. Checked after our own build too, or a stamp that never reaches
the image would be a guard that silently passes everything.

assertSignedIn no longer waits out the whole timeout on a refused status
read. /auth/ is capped at two requests a second for the entire suite, a
bare literal at rest.go:242, and a case signing in on two pages spends
that twice; when the read that repaints the panel is the one the limiter
turns down, the widget shows signed out over a session that exists and
no later request will ask again. The short first wait now ends in a
focus handoff, which the widget answers by re-probing, and only then
does the real wait run. A sign-in that genuinely failed still fails,
since the second read finds no state either. That is what
TestComment_AdminPinsAndVerifies and TestComment_BlockedAuthorCannotPost
were failing on in CI while passing locally.

signInAnon takes the page for that reason, and its callers pass it.

* Cover iframe geometry, the embed contract and five deployment modes

Seventeen cases for the parts of the widget that broke repeatedly and
that nothing here could see, plus the areas jest was the only check on.

Geometry is the biggest of them. The widget measures its own document
and posts the number for the parent to apply, and every way that has
gone wrong is invisible to assertions about elements, which read the
same whether the frame is right, twice too tall or a strip. So: the
first height the parent is given describes rendered content and not the
preloader, the frame matches the document it holds and does not stand
24px taller, no_footer leaves the last comment inside the frame, and the
frame follows the sign-in dropdown and the growing textarea and comes
back down again. Each was verified by reintroducing the defect it covers
and watching it fail: a 63px report before the real one, six pixels of
body padding, and a frame sized under the content.

The embed surface is the other half the widget cannot see. An element
placeholder gives way to exactly one iframe carrying the embed's own
marker, a second createInstance reuses it, destroy takes it away with
its handles, and a theme change after load reaches both the element and
the document. A page that posts a message of its own, which is all
embed.ts's own title observer does, no longer empties an open login
form.

Five configurations that cannot share an instance get one each, since
each changes the widget for every reader: an admin's unlimited edit
window, a session carried in a header and not a cookie, an instance with
no auth provider to offer, anonymous voting, and the notify module,
without which email_notifications is false and the subscribe control
never renders at all. The subscription round trip is the one place a
token from a real message is exchanged for state the server keeps.

Two things surfaced there and are left alone, both said so in place. The
panel confirming an unsubscribe cannot be observed, because the click
changes the step and the dropdown closes on an element no longer in the
rerendered view, which the component notes as its own awkwardness; the
case asserts the request and the answer, which is what decides whether
the reader still gets mail. And the widget takes the subscribed state
from user.email_subscription, absent from what it hydrates the user with
on the next load, so after a reload it offers to subscribe somebody who
already is.

simple_view needs no instance, being a query parameter, so both branches
run against the main one. A transient failure of the status probe has a
case too: the session belongs to the server, and one refused answer must
not end it.

The suite runs about four and a half minutes now, so the workflow's own
budget goes to 20m to match the Makefile, well inside the job timeout,
and the workflow stamps the stack it starts the way the Makefile does.
The locale case that loads the widget document directly is renamed for
what it protects, the origin and CSP its chunks are fetched under.

Telegram gets no test: the base URL is formatted inline inside
go-pkgz/auth, so nothing here can point it elsewhere, and the fix
belongs upstream in v1 and v2 both. Reported as #2208.

Three things CI found that a laptop cannot. The anonymous sign-in form
validates its input against pattern="[\p{L}\d\s_]+", and E2E_RUN_ID is
"<run id>-<attempt>" on a runner, so every username built from it
carried a hyphen the browser refused to submit: no request was made and
the case waited out its timeout on a panel that was never going to
change. Names are built by anonName now, which drops what the pattern
does not allow and adds the pid, so a second run against a surviving
stack does not meet its own blocked and verified users. signInAnon waits
for the request the submit makes, so the next such refusal fails as
itself.

The admin instance takes its admin from an email address, not a name.
remark42 hashes an anonymous id from the name and the client address
together, to tell apart two people picking the same name, so the id
written into ADMIN_SHARED_ID belonged to nobody on a runner and the
instance had no admin at all: the countdown stayed, and the backend
refused the edit. An email id is sha1 of the address, which is the same
everywhere.

The subscription case clears its own precondition and confirms through
the page's session. The dev user is shared and a subscription outlives
the run, so the panel opened on the subscribed step; and the panel moves
to that step while its token textarea is still on screen, leaving no
moment at which the control to submit it exists.

Three settings of remark_config get cases of their own, none having had
any: __colors__, which is the one setting that travels through
window.name and not the query string, so nothing else in the suite would
notice the path going; the url override, which is how a canonical
address keeps one conversation across pages that differ; and the
subscription controls an integrator turns off, with the both-shown case
as the control.

Writing the url case turned up a backend defect, reported as #2204 and
not fixed here: a thread url containing "&" cannot be commented on at
all. Sanitize runs the locator
through SanitizeAsURL, which round-trips it through bluemonday, so the
url is stored html-escaped; the bucket is created under the escaped key,
the read-back uses the real one and answers 500, and every later find,
count and feed asks for the real url and is told the thread is empty.
Any page addressed with two query parameters is affected. The case uses
a single-parameter url for that reason.

Five more from the same audit, none needing a service. Collapsing a
thread shrinks the frame, which every other geometry case would miss:
they all assert growth, and a widget that only grew would satisfy them
while leaving a hole under each collapsed thread. Voting gains the
direction nothing covered, downvoting and its survival of a reload, and
the rule the other vote cases work around, that your own comment offers
no buttons and the backend refuses the vote anyway.

A vote with the X-XSRF-TOKEN header stripped has to be refused. That
check is why a document navigation, an iframe src among them, is always
anonymous and why the widget hydrates its user over XHR, so anything
designed around that wants it pinned.

An unrecognized locale has to render English, which is loadLocale's only
observable guarantee: it falls back the same way for a name it does not
know and for a chunk it cannot fetch. And a comment's timestamp has to
be the reader's own, which is the one part of rendering that cannot move
to the server, asserted from a context in Kiritimati against the same
Intl the widget uses.

A host page on an origin the widget is not served from, which is the
separate-domain setup the manuals describe and the configuration readers
actually hit problems with. Every other host page here is served by
remark42 itself, so the cross-site path was never taken: an nginx on its
own name and port serves e2e/hostsite, and the case asserts the frame is
revealed, which means its document loaded and reported itself inited
across the origin boundary, and that the thread it renders is the one
the page's address names. Signing in is left out on purpose, an embedded
cookie needing SameSite=None, which browsers take only as Secure, and
this stack speaks http; that is #1139 and not something a case here can
settle.

The other half is ALLOWED_HOSTS. The no-provider instance names only
itself, so a page elsewhere embedding it is refused by the browser, the
document never runs, and the reveal comes from the widget's own fallback
five seconds later. Both directions are worth holding: without the
fallback a mistyped host leaves a permanently invisible widget with
nothing to say why, and without the refusal the setting does nothing.

The host page's title reaching the stored comment gets a case, the path
running the other way from everything else here: the page posts its
title into the widget, the widget sends it with the comment, and it is
what a feed and the admin listing show. Set after the widget is up, so
it covers the observer embed.ts installs and not the value read at boot.

max_shown_comments has no case. The setting reaches the widget, appears
in the iframe's query string and changes nothing: four comments render
with it set to two, which is #812, still open, where the reproduction is
now recorded. A case for it would be red on master.

The README gains what the suite cannot reach. Every service here speaks
http, so anything the browser gates on the page protocol is invisible: a
Secure cookie, anything keyed on window.location.protocol, and the
SameSite=None with Secure and Partitioned form that is the only one an
embedded frame can still use. That is not hypothetical, setAuthCookie
having decorated its cookies with __Host- on any https page and survived
precisely because nothing here runs on one. The cross-origin case names
the assertion to add if the stack ever gets TLS, which is the reload:
the widget holds its token in memory for the life of a page, so signing
in and posting without reloading passes while persistence is broken.

And the trap waiting for whoever acts on that: playwright's own default
--disable-features argument carries ThirdPartyStoragePartitioning, and it
beats both --test-third-party-cookie-phaseout and
--block-third-party-cookies passed through Args, so a run meaning to
prove the third-party case keeps an ordinary third-party cookie exactly
as it would with no flags at all. IgnoreDefaultArgs is the lever, and a
blocking run has to assert a control before anything it reports can be
believed. None of it reaches the widget's own storage fallback either:
IS_STORAGE_AVAILABLE stays true with partitioning enforced, chromium
partitioning localStorage instead of denying it, so comments.html needs
webkit and not a flag.

The downvote case now actually corrects. It claimed the score ends where
the second vote leaves it and never cast one, so #728, a reader taking a
vote back, could break with it green. It also turns out the opposite
vote takes the first one back instead of flipping it, so the score
returns to zero and never reaches +1, which is what the case asserts,
before and after a reload. Renamed for what it covers.
2026-08-22 11:59:37 -05:00
UmputunandGitHub a82dc8d3f1 Restore the legacy /web/*.js URLs and fix iframe reuse (#2192)
* Serve the legacy /web/*.js names from their .mjs siblings

The build emitted <name>.js alongside <name>.mjs until the two compilations
were collapsed into one. Dropping the second compilation was right, but it
removed URLs the project itself had published: the v1.16.4 SPA documentation
named /web/embed.js directly and its loader snippet requested .js. Pages that
hard-coded those names now 404 with no deprecation.

webFiles.Open retries a missing .js against the .mjs sibling. The bundles
contain no import or export, so the same bytes serve both names. The retry
runs only once both sources report the name missing, so a real .js still
wins, and an unreadable sibling reports its own error rather than being
flattened into the requested file's 404.

Related to #2178

* Reuse only the comments iframe embed created

createInstance took root.firstElementChild as its iframe, so anything a page
left inside #remark42 was adopted instead. A <noscript> fallback became the
"iframe", createIframe never ran, and the height messages went to an element
that cannot show comments.

That also defeats the placeholder support, which promises content in the root
is cleared once the iframe reports inited: a text placeholder works, but any
element placeholder is mistaken for the iframe, so inited never arrives and
the cleanup never runs.

The iframe now carries data-remark42-iframe and the lookup is scoped to a
direct child, so a second createInstance still reuses it while nothing else
in the root can be adopted.

Related to #1990

* Assert the backup contents rather than the compressed size

TestBackup_MakeBackup and TestBackup_Do pinned the gzip output at 52 bytes,
which ties them to the exact output of compress/flate. The same input encodes
to 57 bytes on go 1.27, so both fail for anyone building on a toolchain newer
than the one CI pins.

They now read the backup back and compare it against what the exporter wrote,
which is what the tests were reaching for and does not move with the
compressor. The payload is a shared constant so the two cannot drift.
2026-08-22 03:09:40 -05:00
Dmitry VerkhoturovandGitHub e3d1d0e23e Create the e2e trace directory before writing a trace (#2194)
`newPageOn` writes a trace into `traces/` when a test fails, and never created
that directory. It is gitignored, so a fresh checkout does not have it.

Traces were not in fact being dropped: the driver creates the parent of the
trace path itself, checked against the version this module pins rather than
assumed. The directory is created here anyway because nothing in the suite
states or tests that dependency, and the missing directory has been raised in
review on #2180 and again on #2193, each time needing the driver checked before
it could be answered.

One visible difference on a fresh checkout: the directory now arrives at 0750
rather than the 0755 the driver's own mkdir leaves, both measured. It runs only
on a test that has already failed, and logs its error rather than swallowing
it, matching the Stop call below it.
2026-08-22 02:47:33 -05:00
Dmitry VerkhoturovandGitHub 49bf83b09c Address the review follow-ups from #2188, #2189 and #2190 (#2193)
* Read the collapsed-threads key through getJsonItem

`getFromLocalStorage` parsed the stored string directly, so anything
malformed under `__remarkCollapsed` threw out of `restoreCollapsedThreads`.
That call sits in `remark.tsx` ahead of the `render`, so the throw took the
whole widget with it: the reader was left on the preloader, over a view
preference.

`getJsonItem` in `common/local-storage.ts` already wraps a parse of a
localStorage key and returns null on failure, and null is a shape the check
below already reads as empty. The rest of that function is total against
whatever the browser holds, and the bare parse was the one way in.

* Stop retrying a failed e2e test in CI

The suite went in with one gotestsum rerun. It has no failures on record to
justify that: 31 CI runs since it landed, all green, and no rerun report has
ever been produced. A retry is what turns an intermittent regression into a
green build, and while the suite is this young its own failures are the
evidence worth keeping.

`E2E_RUN_ID` stays. It stamps the threads a run works on with the CI run id, so
a thread url in a trace or a log names the run it came from. It carries no data
across: the stack is disposable, and a local run under the same id gets those
urls on an empty database.

* Stop two chooseUnusedPort comments claiming collisions cannot happen

All four copies listen on :0, read the assigned port, close the listener
and bind later, so nothing holds the number across that gap and another
binary can take it. The copies in app/cmd and app/rest/api call a collision
very unlikely, which is accurate; the ones in app and the example module
said binaries never land on the same number, which is not, and a comment
ruling out a port collision is what would send the next person chasing one
somewhere else. All four now read the same.

Closing the window rather than describing it means the server binding :0
itself and reporting the address it got, which is a larger change.
2026-08-22 02:23:56 -05:00
Dmitry VerkhoturovandGitHub 0b651dddd4 Make backend tests wait on conditions instead of durations (#2190)
* Make backend tests wait on conditions instead of durations

The backend workflow has a long tail of runs that fail once and pass on
a rerun. Every one of them comes down to a test assuming an operation
finishes within some duration rather than waiting for the state it
needs. Three were reproducible and each was reproduced against the old
code before being changed: TestServerAuthHooks minted a token that lived
one second and never tested expiry, so a slow runner turned the first
POST into a 401; TestServerApp_AnonMode saw "connection refused" because
waitForHTTPServerStart returned silently after three seconds and left a
later assertion to fail with something unrelated; TestFsStore_Cleanup
slept 200ms against a 300ms ttl that Cleanup widens to 400ms with its
commit grace, so roughly 100ms of stall collected an image meant to
survive.

Fixed sleeps before asserting on asynchronous work are replaced with
polls on the condition itself, using require.Eventually and
require.EventuallyWithT, and require.Never where the assertion is that
something did not happen. Polling closures assert on the CollectT they
are handed rather than on t, since testify runs them on another
goroutine, and polls that issue HTTP requests stay under the rate limit
on the routes they poll through.

Where a test needs time to have passed, the clock input is pinned
instead: staging ages are stamped with os.Chtimes on both sides of the
cleanup boundary right before each call, which also makes the 100ms
commit grace an exact case rather than something no assertion reaches,
and the RSS tests set store.Comment.Timestamp explicitly rather than
racing the wall clock into the first 100ms of a second so pubDate
matches.

chooseUnusedPort takes a port from the kernel's ephemeral range. Picking
at random out of a fixed 10000-port window let two package binaries,
which go test ./... runs concurrently, land on the same number between
the probe closing and the server binding. The start helpers fail naming
the port they waited on, and the SSL tests wait on the redirect port as
well as the TLS one.

Arbitrary budgets that nothing tests are gone: ten HTTP clients with a
one-second timeout against bolt-backed import and export, the "should
take about 100msec" assertions, and a one-second bound on noticing an
already cancelled context. Shutdown stays bounded at ten seconds so a
hang is still caught.

Two assertions get stronger. TestServerAuthHooks accepted 403 or 401
from a blocked user, an alternative that existed only because the short
token could expire mid-test; it is deterministically 403 now.
TestAdmin_BlockedList asserted two users blocked while one carried the
same 150ms ttl the next step waits to lapse, so the halves raced each
other.

goleak stops reporting the regexp2 clock goroutine, which chroma pulls
in for syntax highlighting and which lives for up to a second after the
last match with a timeout; it ends on its own but a binary finishing
inside that window was reported as leaking, and this suite now finishes
sooner. The ignore for net/http.(*Server).Shutdown goes the other way:
it no longer matches anything, with both packages run fifteen times each
under CPU oversubscription to confirm.

Two gaps the change would otherwise have opened are covered directly
rather than left to the side effects that used to cover them. The
one-second token was the only thing exercising the authenticator's
ClaimsUpd hook on refresh, so TestServerApp_ClaimsUpd now calls the hook
itself and checks admin, blocked, email and restricted-name
impersonation, including the two pass-through cases. Lifting the
open-route limit removed the last incidental exercise of the rate
limiter, so TestRateLimiter drives a burst past the allowance and checks
the refusals and that the limit is per client. Both run without a wall
clock, and both were confirmed to fail when the behaviour they cover is
removed.

Production code is untouched. The two sleeps outside test code, the 429
backoff in cmd/cleanup.go and the submit poll in store/image/image.go,
are left alone: no CI failure implicates them.

Test sleeps drop from 67 to 21, all of them either inside a
testing/synctest bubble or a poll interval. The suite runs in about 22
seconds instead of 46, mostly because
TestPublic_FindCommentsCtrl_ConsistentCount no longer paces a hundred
subtests with an 80ms sleep each to stay under the open route limit. The
300s per-package budget now matches across both workflows, the race_test
target and the documented command, and CLAUDE.md records the convention.

with '#' will be ignored, and an empty message aborts the commit. # #
Date: Sat Aug 22 01:12:31 2026 +0100 # # interactive rebase in progress;
onto 7c312da1 # Last command done (1 command done): # reword deb6cbf1 #
Make backend tests wait on conditions instead of durations # Next
command to do (1 remaining command): # reword 262e6dc2 # Apply go fix
under Go 1.27 # You are currently editing a commit while rebasing branch
'fix/backend-test-flakiness' on '7c312da1'. # # Changes to be committed:
.github/workflows/release.yml # modified: CLAUDE.md # modified: Makefile
modified: backend/_example/memory_store/server/rpc_test.go # modified:
backend/app/cmd/import_test.go # modified:
backend/app/cmd/server_test.go # modified: backend/app/main_test.go #
modified: backend/app/rest/api/admin_test.go # modified:
backend/app/rest/api/middleware_test.go # modified:
backend/app/rest/api/migrator_test.go # modified:
backend/app/rest/api/rest_private_test.go # modified:
backend/app/rest/api/rest_public_test.go # modified:
backend/app/rest/api/rest_test.go # modified:
backend/app/rest/api/rss_test.go # modified:
backend/app/rest/proxy/image_test.go # modified:
backend/app/store/image/fs_store_test.go # modified:
backend/app/store/service/service_test.go # modified:
docs/backlog/api-tests-deadlock-on-macos.md #

* Apply go fix under Go 1.27

Go 1.27 extends go fix with the modernizers, so `go fix ./...` now
rewrites patterns the language has since replaced. Running it across all
three modules produces this: legacy sync/atomic calls on plain integers
become the atomic types (notify.Service.closed, image.Service.term and
submitCount, and several test counters), reverse index loops become
slices.Backward, a Split-then-index becomes strings.Cut, counted loops
become range over an int, and interface{} becomes any in the e2e suite.

The example module needed no changes. The e2e module is behind a build
tag, so it only matches with `go fix -tags e2e ./...`.

One knock-on: prealloc can see the bound of a loop once it is written as
range over an int, so the slice it feeds is now preallocated.

with '#' will be ignored, and an empty message aborts the commit. # #
Date: Sat Aug 22 01:32:09 2026 +0100 # # interactive rebase in progress;
onto 7c312da1 # Last commands done (2 commands done): # reword deb6cbf1
262e6dc2 # Apply go fix under Go 1.27 # No commands remaining. # You are
currently editing a commit while rebasing branch
'fix/backend-test-flakiness' on '7c312da1'. # # Changes to be committed:
backend/app/migrator/native.go # modified: backend/app/notify/notify.go
backend/app/rest/api/rest_private_test.go # modified:
backend/app/store/comment.go # modified:
backend/app/store/image/image.go # modified:
backend/app/store/service/service_test.go # modified:
backend/app/store/service/title_test.go # modified: e2e/e2e_test.go #
modified: e2e/widgets_test.go #
2026-08-21 22:17:44 -05:00
Dmitry VerkhoturovandGitHub b6975af63c Fix collapsed threads not restoring, and the clock skew correction (#2188)
* Fix collapsed threads not restoring, and the clock skew correction

Collapse state was kept as a flat list of `siteID_url_commentID` strings
and read back by splitting on `_`. Any underscore in the url, the site id
or the comment id made the pieces impossible to tell apart, so a page
whose url contains one lost its collapsed threads on every reload, and one
page's entries could be read or deleted as another's: `/post` matched
everything stored for `/post_2`, and a site id of `blog` matched `blog_ru`.
No separator fixes that, since every candidate can occur inside the values,
so the ids are now nested under the site and the url instead. Anything
stored in the old shape reads as empty: collapsed threads are a view
preference, and re-expanding them once is not worth a migration.

The e2e suite had been stripping underscores out of its own thread urls to
work around this, which left its collapse test unable to fail on the bug it
covers. That workaround is gone, and the test now fails without this fix.

`serverClientTimeDiff` was written in seconds and added to an epoch in
milliseconds, so the correction it exists to apply was a thousandth of the
real skew. It is now milliseconds, and named for the unit.

A response with no usable `date` used to fall back to a zero timestamp,
which already made the "skew" about twenty days and would have made it
fifty-five years once the units were right. Nothing is stored now unless
the reading is plausible, since `Date.parse` is lenient enough to turn junk
into a date and let an absurd value through the branch that parses.

The score tooltip reports controversy again when there is any. It has been
dead since the vote component was rewritten in 0e4ae6e0, which moved the
score into its own component and left the line behind commented out; the
value has been passed in and dropped ever since. Unlike the original it
stays out of the way when there is no controversy, which the backend sends
as an absent field rather than a zero.

* Drop the nested frontend dockerignore

Docker reads `.dockerignore` from the build context root only, and nothing
builds from `frontend/`: every context in the repo is the repository root,
apart from the site, which has its own. There is no Dockerfile under
`frontend/` any more either. So the file was never consulted, and both
lines it carried, `/.vscode/` and `/.idea/`, are already in the root
`.dockerignore` verbatim.
2026-08-21 22:12:19 -05:00
Dmitry VerkhoturovandGitHub 4fca268dc6 Pin staging ages in TestFsStore_Cleanup instead of sleeping (#2191)
The test slept 200ms, ran Cleanup with a 300ms TTL and then asserted the
second and third staged images survived. Cleanup collects anything older than
the TTL plus a 100ms commit grace, and the second image was already 300ms old
by then, so a runner that stalled ~100ms anywhere in the setup aged it past
the line and the assertion failed with "file on staging".

Age comes from the file's modification time, so the test now sets it with
os.Chtimes on both sides of the boundary immediately before each Cleanup call:
the image meant to be collected is backdated an hour, the ones meant to
survive are stamped at now. That leaves no window for a stall to age a file
into the wrong bucket, and drops 600ms of sleeping.

Verified by injecting a stall into the setup: 250ms reproduces the failure on
the current code, while the version here survives 2s.
2026-08-21 22:06:58 -05:00
Dmitry VerkhoturovandGitHub a0879b2336 Measure the iframe reveal budgets from inside the page (#2189)
The three reveal tests timed their budgets from before `page.Goto`, so a
slow navigation was spent against a window that belongs to the iframe. In
`TestIframe_StaysHiddenUntilTheDocumentReportsInited` that made the test
vacuous rather than flaky: on a navigation between 2.5 and 5 seconds the
loop bounding the visibility assertion had no budget left, ran zero times,
and the test passed having asserted nothing. Reproduced by delaying the
demo document by three seconds, where the assertion ran 0 times before and
runs 23 after. The timeout test had the mirror of it, with navigation
counting toward the lower bound that exists to catch a shortened fallback.

An init script now records, in the page, when the widget's iframe element
enters the document and when its visibility first flips. `create-iframe.ts`
arms its fallback a moment earlier, on the detached element, so these read
a shade short and every bound is conservative in the same direction.

Both bounds were also wider than the thing they guard. The hidden window
now runs almost to the fallback rather than half of it, and the lower bound
sits just under it rather than at three quarters, which a fallback
shortened to four seconds used to clear.

CI reruns a failing test once rather than failing the build on the first
flake. A browser suite has a floor no amount of care removes, and one flake
failing the build is what stops people trusting the suite. Once rather than
twice, because a rerun stops at the first pass and each further attempt
only widens the window where a real intermittent regression is absorbed.
What needed a rerun is written to a report and uploaded with the traces,
which are kept whether or not the job went green: a run that recovered on
the rerun is exactly the one whose evidence used to be discarded.
2026-08-21 22:05:51 -05:00
Dmitry VerkhoturovandGitHub 7c312da199 Stop the Telegram paragraph rendering with spaces in Japanese and Chinese (#2187)
`telegram-link.tsx` assembles that paragraph from five separate messages
with the anchor and the QR clause in the middle, joining them with a
hardcoded space. Japanese and Chinese do not put spaces between words, so
the assembled sentence carried them mid-clause: `通过 此链接 或扫描二维码
打开 Telegram,` separated a preposition from its object and an adverbial
phrase from its verb. The separator now comes from the locale and is
empty for `ja`, `zh` and `zh-tw`. Korean keeps its spaces, because Korean
uses them, as do Thai's phrase boundaries.

The locale is matched exactly as `loadLocale` matches it. Comparing case
insensitively would have been worse than the bug: `remark_config.locale`
is forwarded verbatim and `loadLocale` is case sensitive, so a
conventional `zh-TW` loads the English catalogue, and a lowercased
comparison would then join English words with nothing between them. The
test covers that case alongside `ja` and `en`, and fails if either the
comparison loosens or the separator stops depending on the locale.

Macedonian labelled the replies feed as comments. `subscribeByRSS.replies`
carried `Коментари`, the same value as `user.comments`, in a catalogue
whose two reply strings are both `Одговори`. That option subscribes to
`/rss/reply?user=`, which `UserReplies` documents as comments replied to
that user, so the feed is replies.

`auth.user-not-found` is removed. It reached every catalogue but could not
render: the only dynamic path to it is `messages[invalidReason]`, and
`invalidReason` comes from `getTokenInvalidReason`, which returns
`expiredToken`, `invalidToken` or null, or from a backend error string,
and the backend emits nothing matching. Catalogues go from 181 keys to
180.
2026-08-21 19:17:49 -05:00
Dmitry VerkhoturovandGitHub a5b2fe3cfc Consolidate the frontend toolchain onto babel, and ship one bundle (#2178)
Four upgrades that were finished but never merged, the compiler collapse
they enable, and the dependency sweep that follows. Direct
devDependencies go from 78 to 60 and dependencies from 10 to 9.

Three were doing the same job: `ts-loader` stripped types in webpack,
`babel-loader` did everything else, and `@swc/jest` repeated both for the
tests with its own copy of the JSX settings. Babel is the one that
survives, because the `data-testid` stripper has no equivalent elsewhere.

`ts-loader` ran `transpileOnly: true`, so it only stripped types, which
`@babel/preset-typescript` does; `fork-ts-checker-webpack-plugin` was
already what type-checks. Jest runs `babel-jest` against the same
`.babelrc.js` the bundle uses, passed as `configFile` because a
file-relative babel config does not reach the `node_modules` packages in
`transformIgnorePatterns`, and `jest.config.mjs` is plain ESM because a
`.ts` config is compiled against `tsconfig.json`, whose
`verbatimModuleSyntax` rejects ESM syntax in a file the package has not
declared as a module.

That removes `ts-loader`, `@swc/jest` and `@swc/core`. The last was
pinned to 1.2.205 from 2022 with no way forward, because newer builds
emit non-configurable exports and break `jest.spyOn` across 13 suites.

Babel compiles a file at a time with no type information, so it cannot
tell a type-only import from a real one and keeps the module. One line,
`import { boundActions } from './connected-comment'`, pulled the whole
redux store into `last-comments.mjs` and doubled it. `verbatimModuleSyntax`
and `@typescript-eslint/consistent-type-imports` mark them properly; the
statement has to be a separate `import type`, since verbatim semantics
keep an inline `import { type X }` and load the module anyway.

The legacy and modern compilations produced the same bytes. Both read the
same browserslist query, `defaults, not IE 11, not samsung 12` resolves to
chrome 109 and up, and nothing in the source needs transforming for that
set, so 28 of the 29 output pairs were byte-identical.

That made the module/nomodule switch worse than redundant: it served the
`.js` file to browsers with no ES module support, and those files carried
`??`, `?.` and class fields, so the fallback handed its own audience a
syntax error. There is now one bundle, always loaded as a module, in the
five templates and in the seven `site/` documents integrators copy from.
A production build emits 29 files rather than 58, in about 3 seconds
rather than 17. Two of those documents did not work at all beforehand:
the SPA snippet could not parse, and the subdomain example had an
unterminated string.

`@babel/core` 8 declares `^22.18 || >=24.11` and `size-limit` 13 declares
`^22.18 || ^24 || >=26`, so 20 was below the floor of two things installed
here; pnpm only warns, which is why every build passed. All seven places
the frontend pins it move together. `site/` is untouched: it builds with
yarn and eleventy and installs neither.

`eslint --print-config` before and after gives 173 active rules on an
application file against 172, and 172 on a spec file and a plain JS file
against 171. What is gone is three `flowtype` rules with no Flow here,
`no-new-object` and `no-new-symbol` whose upstream replacements are on,
`react/forbid-foreign-prop-types` with no propTypes anywhere, and, on TS
only, `no-useless-constructor`, whose typescript-eslint version is on at
error. `@babel/core` is pinned to 8 across the workspace because
`@jest/transform` and `istanbul-lib-instrument` depend on 7 outright; a
second scoped override holds `eslint-config-preact` on 7, since its
`@babel/eslint-parser` loads babel 7 syntax plugins.

`fast-async` rewrote every async function into nodent promise chains,
calls babel's `transform` synchronously, which babel 8 removed, and every
browser in the target list runs async natively. `prefresh` blew its stack
on `createContext` under babel 8 with no newer release to move to, which
compiled `intl.tsx` and `store/context.tsx` into throwing stubs, so
`pnpm dev:app` could not run the widget at all. `core-js` is not injected
now that `useBuiltIns` is gone, `postcss-custom-properties` was reached
directly although nothing declared it and resolved only through pnpm's
private hoist directory, and `cssnano` ran in both postcss chains although
`CssMinimizerPlugin` already uses it.

`pnpm lint`, `pnpm test` and `pnpm build` now work from `frontend/` as
`CLAUDE.md` and the contributing guide have always said they do; the
workspace root defined none of them.
2026-08-21 19:13:25 -05:00
Dmitry VerkhoturovandUmputun 7ee3a0da48 Tidy the example module for the testify bump
Dependabot updates `backend/` only, so the example module that replaces
it with `../../` keeps the old versions as indirect entries and the
`test examples` job fails with `go: updates to go.mod needed`.

Beyond testify itself this picks up the yaml module move, from
`gopkg.in/yaml.v3` to `go.yaml.in/yaml/v3`, and drops two indirect
entries nothing needs any more.
2026-08-21 18:44:10 -05:00
dependabot[bot]andUmputun 4aaba0fb61 chore(deps): bump github.com/stretchr/testify
Bumps the go-modules-updates group in /backend with 1 update: [github.com/stretchr/testify](https://github.com/stretchr/testify).


Updates `github.com/stretchr/testify` from 1.12.0 to 1.12.1
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-version: 1.12.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-21 18:44:10 -05:00
Dmitry VerkhoturovandGitHub fb7b6c2cdd Serve the build-independent web assets from the backend (#2181)
* Serve the build-independent web assets from the backend

`privacy.html`, `markdown-help.html` and the `400x400.jpeg` it embeds carry
no template variable, link no script or stylesheet, and are imported by
nothing in the widget. They now live in `backend/app/webassets/assets`,
embedded there, and are served under `/web` alongside the frontend build.

`/web` reads the frontend build first and falls back to them, which is what
lets an operator replace one by dropping a file into `--web-root`. That is
what `privacy.html` needs: it describes remark42.com, while the
authorization guide tells operators to hand its URL to Google and Facebook
as their own application's privacy policy.

Only a missing file falls through. An unreadable file in the web root keeps
reporting as unreadable rather than being silently replaced by the embedded
copy, and a name the filesystem rejects reports as missing rather than as a
server error, both matching what `http.Dir` did.

The dev server serves the same directory, so the Markdown help link in the
comment form resolves on the dev port as well as in production.

The two pages are served as they are written. `markdown-help.html` was
minified before, and its formatted inline stylesheet is most of its 8.5 kB;
that is 2.4 kB more over the wire, behind the hour-long cache header the
file server already sets.

Drops `copy-webpack-plugin`, which had no other pattern, and the stylelint
entries that only ever matched these files.

* Make pnpm dev:app start again

The dev server has been failing to start on two counts, so the flow the
contributing guide documents does not run at all.

`webpack-cli` 4 drives `webpack-dev-server` 5 through the argument order
of an older major, handing it the compiler where it expects the options
object. It rejects that against its schema and exits, complaining about an
unknown `_assetEmittingPreviousFiles` property, which is a field of the
compiler. `webpack-cli` 7 is the release that declares
`webpack-dev-server` 5 as a peer.

Past that, `http-proxy-middleware` resolves to 4.1.1, which no longer
accepts the two-argument call `webpack-dev-server` makes, so the `/api`
and `/auth` proxies throw on startup. It is pulled in by the security
override for CVE-2025-32996, the only override in the file with no upper
bound: `>=2.0.10` matches every later major. Bounding it to the 2.x line
keeps the fix and the API `webpack-dev-server` calls.

With both in place `pnpm dev:app` serves the widget and the pages under
`/web` on port 9000.
2026-08-21 18:43:06 -05:00
dependabot[bot]andUmputun 123b9328d9 chore(deps): bump alpine in /site in the site-image-updates group
Bumps the site-image-updates group in /site with 1 update: alpine.


Updates `alpine` from 3.22 to 3.24

---
updated-dependencies:
- dependency-name: alpine
  dependency-version: '3.24'
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: site-image-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-21 18:20:34 -05:00
dependabot[bot]andUmputun 2bfad021e3 chore(deps): bump github.com/mxschmitt/playwright-go
Bumps the go-modules-updates group in /e2e with 1 update: [github.com/mxschmitt/playwright-go](https://github.com/mxschmitt/playwright-go).


Updates `github.com/mxschmitt/playwright-go` from 0.6201.0 to 0.6201.1
- [Release notes](https://github.com/mxschmitt/playwright-go/releases)
- [Commits](https://github.com/mxschmitt/playwright-go/compare/v0.6201.0...v0.6201.1)

---
updated-dependencies:
- dependency-name: github.com/mxschmitt/playwright-go
  dependency-version: 0.6201.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-21 18:20:27 -05:00
Dmitry VerkhoturovandGitHub 4c9ef37cf1 Move the site from eleventy to hugo (#2179)
* Move the site from eleventy to hugo

The site is built by a single static binary. No node, no package manager,
no lockfile, and the toolchain it needed is gone: eleventy, tailwind,
postcss, markdown-it and its three plugins, date-fns, prism, npm-run-all,
cross-env and html-minifier-terser.

Hugo covers most of that itself. Chroma replaces prism, goldmark replaces
markdown-it, `--minify` replaces html-minifier-terser, and fingerprinted
asset URLs replace the cache-busting `version` shortcode that stamped
`Date.now()` into every stylesheet link.

`assets/styles.css` is hand-written, since tailwind was the only reason
left to keep a package manager. The palette and the light and dark values
are custom properties at the top of the file; the minified stylesheet is
15 kB against tailwind's 46 kB, and the whole build 1.0 MB against 1.2 MB.
It was matched to the old one by comparing computed styles rather than by
eye, which is how the heading weights and line heights, the list marker
colour, and the home page heading and sign-off were caught: the last of
those had been carried by tailwind utilities written into the markup.

The `::: note` container becomes a `note` shortcode taking the emoji to
show. Its closer needs a blank line after it, because a shortcode is not
a block rule the way `markdown-it-container` was, and without one goldmark
keeps the callout inside the open paragraph. The `overflow-x` wrapper
around tables and the heading anchors are goldmark render hooks.

Syntax guessing is off. Chroma detected a systemd unit file as gdscript
and a chat transcript as mysql, and colouring a snippet as the wrong
language is worse than not colouring it. The two chroma themes are scoped
to opposite sides of the theme switch rather than layered, because they do
not declare the same properties on the same tokens: github gives Error a
background github-dark never overrides, and styles Punctuation where
github-dark leaves it alone. Layered, either leaves a light value applying
on a dark page.

`[frontmatter] lastmod` resolves through git, then front matter, then file
modification time. Without that chain `.Lastmod` falls back to `.Date`,
which is zero when a page carries no date, and every page reads
`Jan 01, 0001`. `enableGitInfo` is off because the image build context is
`site/` alone, where hugo fails hard rather than degrading;
`HUGO_ENABLEGITINFO=true` gives real per-page commit dates locally.

Three fixes fall out of the move rather than being sought:

- `/docs/` redirected nowhere. The stub was a markdown file whose
  permalink was a template expression while `markdownTemplateEngine` was
  false, so it never rendered and the URL 404'd. It is an alias now
- `/docs/contributing/` pointed at `/docs/contributing/development/`,
  which has never existed. It points at the backend page
- the 404 page was built to `/404/` and nothing served it. Hugo writes it
  to `/404.html` and reproxy is told to use it

The mobile documentation menu is a checkbox and label. `visibility: hidden`
on the checkbox, which is what the old `invisible` utility set, takes it
out of the tab order, and a label is not focusable on its own, so the menu
could not be opened from the keyboard at all. The checkbox is clipped
rather than hidden, and its label shows a focus ring.

Content is unchanged. Every code block on every page is byte-identical to
the eleventy output; the only prose difference is that two example values,
`mysite.com` and a quoted `https://demo.remark42.com`, are no longer
turned into links, goldmark's linkify being narrower than markdown-it's.

`backend/README.md` and `frontend/apps/remark42/README.md` are symlinks
into the docs tree and follow it to `site/content/`, as does the path
`release.yml` watches. `frontend/CLAUDE.md` described the site as a node
and yarn project in four places.

* Keep the heading anchors markdown-it generated

Goldmark strips punctuation markdown-it kept, so 22 headings holding a
dot, slash, apostrophe, question mark, bracket or em dash would take a new
id and any link into one from outside the repository would stop resolving.

Those headings carry their previous id as well, as an empty target emitted
ahead of the heading by the render hook, from a map of content path to old
anchor in `data/anchor_aliases.json`. The map was built by matching
heading text between the two builds rather than by position, so it
survives a heading being added or moved.

The hook rather than markdown, because goldmark's `{#id}` attribute syntax
cannot express these: it accepts dots, apostrophes and em dashes but
treats a slash, a question mark, a bracket or a percent sign as heading
text, which is 11 of the 22. The ids are stored percent-decoded, since a
browser decodes a fragment before matching, so `#children%E2%80%99s-privacy`
finds `children’s-privacy`. Verified by navigating to the awkward ones
against the built image and measuring where the page settles: each lands
112px down, which is the header offset the target carries.

Three pages carried no title, so the docs template rendered an empty `<h1>`
above the heading their markdown already had. They take their titles from
that heading text, so neither the wording nor its anchor changes, and the
template's `<h1>` carries an id. One in-page link pointed at an anchor
goldmark no longer generates.

The heading render hook emits no permalink anchor. The one it replaced was
an empty `<a href>` with `pointer-events: none`, so it could not be
clicked, and its only job was a `::before` spacer that `scroll-margin-top`
on the heading already does. Being an `<a href>` it stayed in the tab
order, so every heading was an unexplained keyboard stop: eight on the
installation page alone. Fragment navigation still lands 112px down, clear
of the fixed header.

* Harden the site image build and its CI

The architecture guard could not fire. `${TARGETARCH:-amd64}` defaulted
before the `unsupported arch` branch was reachable, so a build without
buildkit put an amd64 hugo inside an aarch64 image and ran only because
Docker Desktop emulates it. Reproduced with `--build-arg TARGETARCH=`:
`/etc/apk/arch` reported aarch64 and `hugo version` linux/amd64. An empty
value is an error now. `Dockerfile.dev` had the same defect and no smoke
step to catch it, so it would have failed at `compose up`.

The hugo tarball is verified against the release's own `checksums.txt`,
and the match is asserted present before it is used: piping grep straight
into `sha256sum -c` left the guarantee resting on what the checker does
with empty input. Busybox exits 1 there, so it did fail closed, but
nothing in the line said so. Verified against a checksums file that does
not list the tarball: the build stops before the install.

Hugo exits 0 on an empty content tree and emits a two-page shell, which
would have been copied, pushed and deployed. The build asserts the home
page and a docs page exist.

`site/**` pull requests were never built. The only building job is gated
on `github.ref == 'refs/heads/master'`, so on a pull request every job
skipped and rendered in the checks list the same way a pass does, and the
image was first built on the run that also deploys it. A `validate` job
builds it with `push: false`, needing no secrets so it works on a fork.

`.github/dependabot.yml` watched `/site` for npm packages that are gone.
That entry is a docker one, which tracks the alpine base. It does not
track the hugo pin and cannot: the docker ecosystem reads `FROM`
references, and `ARG HUGO_VERSION` is a bare string in a download URL, so
that one is a manual bump and `site/README.md` says so.

`Dockerfile.dev` carries a `COPY`, so the dev image works without the
compose bind mount, and compose runs as the invoking user rather than
root, which on linux left root-owned `public/` and `resources/` in the
checkout.

Recorded in the backlog: `master` has `required_status_checks` off with an
empty check list, so the new job surfaces a red X and does not block a
merge. That is a settings decision rather than a code fix.
2026-08-21 18:05:56 -05:00
Dmitry VerkhoturovandGitHub ff77f41a3a Move the e2e suite to Go and playwright-go (#2180)
* Move the e2e suite to Go and playwright-go

The seven playwright tests in `frontend/e2e` become twenty in `e2e/`, a
separate Go module driving the same browsers through playwright-go. The
npm project, its lockfile entries, its prettier config and
`Dockerfile.e2e` go with it, leaving `frontend/` a single-member
workspace.

The suite covers posting with markdown, replying and the nesting that
implies, editing inside the deadline and the backend refusing one outside
it, deleting, voting with the optimistic score observed mid-flight and
rolled back on failure, changing the sort, collapse persistence across a
reload, dev, anonymous and email sign-in end to end, the profile iframe,
and the two scripts that render into the host page rather than the
widget's own frame.

The rendering tests run in chromium, firefox and webkit. The rest sign in,
sign-in needs the dev oauth2 provider, and reaching that by name from the
host is chromium-only, so they run there alone.

`compose-e2e-test.yml` runs remark42, a second instance with a short edit
window so that path does not need a five-minute test, and mailpit, which
catches the email verification message the suite reads back. Everything
binds to the loopback interface: the stack holds a known secret and an
admin shared id, and `go test` can start it unattended. The tests run on
the host rather than in a container.

Three settings there exist for the tests rather than for realism.
`REMARK_URL` uses a hostname because the dev oauth2 server binds whatever
host it reads out of it, and a loopback bind inside a container cannot be
published. `UPDATE_LIMIT` is raised because the default of 0.5/sec rejects
any test posting twice in a row. The suite also paces its own `/auth/`
calls, which are capped at 2/sec by a bare literal in `rest.go` rather
than by a setting.

Each test gets its own comment thread from a query string on the demo
page, so nothing has to reset the database between runs.

CI gains a vet and lint job for the module, since the build tag keeps it
out of a plain `go test ./...`, and uploads a browser trace for any test
that fails.

`e2e/README.md` carries the rest: how to run it, what the stack is for,
and the widget behaviour the assertions have to work around.

* Update golangci-lint to 2.13.1 in the backend workflow

The pin sat three minors behind what the linter installs locally, so CI
checked the backend with an older set of rules than anyone running it by
hand. 2.10.1 also fetches its config schema over the network on every
`config verify`, which is a failure mode with no bearing on the code.

Both targets are clean on 2.13.1, `backend/app` and the memory_store
example.
2026-08-21 17:53:12 -05:00
Dmitry VerkhoturovandGitHub 1bb002348a Complete and correct every translation catalogue (#2177)
* Fix wrong and missing translations across 17 locales

`errors.8` is `ErrReadOnly` (`backend/app/rest/httperrors.go:29`), but 13
catalogues carried a copy of `errors.7`, which is `ErrUserBlocked`. A
reader who simply hit a read-only thread was told they had been blocked,
in Belarusian, Bulgarian, Brazilian Portuguese, German, Finnish, French,
Japanese, Polish, Russian, Turkish, Ukrainian, Vietnamese and Simplified
Chinese. Each now says the page is read-only, in the terminology that
catalogue already uses for its read-only badge.

Czech had an off-by-one: `errors.19`, restricted words, carried the text
of `errors.18`, file not found, and `errors.18` was left in English. So a
comment caught by the word filter reported a missing file. Both rewritten.

Also corrected, all of the same class:

- `de` `vote.downvote` had a leading space
- `mk` had dropped `{shortcut}` from the bold, italic and link tooltips,
  losing the keyboard hints
- `fi` `comment.pin` was "Sitoo", which means "it binds", and `comment.unpin`
  followed from it; `errors.forbidden` was misspelled "Kieletty."
- `be` `errors.failed-fetch` ended in a stray "r"
- `zh-tw` `authPanel.read-only` wrote 唯獨 for 唯讀
- `ja` `errors.conflict` used 相衝, which is not Japanese usage
- `it` `auth.symbols-restriction` misspelled "numberi" and used "username"
  where `auth.username` now says "Nome utente"

Nine Finnish strings, `it` `auth.username` and `zh-tw` `comment.time` were
left in English; `comment.time` is a format string and now matches the
other CJK locales at `{day} {time}`.

No English source string changes, and every runtime placeholder is
preserved. Reviewed by two independent passes, which between them reworded
five of these and found four of the pre-existing defects above.

* Complete every translation catalogue

No locale carries English text any more, and the Telegram authorisation
paragraph now reads as a sentence in all 23 of them.

That paragraph is assembled in `telegram-link.tsx` from five separate
keys with the link and the QR clause in the middle, so a catalogue that
translates each key in isolation produces word salad in any language
whose verb does not sit where English puts it. Japanese rendered as
"テレグラムを開く リンクで または QR コードをスキャン そこで..." and Korean, Persian,
Traditional Chinese and Czech had the same break. Each of those now
splits the sentence at the point its own grammar wants, and every
catalogue was checked by rendering both the wide and the narrow layout,
since the QR clause only appears above 768px.

Ten catalogues also had `auth.telegram-link`, which is the anchor text
and reads "by the link", left as "Telegram bot" from an older source
wording, so the paragraph named the bot twice and never said what the
link was for. Six of the block's keys had never been translated at all
in the 16 locales that ship it, and `auth.telegram-message-1/2/3` were
English in nine. None of these were visible to a check for "value equals
the English string", because none of them equalled it.

Two of the six untranslated keys are word for word the English of their
`subscribeByEmail` siblings, so each catalogue's own existing wording was
reused rather than a second phrasing invented for the same sentence.

Quotation marks in that paragraph now follow each language rather than
the English source: «» for be, ua, ru, fa, fr and ar, „“ for bg, cs, de
and mk, „” for pl and ro, ”” for fi, 「」 for ja and zh-tw.

The button the paragraph tells the reader to press is Telegram's own, and
Telegram ships no interface translation for Japanese, Thai or Vietnamese,
so those three now name it the way Russian and Traditional Chinese
already did, with the Latin label alongside the translated one.

Also swept and fixed: Vietnamese "Bằng đã huỷ đăng kí" for "Bạn", Spanish
"ó" for "o", Thai "คลิ๊ก" for "คลิก", an unclosed quotation mark in Arabic
`commentForm.upload-file-fail`, German alternating between tippen and
klicken for the same action, Czech infinitive "Otevřít" where the rest of
the paragraph is imperative, Macedonian "СО ЛИНК" in caps, French
"Sélectionner" on a button the text calls "Vérifier", a missing space
after a full stop in `ua` `errors.9`, and double spaces in `mk`, `pl` and
`vi`.

Left identical to English on purpose, because the word is the same in
that language: `RSS` and `Telegram` everywhere, "Email" in be, it, pl,
ro, ua and vi, "Site" in bp, fr and ro, and "Conflict." in ro. German and
Turkish do not use bare "Site" and say "Website" and "Web sitesi".
2026-08-21 17:42:41 -05:00
Dmitry VerkhoturovandGitHub fc4e10573c Replace react-intl and remove React from the widget (#2176)
Second and final step of #2166. `react-intl` is replaced by
`app/common/intl.tsx`, a small i18n binding over Preact context, and with
`react-redux` already gone nothing holds the React compatibility alias.

React is now absent from the lockfile, the installed tree, the config and
the bundles: `react`, `react-dom`, `react-intl`, `@types/react`,
`@preact/compat`, `use-sync-external-store` and `intl-messageformat` are
all gone, along with the `paths` entries in `tsconfig.json` and three
babel-loader excludes. Runtime dependencies go from 15 to 10.

`preact/compat` goes too, which matters more than its 3.8 kB. Importing
it anywhere installs hooks on preact's shared `options` that remap
`onFocus`/`onBlur` to `focusin`/`focusout` for every element and make
`@testing-library/preact` rewrite `change` to `input`, the two bugs
behind #2166, still live until now. `Button` was wrapped in `forwardRef`
with no caller passing one, and `TextareaAutosize` now takes its ref as
an ordinary prop. The workaround in `sort-picker.spec.tsx` is gone with
them, since `fireEvent.change` reaches a `<select>` again.

Gzipped, against master: `remark.mjs` 76.17 kB to 56.47, `last-comments.mjs`
37.97 to 18.26, `deleteme.mjs` 14.51 to 8.42. The limits move with them and
keep more relative headroom than master shipped.

### The binding

`IntlProvider`, `useIntl`, `createIntl`, `defineMessages`,
`FormattedMessage` and `IntlShape`. 32 files change only their import.

The export names copy react-intl's deliberately: `formatjs extract` finds
messages by recognising `defineMessages`, `FormattedMessage` and
`intl.formatMessage` in the AST rather than by import source, so renaming
one silently empties the catalogue. `frontend/CLAUDE.md` records that,
along with the destructive part: `translation:generate` would then strip
the unextracted keys from all 24 catalogues and the check would pass.

A message the binding cannot parse falls back to the message in the
source: a broken, unhandled or nested tag, a brace that is not a
well-formed placeholder, and a placeholder naming a value the caller did
not supply. `mk.json` and `th.json` carried broken markup and rendered in
English; both are repaired, so a catalogue sweep over every locale can now
require well-formed markup with no exceptions listed.

`translation:check` gained the validation that would have caught them when
they were proposed: a translation's tags have to be well-formed pairs of the
names the English string uses, with no attributes, and its placeholders have
to be ones the English string provides. Leaving a tag or a placeholder out
stays allowed. Run against master's catalogues it reports both.

### enzyme

`@types/enzyme` was the last thing pulling `@types/react`, so React could
not leave while enzyme stayed. Its three test files move to
`@testing-library/preact`, which now has no rival: `@testing-library/preact-hooks`
had one import left and its own unmet peer warning. `intersection-observer`
was a runtime dependency nothing imported, and the `cheerio` override lost
its last dependent with enzyme.

Enzyme's `.find(X).prop()` threw unless exactly one node matched, so the
converted tests assert node counts explicitly to keep that.

### Verified

All 181 message ids formatted across all 24 catalogues through both real
react-intl and this binding: 4344 comparisons, no differences. From a wiped
`node_modules`: `pnpm install --frozen-lockfile`, `pnpm lint`,
`pnpm type-check`, `pnpm test` (392 tests, 42 suites), `pnpm build`,
`pnpm size-check`, `pnpm translation-check`.
2026-08-21 02:30:26 -05:00
Dmitry VerkhoturovandGitHub a91e322d5c Replace react-redux with a preact context binding (#2175)
* Replace react-redux with a preact context binding

One of the two packages holding the @preact/compat alias in place, and
the contained one: the store is plain redux, and the only react-redux
import inside it was a single line re-exporting typed hooks.

* Drop the now-unused react-redux types

* Subscribe before paint and check once on subscribe

Previously, useSelector subscribed to the store inside useEffect, which
runs after paint. A dispatch landing between render and that effect was
never delivered, since the listener did not exist yet, so the component
kept rendering a stale value until some later unrelated dispatch happened
to differ from the stale ref.

Subscribing in useLayoutEffect narrows the window to before paint, and
running the check once immediately on subscribe closes it, which is what
react-redux does for the same reason.

Adds the first tests for the binding, one of which fails without this
change: the store holds 1 while the DOM still shows 0.

* Only re-check on subscribe when the state actually moved

The subscribe-time check ran unconditionally, so it re-ran the selector
at mount. A selector building a fresh object fails Object.is against the
value the render already computed, which forced a second render of every
connected component: ConnectedRoot and every ConnectedComment, so around
201 extra renders for a 200-comment thread.

Reducers return a new root object on every change, so an unchanged state
reference means no dispatch was missed and the check has nothing to find.
Comparing against the state the render used keeps the property the check
exists for while dropping the extra render.

The race test still exercises the guarded path, since its dispatch
produces a new state object, and a new test pins the mount case: it fails
without the guard.

Raised by umputun in review.
2026-08-21 00:47:08 -05:00
Dmitry VerkhoturovandUmputun 931f2db4e3 Drop turbo
CI never invoked it, and after #2172 removed the four api scripts its
only remaining job was orchestrating one script in one package.
2026-08-20 18:12:34 -05:00
Dmitry VerkhoturovandGitHub b8f6dc5f91 Require node 20 and record every place the version is pinned (#2168)
* Require node 20 and record every place the version is pinned

The declared floor was >=18 while CI, Docker and both .nvmrc files had
been on 20 since the pnpm 8 to 10 migration, and transitive dependencies
now require 20.18.1. The docs had drifted further still, telling
contributors to install Node 16 and PNPM 8.

* Set the node floor to the strictest dependency and keep one checklist

undici needs >=20.18.1, so a bare >=20 advertised support for 20.0 to
20.18.0, which fail dependency engine checks. frontend/CLAUDE.md already
carried a pinning checklist, so the new entries fold into it rather than
starting a rival list in the root file.

* Keep the node floor at the major, not a patch version

engines.node states the major we support. Individual dev dependencies
can be stricter within it, and chasing those patch floors into engines
and the docs would turn every lockfile refresh into a docs change.
2026-08-20 18:12:30 -05:00
Dmitry VerkhoturovandGitHub b03dc366f9 Update preact to 10.29.8 (#2163)
* Update preact to 10.29.8

Also moves TypeScript to 5.9, which preact 10.29 typings require, and the
compat and testing library pins that go with it. Type checking resolves
JSX from preact via the automatic runtime; the bundle keeps the classic
transform so babel still strips test ids.

* Move babel to the automatic JSX runtime and refresh frontend notes

Leaving babel on the classic h pragma while tsconfig used the automatic
runtime meant a tsx file without an h import would type-check and lint
clean, then throw at runtime, since eslint-config-preact disables
react/react-in-jsx-scope and no-undef is off.

* Address review findings on the preact upgrade

Forward the textarea ref with useImperativeHandle so it clears on unmount
and lands during commit rather than after paint. Pair typescript-eslint
with the TypeScript it now has to parse. Use the preact namespace types
rather than the deprecated JSX aliases, and drop the redundant type
re-declarations the element-specific interfaces already provide.

* Drive the focus tests through real DOM focus and blur

Dispatching a synthetic focusin hard-coded preact/compat's internal
alias for onFocus. Calling focus() and blur() exercises the sequence a
browser produces and stays correct if that mapping changes.

* Raise the two bundle limits the preact upgrade pushes past

CI measures remark.mjs at 78024 bytes against a limit size-limit reads as
78000, so it failed by 24. last-comments.mjs had 36 bytes of headroom and
would have tripped on the next change.

* Regenerate the lockfile after the rebase

The rebase resolution left it missing the @typescript-eslint entries, so
every CI job failed at pnpm install --frozen-lockfile.
2026-08-20 02:31:43 -05:00
Umputun 36062de0e7 docs: drop the npm deprecation backlog item
remark42 deferred work belongs in the pull request response where paskal and
akellbl4 will see it, not in a file.
2026-08-20 02:31:15 -05:00
Umputun 90766d6637 ci: add umputun as a frontend code owner
frontend/* required @akellbl4 or @Mavrin, so umputun could not satisfy the
code-owner rule on any frontend pull request. #2172 needed an admin override
and #2163 could not use one, because GitHub routes stacked pull requests
through the async merge endpoint, which applies no override.
2026-08-20 02:31:15 -05:00
Umputun a1dbb2cb92 ci: run frontend checks on any frontend change
The path filter matched only frontend/apps/remark42/**, so a change to the
workspace root ran nothing: no lint, type-check, tests or size-limit, and no
docker build either since docker.yml waits on this workflow by name.

#2160 rewrote pnpm-lock.yaml and the override block, and #2172 removed a
workspace package and its CI workflow. Neither ran a single frontend check on
its PR or on master.
2026-08-19 23:53:52 -05:00
Dmitry VerkhoturovandGitHub d370b78613 Drop the @remark42/api package (#2172)
* Drop the @remark42/api package

It cannot authenticate anyone: clients/auth.ts exposes only anonymous,
email and telegram, with no OAuth method, and the fetcher never sets
credentials so its cookie auth cannot work cross-origin. Nothing in the
repo consumes it, no third-party consumer exists, and npm has served an
alpha from July 2022 that CI never publishes.

* Drop the removed workflow from the pnpm pinning checklist

frontend/CLAUDE.md still counted ci-frontend-api.yml among the places the
pnpm version is pinned, and stated a fixed total that no longer holds.
2026-08-19 23:28:52 -05:00
Umputun 439ccfa83c docs: note @remark42/api is still published and undeprecated on npm 2026-08-19 23:19:15 -05:00
Dmitry VerkhoturovandUmputun 29627f4bf0 Raise site resolution floors to clear remaining advisories
Both floors are bounded on the upper side, as an open-ended resolution
lets yarn cross a major version.
2026-08-19 03:39:33 -05:00
Dmitry VerkhoturovandUmputun 164eb89c60 Raise pnpm override floors to clear all frontend advisories
All 23 open Dependabot alerts against frontend/pnpm-lock.yaml resolve to
packages whose override floor sat below the patched release. Every floor
now carries an explicit upper bound, as an open-ended floor lets pnpm
resolve across a major version.
2026-08-19 03:39:25 -05:00
Dmitry VerkhoturovandUmputun 09110c792f Bump backend Go modules to latest
Updates every backend dependency with a newer release available, and
tidies the example module alongside as any change to backend/go.mod
requires.
2026-08-19 03:39:11 -05:00
3f5b3cdd98 feat: add configurable SMTP HELO hostname (#2146)
* feat: add configurable SMTP HELO hostname

Allow the SMTP HELO/EHLO hostname to be configured separately from
the SMTP server hostname.

This is useful when the SMTP server requires clients to identify
themselves with a fully qualified hostname different from the server
address.

* chore: remove vendored dependency changes

* Bump go-pkgz/notify to v1.4.0 and document SMTP_HELO_HOST

The HELOHost field lands in go-pkgz/notify v1.4.0, so the branch needs the
bump to compile; v1.3.0 in master has no such field. The example module is
tidied alongside, as any change to backend/go.mod requires.

Documents the parameter in the parameters table and, separately, in the email
setup page: what it does, that leaving it unset keeps the previous `localhost`
greeting, and the case it exists for, a relay refusing the greeting under
Postfix `reject_non_fqdn_helo_hostname`.

Also records the current limit: verification emails for email authentication
go through go-pkgz/auth's own sender, which has no equivalent setting, so the
greeting there is unchanged.

* Bump go-pkgz/auth to v2.2.0 and apply SMTP_HELO_HOST to verification email

The verification email sender had no way to set the greeting, so a relay that
refuses the HELO would accept notifications and still reject sign-in emails.
EmailParams gains HELOHost in go-pkgz/auth v2.2.0, so the same SMTP_HELO_HOST
now drives both paths.

The example module is tidied alongside, as any change to backend/go.mod
requires.

---------

Co-authored-by: oli <someone@somewhere.tld>
Co-authored-by: Dmitry Verkhoturov <paskal.07@gmail.com>
2026-08-19 02:52:39 -05:00
dependabot[bot]andUmputun 43fccf3bc9 chore(deps): bump the github-actions-updates group across 1 directory with 3 updates
Bumps the github-actions-updates group with 3 updates in the / directory: [actions/setup-go](https://github.com/actions/setup-go), [pnpm/action-setup](https://github.com/pnpm/action-setup) and [actions/setup-node](https://github.com/actions/setup-node).


Updates `actions/setup-go` from 6 to 7
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v6...v7)

Updates `pnpm/action-setup` from 6.0.9 to 6.0.10
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v6.0.9...v6.0.10)

Updates `actions/setup-node` from 6 to 7
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/setup-node
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 6.0.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-19 00:49:19 -05:00
Dmitry VerkhoturovandGitHub 1f34984dab Bump go-pkgz/rest to v1.24.0 and opt in to wildcard origins with credentials (#2157)
rest.CORS refuses "*" together with credentials since go-pkgz/rest#52, so the
bump and the option have to land together: the option does not exist in v1.22.0
and the panic fires at construction, inside routes(), which makes it a startup
failure rather than a request-time one.

The wildcard stays. The comment widget is embedded on arbitrary third-party
sites, so the set of origins is not knowable, which is why the escape hatch was
asked for upstream instead of accepting the panic. What it costs is unchanged
and now written next to the call: any site a signed-in user visits can read
authenticated responses, so state-changing requests have to keep being protected
by something other than the origin, X-XSRF-Token today.

The example module is tidied in the same commit, as it reaches go-pkgz/rest
through the replace directive and its indirect graph would otherwise keep the
old pin and fail the readonly module check in CI.

The bump also carries testify to v1.12.0, which drops go-spew and go-difflib
from the module graph.
2026-08-19 00:33:13 -05:00
Dmitry VerkhoturovandGitHub 455d770899 ci: track the latest Go 1.25 patch instead of pinning one (#2156)
govulncheck fails on master against the 1.25.12 pin with seven stdlib
advisories, all fixed in 1.25.13: GO-2026-5026, GO-2026-5972, GO-2026-6088,
GO-2026-6089, GO-2026-6090, GO-2026-6091 and GO-2026-6218, across crypto/tls,
encoding/asn1, encoding/xml, html/template, net/http and net/url.

Pinning the next patch would only move the problem to the following advisory,
as it did in 76d0cc2c. setup-go accepts a minor-only spec, so "1.25" resolves
to a patch on that line at run time. Staying on 1.25 rather than "stable"
keeps a move to a new minor a deliberate change, matching the `go 1.25.0`
directive in both go.mod files.

check-latest is required with it: by default setup-go uses the patch already
cached on the runner image, so a minor-only spec alone would keep resolving
to whatever that image ships, currently 1.25.12, and the scan would stay red.
2026-08-19 00:31:22 -05:00
Dmitry VerkhoturovandUmputun bf67c251c5 docs(claude): require an example tidy on any go.mod change
Previously the rule was scoped to "updating Go modules", which reads as
version bumps only. Adding or removing a dependency, or changing the `go`
directive, puts the example module out of step in exactly the same way, and
the failure is the same `test examples` step reporting "updates to go.mod
needed".

Also records that Dependabot Go module PRs need this, since the bot updates
`backend/` alone.
2026-08-18 20:27:21 -05:00
Umputun cebba4cee4 docs: add backlog item for the macOS api test deadlock 2026-08-18 20:24:30 -05:00
Umputun 35a389cb75 fix: bump golang.org/x/image to v0.45.0 for GO-2026-6222
Excessive memory allocation during VP8L decoding, reachable from
app/store/image/image.go:333 where image.Decode runs on uploaded data.
The existing DecodeConfig dimension guard doesn't cover it, since the
over-allocation happens during decode rather than from declared dimensions.

Tidies backend/_example/memory_store in the same commit: it carries the
backend's deps as indirect entries and would otherwise fail the example CI step.
2026-08-18 20:24:30 -05:00
Umputun a725d990ed docs: add backlog item for the CORS wildcard credentials opt-in 2026-08-18 18:58:07 -05:00
Dmitry VerkhoturovandUmputun fdfce6495c Remove the widget body padding and the surplus reported height
Previously the widget document had `padding: 6px` on the body, so every
embedded widget sat 6px inside its container and could not align flush with
the host layout. `updateIframeHeight` then reported
`document.body.offsetHeight + 12`, but the body is `box-sizing: border-box`
and `offsetHeight` already includes padding, so the addition double-counted
it.

Measured against the deployed widget: the content needs 20610px, the body
reported 20622px with the padding, and the parent was told 20634px, leaving
24px of empty space below every embed on top of the horizontal inset.

Removing the padding does not clip anything. With it at zero, offsetHeight,
body scrollHeight and documentElement scrollHeight all agree, and the last
child carries no bottom margin, so no margin collapses through the body edge.

Resolves #1487.
2026-08-18 18:46:05 -05:00
Dmitry VerkhoturovandUmputun 8801903d01 Derive host from the page URL on self-served pages
Previously the pages Remark42 serves from /web/ carried a build-time host.
The `{% REMARK_URL %}` placeholder is substituted during the image and
release-asset builds, both of which write `http://127.0.0.1:8080`. Docker
rewrites it again at container start from REMARK_URL, but a release binary
has no equivalent step, so it serves demo, counter, last-comments and
deleteme pages pointing at the visitor's own loopback address. `counter.ejs`
additionally had that address hardcoded in two "note" links, which no
substitution touched.

These pages are served by Remark42 itself, so the host is whatever origin and
path prefix delivered them. Deriving it from `location` is correct at the root
and under a path prefix alike, and needs no build-time value. Sibling links
are now relative for the same reason.

`site_id` is left as a literal so the startup substitution in docker-init.sh
keeps matching it.

Reported by @andreas-hempel.

Resolves #1996.
2026-08-18 18:45:59 -05:00
Dmitry VerkhoturovandUmputun 8bcfd9e456 Close the login dropdown only on a genuine outside click
Previously any message reaching the widget closed the Sign In dropdown,
because the handler returned early only for a clickOutside payload while
closing was disabled and fell through to closing in every other case. The
embedding page posts hash, title and theme messages of its own, and
`embed.ts` installs a MutationObserver on the host page title that posts on
every mutation, so a host page whose title changes closes an open login form
and discards whatever was typed into it. Browser extensions that post into
the page have the same effect.

After this change the dropdown closes only for a clickOutside payload from
`window.parent`. Reproduced against the deployed demo: with the form open and
filled, a single `document.title` assignment on the host page removed it,
while three seconds of inactivity did not.

Resolves #2139.
2026-08-18 18:45:54 -05:00
Dmitry VerkhoturovandUmputun 5b37a583ce Stop Dependabot npm updates, including security updates
Previously the npm entries carried only open-pull-requests-limit: 0, which
bounds version updates and leaves security updates unlimited, so npm pull
requests kept arriving from Dependabot alerts. The ignore option applies to
both kinds, so a blanket ignore per npm entry is what actually stops them.

Go modules and GitHub Actions updates are unchanged.
2026-08-18 18:45:49 -05:00
Dmitry VerkhoturovandUmputun 29b5f88a1c Document Microsoft supported account types, drop deprecated Twitter example
Previously the Microsoft setup instructions did not mention supported
account types. Remark42 authenticates against the `common` endpoint by
default, which only accepts an application registered for both work or
school accounts and personal Microsoft accounts, so an application created
with any other value fails to authenticate with no hint as to why.

The reproxy manual still configured `AUTH_TWITTER_CID` and
`AUTH_TWITTER_CSEC` in its example, which have been deprecated and
non-functional since 1.14.0.

Resolves #1823.
2026-08-18 18:24:36 -05:00
Umputun 287aef4dfb docs: add backlog items for site PR validation and frontend js-yaml overrides
site/** pull requests get no build validation: ci-site.yml declares a
pull_request trigger but gates its only build job to master and tags, so
a bad site lockfile first fails on the post-merge run that deploys.

frontend pnpm override floors still admit js-yaml 3.15.0 and 5.2.0,
leaving three open advisories including the one PR 2141 closed for site/.
2026-08-11 10:54:15 -05:00
dependabot[bot]andUmputun d06aa6771c chore(deps): bump js-yaml from 3.15.0 to 3.15.1 in /site
Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.15.0 to 3.15.1.
- [Changelog](https://github.com/nodeca/js-yaml/blob/3.15.1/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/3.15.0...3.15.1)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 3.15.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-11 10:36:46 -05:00
Dmitry VerkhoturovandUmputun a54d2d2756 Cache per-user flag lookups in comment listings
alterComment issued two engine.Flag calls (Blocked, then Verified) for
every comment, so a listing of N comments triggered up to 2N BoltDB read
transactions even when many comments shared the same author. Find,
FindSince, User and Last all funnel through it.

Add a userFlagCache that memoises blocked/verified results by site and
user for the duration of a single listing, so repeated authors are
looked up once. alterComment keeps its signature for single-comment
callers (Get) by using a fresh cache; the batch paths share one.
2026-07-11 02:29:17 -05:00
Umputun e575066ea9 revert(ci): restore two-build docker.yml publish
#2122 collapsed the per-registry builds into one build with two type=image
outputs and a single steps.build.outputs.digest. With build-push-action's
default provenance attestation, that digest does not resolve at ghcr, so the
multi-arch manifest step fails ("ghcr.io/...@sha256:...: not found"). Restore
the separate build-ghcr / build-dockerhub steps so each registry gets its own
digest. The ci-build.yml type=gha cache change from #2122 is kept.
2026-07-11 02:18:00 -05:00
Dmitry VerkhoturovandUmputun 2544d80f98 Return 400 for export of an unknown site
exportCtrl mapped every export failure to 500 Internal Server Error, so
requesting a backup for a non-existent site (e.g. wrong -s/--site) came
back as a misleading 500 instead of a client error — inconsistent with
the rest of the admin/public API, which returns 400 + ErrSiteNotFound
for site-lookup failures.

Add an engine.ErrSiteNotFound sentinel (wrapped at the bolt db-lookup so
the existing "site %q not found" message is unchanged) and map it to 400
+ rest.ErrSiteNotFound in exportCtrl; genuine internal failures (gzip
close/write) still return 500.
2026-07-11 02:10:42 -05:00
Dmitry VerkhoturovandUmputun 5c0798fe10 docs(claude): document milestone + issue-label conventions
Milestones: one vX.Y.Z per release; decide a PR's release by whether its merge
commit is contained in a release tag (git tag --contains), not by dates; issues
get a milestone only when closed by a code change. Plus the issue-label taxonomy
(type / area / priority / contribution / resolution).
2026-07-11 02:08:40 -05:00
Umputun 76d0cc2cf6 fix(ci): pin go-version to 1.25.12 for GO-2026-5856
setup-go with go-version "1.25" resolved to 1.25.11, which govulncheck flags for
GO-2026-5856 (ECH privacy leak in crypto/tls, fixed in go1.25.12). Pin the exact
patch in ci-backend.yml and release.yml so the vuln scan passes and release
binaries build on the fixed toolchain.
2026-07-11 02:05:58 -05:00
Dmitry VerkhoturovandUmputun 51b6a7e890 Modernise Docker build workflows
ci-build.yml used the legacy actions/cache + /tmp/.buildx-cache local
cache with a manual rotate step. Switch it to buildx type=gha cache
(separate scopes for the main and example images), dropping the
actions/cache and rotate-cache steps.

docker.yml built each image twice per platform — one build-push-action
call per registry. Build once and push the same content-addressed image
to both ghcr.io and DockerHub via multiple outputs; the single build
digest is identical for both registries, so digest export is simplified
accordingly. The multi-arch manifest merge is unchanged.
2026-07-11 02:00:52 -05:00
Dmitry VerkhoturovandUmputun 8c5e82bd16 Drop armv7 build target and remove dead golangci settings
The published multi-arch Docker manifest is amd64+arm64 only, but
GoReleaser and the Makefile dockerx target still built linux/arm/v7
binaries with no matching image. Drop the armv7 target (goarch arm +
goarm 7) from .goreleaser.yml and linux/arm/v7 from the Makefile so
shipped binaries match the images; other platforms are unchanged.

Also remove the goconst and lll settings blocks from
backend/.golangci.yml — neither linter is in the enable list, so the
settings were inert.
2026-07-11 01:57:54 -05:00
Dmitry VerkhoturovandUmputun 95f59213e5 Make notify drop tests deterministic
TestService_WithDrops and TestService_SubmitVerificationWithDrops
submitted three items into a size-1 queue and asserted at least one was
dropped, relying on the single consumer not draining the queue between
submits. synctest does not fully pin this because the MockDest send path
does real logging I/O outside the bubble, so under CI's -race scheduling
the consumer occasionally drained all three, delivering everything and
failing the "<= 2" assertion (~0.2% of CI runs).

Add an optional gate channel to MockDest so a destination blocks in
Send/SendVerification until released. The tests now submit one item (the
consumer picks it up and blocks on the gate), fill the size-1 queue,
submit an overflow item that is dropped, then release the gate — the
drop is deterministic regardless of scheduling. Assert exactly two
delivered instead of "at most two".
2026-07-11 01:56:10 -05:00
Dmitry VerkhoturovandUmputun 503f5cacb0 Add workflow to validate compose files
Compose files were not covered by any CI workflow, so a malformed change
to docker-compose.yml or a compose-*.yml could merge unnoticed. Add a
workflow that runs docker compose config on every tracked compose file
(vendored ones excluded) on changes to any of them.
2026-07-11 01:54:21 -05:00
Dmitry VerkhoturovandUmputun d1f8cf412b Add govulncheck scan to backend CI
Nothing in CI guarded against known vulnerabilities in the Go
dependency tree. Add a vulncheck job that runs govulncheck over the
backend module on every backend change. The version is pinned rather
than tracking latest for reproducible runs. Current tree scans clean.
2026-07-11 01:52:13 -05:00
Dmitry VerkhoturovandUmputun db9d8703ef Fix flaky TestPublic_FindCommentsCtrl_ConsistentCount
The test decided the expected HTTP status with strings.Contains(tc.params,
"=bad"), but comment IDs are random UUIDs. When one started with "bad"
(e.g. offset_id=bad49e60-...), the param string contained "=bad" and the
case was wrongly expected to return 400 while the handler correctly
returned 200, failing the run about 0.2% of the time.

Identify bad-request cases by their error response body instead, which
is deterministic per case and independent of the generated IDs.
2026-07-11 01:50:36 -05:00
Dmitry VerkhoturovandUmputun 8f61ec691b Fix comment-tree pagination under-filling exact-fit subtrees
The limit() boundary used >=, so a subtree that fit the page exactly was
treated as overflow and dropped — under-filling the final page (e.g.
limit=5 over subtrees 3,2 returned only the first subtree, 3 comments,
instead of both). Change it to > so an exact-fit subtree is included;
the first node is still always returned in full and larger subtrees
still overflow to the next page.

Adds table tests for MakeTree's limit/offset pagination and countReplies,
and updates the /find consistent-count expectations for the corrected
boundary.
2026-07-11 01:48:45 -05:00
Dmitry VerkhoturovandUmputun 07f6b9a0a0 Remove obsolete version key from compose files
The top-level version: "2" field is ignored by modern Docker Compose,
which warns about it on every invocation. Drop it from docker-compose.yml
and the compose-dev-backend, compose-dev-frontend and compose-e2e-test
files.
2026-07-11 01:33:06 -05:00
Dmitry VerkhoturovandUmputun 98e4f03091 Run backend CI on PR updates and freeze frontend lockfile installs
ci-backend.yml had pull_request: types: [opened, reopened], which
excludes synchronize, so pushes to an open PR branch did not re-run
backend tests, lint or coverage and a broken follow-up commit could land
after the first green run. Drop the types filter so all default
pull_request events trigger the workflow.

ci-frontend.yml and ci-frontend-api.yml now install with
pnpm install --frozen-lockfile instead of pnpm i, matching release.yml
and preventing silent lockfile drift in CI.
2026-07-11 01:31:14 -05:00
Dmitry VerkhoturovandUmputun f8f2becb4b Fix dropped notification errors and switch to errors.Join
notify/email.go accumulated multi-recipient errors with
multierror.Append(fmt.Errorf(...)) instead of
multierror.Append(result, ...), so the accumulator was overwritten each
iteration and only the last failing recipient's error survived; earlier
failures were silently dropped. The telegram notifier did it correctly.

Replace hashicorp/go-multierror with the stdlib errors.Join everywhere
it was used (notify/email.go, notify/telegram.go, rest/api/rest_private.go,
store/service/service.go, store/image/image.go and store/engine/bolt.go),
which fixes the bug and drops the direct dependency. It stays indirect
because go-pkgz/lcw/v2 still imports it. A regression test in
email_test.go now sends two failing recipients and asserts both errors
are reported.
2026-07-11 01:28:31 -05:00
Umputun 6e7820d2b7 fix(frontend): remove white flash on comments iframe load
on dark host pages the widget flashed an opaque white rectangle while loading.
the iframe element carries color-scheme from the theme param, but its document
had none until remark.tsx ran, and a mismatched color-scheme makes the embedded
canvas opaque instead of transparent. broken since #2023 added the element-side
color-scheme to fix a firefox dark-mode bug.

set the document's color-scheme from the theme param in an inline head script,
before first paint, using the same rule as create-iframe.ts. that closes the long
window but not the surface browsers paint before the document is parsed, which
webkit renders white and chromium hides behind paint holding. so also create the
iframe hidden and reveal it when the document posts inited, with a timeout
fallback so a failed bootstrap cannot leave the widget invisible.

the reveal lives in createIframe rather than embed.ts so the profile modal, the
other caller, gets it too. that modal focuses its iframe on open, and a hidden
element cannot take focus, so focus now fires from the reveal instead of a timer.

covered by a unit test for the reveal paths and the event.source guard, and by
e2e for the document's color-scheme and the iframe's visibility before inited,
after inited, and after the fallback.
2026-07-09 22:10:48 -05:00
Umputun 3e63d72852 fix(ci): build docker images on frontend-only master pushes
the docker workflow chained off the backend workflow only, and backend has a
backend/** path filter. master pushes touching just frontend/apps or the docker
files never triggered docker.yml, so no master image was published and
remark42.com was not redeployed. broken since the build workflow was split in
#1977.

listen to workflow_run from both backend and frontend, and add Dockerfile,
docker-init.sh and .dockerignore to the backend workflow paths to restore the
path coverage the old build workflow had.
2026-07-09 20:03:26 -05:00
Dmitry VerkhoturovandUmputun a8dd527c45 Fix comments iframe collapsing to preloader height on load
On mount ConnectedRoot immediately reported the iframe height to the
parent page while the app was still showing the global preloader, so the
parent shrank the iframe from its initial size to ~63px and then grew it
back step by step as content rendered. On pages with many comments this
reads as the widget blinking several times before loading (reported for
radio-t.com). The June frontend dependency refresh (#2091) shifted
render/effect timing enough to make the premature measurement happen on
every load rather than only on slow connections.

Move the height reporting into Root and start it in the setState callback
that replaces the preloader with real content: the first height message
now always describes rendered content, the iframe never shrinks below it,
and subsequent ResizeObserver updates only grow the frame as comments
arrive. Also adds the previously missing observer disconnect on unmount.

Verified by instrumenting the embed with a height-message listener:
master sent 63px then 316px on an empty test page (v1.16.1 sent a single
316px); with this fix the first message is 316px again.
2026-07-09 17:28:40 -05:00
Dmitry VerkhoturovandUmputun e62b3c830d fix(trusted-proxy): warn on catch-all, cover more cases, trim wording
Follow-up to the review notes on #2116:
- warn at startup when --trusted-proxy contains a catch-all (0.0.0.0/0 or ::/0),
  which trusts every peer and re-opens the bypass - mirrors the unset-case warning
- realIPMiddleware tests: cover the unparseable-peer and trusted-peer-without-header
  branches, and make the observed values per-call so subtests don't share closure locals
- trim the flag description and shorten the startup warning to the terse [WARN] style
2026-07-09 15:05:05 -05:00
Dmitry VerkhoturovandUmputun b1502801fa fix: add --trusted-proxy to gate client-IP forwarding headers
Rate limiting and (with --votes-ip) vote de-duplication key on the client IP,
recovered from forwarding headers (X-Real-IP / X-Forwarded-For / CF-Connecting-IP)
when behind a reverse proxy. Those headers were accepted from any client, so a
caller could set them to change its apparent IP.

Add --trusted-proxy / TRUSTED_PROXY (comma-separated CIDR/IP): forwarding headers
are honored only when the direct peer is a trusted proxy; other peers keep their
real socket address. Unset preserves the previous trust-all behavior (with a
startup warning) so existing deployments keep working on upgrade.

Docs: a 'Trusted proxies and client IP' section with per-topology guidance, plus a
note in the nginx manual.
2026-07-05 17:47:19 -05:00
Dmitry VerkhoturovandUmputun 2e3a680ca4 fix(deleteme): surface real avatar-store errors, tolerate only not-found
Bumps go-pkgz/auth to v2.1.5, which adds avatar.ErrNotFound. deleteMeRequestCtrl's
avatar removal was best-effort (log and continue on any error) because before the
sentinel there was no portable way to tell an already-removed avatar from a genuine
failure. It now tolerates only errors.Is(err, avatar.ErrNotFound) - keeping the
repeated-request idempotency - and surfaces any other store failure as 500.
2026-07-05 17:28:01 -05:00
Dmitry VerkhoturovandUmputun d8b7f7530c fix: remove user avatar on deleteme request
The delete_me token built by deleteMeCtrl omitted the user's Picture, so the
avatar-removal branch in deleteMeRequestCtrl never ran for real requests and
avatars survived account deletion. Carry Picture in the token so the stored
avatar is removed when the request is processed.

Make the removal best-effort: the avatar stores report an already-missing
avatar as an error with no distinguishable sentinel, and the user's data is
already deleted at that point, so a missing avatar (e.g. a repeated request)
no longer fails the whole deletion with a 400.

Only remove a well-formed avatar id ("<hash>.image") so a malformed picture
can't make a filesystem-backed store target an unexpected path.
2026-07-03 15:40:31 -05:00
Dmitry VerkhoturovandUmputun b33025a76f feat(api): adopt enforcing rest.Timeout, drop local cooperative timeout
go-pkgz/rest v1.22.0 ships an enforcing Timeout middleware (net/http.TimeoutHandler
style): it runs the handler with a deadline and returns 504 at the deadline even if
the handler ignores the context - unlike the local cooperative timeout, which only
cancelled the context and never actually stopped a stuck handler.

Replace the local timeout with rest.Timeout on every route with a bounded response.
The streaming and long-polling routes are deliberately left without it, since the
enforcing timeout buffers the whole response in memory and aborts at the deadline:
- GET /api/v1/userdata and GET /api/v1/admin/export stream gzipped exports
- GET /api/v1/admin/wait long-polls for up to 15m
- POST /api/v1/admin/import[/form] and /remap ingest large uploads

Delete the local timeout middleware and its test; the enforcing behaviour is covered
by go-pkgz/rest. TestRouteTimeout locks the enforcing-vs-exempt contract in this build.
2026-07-03 15:40:10 -05:00
Dmitry VerkhoturovandUmputun c48254a994 chore(deps): bump go-pkgz/rest to v1.22.0, drop local CORS Vary workaround
v1.22.0 includes the preflight Vary fix (https://github.com/go-pkgz/rest/pull/44):
rest.CORS now adds Vary: Access-Control-Request-Method and
Access-Control-Request-Headers on preflight itself, making the local wrapper
that added them redundant. corsMiddleware now returns rest.CORS directly;
TestCorsMiddleware still asserts those preflight Vary headers, now supplied
upstream.

Also tidies the _example/memory_store module for the new version.
2026-07-03 15:40:10 -05:00
Dmitry VerkhoturovandUmputun 3fc5d6b970 fix: make user deletion idempotent for users without comments
deleteUser now succeeds for a user who has no comments (e.g. one who only logged
in) instead of failing on the missing per-user bucket. In hard mode the per-user
bucket is deleted, tolerating bbolt's ErrBucketNotFound so a bucket left behind by
an earlier partial removal is still removed; the comment-deletion failure path now
wraps the actual error.

Because the engine cannot distinguish a valid login-only user from a never-existed
one, deletion is idempotent: /admin/deleteme returns 200 for an unknown (but validly
signed) token rather than 400. The deleteme test is updated to this contract, engine
tests cover hard and soft deletion of login-only and unknown users, and the API docs
note the idempotent behaviour.
2026-07-01 15:05:27 -05:00
Fredrik AppelrosandUmputun 380aa3c828 Allow deleteUser to be called on users with no comments 2026-07-01 15:05:27 -05:00
Fredrik AppelrosandUmputun b6bc8ba675 Fix error handling in deleteUser function to return the correct error when deleting a user bucket. 2026-07-01 15:05:27 -05:00
Dmitry VerkhoturovandGitHub c5121fd402 refactor(api): replace go-chi/chi router with go-pkgz/routegroup (#2103)
Migrate the REST router off go-chi/chi onto go-pkgz/routegroup (backed by the
stdlib http.ServeMux), removing the last use of go-chi from the backend:

- rest.go routes() builds the tree with routegroup (Mount/Group/Route/With) and
  net/http method+path patterns instead of chi's Get/Post/Route/Mount helpers
- chi.URLParam(...) -> r.PathValue(...) in the admin, public and private handlers
- rest_public_test.go loadPictureCtrl test uses routegroup + http.ServeMux
- rest_test.go: add TestRest_FileServerStaticAssets (bare /web -> /web/ redirect,
  cache headers, 404, directory-listing block) and update the path-traversal test
  for ServeMux normalising a literal ".." (encoded traversal is still rejected
  by the handler)
- drop go-chi/chi from go.mod, go.sum and vendor; update the CLAUDE.md reference
2026-07-01 15:04:34 -05:00
Dmitry VerkhoturovandUmputun fff9127976 fix: correct no-providers message grammar, translate it, and cover both branches
Reword "May be" to "Maybe" in the auth.no-providers message and run
translation:generate to register the key in every locale dictionary, then
replace the English placeholders with proper translations for each locale.
Add a test asserting the error is hidden when providers are configured.
2026-06-30 18:23:21 -05:00
Eugene OrlovandUmputun 406df022ba fix: ui error when no auth providers configured 2026-06-30 18:23:21 -05:00
Dmitry VerkhoturovandUmputun 6840a46ac9 Replace go-chi/cors with go-pkgz/rest CORS
Swap the go-chi/cors middleware for rest.CORS (already a dependency),
removing the go-chi/cors module entirely. Behaviour-preserving:
- with AllowedOrigins "*" and credentials enabled, both reflect the request
  Origin into Access-Control-Allow-Origin (a literal "*" is invalid with
  credentials)
- preflight responses also vary on Access-Control-Request-Method/-Headers, not
  just Origin, matching go-chi/cors so caches don't reuse a preflight response
  across different requests

Extract the config into corsMiddleware() in middleware.go and add
TestCorsMiddleware covering origin reflection, credentials, preflight
methods/headers/max-age/Vary, and the no-Origin case. go-chi/cors dropped
from go.mod; the chi router stays until the router migration. go test -race,
vet, golangci-lint, govulncheck clean.
2026-06-30 17:40:03 -05:00
Dmitry VerkhoturovandUmputun 6a50ffd88a Use stdlib http.ServeMux instead of chi in cleanup_test
The cleanup command test builds a self-contained mock HTTP server with
only static routes (and {id} patterns read via r.URL.Path, not URLParam),
so chi.NewRouter is unnecessary — http.NewServeMux (Go 1.22 routing) covers
it. Independent of the main router; drops the chi import from app/cmd.

go test -race and golangci-lint clean.
2026-06-30 17:39:37 -05:00
Dmitry VerkhoturovandUmputun f7dbdae26c Consolidate request middlewares into middleware.go
Pure relocation, no behaviour change: gather all request-scoped middlewares
and their tests into dedicated files instead of scattering them across
rest.go and ssl.go.

  funcs -> app/rest/api/middleware.go:
    timeout (from ssl.go); rejectAnonUser, matchSiteID, cacheControl,
    apiCSPMiddleware, securityHeadersMiddleware, subscribersOnly,
    validEmailAuth, rateLimiter (from rest.go)
  tests -> app/rest/api/middleware_test.go:
    TestTimeout (from ssl_test.go); TestRest_rejectAnonUser,
    TestRest_cacheControl, TestRest_apiCSP, TestRest_securityHeaders,
    TestRest_subscribersOnly, Test_validEmailAuth, TestRest_matchSiteID
    (from rest_test.go)

go test -race, vet, golangci-lint and govulncheck clean; example builds.
2026-06-30 17:15:39 -05:00
Dmitry VerkhoturovandUmputun f4b236c66a Replace chi middleware.Timeout with the timeout helper, drop chi/middleware
middleware.Timeout was the last use of go-chi/chi/v5/middleware (RealIP, the
other user, landed in #2099). Swap it for the local timeout helper (context
deadline + 504 on deadline, matching chi exactly; covered by TestTimeout),
which removes the go-chi/chi/v5/middleware package from the vendor tree.

The go-chi/chi module stays in go.mod — the router (chi.NewRouter etc.) still
uses it, so go.mod only shrinks after the router migration. Build, vet, race
tests and golangci-lint clean.
2026-06-30 17:15:04 -05:00
Dmitry VerkhoturovandUmputun 17365f4304 Replace chi middleware.RealIP with rest.RealIP on the main router
Behaviour-preserving swap: rest.RealIP sets r.RemoteAddr from
X-Real-IP / X-Forwarded-For like chi's middleware.RealIP, removing chi from
the RealIP path without changing the trust model (GHSA-56x6-q882-mf27 stays
present, to be fixed separately). chi/middleware stays imported for Timeout;
whichever of this PR and the Timeout PR (#2097) merges last drops the import.
Drop-in to a tested rest middleware; covered by existing api router tests.
2026-06-30 16:38:12 -05:00
Dmitry VerkhoturovandUmputun 0b6eea68a1 Replace chi middleware.NoCache with rest.NoCache on the main router
go-pkgz/rest.NoCache is borrowed from chi's middleware.NoCache and behaves
identically: same no-cache response headers and the same stripping of
conditional request headers (If-None-Match etc.), which the image-proxy
etag logic relies on. Drop-in swap, chi router left in place.

chi/middleware stays imported for RealIP and Timeout. Build, vet, race
tests and golangci-lint clean.
2026-06-30 16:27:01 -05:00
Dmitry VerkhoturovandUmputun b19e6269c1 Replace chi middleware.Throttle with rest.Throttle on the main router
Drop-in swap of the global concurrency limiter (go-pkgz/rest.Throttle has
the same signature and semantics as chi's middleware.Throttle), with the
chi router left in place. First of the per-middleware swaps that chip away
at go-chi/chi/v5/middleware before the router itself is migrated.

chi/middleware is still imported for RealIP/Timeout/NoCache; build, vet,
race tests and golangci-lint clean.
2026-06-30 16:26:28 -05:00
Dmitry VerkhoturovandUmputun bb6d1450f1 Migrate ssl.go TLS routers from chi to routegroup
First step of the go-chi -> go-pkgz/routegroup migration. The HTTP->HTTPS
redirect and ACME http-01 challenge routers are small, self-contained
http.Handlers separate from the main API router, so they move cleanly:

- chi.NewRouter() -> routegroup.New(http.NewServeMux())
- middleware.Throttle -> rest.Throttle (same concurrency-limit semantics)
- middleware.Timeout -> local timeout helper (context deadline, mirrors chi)
- drop middleware.RealIP: these routers do redirect/challenge only, with no
  per-IP logic, so the spoofable header trust is simply removed here
- return http.Handler instead of chi.Router (callers already take http.Handler)

chi stays a dependency (still used by the main API router); this only removes
its use from ssl.go. go test -race, vet, golangci-lint and govulncheck clean.
2026-06-30 16:26:03 -05:00
dependabot[bot]andUmputun 8318f89dde chore(deps): bump the github-actions-updates group across 1 directory with 4 updates
Bumps the github-actions-updates group with 4 updates in the / directory: [actions/checkout](https://github.com/actions/checkout), [actions/cache](https://github.com/actions/cache), [pnpm/action-setup](https://github.com/pnpm/action-setup) and [codecov/codecov-action](https://github.com/codecov/codecov-action).


Updates `actions/checkout` from 6 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

Updates `actions/cache` from 5 to 6
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v5...v6)

Updates `pnpm/action-setup` from 6.0.4 to 6.0.9
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v6.0.4...v6.0.9)

Updates `codecov/codecov-action` from 6 to 7
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 6.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-updates
- dependency-name: codecov/codecov-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-30 15:57:05 -05:00
Dmitry VerkhoturovandUmputun 3e18681ca7 Sanitize comment text in email notifications (GHSA-74pc-3r2m-ppx3)
Email notification templates rendered the comment HTML via text/template,
so the store-level UGC sanitizer's permitted <a> and <img> tags reached
the email body verbatim. An authenticated user could plant phishing links
and remote tracking pixels in notification emails sent from the legitimate
remark42 address.

Switch notify to html/template (auto-escaping every non-HTML field) and
add a stricter email-only bluemonday policy that drops <a> and <img> while
keeping basic text formatting; the sanitized comment HTML is passed as
template.HTML. Add regression tests asserting links and images are stripped
while anchor text and formatting survive.
2026-06-30 15:56:36 -05:00
UmputunandGitHub 11d8a978a2 Merge pull request #2094 from umputun/fix/eleventy-outputpath-guard
Guard against falsy outputPath in eleventy htmlmin transform
2026-06-30 15:56:08 -05:00
Dmitry Verkhoturov d7fe27cb97 Guard against falsy outputPath in eleventy htmlmin transform
Eleventy passes a falsy outputPath to transforms for templates rendered
without a written file (e.g. permalink: false); calling .endsWith on it
would throw. Skip minification in that case instead. Pre-existing latent
issue surfaced by Copilot review on #2091.
2026-06-30 20:59:57 +01:00
UmputunandGitHub 7fee12a978 Merge pull request #2091 from umputun/deps/update-frontend
Update frontend and site dependencies to latest, bump pnpm to 10, clear audit alerts
2026-06-30 14:23:48 -05:00
Dmitry Verkhoturov fc3d93c398 Add frontend/CLAUDE.md documenting dependency-update gotchas
Captures what isn't obvious from the diff alone: the ten places a
node/pnpm version is pinned and must move together (including .nvmrc,
which CI never reads and is how the node-16 drift in this PR's first
push went unnoticed), the pnpm-10 layout pins, the msw 1->2 migration,
the deliberately held-back majors, and the abandoned html-minifier
replacement. Written so the next dependency bump doesn't repeat the
same gaps.
2026-06-30 20:11:53 +01:00
Dmitry Verkhoturov 4baf0f4260 Close remaining node/pnpm version drift after the pnpm 10 bump
- frontend/.nvmrc was still pinned to 16, left behind by the node 16->20
  bump everywhere else (Dockerfile, CI matrices). A contributor running
  'nvm use' in frontend/ would land on node 16, which cannot even run
  pnpm 10 (requires node >=18) -- CI never reads .nvmrc, so this was
  invisible to every check.
- pnpm/action-setup 'version: 10' floated the patch release in CI,
  inconsistent with the exact 10.10.0 pin now used in Dockerfile,
  Dockerfile.e2e and packageManager. Pinned all ten occurrences across
  ci-frontend.yml, ci-frontend-api.yml and release.yml to 10.10.0.
2026-06-30 20:10:43 +01:00
Dmitry Verkhoturov b72030114c Address Copilot review feedback on #2091
- Pin pnpm to the exact version (10.10.0) when installing it in the
  production Dockerfile, matching packageManager and Dockerfile.e2e,
  instead of a floating major that can drift the lockfile behaviour.
- Fix mockEndpoint's array header handling in the api test utility:
  append each value instead of joining with a comma, which is how
  multi-value headers (e.g. set-cookie) are actually represented.
- Update apps/remark42's engines to node >=18 / pnpm >=10, matching
  the pnpm 10 requirement instead of the stale node 16 / pnpm 8 range.
2026-06-30 20:05:42 +01:00
Dmitry Verkhoturov 8626e4181f Fix CI for node 20 / pnpm 10: e2e Playwright image and jest arg forwarding
- frontend/Dockerfile.e2e: bump base image to mcr.microsoft.com/playwright:
  v1.61.1-noble to match the Playwright 1.61.1 npm bump (browser revision
  mismatch was failing all e2e specs), and corepack pnpm@8 -> pnpm@10.10.0 to
  match the pnpm bump and the v9 lockfile.
- release.yml validate: pnpm 10 forwards 'test -- --runInBand' literally as
  'jest -- --runInBand' (treated as a path pattern, 0 tests). Drop the extra
  separator: 'pnpm test --runInBand'.
2026-06-30 19:53:10 +01:00
Dmitry Verkhoturov d274724c08 Update site dependencies and clear all yarn audit alerts
yarn audit: 0 vulnerabilities (was 52 findings). Site builds via eleventy +
tailwind on node 20.

Direct bumps: markdown-it 14.2, cross-env 10, date-fns 4.4, prettier 3.9,
@tailwindcss/typography 0.5.20, @11ty/eleventy-plugin-syntaxhighlight 5.0.2.
Replaced abandoned html-minifier (unpatched ReDoS, no fix released) with the
maintained html-minifier-terser fork; the .eleventy.js htmlmin transform is now
async. Transitive vulns patched via yarn resolutions. js-yaml resolves to 3.15.0
(3.x backport) which keeps gray-matter working.

Held: tailwindcss 3.4 (tailwind 4 is a config rewrite) and @11ty/eleventy 2
(eleventy 3 is an ESM migration) - both invasive.

Build output verified against a clean master build: every HTML page differs only
by the build-time ?v= cache-bust query; style.css differs only by an equivalent
refactor of @tailwindcss/typography's prose kbd-shadow variables (same rendered
result). Functionally identical.
2026-06-30 19:47:25 +01:00
Dmitry Verkhoturov f5ccfaa0e1 Update frontend dependencies to latest, bump pnpm to 10, clear all npm audit alerts
pnpm 8.15.9 -> 10.10.0 (packageManager + lockfile regenerated to v9). Frontend
CI (ci-frontend.yml, ci-frontend-api.yml, release.yml) and the production
Dockerfile bumped from node 16 + pnpm 8 to node 20 + pnpm 10 (pnpm 10 requires
node 18+). pnpm audit: no known vulnerabilities (was 63 alerts).

packages/api: bumped to latest including the major test stack - vitest 4, jsdom
29, @vitest/coverage-v8 4, @typescript-eslint 8.62, typescript 5.9, prettier
3.9, @types/node 26, and msw 1 -> 2. Migrated tests/test-utils.ts to the msw 2
http/HttpResponse API (capturing a compatible request shape) and made test base
URLs absolute so node 20's native fetch is intercepted; added the jsdom base
URL. type-check:api, lint:api and coverage:api (45 tests) all pass.

apps/remark42: safe in-major bumps (webpack 5.108, postcss, mini-css-extract,
html-webpack-plugin, ts-loader, webpack-dev-server 5.2.5, core-js, clsx 2,
lodash-es 4.18, dotenv 17, @types/*). Transitive vulns patched via
pnpm.overrides. type-check, lint, build, jest coverage (299 tests) and
translations all pass.

pnpm 10's stricter layout required a few pins to keep the app's preact-compat
setup compiling: preact 10.6.2 (override), react-intl 6.0.5 and
@testing-library/preact 3.2.2 (newer types break the build), tsconfig paths for
preact, @types/minimatch 5.1.2 (6.x is an empty stub) and cheerio 1.0.0-rc.12
(1.2 is ESM and breaks jest 28). Held: react/react-dom (preact compat alias),
babel 7, eslint 8, stylelint 14, jest 28, typescript 4.7 (app),
redux/react-redux - majors that change the bundle or need a config migration.

Build output verified against a clean master build: apps/remark42 output is
functionally identical (the only diffs are webpack module-id numbering and
css-module class tokens from the webpack/css-loader bump; all HTML, CSS values
and translations byte-identical).
2026-06-30 19:47:25 +01:00
UmputunandGitHub c8832e708c Merge pull request #2088 from umputun/deps/update-backend
Update backend dependencies to latest
2026-06-30 12:41:13 -05:00
Dmitry Verkhoturov 07c7926453 Update backend dependencies to latest
Update all Go modules in backend/ and backend/_example/memory_store/ to
their latest versions (chroma 2.27, go-redis 9.21, bbolt 1.5, slack 0.27,
golang.org/x/* and others); re-tidy and re-vendor, keep the example module
in sync.

Hold github.com/go-chi/chi/v5 at v5.2.5: v5.3.0 deprecates
middleware.RealIP (IP-spoofing advisories). Switching off RealIP changes
how the client IP is derived for rate limiting and votes, which is a
security decision better made on its own rather than inside a dependency
bump.

go test -race, go vet, golangci-lint and govulncheck all clean on both
modules.
2026-06-30 18:35:31 +01:00
UmputunandGitHub 34ed97b7a6 Merge pull request #2056 from umputun/dependabot/npm_and_yarn/frontend/postcss-8.5.10
chore(deps-dev): bump postcss from 8.4.14 to 8.5.10 in /frontend
2026-06-01 22:10:26 -05:00
UmputunandGitHub 0868b70fa9 Merge pull request #2063 from umputun/dependabot/npm_and_yarn/frontend/webpack-dev-server-5.2.4
chore(deps-dev): bump webpack-dev-server from 4.9.3 to 5.2.4 in /frontend
2026-06-01 22:10:21 -05:00
Umputun 589e956ade fix: handle REST shutdown before server start 2026-06-01 19:55:14 -05:00
Paul MineevandUmputun a21044738d fix typo in file name 2026-05-28 17:53:37 -05:00
Dmitry VerkhoturovandGitHub 929c06d957 site: fetch latest version client-side instead of embedding at build time (#2072)
* site: fetch latest version client-side instead of embedding at build time

The header version badge was filled in by site/src/data/github.js calling
the GitHub releases API at Eleventy build time and baking data[0].tag_name
into every page. This had three failure modes:

1. Layered cache: Buildx caches the yarn build layer; on a release-triggered
   workflow nothing under ./site changes, so the cached HTML (with the
   previous tag baked in) gets shipped. v1.16.0 went out and remark42.com
   kept showing v1.15.0 until a separate site/ commit landed and naturally
   invalidated the COPY layer.

2. Tag mismatch: the deploy pulls ghcr.io/umputun/remark42-site:master,
   but release events build :v1.16.0 and :latest only. A cache-skip
   workflow tweak wouldn't even reach the served image.

3. API propagation race: the workflow fires ~2s after release publish,
   so even with cache disabled the API might still return the previous
   tag from a stale read replica.

All three vanish if the version is fetched in the browser. GitHub serves
the /releases/latest response with Cache-Control: public, max-age=60 so
per-visitor cost is bounded; failures fall through silently and the
badge stays empty rather than wrong.

Changes:
- header.njk: replace {{ github.latestVersion }} with a
  <span data-remark42-version></span> placeholder.
- inline.js: add a fetch of /releases/latest that fills any
  [data-remark42-version] element on the page. fallback is no-op on any
  network/parse failure.
- delete site/src/data/github.js (Eleventy data file is no longer used).
- drop node-fetch from devDependencies (was used only by github.js).

* site: address PR review on version badge fetch

- gate DOM update on DOMContentLoaded — inline.js is loaded sync in <head>,
  so a cache-hit fetch can resolve before the placeholder span is parsed.
- hide placeholder span by default (`hidden`) so a failed/blocked fetch
  doesn't leave a 0.5rem stray gap before the github icon.
- log fetch failures (rate limit, offline, blocked) instead of silently
  swallowing — matches the prior behaviour of build-time github.js.

* site: cache latest version in sessionStorage with 1h TTL

avoids hitting the GitHub API on every page load — repeated navigations
within a tab read from sessionStorage instead. TTL caps stale display at
1h for very long-lived tabs. cleared on tab close, so each new session
fetches once and reuses the result throughout.

* site: address PR review on header & version fetch

Copilot review on the cache commit raised three points:

1. inline.js had a hard-coded `https://api.github.com/repos/umputun/remark42`
   while the templates use `site.githubUrl`. Rename inline.js → inline.njk
   so nunjucks evaluates it, add `githubApiUrl` to site.json, and template
   the fetch URL from it. One place to update if the repo ever moves.

2. header.njk aria-label said "Remark42's GitHub Repository" but the link
   target is `/releases`. Change to "{{ site.name }} releases on GitHub"
   so screen readers describe the actual destination.

3. console.warn on fetch failure (kept after umputun's prior review noted
   the trade-off): addressed in the PR description, no code change.

* site: actually template fetch URL via site.githubApiUrl

Copilot's second pass caught that 7697dcf3 added site.githubApiUrl,
renamed inline.js → inline.njk, and pointed head.njk at the .njk file
— but the fetch() call itself was never changed to use the template
variable. Build output looked correct because the literal hard-coded
URL happened to match what {{ site.githubApiUrl }} would expand to.

* site: normalise Nunjucks spacing in header.njk

{{ site.githubUrl}}/releases → {{ site.githubUrl }}/releases. Cosmetic
only; matches the spacing used everywhere else in the templates.
2026-05-28 13:10:35 -05:00
Dmitry VerkhoturovandGitHub 198efddb54 fix(frontend): no_footer scrollbar regression introduced in v1.16.0 (#2076)
* fix(frontend): no_footer scrollbar regression introduced in v1.16.0

Two unrelated changes in v1.16.0 combined to surface a scrollbar in
no_footer=true mode:

1. c26f45e5 removed the deprecated `scrolling="no"` iframe attribute
   on the grounds that "overflow is already hidden via CSS". That CSS
   (`overflow: hidden` in createIframe styles) is on the iframe ELEMENT
   in the parent page; it has no effect on the iframe DOCUMENT's own
   scrollbars. The spec-correct replacement is `overflow: hidden` on
   the iframe document's body — added here to global.css.

2. The negative `margin-bottom: -24px` on `.thread:last-child` was a
   trick to tighten the gap to the footer (combined with the footer's
   `margin-top: 48px` it collapsed to a 24px net gap). With no_footer
   the negative margin had no positive-margin sibling to collapse
   against and instead propagated up through .root, leaving body
   ~24px shorter than the visual content. The iframe height calc
   (`body.offsetHeight + 12`) then sized the iframe below the visible
   bottom of the last thread → scrollbar.

   Replace the negative-margin trick with a straight `margin-top: 24px`
   on `.copyright`. Same 24px visual gap when the footer is shown, no
   propagation when it isn't. The mix={styles.thread} on Thread becomes
   a dead reference and is dropped.

Closes #2073

* fix(frontend): drop dead Thread.mix prop after root.tsx removed its only caller

Both Copilot and umputun flagged this in PR review: after the parent
commit on this branch dropped `mix={styles.thread}` from root.tsx, the
`mix?: string` prop and the corresponding entry in the clsx() call in
thread.tsx are dead code — no caller passes it (the recursive Thread
render in thread.tsx:82 never did either). Remove the prop, the
destructure, and the clsx entry.
2026-05-28 13:02:25 -05:00
Dmitry VerkhoturovandGitHub 39408dffe8 fix: parameter docs + --help text inconsistencies (audit) (#2077)
* fix: address parameter docs and --help text inconsistencies

Audit findings from comparing site/src/docs/configuration/parameters/
against the backend flag tags.

Docs (parameters/index.md):
- image.bolt.file default was `/var/pictures.db` (absolute, looks like
  a system path); actual default is `./var/pictures.db` (relative,
  under the working dir).
- notify.webhook.template default was shown as
  `{"text": {{.Text | escapeJSONString}}}` — both the function name
  doesn't exist and the unescaped pipe inside the table cell broke the
  Description column count for that row. Real default is the literal
  `{"text": "{{.Text}}"}`.
- "Custom OAuth2 integration currently supports only one custom
  provider at a time" was a free-standing paragraph wedged between two
  table rows. kramdown terminated the table on that paragraph and
  restarted a new headerless table for the rest of the rows. Moved it
  to its own subsection after the table so the table stays contiguous.

Backend --help text (server.go):
- allowed-hosts description ended with a stray double apostrophe in
  `CSP 'frame-ancestors''` (typo).
- Deprecated auth.email.{port,passwd,user,tls} flag descriptions were
  shuffled — port said "SMTP password", passwd said "SMTP port", user
  said "enable TLS", tls said "SMTP TCP connection timeout". Fixed
  each to match the flag it's actually describing. Docs already had
  the correct descriptions for these deprecated flags.

* fix: webhook template flag default override masking safe fallback

Copilot flagged the audit's "real default" claim and was right. server.go:286
had default:"{\"text\": \"{{.Text}}\"}" — the literal, JSON-unsafe template
that produces invalid JSON if a comment contains a quote or newline. The
notify package (webhook.go:50) has a safer fallback:

    if params.Template == "" {
        params.Template = webhookDefaultTemplate
    }

where webhookDefaultTemplate is {"text": {{.Text | escapeJSONString}}}. But
go-flags applies its default tag at parse time, so the field is never empty
when the user omits --notify.webhook.template, and the safer fallback never
runs.

Drop the unsafe default tag so the webhook package's escapeJSONString-based
default takes effect. Also:
- fix the --help description (was "webhook authentication template", but
  it's a payload template, not an auth one; same for headers).
- update parameters/index.md to document the actual safe default
  ({{.Text | escapeJSONString}}); escape the cell's | as \| so kramdown
  doesn't treat it as a column separator.
- typo: "bellow" -> "below" in the headers env-delim comment.
2026-05-28 12:56:33 -05:00
Dmitry VerkhoturovandUmputun 6961dc24e5 docs: close backtick in smtp.login_auth default cell
opening backtick had no closer in the Default column, so kramdown saw a
broken cell and stopped rendering the parameters table — every row from
smtp.login_auth onward (~40 rows) rendered as raw pipe-delimited text
instead of HTML table cells. closes #2074
2026-05-26 14:12:07 -05:00
Umputun e8b9d70061 docs(site): bump remark42 image tag to v1.16.0 in kubernetes manual
the kubernetes deployment example pinned ghcr.io/umputun/remark42:v1.14.0,
two releases behind.
2026-05-22 16:15:37 -05:00
UmputunandGitHub 556e0a70d5 chore(release): build binary artifacts with GoReleaser (#2070)
replace the Docker artifact build with GoReleaser config and a tag release workflow. Keep local artifact builds snapshot-only and clean generated frontend embed files after release runs.
2026-05-22 13:24:32 -05:00
Dmitry VerkhoturovandGitHub 0e20861419 fix(security): reject non-image content-types in image proxy and /picture/ to prevent stored XSS (#2067)
* fix(security): reject non-image content-types in image proxy and /picture/ to prevent stored XSS

The /api/v1/img proxy and /api/v1/picture/{user}/{id} endpoints emitted
http.DetectContentType on the served bytes as the response Content-Type. A
controlled upstream serving Content-Type: image/png with an HTML body passed
the upstream check (only the response header was inspected, not the body),
and the body bytes then sniffed back to text/html — so the proxy served the
attacker's HTML from the remark42 origin. Browsers honoured the declared
text/html and executed the response as a document with access to cookies and
CSRF tokens. Affected from v1.6.0 (April 2020) through v1.15.0; verified live
via published docker images.

Layered defense applied to both handlers:

- rest.SafeImgContentType (in backend/app/rest/) validates sniffed content
  against a strict allowlist: image/png, image/jpeg, image/gif, image/webp,
  image/bmp, image/x-icon. Anything else (HTML, XML, SVG, plain text,
  octet-stream, or any future image type the stdlib sniffer may learn) is
  rejected with no body echo. SVG is implicitly excluded — it sniffs as
  text/xml or text/plain, never image/svg+xml, and SVG can execute scripts
  when navigated to top-level. The previous octet-stream → image/* fallback
  is gone.
- Per-endpoint Content-Security-Policy override sets
  "default-src 'none'; sandbox; frame-ancestors 'none'" on every response
  (success, 304, or error). Sandbox neuters scripts even if Content-Type
  ever regresses. The same policy is also applied to all /api/v1/* via
  apiCSPMiddleware as defense-in-depth.
- Content-Disposition: inline; filename="image" frames the response as a
  file rather than a renderable document.
- /picture/ rejection paths set Cache-Control: no-store so 4xx responses
  are never cached.

The defense headers and the strict ETag matcher are extracted as
rest.SetImageDefenseHeaders and rest.EtagMatches in the shared rest package
(consumed by both proxy/image and api/rest_public — no package cycle).

The /api/v1/img path additionally bumps the ETag to a versioned `"v2:..."`
so revalidating clients (top-level navigation, Ctrl+R, intermediaries) get
a fresh 200 instead of a 304 against poisoned pre-fix cached HTML.

DELIBERATE TRADEOFF: Cache-Control on /api/v1/img success responses remains
max-age=2592000 (30 days), unchanged from before. An aggressive "force
revalidate on every reuse" policy was prototyped during review but reverted
because the perf cost (a server round-trip on every image view, even with
304 saving the body bytes) outweighed the corner-case mitigation. The
realistic exposure of cache carryover is narrow: cache carryover only
affects users who navigated top-level to an attacker URL pre-fix and still
have it in their local cache — the normal <img> embed path cached text/html
but never executed it. Local browser caches that hold pre-fix bytes
continue to serve them until their 30-day TTL expires or are evicted under
memory pressure. The ETag bump reaches all clients that DO revalidate
during the cached lifetime (Ctrl+R, intermediaries, post-expiry use); for
the rest, exposure self-limits via cache expiry. Operators running a
CDN/edge cache in front of remark42 should purge /api/v1/img after deploy.

The /api/v1/img handler short-circuits on a matching current-version
If-None-Match before any store Load or upstream fetch, returning a bodyless
304 with the defense headers set. Safe because the 304 carries no body and
the client's cached bytes came from a prior validated 200; an attacker
fabricating an etag value can only short-circuit fetches for URLs they
themselves crafted. This avoids upstream DoS amplification when clients
revalidate on hot comment pages.

The /api/v1/img route was moved from the "open routes" group (which uses
middleware.NoCache, stripping If-None-Match from incoming requests) to the
"open routes, cached" group alongside /picture/ and /qr/telegram so the
304 revalidation path is no longer broken upstream of the handler.

The /picture/{user}/{id} endpoint does not need the v2 etag prefix. Upload
validates input format via readAndValidateImage and the serve path
re-validates the stored bytes via rest.SafeImgContentType. Bytes within
the resize dimension limits are preserved verbatim, so the browser defense
relies on the response headers (validated Content-Type + nosniff + strict
CSP + Content-Disposition: inline), not on byte normalization.

Global CSP: font-src data: → font-src 'none'. Audit confirmed no @font-face,
no base64 fonts, no icon-font library in the bundle. Drops an unnecessary
attack surface; no behavioural change.

Tests: TestImage_ContentTypeHandling table-tests a real PNG and attack
shapes (HTML claimed as image/png, image/jpeg, image/gif, image/svg+xml,
image/webp; svg with onload; html fragment; polyglot PNG+HTML), proving
the defense holds across arbitrary upstream Content-Type variation.
Polyglot case is intentionally served as image/png — the browser cannot
execute the trailing HTML when the response type is image/png with nosniff.
TestImage_ContentTypeHandling_CacheHit exercises the cache-hit branch with
attacker bytes preloaded into the store. TestImage_PerRequestRevalidation
alternates upstream PNG/HTML across four proxy calls to prove no trust
accumulates between requests. TestImage_RoutesUsingCachedImage asserts
cache-poisoning is caught at serve time. TestImage_EtagVersioned asserts
the v2 prefix invalidates pre-fix etags AND that the revalidation 304
triggers no store Load. TestImage_RevalidationSkipsIO proves the
short-circuit works even with no upstream reachable. TestSafeImgContentType
covers the allowlist directly. TestRest_LoadPictureDefenseHeaders and
TestRest_LoadPictureRejectsNonImage exercise the /picture/ endpoint.
TestRest_apiCSP covers the strict CSP middleware on JSON API + RSS routes;
TestRest_securityHeaders confirms /web/ HTML pages keep the global CSP.

Verified end-to-end against the dev docker image: the original demo URL
(arbitrary HTML claimed as image/png) now returns 415 application/json with
CSP/nosniff/Content-Disposition set, no XSS in the browser.

* fix(security): set Cache-Control: no-store on image-proxy error paths, sync stale route comment

Addresses two review comments on #2067:

1. Cache-Control: max-age=2592000 and Etag were set before the
   load/download/validation block, so 404/400/415 error responses inherited
   the 30-day cache TTL and the versioned etag — a transient failure (or an
   intentionally triggered 415) would be pinned in browser/intermediary
   caches for that TTL, keeping users locked out even after the underlying
   cause was resolved. Now: etag is computed but not set as a header until
   after validation succeeds; error paths route through sendImageProxyError
   which sets Cache-Control: no-store and never sets Etag. The 304
   short-circuit still sets both because that path serves the same validated
   content the client already has cached.

2. The comment at rest.go:282 still described the prototyped
   no-cache/must-revalidate Cache-Control policy that was reverted before
   the PR landed. Updated to match the actual 30-day max-age behavior.

Tests: TestImage_ContentTypeHandling now asserts reject paths carry
Cache-Control: no-store and have no Etag header, and accept paths carry
the max-age=2592000 + v2: etag.
2026-05-20 22:37:25 -05:00
Dmitry VerkhoturovandUmputun 8224626ed4 fix(image): reject decompression-bomb dimensions before raster decode
readAndValidateImage caps the byte size of incoming images but the resize()
helper that follows still called image.Decode unconditionally, allocating
pixel memory proportional to the *declared* image dimensions. A ~100 KB
compressed PNG or GIF that declares 65535x65535 px forces image.Decode to
allocate ~17 GB of raster, OOMing the service on a single comment upload
(or on the proxy's CacheExternal path when caching a malicious upstream).

Hardening:

- maxImagePixels = 16 MP constant. Covers any realistic image (~4096x4096)
  while bounding peak allocation.
- resize() now runs image.DecodeConfig first (cheap, no pixel allocation)
  to read declared width/height before any full decode.
- Multiplication of width × height uses int64 to defeat 32-bit overflow
  (GOARCH=386, 32-bit arm): on those targets, int(cfg.Width)*int(cfg.Height)
  could wrap below maxImagePixels and bypass the cap. GIF's 16-bit logical
  screen and JPEG's 16-bit SOF dimensions both reach this if int-multiplied.
- Bytes exceeding the cap, or non-image input that fails DecodeConfig,
  return nil. prepareImage propagates the rejection as a clear error
  instead of storing the malformed/oversized data verbatim.
- The no-resize-needed path returns the validated original bytes verbatim
  so animated GIFs round-trip without being flattened to a single frame.

The DecodeConfig precheck applies even when MaxWidth/MaxHeight are 0
(resize disabled) — the dimension cap is unconditional defense-in-depth.

Two adjacent fixes surfaced by the new resize contract:

1. readAndValidateImage previously did `data[:512]` without a bounds check,
   panicking on any body shorter than 512 bytes. Now bounded with min().
2. image/webp was listed as an allowed format but no WebP decoder was
   registered, so DecodeConfig would refuse legitimate WebP uploads. Added
   `_ "golang.org/x/image/webp"` (already in go.mod via x/image/draw) so
   the registered decoders match the allowlist.

Tests:

- TestService_resizeRejectsDecompressionBomb builds a 14-byte GIF87a header
  declaring 65535x65535 and asserts resize() refuses it both at the unit
  level and through SaveWithID end-to-end (no store write).
- TestService_SaveWithIDShortPayload regression-tests the short-body panic.
- TestService_SaveWithIDWebP regression-tests WebP round-trip through
  prepareImage with the new DecodeConfig requirement.
- TestService_resize subtests updated to assert non-image bytes are now
  refused (previously the helper fell back to returning the raw bytes
  verbatim, letting malformed content reach the store).
2026-05-20 21:48:23 -05:00
Dmitry VerkhoturovandUmputun 45c17a913f chore(deps): bump go modules in backend and example
Backend (backend/go.mod):
- github.com/go-pkgz/auth/v2 v2.1.2 → v2.1.4
- github.com/klauspost/compress v1.18.5 → v1.18.6
- github.com/redis/go-redis/v9 v9.18.0 → v9.19.0
- github.com/slack-go/slack v0.21.1 → v0.23.1
- golang.org/x/crypto v0.50.0 → v0.51.0
- golang.org/x/image v0.39.0 → v0.40.0
- golang.org/x/net v0.53.0 → v0.54.0
- golang.org/x/sys v0.43.0 → v0.44.0
- golang.org/x/text v0.36.0 → v0.37.0

Example (backend/_example/memory_store/go.mod):
- golang.org/x/crypto v0.50.0 → v0.51.0
- golang.org/x/image v0.39.0 → v0.40.0
- golang.org/x/net v0.53.0 → v0.54.0
- golang.org/x/sys v0.43.0 → v0.44.0

Transitive cleanup: github.com/dgryski/go-rendezvous is no longer required
after redis/go-redis bump and gets pruned by `go mod tidy`.

`go mod tidy` + `go mod vendor` run on both modules. Both build with -race
and full test suites pass.
2026-05-20 20:09:47 -05:00
Umputun f3a7dea1f1 docs: offer github private vulnerability reporting in security policy
Mention the "Report a vulnerability" button (GitHub private vulnerability
reporting) alongside the existing email contact, now that private reporting
is enabled on the repository.
2026-05-20 13:41:06 -05:00
dependabot[bot]andGitHub e8c106f06b chore(deps-dev): bump webpack-dev-server in /frontend
Bumps [webpack-dev-server](https://github.com/webpack/webpack-dev-server) from 4.9.3 to 5.2.4.
- [Release notes](https://github.com/webpack/webpack-dev-server/releases)
- [Changelog](https://github.com/webpack/webpack-dev-server/blob/main/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-server/compare/v4.9.3...v5.2.4)

---
updated-dependencies:
- dependency-name: webpack-dev-server
  dependency-version: 5.2.4
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-19 19:47:06 +00:00
dependabot[bot]andGitHub 54b7b3fdd4 chore(deps-dev): bump postcss from 8.4.14 to 8.5.10 in /frontend
Bumps [postcss](https://github.com/postcss/postcss) from 8.4.14 to 8.5.10.
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.4.14...8.5.10)

---
updated-dependencies:
- dependency-name: postcss
  dependency-version: 8.5.10
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-06 06:54:50 +00:00
UmputunandGitHub c0636f204e Merge pull request #2053 from umputun/dependabot/github_actions/github-actions-updates-512a575e1a
chore(deps): bump pnpm/action-setup from 5.0.0 to 6.0.4 in the github-actions-updates group
2026-05-06 01:53:15 -05:00
UmputunandGitHub c418f8ec00 Merge pull request #2052 from umputun/dependabot/go_modules/backend/go-modules-updates-47fdc5c9f4
chore(deps): bump the go-modules-updates group in /backend with 2 updates
2026-05-06 01:53:10 -05:00
dependabot[bot]andDmitry Verkhoturov e9ad5dcc09 chore(deps): bump the go-modules-updates group
Bumps the go-modules-updates group in /backend with 2 updates: [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) and [github.com/go-pkgz/auth/v2](https://github.com/go-pkgz/auth).

Updates `github.com/alecthomas/chroma/v2` from 2.23.1 to 2.24.1
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.23.1...v2.24.1)

Updates `github.com/go-pkgz/auth/v2` from 2.1.2-0.20260421203319-686683f19cf7 to 2.1.2
- [Release notes](https://github.com/go-pkgz/auth/releases)
- [Commits](https://github.com/go-pkgz/auth/commits/v2.1.2)

---
updated-dependencies:
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-version: 2.24.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/auth/v2
  dependency-version: 2.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-01 22:52:12 +01:00
dependabot[bot]andGitHub d072f34a67 chore(deps): bump pnpm/action-setup in the github-actions-updates group
Bumps the github-actions-updates group with 1 update: [pnpm/action-setup](https://github.com/pnpm/action-setup).


Updates `pnpm/action-setup` from 5.0.0 to 6.0.4
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v5.0.0...v6.0.4)

---
updated-dependencies:
- dependency-name: pnpm/action-setup
  dependency-version: 6.0.4
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-01 13:46:19 +00:00
Dmitry VerkhoturovandGitHub a4c5e17bbb Probe /auth/status from frontend to avoid 401 on /user (closes #1188) (#1763)
* Probe /auth/status from frontend to avoid 401 console noise on /user

GET /api/v1/user requires auth and returns 401 for anonymous visitors,
which the browser logs to console even when JS catches it. Probe
/auth/status first (always 200), then fetch /user only when logged in.
Stale auth cookies are cleared when status reports "not logged in" to
preserve the cleanup-on-probe behaviour previously triggered by /user 401.

Closes #1188.

* Don't clear auth cookies when /auth/status probe itself fails

A transient network/5xx on the /auth/status probe used to fall through
into the cookie-clear branch and silently log the user out on the next
page load. Distinguish "probe failed" (null) from explicit "not logged
in"; only the latter clears JWT/XSRF cookies. Lock the distinction with
a negative assertion in the probe-failure test.

Also align packages/api prettier config with apps/remark42 (trailingComma: 'es5')
so future edits don't sweep unrelated trailing commas into the diff.
2026-04-30 19:32:49 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8d9290ea1f Bump picomatch from 2.3.1 to 2.3.2 in /site (#2028)
Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 2.3.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:44 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
07d89202b4 chore(deps): bump liquidjs from 10.19.0 to 10.25.6 in /site (#2050)
Bumps [liquidjs](https://github.com/harttle/liquidjs) from 10.19.0 to 10.25.6.
- [Release notes](https://github.com/harttle/liquidjs/releases)
- [Changelog](https://github.com/harttle/liquidjs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/harttle/liquidjs/compare/v10.19.0...v10.25.6)

---
updated-dependencies:
- dependency-name: liquidjs
  dependency-version: 10.25.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:41 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5d6599237d Bump the github-actions-updates group across 1 directory with 7 updates (#2034)
Bumps the github-actions-updates group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) | `3` | `4` |
| [pnpm/action-setup](https://github.com/pnpm/action-setup) | `4.2.0` | `5.0.0` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5` | `6` |
| [docker/login-action](https://github.com/docker/login-action) | `3` | `4` |
| [docker/build-push-action](https://github.com/docker/build-push-action) | `6` | `7` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `6` | `7` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7` | `8` |



Updates `docker/setup-buildx-action` from 3 to 4
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

Updates `pnpm/action-setup` from 4.2.0 to 5.0.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v4.2.0...v5.0.0)

Updates `codecov/codecov-action` from 5 to 6
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v5...v6)

Updates `docker/login-action` from 3 to 4
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

Updates `docker/build-push-action` from 6 to 7
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

Updates `actions/upload-artifact` from 6 to 7
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v6...v7)

Updates `actions/download-artifact` from 7 to 8
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7...v8)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: codecov/codecov-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:38 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b13b737461 Bump handlebars from 4.7.8 to 4.7.9 in /site (#2030)
Bumps [handlebars](https://github.com/handlebars-lang/handlebars.js) from 4.7.8 to 4.7.9.
- [Release notes](https://github.com/handlebars-lang/handlebars.js/releases)
- [Changelog](https://github.com/handlebars-lang/handlebars.js/blob/v4.7.9/release-notes.md)
- [Commits](https://github.com/handlebars-lang/handlebars.js/compare/v4.7.8...v4.7.9)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-version: 4.7.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:27 -05:00
Dmitry VerkhoturovandGitHub c9ba8520c7 fix(auth): close OAuth open-redirect by wiring AllowedRedirectHosts (#2049)
* fix(auth): close OAuth open-redirect by wiring AllowedRedirectHosts

Bump go-pkgz/auth/v2 to master (v2.1.2-0.20260421203319-686683f19cf7)
which carries the `from` redirect validator from go-pkgz/auth#275.

The library default with a nil AllowedRedirectHosts is permissive
(preserves legacy behavior for existing consumers on a dep bump), so
just bumping the dep leaves remark42 vulnerable — a crafted
/auth/<provider>/login?from=https://evil.example.com/... still issues
the 307 to the attacker host after the user completes legitimate
OAuth. Verified end-to-end against a local dev-auth instance before
and after this commit.

Wire Opts.AllowedRedirectHosts in getAuthenticator to the operator's
existing --allowed-hosts config, stripping the CSP "self" sentinel
which is not a real hostname. RemarkURL's own host is always implicit
per the library contract, so a default single-site deployment gains
the protection with no config change. Multi-host embeds work as soon
as their embedding hosts are added to AllowedHosts (they already need
to be there for CSP frame-ancestors).

Refreshed vendor tree to match the new module version.

* chore(lint): suppress G703 false positives on image Save

CI's newer gosec flags os.MkdirAll/os.WriteFile in FileSystem.Save with
G703 because id flows in from the caller. id is validated at the HTTP
layer (safePictureSegment in rest_public.go) and dst is derived via
f.location — not a real traversal. Targeted //nolint with reason.

* fix(auth): normalise AllowedRedirectHosts entries + add unit test

Address Copilot review on PR #2049. The previous closure passed raw
s.AllowedHosts entries straight to the auth library, but --allowed-hosts
holds CSP frame-ancestors source expressions: scheme-prefixed values
(https://blog.example.com), entries with ports, and wildcards
(*.cdn.example.com) are all valid there but the auth library compares
against u.Hostname() and would silently drop them — breaking legitimate
redirects on multi-host deployments.

Extract getAllowedRedirectHosts that:
* trims whitespace, drops empty / 'self' / "self" / wildcard entries
* prepends https:// if scheme missing then url.Parse to extract Hostname
* logs a warning on parse failure rather than poisoning the allowlist

Wire the closure in getAuthenticator to call the helper.

Test_getAllowedRedirectHosts covers all the edge cases Copilot flagged
(scheme stripping, port handling, self spellings, wildcards, empty,
mixed real-world).

* fix(auth): preserve explicit port in AllowedRedirectHosts + clarify fs_store nolint

Address Copilot follow-up on PR #2049:

* getAllowedRedirectHosts stripped explicit ports via u.Hostname(), which
  broadened the allowlist. The auth validator checks both Hostname() and
  Host, so an entry like admin.example.com:8443 can and should be kept
  host:port — allowing only that port, not any. Emit u.Host when
  u.Port() != "", u.Hostname() otherwise. Updated tests.

* fs_store Save nolint rationale said "id validated at HTTP layer", but
  Save is reached via image.Service.Save and SaveWithID (cache), neither
  of which is HTTP validation. id is actually a server-generated hash in
  both paths. Updated the comment.
2026-04-21 19:09:26 -05:00
Dmitry VerkhoturovGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>umputun
ee782785f0 test: use testing/synctest to eliminate wall-clock sleeps (#2048)
Go 1.25's testing/synctest package (GA) provides a fake clock bubble
for deterministic goroutine and timer testing. Convert tests that
waited on real-time durations to use synctest, removing most wall-clock
time.Sleep workarounds.

Converted (11 tests, 9 files):
- notify/notify_test.go — all tests, replaced 17 time.Sleep(110ms) with synctest.Wait()
- store/service/service_test.go — VoteSameIPWithDuration, UserReplies, submitImages,
  ResubmitStagingImages, deleteImagesOnCommentDelete
- store/image/{image,bolt_store}_test.go — Cleanup, Submit, SubmitDelay
- store/engine/bolt_test.go — FlagListBlocked
- providers/telegram_test.go — DispatchTelegramUpdates
- migrator/backup_test.go — TestBackup_Do
- _example/memory_store/accessor/data_test.go — FlagListBlocked

Simplifications along the way:
- notify/notify_mock.go: dropped the 10ms time.After delay and
  ctx.Done select in MockDest — the artificial I/O simulation is
  pointless and blocked synctest.Wait from draining the queue
- Removed three dead-code time.Sleep(1s) calls in EditCommentDurationFailed,
  EditCommentAdmin, and Info tests: prepopulated comments from 2017
  already exceed any EditDuration/ReadOnlyAge under real clock, making
  the sleeps meaningless
- UserReplies: replaced the Eventually+Sleep+mutex polling with a
  direct time.Sleep under fake clock

Skipped (incompatible with synctest):
- fs_store_test.go: relies on OS file mtime (real wall clock)
- rss_test.go: needs real wall-clock second boundary for pubDate
- admin/rest_private/rest_public tests: httptest network I/O
- cmd/server_test.go: real HTTP server startup polling

Notes on quirks encountered:
- synctest.Wait() does NOT advance fake time, contrary to what one
  might expect. It only returns once all other bubble goroutines are
  durably blocked. To advance the fake clock, the test goroutine must
  itself call time.Sleep
- BoltDB keys the "last" bucket by comment.Timestamp nanosecond string.
  Rapid b.Create calls under frozen fake time produce identical keys
  and overwrite each other. TestService_UserReplies adds
  time.Sleep(time.Nanosecond) between Creates to advance the clock
- Bolt image Cleanup uses strict age > ttl. Under fake time the
  age-ttl delta is exactly zero at the boundary, so subtract 1ms from
  the passed ttl to stay strictly under

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: umputun <535880+umputun@users.noreply.github.com>
2026-04-18 02:44:21 -05:00
Dmitry VerkhoturovandUmputun 3b1d7be6fc fix(safehttp): clone http.DefaultTransport, sharpen Image.Transport contract
Address review feedback on PR #2044.

safehttp.Transport():
* Clone http.DefaultTransport instead of building a bare &http.Transport{} so
  Proxy, ForceAttemptHTTP2, MaxIdleConns, IdleConnTimeout, TLSHandshakeTimeout
  and ExpectContinueTimeout are inherited (the bare struct loses them all).
  Verified by new TestTransport_PreservesDefaultTransportSettings.
* TestTransport_AllowsPublic: bound the dial of TEST-NET-3 with a 100ms
  context so the test does not depend on real-world routing of 203.0.113.0/24,
  and drop the dead dialer var.

proxy/image.go:
* Document Image.Transport contract: nil installs safehttp.Transport (SSRF-safe);
  caller-supplied transport is the caller's responsibility.
* Replace the misleading "SSRF mitigated by safehttp.Transport" nolint comments
  with one that points at the documented contract above.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun e98657a88a chore(lint): cap multipart upload size and suppress remaining gosec G70x
Address all golangci-lint v2.10.1 (CI's version) findings:

* Add http.MaxBytesReader hard cap to ParseMultipartForm sites in
  rest_private.savePictureCtrl (32MB) and api/migrator (256MB) — fixes
  G120 by bounding total request body before form parsing.

* Suppress G70x in CLI subcommands cmd/{backup,cleanup,import,remap}.go:
  all four issue HTTP requests against operator-supplied RemarkURL/CLI
  flags, never user input. Each suppression carries a one-line reason.

* Suppress G122 in image fs_store cleanup walk: staging directory tree
  is server-only, no untrusted symlinks land there.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun a96bddcb8d chore(lint): suppress gosec G70x false positives in admin/CLI paths
CI's golangci-lint v2.10.1 (newer rule set than my local 2.11.4) flags
four G70x cases the previous run missed. All are false positives:
backup.go and cleanup.go drive HTTP requests against the operator's own
RemarkURL from CLI flags (not user input); migrator.go removes a temp
file whose name was returned by os.CreateTemp (server-controlled). Add
targeted //nolint:gosec comments naming the reason at each site.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun 5ff5059db3 chore(lint): re-enable gosec G703/G704/G705 with targeted suppressions
Commit aca0cff3 silenced the path-traversal, SSRF and XSS taint rules
project-wide as "false positives" while fixing image-proxy SSRF. With
the path-traversal and TitleExtractor SSRF gaps now closed, restore the
rules so future regressions get flagged. The four genuine false positives
that remain (image proxy http.NewRequest, QR png Write, two RSS XML
Writes) get individual //nolint:gosec comments naming the reason.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun ff85bbc5ea fix(ssrf): apply ssrf-safe transport to TitleExtractor
The image proxy got an ssrfSafeTransport in commit aca0cff3 that resolves
DNS first, blocks any IP in private/reserved CIDRs, then dials by IP to
defeat DNS rebinding. The TitleExtractor used to construct comments'
PostTitle from Locator.URL — a user-supplied field — was missed by that
fix and kept using http.DefaultTransport. The hostname allowlist there
checks the parsed URL host but never the IP it resolves to, so a domain
suffix-matching an allowed host (or 127.0.0.1 itself when AllowedHosts
is empty) reaches the metadata service or any other internal endpoint.

The same gosec rule (G704) was excluded globally in .golangci.yml as part
of aca0cff3, so this gap was not caught by the linter either.

Extract the transport into a new safehttp package so it lives in one
place and can be reused, then pass safehttp.Transport() into the
TitleExtractor's http.Client at construction (cmd/server.go). The image
proxy switches to safehttp.Transport() too — same behaviour, no longer
duplicated.

Reproduction in title_test.go uses the production-style client to hit
an httptest.Server (always 127.0.0.1) and asserts the dialer refuses
even though "127.0.0.1" is in the allowed-domains list. A control case
shows the same setup without safehttp.Transport returns the page —
making the original vulnerability explicit.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun 5d88c1b2fa fix(api): drop QR-write nolint dup + trim dead .. check
Address PR #2045 review (umputun):

* The //nolint:gosec on telegramQrCtrl's w.Write(png) was byte-identical
  to the same line in #2044 (gosec-rule restoration). Drop it here so
  the two PRs do not conflict; #2044 owns it.
* `seg == ".."` in safePictureSegment was already covered by the
  strings.Contains(seg, "..") check two lines down — trim and add an
  inline comment so the cover-by-superset is explicit.
2026-04-18 02:15:53 -05:00
Dmitry VerkhoturovandUmputun 114a1be2e9 fix(api): reject control characters in /picture URL segments
Address PR #2045 review feedback (Copilot #2045-1). The previous
safePictureSegment allowed CR/LF/TAB through, so a request such as
GET /api/v1/picture/dev%0Auser/abc.png would inject literal newlines
into the access log line ("GET - /api/v1/picture/dev\nuser/abc.png ...")
— a log-forgery primitive against any operator parsing those logs.

Reject any unicode.IsControl rune in either segment (NUL was already
caught via strings.ContainsAny). New TestRest_LoadPictureRejectsControlCharsInSegment
covers LF, CR, TAB, NUL across both segments.
2026-04-18 02:15:53 -05:00
Dmitry VerkhoturovandUmputun 59c92f8c4d fix(api): reject path traversal and sanitise error in /picture/{user}/{id}
The unauthenticated GET /api/v1/picture/{user}/{id} handler concatenated the
two URL params verbatim into a filesystem path via path.Join, so a request
like /api/v1/picture/../remark.db resolved to <base>/../remark.db, escaping
the image directory. With Partitions=0 (a documented option) this is a
direct arbitrary-file read; with the default Partitions=100 the constructed
path lands in a CRC-derived subdirectory but the server still leaks the
internal filesystem path back to the unauthenticated caller via the JSON
error body — confirmed against demo.remark42.com (master-80c12a3) which
returned `stat /var/folders/.../staging/.../remark.db` for `..` requests.

Validate both URL segments via safePictureSegment (no traversal markers,
no path separators, no NULs) at the handler entry, and replace the raw
storage error with a generic "image not found" response. The original
error is logged for operators.

Reproduction test asserts that ../remark.db, foo/..%2Fremark.db and
%2E%2E/remark.db all return 400 with no internal path leaked.
2026-04-18 02:15:53 -05:00
Dmitry VerkhoturovandUmputun ddcb2c7b5f test(store): use time.UTC in test fixtures to be timezone-agnostic
The store tests stored timestamps with time.Local in their fixtures and
asserted equality against returned values that the engine round-trips
through UTC. assert.Equal compares zone identity, so on UTC machines
(CI, most cloud envs) Local==UTC and the tests passed; on a developer
machine in any other timezone (here BST, UTC+1) TestService_Put,
TestService_List, TestBoltDB_InfoPost, TestBoltDB_InfoList and several
others would fail with same wall-clock numbers but mismatched zones.

Replace time.Local with time.UTC across store/comment_test.go,
store/formatter_test.go, store/service/service_test.go,
store/engine/bolt_test.go, store/engine/engine_test.go. Production code
is untouched.
2026-04-17 19:38:11 -05:00
Dmitry VerkhoturovandUmputun f8ba38779b fix(api): require explicit ?site= in matchSiteID middleware
matchSiteID guarded most authenticated and admin routes with
`if siteID != "" && user.SiteID != siteID`. Dropping the ?site= query
parameter made the check no-op and any authenticated user passed the
middleware. Downstream handlers fell back to reading site from the JSON
body or just used the empty string, so on email/telegram subscribe
endpoints (which read site from body) a user authenticated to siteA
could perform actions targeting siteB without the cross-site guard
ever firing.

Require ?site= to be present and to match user.SiteID. Body-only site
flows are still supported provided the URL also carries the matching
?site= — both must agree, which removes the bypass and keeps the
declared site visible to the middleware.

Reproduction TestRest_matchSiteID enumerates four cases (matching,
mismatched, missing, empty). Existing test calls that relied on the
implicit pass had to add ?site=remark42 to the URL: the addComment
helper now derives the param from c.Locator.SiteID, picture upload
URL gets the param explicitly, and the email/telegram subscribe table
adds it to every endpoint. The negative cases that previously asserted
StatusBadRequest from the handler now correctly assert StatusForbidden
from the middleware.
2026-04-17 19:35:50 -05:00
AlexMa233andGitHub 94d1f6e224 feat: custom oauth2 provider (#2006)
* feat: add configurable custom OAuth2 provider and icons

* fix: reserve built-in custom provider names

* fix: add nolint directive for sha1 import

* fix: harden custom oauth provider validation
2026-04-16 23:10:05 -05:00
Adán Román RuizandUmputun ba3df171d1 #2025 Fix typo in Spanish localization for sort-by 2026-04-14 16:11:30 -05:00
Amir MohamadandGitHub 7ec5af8068 Fix Firefox dark mode white background on comment iframe (#2023)
* fix(embed): set color-scheme on iframe to fix Firefox dark mode

Firefox renders a white background in dark mode when color-scheme is 'none' on the iframe. Set color-scheme to match the active theme on both the outer iframe element and the inner document root, so Firefox uses the correct rendering mode from the start and on theme changes.

* fix(embed): default iframe color-scheme to light when no theme set

Changes the fallback from 'light dark' to 'light' to match the inner document's default behavior, which always defaults to light when no theme is specified.
2026-04-14 15:35:41 -05:00
Dmitry VerkhoturovandUmputun fc6f15534e fix(frontend): preserve orig verbatim in edit textarea (#2040)
The edit textarea was running `data.orig` through the browser's HTML
parser via a detached `<span>.innerHTML` to "decode entities", which
turned user-typed `&lt;`/`&gt;` into real `<`/`>`. On save, blackfriday
then saw a real `<script>` tag, bluemonday stripped it, and the comment
body collapsed to an empty string.

The decode block predates commit 243c835 (2022) which stopped the
backend from sanitising `orig` with bluemonday. Before 243c835, orig
came back HTML-escaped from the API and the frontend compensated.
After 243c835 the backend stores and returns orig byte-for-byte, but
the frontend decode was never removed — so it has been silently
corrupting user input containing entities for ~3.5 years.

The backend contract is clear: `orig` is the raw user input, never
rendered as HTML. The frontend should echo it back into the textarea
unchanged. This change removes the decode and adds 45 table-driven
regression tests covering entity round-trips, unicode edge cases,
and markdown constructs.
2026-04-12 11:55:24 -05:00
Dmitry VerkhoturovandUmputun 80c12a3f10 chore(deps): update Go modules
Bump Go dependencies in both backend/ and backend/_example/memory_store.

Notable updates:
- github.com/go-pkgz/lgr v0.12.1 -> v0.12.3
- github.com/klauspost/compress v1.18.2 -> v1.18.5
- github.com/PuerkitoBio/goquery v1.11.0 -> v1.12.0
- github.com/montanaflynn/stats v0.7.1 -> v0.9.0
- github.com/redis/go-redis/v9 v9.17.2 -> v9.18.0
- github.com/slack-go/slack v0.17.3 -> v0.21.1
- go.mongodb.org/mongo-driver v1.17.6 -> v1.17.9
- golang.org/x/crypto v0.48.0 -> v0.50.0
- golang.org/x/net v0.49.0 -> v0.53.0
- golang.org/x/image v0.36.0 -> v0.39.0
- golang.org/x/sys v0.41.0 -> v0.43.0
- golang.org/x/{oauth2,sync,text} minor bumps

Key markdown/sanitisation libs (bluemonday v1.0.27,
alecthomas/chroma/v2 v2.23.1, russross/blackfriday/v2 v2.1.0,
Depado/bfchroma/v2 v2.0.0) are already at the latest available
versions and were not bumped.

Verified the Chroma span-class allowlist regex in
backend/app/store/comment.go:128-131 is still fully in sync with
chroma/v2 types.go StandardTypes map (86 classes, byte-equal after
sorting). The inline comment references commit c263f6f which is
stale (Chroma is at v2 now), but the class list content is current.

Ran `go mod tidy` + `go mod vendor` + full race test suite on both
modules. All green. Added a reminder in CLAUDE.md that updating
backend/ Go modules also requires `go mod tidy` in
backend/_example/memory_store since the example module uses a
local replace directive and inherits indirect deps from the main
module.
2026-04-12 11:52:57 -05:00
UmputunandGitHub bea67f0136 Merge pull request #2032 from umputun/dependabot/go_modules/backend/_example/memory_store/golang.org/x/image-0.38.0
Bump golang.org/x/image from 0.36.0 to 0.38.0 in /backend/_example/memory_store
2026-04-04 23:19:12 -05:00
dependabot[bot]andGitHub 8b9e5c6c8c Bump golang.org/x/image in /backend/_example/memory_store
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.36.0 to 0.38.0.
- [Commits](https://github.com/golang/image/compare/v0.36.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/image
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-30 16:29:36 +00:00
Dmitry VerkhoturovandGitHub 06436ff9b0 Migrate batch 1 components from BEM to CSS Modules (#2014)
* feat: migrate batch 1 components from BEM to CSS Modules

Migrate 8 components from BEM to CSS Modules:
- button (7 BEM files -> 1 module)
- dropdown (7 BEM files -> 1 module)
- thread (3 BEM files -> 1 module)
- auth-panel (2 BEM files -> 1 module)
- dropdown-item, list-comments, subscribe-by-rss, settings (from batch 0 PR #2013)

Consolidates 19 BEM CSS files into 8 CSS Module files. Uses clsx for
conditional class composition, replacing bem-react-helper's b() calls.
Class naming follows the established convention: BEM block = .root,
elements = camelCase, modifiers = camelCase.

Visual regression verification on built artefacts:
- remark.css: 43,779 -> 43,299 bytes (480 bytes smaller)
- last-comments.css: 18,792 -> 18,776 bytes (16 bytes smaller)
- remark.js: 256,709 -> 304,837 bytes (48KB larger, expected: CSS Module
  classname mappings now live in JS instead of plain strings)
- Dark theme: pixel-identical (zero difference)
- Light theme: pixel-identical (0.21% diff is the native demo page
  "Toggle theme" button, not any remark42 widget element)

Also updates CLAUDE.md CSS guideline to reflect the migration status.

* Migrate remaining BEM components to CSS Modules (final batch)

Migrate the last 4 BEM components to CSS Modules, completing the
migration and removing bem-react-helper from the project entirely.

Components migrated:
- subscribe-by-email (1 BEM CSS file -> 1 module)
- comment-form + markdown-toolbar (20 BEM CSS files -> 2 modules)
- comment (19 BEM CSS files -> expanded existing module)
- root (10 BEM CSS files -> expanded existing module)

Consolidates ~50 BEM CSS files into 4 new + 2 expanded CSS Module files.
Removes bem-react-helper dependency — all components now use clsx for
conditional class composition.

Dead CSS cleanup during migration:
- Orphaned comment-actions selectors in comment theme CSS (already migrated)
- Dead BEM modifiers: comment_disabled, comment_pinned, comment_guest
- Dead element: comment__user-id (CSS existed but never used in TSX)
- Dead button type classes: comment-form__button_type_preview/_send
- Dead mix values: auth-email-login-form__back-button, comment-form__email-dropdown

Key implementation details:
- comment_highlighting stays global via :global() (imperatively added by classList)
- Bare .dark/.light theme class preserved on root wrapper (8+ modules depend on it)
- raw-content.css kept as global utility CSS (syntax highlighting)

Visual regression verification on built artefacts:
- remark.css: 43,779 -> 36,106 bytes (-17.5%)
- last-comments.css: 18,792 -> 13,955 bytes (-25.7%)
- remark.js: 256,709 -> 253,637 bytes (-1.2%)
- last-comments.js: 121,726 -> 120,795 bytes (-0.8%)
- Total: 441,006 -> 424,493 bytes (-3.7%)
- Screenshot comparison: pixel-identical across light/dark themes
2026-03-25 16:53:32 -05:00
Dmitry VerkhoturovandGitHub b888a53759 Migrate dropdown-item, list-comments, subscribe-by-rss, and settings from BEM to CSS Modules (#2013)
Consolidate legacy BEM CSS files into CSS Modules for 4 components:
- dropdown/__item: 1 CSS file → dropdown-item.module.css
- list-comments: 1 CSS file → list-comments.module.css (removed unused
  comments-list class that had no CSS rules)
- comment-form/__subscribe-by-rss: 1 CSS file → subscribe-by-rss.module.css,
  removed dead titleClass prop and dead __rss-link directory
- settings: 10 CSS files → settings.module.css, removed dead
  .settings__blocked-users-username CSS rule

Built artefact comparison (master vs branch):
- 83 of 89 files in /srv/web/ are byte-identical (all locale bundles,
  SVGs, HTML pages unchanged)
- 6 files differ: remark.css/js/mjs and last-comments.css/js/mjs
- CSS changes are class name hash shifts (e.g. G_A → H_A) caused by
  webpack's module ordering, plus 3 new var() fallback values added
  by the CSS modules build; all property:value pairs are preserved
- JS changes are minified variable name shifts (O ↔ A, I ↔ L) from
  changed import order; no logic changes
- Visual comparison (pixel-by-pixel screenshots of both light and dark
  themes on the demo page) shows 0 different pixels
- Bundle sizes: remark.css -626 bytes, remark.js -512 bytes,
  last-comments.css -16 bytes (dead CSS removed)
2026-03-25 16:42:44 -05:00
Dmitry VerkhoturovandGitHub c26f45e55e Clean up deprecated CSS and fix silent CSS bugs in frontend (#2012)
* frontend: remove deprecated iframe attrs and non-standard CSS

Three separate cleanups:

1. remove deprecated HTML attributes from iframe creation (create-iframe.ts)
   - frameborder="0": deprecated since HTML5; border is already set to none via CSS
   - allowtransparency="true": non-standard Microsoft attribute never in any spec;
     transparency is handled by body { background: transparent } in CSS instead
   - scrolling="no": deprecated since HTML5; overflow is already hidden via CSS
   - horizontalscrolling/verticalscrolling: non-standard IE-era attributes with
     no effect in modern browsers; remove without replacement

2. replace allowtransparency with explicit CSS (global.css)
   - add background: transparent to body; this is the spec-correct way to make
     an iframe document transparent, as documented by MDN

3. drop -moz-touch-enabled media query prefix (5 comment CSS files)
   - -moz-touch-enabled was a Firefox-only non-standard media feature removed
     in Firefox 58 (2018); pointer: coarse is the standard equivalent and was
     already present as the second condition in every query, so removing the
     dead -moz prefix reduces the media query to just (pointer: coarse)

note: colorScheme: 'none' in create-iframe.ts is intentionally left unchanged;
it is tracked by #1430 and requires a broader color-scheme implementation

* frontend: fix CSS bugs and replace deprecated properties

Bugs fixed:

- comment-votes.module.css: add missing comma between transition values;
  without it the shorthand was invalid and colour transitions on vote
  buttons were silently ignored

- icon-button.module.css: fix "transfrom" typo (should be "transform");
  the misspelling made the transition declaration a no-op, so the hover
  scale animation jumped instantly instead of easing

- auth.module.css: remove doubly-nested rgb(rgb(var(…))) call; the outer
  rgb() rejected the inner rgb() result, so the .title element's colour
  fell back to inherited instead of the intended --secondary-text-color

Deprecated properties replaced:

- comment-form__markdown-toolbar.css: replace deprecated clip: rect()
  with clip-path: inset(50%); clip was deprecated in CSS Masking Level 1

- raw-content.css: replace word-wrap with overflow-wrap; word-wrap was
  renamed in CSS Text Level 3, all current browsers support overflow-wrap

- global.css: remove redundant literal-colour fallback lines before
  var() declarations in .preloader and .preloader_view_iframe; the var()
  calls already have inline fallback values (e.g. var(--color6, #fff)),
  making the preceding duplicate property and its stylelint-disable
  comment unnecessary since IE11 EOL

* move border:none from inline style to widget__comments-frame class
2026-03-25 16:42:40 -05:00
Dmitry VerkhoturovandGitHub ba7c3aed94 refactor: modernise Go code with go fix and manual improvements (#2027)
Apply go fix ./... analysers (Go 1.26) across backend and examples:
- interface{} → any (type alias, no behaviour change)
- for i := 0; i < N; i++ → for range N / for i := range N
- slices.Contains / slices.ContainsFunc replacing manual loops
- strings.SplitSeq replacing strings.Split in range (avoids allocation)
- strings.CutPrefix replacing HasPrefix+TrimPrefix
- min() replacing manual if/else
- fmt.Appendf replacing []byte(fmt.Sprintf(...))
- strings.Builder replacing string += concatenation
- wg.Go(func(){}) replacing wg.Add(1)/go/wg.Done() pattern
- removed redundant ii := i loop variable copies (unnecessary since Go 1.22)

omitempty on struct-typed JSON fields: go fix removed omitempty from
struct-typed fields (time.Time, PostInfo, UserDetailEntry) because
encoding/json's omitempty never applied to struct types — it was always
a no-op. Kept as bare tags (no omitzero replacement) to preserve the
existing serialisation behaviour.
2026-03-25 16:42:37 -05:00
UmputunandGitHub 8aafc8fcd7 Merge pull request #2020 from paskal/ci/add-pnpm-cache
ci: add node dependency caching
2026-03-16 01:38:00 -05:00
Dmitry Verkhoturov ab9e6675cf fix type check failure in @remark42/api package
Add skipLibCheck to skip type checking of .d.ts files in node_modules,
matching the setting already used by the main remark42 app. Fixes
@types/eslint-scope vs @types/eslint type incompatibility.
2026-03-07 21:37:46 +00:00
Dmitry Verkhoturov ed67390dea ci: add pnpm dependency caching via setup-node
Replace manual actions/cache steps with built-in setup-node cache support.
Add cache: pnpm and cache-dependency-path to all setup-node steps in both
ci-frontend.yml and ci-frontend-api.yml. Move pnpm install before setup-node
as required for pnpm caching to work.
2026-03-07 21:22:25 +00:00
Umputun aca0cff399 fix: IPv6 address truncation and image proxy SSRF vulnerabilities
Replace strings.Split(RemoteAddr, ":") with net.SplitHostPort for correct
IPv6 address extraction in vote deduplication and comment IP tracking.

Harden image proxy: add SSRF-safe transport blocking private/reserved IPs
at connection time with DNS rebinding protection, sanitize error messages
to prevent information leakage, add response size limit via io.LimitReader.

Fix shadowed error variables in BlockedUsers, SetTitle, and Delete methods.
Exclude gosec taint analysis false positives at linter config level.
2026-02-28 04:13:07 -06:00
Dmitry VerkhoturovandUmputun f359256489 docs: document placeholder support in the remark42 div (#1990)
Clarify that any content placed inside the `<div id="remark42">` is
automatically removed once the iframe signals it has initialised.
Update all code examples across getting-started, frontend config, and
Astro/Gatsby integration guides to use "Comments loading..." as the
placeholder so the feature is visible by default.
2026-02-22 17:54:43 -06:00
Dmitry VerkhoturovandUmputun 336f17e7b7 Document EDIT_TIME=0 behavior in parameters
Setting edit-time to 0 disables both comment editing and staged image cleanup.
2026-02-22 17:54:19 -06:00
Dmitry VerkhoturovandUmputun 0105bc2314 Drop GitHub token permissions on deploy jobs
Deploy jobs only curl an external updater URL and need no GitHub API
access. Without an explicit permissions block they inherit the workflow
default, which may include contents:write, packages:write, etc.
Setting permissions to {} limits the blast radius if a job is
compromised.
2026-02-21 20:16:00 -06:00
Dmitry VerkhoturovandUmputun 78d6de6bce Add X-Content-Type-Options and Referrer-Policy security headers
Add two missing security headers to the existing securityHeadersMiddleware:

- X-Content-Type-Options: nosniff — prevents browsers from MIME-sniffing
  responses away from the declared Content-Type, stopping e.g. a
  user-uploaded image from being reinterpreted as executable HTML/JS

- Referrer-Policy: strict-origin-when-cross-origin — limits URL information
  leaked in the Referer header on cross-origin requests to just the origin
  (no path), and sends nothing at all on HTTPS-to-HTTP downgrades
2026-02-21 20:14:44 -06:00
dependabot[bot]GitHubpaskaldependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
638fa63e81 Bump the go-modules-updates group in /backend with 7 updates (#1995)
* Bump the go-modules-updates group in /backend with 7 updates

Bumps the go-modules-updates group in /backend with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.21.1` | `2.23.1` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.2.3` | `5.2.4` |
| [github.com/go-pkgz/rest](https://github.com/go-pkgz/rest) | `1.20.6` | `1.21.0` |
| [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) | `5.3.0` | `5.3.1` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.46.0` | `0.47.0` |
| [golang.org/x/image](https://github.com/golang/image) | `0.34.0` | `0.35.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.48.0` | `0.49.0` |


Updates `github.com/alecthomas/chroma/v2` from 2.21.1 to 2.23.1
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.21.1...v2.23.1)

Updates `github.com/go-chi/chi/v5` from 5.2.3 to 5.2.4
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.3...v5.2.4)

Updates `github.com/go-pkgz/rest` from 1.20.6 to 1.21.0
- [Release notes](https://github.com/go-pkgz/rest/releases)
- [Commits](https://github.com/go-pkgz/rest/compare/v1.20.6...v1.21.0)

Updates `github.com/golang-jwt/jwt/v5` from 5.3.0 to 5.3.1
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.3.0...v5.3.1)

Updates `golang.org/x/crypto` from 0.46.0 to 0.47.0
- [Commits](https://github.com/golang/crypto/compare/v0.46.0...v0.47.0)

Updates `golang.org/x/image` from 0.34.0 to 0.35.0
- [Commits](https://github.com/golang/image/compare/v0.34.0...v0.35.0)

Updates `golang.org/x/net` from 0.48.0 to 0.49.0
- [Commits](https://github.com/golang/net/compare/v0.48.0...v0.49.0)

---
updated-dependencies:
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-version: 2.23.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.2.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/rest
  dependency-version: 1.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-version: 5.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-version: 0.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-version: 0.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-version: 0.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>

* Run go mod tidy in examples directory

Co-authored-by: paskal <712534+paskal@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: paskal <712534+paskal@users.noreply.github.com>
2026-02-14 19:48:24 -06:00
UmputunandGitHub e3b0d63648 Merge pull request #1999 from umputun/configurable-microsoft-tenant
feat: make Microsoft Entra ID tenant configurable
2026-02-10 22:53:31 -06:00
UmputunandGitHub b38d91cb0f Merge pull request #2000 from umputun/fix/quick-fixes-1946-1991-1996
Fix email encoding, image cleanup CPU spin, and demo template paths
2026-02-10 22:52:40 -06:00
UmputunandGitHub d6d53ff2e0 Merge pull request #2001 from umputun/fix/admin-edit-frontend-1986
Fix frontend not respecting ADMIN_EDIT config
2026-02-10 22:51:32 -06:00
UmputunandGitHub 195becc6ee Merge pull request #2002 from umputun/fix/placeholder-clearing-1990
Clear user placeholder content when comments iframe loads
2026-02-10 22:50:51 -06:00
UmputunandGitHub 5bc5167a31 Merge pull request #2003 from umputun/docs/email-template-variables
Document email template variables and plain-text email setup
2026-02-10 22:50:20 -06:00
UmputunandGitHub 1320b1f055 Merge pull request #1984 from umputun/dependabot/github_actions/github-actions-updates-35b2a8182b
Bump the github-actions-updates group with 3 updates
2026-02-10 22:40:25 -06:00
UmputunandGitHub 283e2c19c7 Merge pull request #1994 from umputun/dependabot/npm_and_yarn/frontend/lodash-es-4.17.23
Bump lodash-es from 4.17.21 to 4.17.23 in /frontend
2026-02-10 22:40:18 -06:00
UmputunandGitHub 55d9e22373 Merge pull request #1997 from umputun/dependabot/npm_and_yarn/frontend/webpack-5.104.1
Bump webpack from 5.73.0 to 5.104.1 in /frontend
2026-02-10 22:40:09 -06:00
Dmitry Verkhoturov 31e20fc26d feat: make Microsoft Entra ID tenant configurable
Add AUTH_MICROSOFT_TENANT env var to allow configuring the Azure AD
tenant for single-tenant Entra ID applications, which cannot use the
default /common endpoint.

Depends on go-pkgz/auth#266

Closes #1998
2026-02-11 00:45:38 +00:00
Dmitry Verkhoturov c2cc2305c1 Document email template variables and plain-text email setup 2026-02-11 00:08:05 +00:00
Dmitry Verkhoturov 4d0bd29b45 Clear placeholder content when comments iframe loads
Remove non-iframe child nodes from the root element once the
iframe signals it has initialised, allowing users to add
loading placeholders that get cleaned up automatically. Fixes #1990
2026-02-10 23:52:22 +00:00
Dmitry Verkhoturov a1215d87d9 Fix frontend not respecting ADMIN_EDIT for comment editing
Add admin_edit field to frontend Config types and use it in
comment component to give admins unlimited edit time and allow
editing comments with replies. Hide countdown timer when
editDeadline is Infinity. Fixes #1986
2026-02-10 23:52:17 +00:00
Dmitry Verkhoturov 3c2679a1d5 Fix hardcoded /web paths in demo template
Use REMARK_URL template variable for widget links so demo page
works with non-root path prefixes. Fixes #1996
2026-02-10 23:52:12 +00:00
Dmitry Verkhoturov 79177e52f9 Fix 100% CPU when EDIT_TIME=0 in image cleanup
When EditDuration is zero or negative, cleanupTTL becomes zero,
causing time.After(0) to fire immediately in a tight loop.
Block on ctx.Done() instead when edit duration is disabled. Fixes #1991
2026-02-10 23:52:12 +00:00
Dmitry Verkhoturov baa615a0d1 Fix NOTIFY_EMAIL_FROM plus sign encoding in mailto URLs
URL-encode e.From in mailto query parameters so that + characters
are preserved instead of being decoded as spaces. Fixes #1946
2026-02-10 23:52:12 +00:00
dependabot[bot]andGitHub e665dcf9e2 Bump webpack from 5.73.0 to 5.104.1 in /frontend
Bumps [webpack](https://github.com/webpack/webpack) from 5.73.0 to 5.104.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Changelog](https://github.com/webpack/webpack/blob/main/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack/compare/v5.73.0...v5.104.1)

---
updated-dependencies:
- dependency-name: webpack
  dependency-version: 5.104.1
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-06 23:31:33 +00:00
Dmitry VerkhoturovandUmputun b7a13a6636 Fix site rebuild on release
The paths filter was applied to tag events, preventing site rebuilds
when releases don't include site changes. Switch to release event
trigger which always fires on new releases, ensuring the site fetches
the latest version from GitHub API.

Closes #1992
2026-02-05 11:06:58 -06:00
dependabot[bot]andGitHub 218570cfad Bump lodash-es from 4.17.21 to 4.17.23 in /frontend
Bumps [lodash-es](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.21...4.17.23)

---
updated-dependencies:
- dependency-name: lodash-es
  dependency-version: 4.17.23
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-21 23:06:53 +00:00
dependabot[bot]andGitHub 4c9a791d6d Bump the github-actions-updates group with 3 updates
Bumps the github-actions-updates group with 3 updates: [actions/cache](https://github.com/actions/cache), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/cache` from 4 to 5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

Updates `actions/upload-artifact` from 5 to 6
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v5...v6)

Updates `actions/download-artifact` from 6 to 7
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-01 11:09:55 +00:00
Umputun cdad560df3 Update backend base image to buildgo-v1.17.0 in Dockerfile for artifacts build 2025-12-24 02:55:38 -06:00
Umputun 307e69e5c1 Bump dependencies
- chroma/v2: v2.20.0 → v2.21.1
- go-pkgz/auth/v2: v2.1.0 → v2.1.1
- go-pkgz/rest: v1.20.4 → v1.20.6
- golang.org/x/* packages to latest

Also exclude "meaningless package names" revive warning in linter config.
2025-12-24 01:48:14 -06:00
Dmitry VerkhoturovandUmputun d5b07d7670 Fix WriteHeader + RenderJSON causing wrong Content-Type header
Replace WriteHeader() + RenderJSON() pattern with EncodeJSON() which
properly sets Content-Type header before writing status code. The
previous pattern caused Content-Type to default to text/plain instead
of application/json, breaking frontend JSON parsing.

Fixes #1979
2025-12-16 13:01:19 -06:00
Dmitry VerkhoturovandGitHub 41b75eba08 Merge pull request #1977 from umputun/docker-native-arm64-runners
Improve GitHub Actions workflows security and performance
2025-12-08 14:46:13 -06:00
UmputunandGitHub 4aa8362de1 Merge pull request #1976 from umputun/ci-native-arm64-runners
Migrate Docker builds to native GitHub ARM64 runners
2025-12-07 16:01:57 -06:00
Umputun dc168f69bf Fix dual-registry Docker builds and drop armv7
- Remove armv7 platform (eliminates QEMU emulation bottleneck)
- Build separately to ghcr.io and DockerHub with distinct digests
- Create registry-specific manifests from corresponding digests
- Update expected digest count from 3 to 2 per registry
2025-12-07 15:58:35 -06:00
Dmitry Verkhoturov 2a4a1591fa Migrate Docker builds to native GitHub ARM64 runners
Replace QEMU emulation with GitHub's native ARM64 runners for faster builds:
- Use ubuntu-24.04-arm for ARM builds instead of QEMU emulation
- Split build job into matrix for parallel platform builds
- Add digest-based workflow for multi-arch manifest creation
- Keep build-test on ARM64 runner for faster PR verification
2025-12-07 21:39:59 +00:00
Dmitry VerkhoturovandUmputun bc612ddf81 Remove redundant frame ancestors log from middleware
The log message was printed on every request when allowed ancestors
were configured, creating unnecessary noise in the logs.
2025-12-04 11:15:06 -06:00
Dmitry VerkhoturovandUmputun 9132a6158b Add display name format to EMAIL_FROM examples in docs
Show users they can use "Display Name"<email@example.com> format
for AUTH_EMAIL_FROM and NOTIFY_EMAIL_FROM settings.
2025-12-04 11:14:32 -06:00
Dmitry VerkhoturovandUmputun 658bf307b1 Add mailto: to CSP frame-src directive
Allow mailto links on the deleteme page to work without being blocked
by Content Security Policy.
2025-12-04 11:13:50 -06:00
Fredrik AppelrosandUmputun c6efcc56a9 Fix deleteme feature
The last step of the data deletion feature–when the admin user visits the link sent by the user requesting to delete their data–was broken due to the deleteme.js script not being loaded.
2025-12-04 11:13:50 -06:00
dependabot[bot]andUmputun bdee00b662 Bump nanoid from 3.3.7 to 3.3.8 in /site
Bumps [nanoid](https://github.com/ai/nanoid) from 3.3.7 to 3.3.8.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.3.7...3.3.8)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-04 11:13:02 -06:00
kumudiaorongandUmputun 3013d03be2 fix: Modify the example to avoid misunderstanding 2025-12-04 11:12:32 -06:00
4a2bb5eda8 #1833 - Toolbar buttons are stuck to the main comment form (#1948)
* 1833 - Toolbar buttons are stuck to the main comment form

* Add readonly and JSDoc to CommentForm textareaId properties

Improve code quality based on review feedback: mark textareaId as
readonly since it should never change after construction, and add
JSDoc to static textareaCounter explaining its purpose.

---------

Co-authored-by: Dmitry Verkhoturov <paskal.07@gmail.com>
2025-12-04 11:12:05 -06:00
d01b738741 Implement function to prune string keeping HTML closing tags (#1870)
* Implement function to prune string keeping HTML closing tags

Fixes #1587

* change const name

remove unneeded comment

* move pruneHTML to separated file

* move const back to telegram.go

* Add unit tests for string array manipulation and HTML pruning

Introduce comprehensive test cases for stringArr methods (Push, Pop, Unshift, Shift, String) to ensure correct behavior and state management. Additionally, add tests for HTML pruning functions (pruneHTML, pruneStringToWord) to validate handling of length constraints and formatting scenarios.

* Improve behavior

* Fix pruneHTML to count visible text only, add parent text pruning

- Fix bug where HTML tags were counted toward the character limit
  instead of only visible text content
- Add pruning for parent comment text in Telegram notifications
- Simplify pruneStringToWord using strings.LastIndex
- Remove unused stringArr type and its tests
- Consolidate and simplify test cases

---------

Co-authored-by: Umputun <umputun@gmail.com>
Co-authored-by: Dmitry Verkhoturov <paskal.07@gmail.com>
2025-12-04 11:11:10 -06:00
Dmitry VerkhoturovandGitHub 564e8ff316 Update go dependencies (#1972) 2025-12-03 19:47:01 -06:00
dependabot[bot]andUmputun b451142790 Bump js-yaml from 3.14.1 to 3.14.2 in /site
Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.14.1 to 3.14.2.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/3.14.1...3.14.2)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 3.14.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 18:18:10 -06:00
dependabot[bot]andUmputun 6d8a0c783b Bump the go-modules-updates group across 1 directory with 11 updates
Bumps the go-modules-updates group with 11 updates in the /backend directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.2` | `1.11.0` |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.15.0` | `2.20.0` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.2.1` | `5.2.3` |
| [github.com/go-chi/cors](https://github.com/go-chi/cors) | `1.2.1` | `1.2.2` |
| [github.com/go-pkgz/jrpc](https://github.com/go-pkgz/jrpc) | `0.3.1` | `0.4.0` |
| [github.com/go-pkgz/lgr](https://github.com/go-pkgz/lgr) | `0.12.0` | `0.12.1` |
| [github.com/go-pkgz/rest](https://github.com/go-pkgz/rest) | `1.20.3` | `1.20.4` |
| [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) | `5.2.2` | `5.3.0` |
| [github.com/stretchr/testify](https://github.com/stretchr/testify) | `1.10.0` | `1.11.1` |
| [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt) | `1.4.0` | `1.4.3` |
| [golang.org/x/image](https://github.com/golang/image) | `0.26.0` | `0.33.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.10.2 to 1.11.0
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.2...v1.11.0)

Updates `github.com/alecthomas/chroma/v2` from 2.15.0 to 2.20.0
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.15.0...v2.20.0)

Updates `github.com/go-chi/chi/v5` from 5.2.1 to 5.2.3
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.1...v5.2.3)

Updates `github.com/go-chi/cors` from 1.2.1 to 1.2.2
- [Release notes](https://github.com/go-chi/cors/releases)
- [Commits](https://github.com/go-chi/cors/compare/v1.2.1...v1.2.2)

Updates `github.com/go-pkgz/jrpc` from 0.3.1 to 0.4.0
- [Release notes](https://github.com/go-pkgz/jrpc/releases)
- [Commits](https://github.com/go-pkgz/jrpc/compare/v0.3.1...v0.4.0)

Updates `github.com/go-pkgz/lgr` from 0.12.0 to 0.12.1
- [Release notes](https://github.com/go-pkgz/lgr/releases)
- [Commits](https://github.com/go-pkgz/lgr/compare/v0.12.0...v0.12.1)

Updates `github.com/go-pkgz/rest` from 1.20.3 to 1.20.4
- [Release notes](https://github.com/go-pkgz/rest/releases)
- [Commits](https://github.com/go-pkgz/rest/compare/v1.20.3...v1.20.4)

Updates `github.com/golang-jwt/jwt/v5` from 5.2.2 to 5.3.0
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.2...v5.3.0)

Updates `github.com/stretchr/testify` from 1.10.0 to 1.11.1
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.10.0...v1.11.1)

Updates `go.etcd.io/bbolt` from 1.4.0 to 1.4.3
- [Release notes](https://github.com/etcd-io/bbolt/releases)
- [Commits](https://github.com/etcd-io/bbolt/compare/v1.4.0...v1.4.3)

Updates `golang.org/x/image` from 0.26.0 to 0.33.0
- [Commits](https://github.com/golang/image/compare/v0.26.0...v0.33.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-version: 1.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-version: 2.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/cors
  dependency-version: 1.2.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/jrpc
  dependency-version: 0.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/lgr
  dependency-version: 0.12.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/rest
  dependency-version: 1.20.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-version: 5.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/stretchr/testify
  dependency-version: 1.11.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: go.etcd.io/bbolt
  dependency-version: 1.4.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-version: 0.33.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 18:17:46 -06:00
notmalicikandGitHub d925584af8 Romanian language added (#1962) 2025-12-03 12:57:07 -06:00
dependabot[bot]andUmputun 22ee7a06d5 Bump playwright from 1.25.0 to 1.55.1 in /frontend
Bumps [playwright](https://github.com/microsoft/playwright) from 1.25.0 to 1.55.1.
- [Release notes](https://github.com/microsoft/playwright/releases)
- [Commits](https://github.com/microsoft/playwright/compare/v1.25.0...v1.55.1)

---
updated-dependencies:
- dependency-name: playwright
  dependency-version: 1.55.1
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:44:51 -06:00
dependabot[bot]andUmputun a311ff7b38 Bump the github-actions-updates group across 1 directory with 6 updates
Bumps the github-actions-updates group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [actions/checkout](https://github.com/actions/checkout) | `4` | `5` |
| [actions/setup-go](https://github.com/actions/setup-go) | `5` | `6` |
| [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) | `6` | `8` |
| [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` |
| [pnpm/action-setup](https://github.com/pnpm/action-setup) | `4.1.0` | `4.2.0` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `5` |



Updates `actions/checkout` from 4 to 5
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

Updates `actions/setup-go` from 5 to 6
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v6)

Updates `golangci/golangci-lint-action` from 6 to 8
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v6...v8)

Updates `actions/setup-node` from 4 to 6
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v6)

Updates `pnpm/action-setup` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v4.1.0...v4.2.0)

Updates `actions/upload-artifact` from 4 to 5
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/setup-go
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: golangci/golangci-lint-action
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/setup-node
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:43:50 -06:00
ClaudeandUmputun 88257931ca Update Google and X (Twitter) logos to latest brand versions
- Update Google logo to use gradient version per new brand guidelines
- Replace Twitter bird logo with X logo
- Add light/dark variants for X logo (like Apple and GitHub)
- Update oauth.consts.ts to use new X logo variants

Fixes #1957
2025-12-03 11:43:25 -06:00
dependabot[bot]andUmputun 588ec169ff Bump golang.org/x/crypto from 0.37.0 to 0.45.0 in /backend
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.37.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.37.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:43:03 -06:00
dependabot[bot]andUmputun d9efa765d1 Bump golang.org/x/crypto in /backend/_example/memory_store
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.37.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.37.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:42:34 -06:00
Dmitry VerkhoturovandGitHub baf0db1947 Get rid of github.com/go-chi/render use (#1919)
Replace go-chi/render with go-pkgz/rest for JSON responses and custom
helpers for HTML/plain text responses.

Key changes:
- Replace render.JSON/render.Status with rest.RenderJSON and explicit
  w.WriteHeader() calls
- Replace render.DecodeJSON with json.NewDecoder().Decode()
- Add SendErrorJSON helper that sets Content-Type header before
  WriteHeader (required since rest.RenderJSON can't set headers after
  WriteHeader is called)
- Add HTMLResponse and PlainTextResponse helpers

Fix export double-execution in migrator.go:
The original code called Export twice - once to io.Discard to check for
errors, then again to actually write. This was wasteful and had a race
condition risk. Now file mode buffers to memory first for atomic
success/failure, while stream mode writes directly with proper error
handling.
2025-12-03 11:41:29 -06:00
UmputunandGitHub 34ea4c3e83 Migrate golangci-lint to v2 and update Go version (#1965)
* migrate golangci-lint to v2 and update go version

- migrated .golangci.yml to version 2 format
- updated go.mod from 1.23.0 to 1.24
- removed deprecated run.timeout configuration

* update to go 1.25 and baseimage v1.17.0

- updated go.mod to go 1.25
- updated Dockerfile to use buildgo-v1.17.0 (go 1.25.0)
- updated Dockerfile to use app-v1.17.0

* fix flaky tests with proper synchronization

- use assert.Eventually instead of fixed sleep in TestService_Many
- wait for webhook before shutdown in TestMain_WithWebhook
- fixes race conditions exposed by Go 1.25 scheduler changes

* fix data race in MockDest.closed field

- add IsClosed() method with proper locking
- add locking to String() method
- use IsClosed() in tests instead of direct field access
- fixes race condition detected by go test -race

* update example go.mod to go 1.25

* update golangci-lint to v2.6.0 for go 1.25 support

* fix linter issue and update CLAUDE.md

- merge conditional assignment in example accessor/data.go
- add reminder in CLAUDE.md to always test and lint examples before committing

* update example Dockerfile to baseimage v1.17.0 for go 1.25
2025-11-02 14:17:39 -06:00
UmputunandGitHub 8112f445f4 Merge pull request #1947 from schnerring/add-backup-docs
docs: improve backup instructions; closes #1613
2025-10-01 02:02:46 -05:00
UmputunandGitHub bd7bbe629f Merge pull request #1953 from diosfera/master
Add Macedonian translation
2025-10-01 02:01:43 -05:00
diosfera ed8b3c314d Delete package-lock.json file 2025-09-21 20:07:12 +02:00
diosfera 51ec396691 Add Macedonian translation 2025-09-17 06:29:51 +02:00
Michael Schnerring ca8ab66812 docs: improve backup instructions; closes #1613 2025-09-04 22:20:18 +02:00
UmputunandGitHub eaa64bac45 Merge pull request #1929 from umputun/fix/auth-send-jwt-header
Fix login persistence with AUTH_SEND_JWT_HEADER enabled
2025-07-06 17:55:09 -05:00
UmputunandGitHub abdca907a7 Merge branch 'master' into fix/auth-send-jwt-header 2025-07-06 15:39:12 -05:00
UmputunandGitHub a9c8cf51a0 Merge pull request #1937 from umputun/docs/telegram-group-notifications 2025-06-16 00:43:37 -05:00
Dmitry Verkhoturov d829a57061 docs: update Telegram configuration with group notification details
Add information about sending notifications to users, groups, and channels.
Document that public group usernames can be used without @ symbol as IDs.

Addresses discussion #1756
2025-06-16 05:19:50 +01:00
UmputunandGitHub 1dffb2f16e Merge pull request #1936 from talentedunicorn/patch-1
Fixed typo
2025-06-06 12:12:56 -05:00
EzeandGitHub 9e2a1da0df Fixed typo 2025-06-07 01:11:29 +08:00
UmputunandGitHub 6fbaa806f0 Merge pull request #1934 from up9cloud/master
Fix typo (rootDissapear should be rootDisappear)
2025-06-06 10:46:29 -05:00
HsüanandGitHub 58acca4dcb Fix profile.spec.tsx typo 2025-06-04 15:18:46 -07:00
HsüanandGitHub 22d21df22b Fix profile.ts typo 2025-06-04 15:17:40 -07:00
Dmitry Verkhoturov ddb490bbc1 Fix login persistence with AUTH_SEND_JWT_HEADER enabled
With AUTH_SEND_JWT_HEADER=true, frontend now properly handles JWT authentication:
- Store JWT token in client-side cookie named 'JWT'
- Extract and store XSRF token from JWT payload
- Set Secure flag automatically when on HTTPS connection
- Update documentation to clarify this behavior

This fixes an issue where login state would be lost after page reload
when using header-based JWT authentication.
2025-04-29 23:45:55 +01:00
UmputunandGitHub 242499787e Merge pull request #1930 from umputun/paskal/update-gopkgz
Update go-pkgz, system modules
2025-04-29 02:31:12 -05:00
Dmitry Verkhoturov fd0799384f Update go-pkgz, system modules
This brings stricter check for auth provider names, slog support.
2025-04-29 08:24:53 +01:00
UmputunandGitHub 61dbf6b1f2 Merge pull request #1915 from umputun/dependabot/go_modules/backend/github.com/redis/go-redis/v9-9.7.3
Bump github.com/redis/go-redis/v9 from 9.7.0 to 9.7.3 in /backend
2025-03-24 21:51:56 -05:00
UmputunandGitHub 4a3c73db31 Merge pull request #1911 from umputun/dependabot/npm_and_yarn/site/prismjs-1.30.0
Bump prismjs from 1.29.0 to 1.30.0 in /site
2025-03-24 21:51:02 -05:00
UmputunandGitHub df510360e6 Merge pull request #1909 from umputun/dependabot/github_actions/github-actions-updates-e0e9667351
Bump pnpm/action-setup from 4.0.0 to 4.1.0 in the github-actions-updates group
2025-03-24 21:50:47 -05:00
UmputunandGitHub a3309516c3 Merge pull request #1916 from umputun/dependabot/go_modules/backend/github.com/golang-jwt/jwt/v5-5.2.2
Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 in /backend
2025-03-24 21:50:33 -05:00
UmputunandGitHub 5e30dc86e1 Merge pull request #1917 from umputun/paskal/golangci_lint_v2
Automatic fix of errors reported by golangci-lint v2
2025-03-24 21:50:20 -05:00
dependabot[bot]andGitHub bbcba5487e Bump github.com/redis/go-redis/v9 from 9.7.0 to 9.7.3 in /backend
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.7.0 to 9.7.3.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/CHANGELOG.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.7.0...v9.7.3)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-25 02:49:28 +00:00
UmputunandGitHub c32e5efdc2 Merge pull request #1918 from umputun/paskal/add_systemd
Add install instructions for setting up Remark42 as a systemd service
2025-03-24 21:48:27 -05:00
UmputunandGitHub af139a7f4c Merge pull request #1910 from umputun/dependabot/go_modules/backend/go-modules-updates-e61953c257
Bump the go-modules-updates group in /backend with 8 updates
2025-03-24 21:48:05 -05:00
dependabot[bot]andDmitry V 89221ff2bc Bump the go-modules-updates group in /backend with 8 updates
Bumps the go-modules-updates group in /backend with 8 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.1` | `1.10.2` |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.14.0` | `2.15.0` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.2.0` | `5.2.1` |
| [github.com/go-pkgz/jrpc](https://github.com/go-pkgz/jrpc) | `0.3.0` | `0.3.1` |
| [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt) | `1.3.11` | `1.4.0` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.31.0` | `0.33.0` |
| [golang.org/x/image](https://github.com/golang/image) | `0.23.0` | `0.25.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.33.0` | `0.35.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.10.1 to 1.10.2
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.1...v1.10.2)

Updates `github.com/alecthomas/chroma/v2` from 2.14.0 to 2.15.0
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Changelog](https://github.com/alecthomas/chroma/blob/master/.goreleaser.yml)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.14.0...v2.15.0)

Updates `github.com/go-chi/chi/v5` from 5.2.0 to 5.2.1
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.0...v5.2.1)

Updates `github.com/go-pkgz/jrpc` from 0.3.0 to 0.3.1
- [Release notes](https://github.com/go-pkgz/jrpc/releases)
- [Commits](https://github.com/go-pkgz/jrpc/compare/v0.3.0...v0.3.1)

Updates `go.etcd.io/bbolt` from 1.3.11 to 1.4.0
- [Release notes](https://github.com/etcd-io/bbolt/releases)
- [Commits](https://github.com/etcd-io/bbolt/compare/v1.3.11...v1.4.0)

Updates `golang.org/x/crypto` from 0.31.0 to 0.33.0
- [Commits](https://github.com/golang/crypto/compare/v0.31.0...v0.33.0)

Updates `golang.org/x/image` from 0.23.0 to 0.25.0
- [Commits](https://github.com/golang/image/compare/v0.23.0...v0.25.0)

Updates `golang.org/x/net` from 0.33.0 to 0.35.0
- [Commits](https://github.com/golang/net/compare/v0.33.0...v0.35.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/jrpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: go.etcd.io/bbolt
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 23:13:46 +01:00
Dmitry Verkhoturov 528242c1ca Add install instructions for setting up Remark42 as a systemd service 2025-03-24 23:09:28 +01:00
Dmitry Verkhoturov edfc5b9d76 Automatic fix of errors reported by golangci-lint v2
- Use strings.ReplaceAll
- Remove redundant internal structure names
2025-03-24 22:46:43 +01:00
dependabot[bot]andGitHub 7e944bfe0d Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 in /backend
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-21 22:26:36 +00:00
UmputunandGitHub e6afc58b34 Merge pull request #1914 from umputun/paskal/add_system_requirements 2025-03-16 14:28:22 -05:00
Dmitry Verkhoturov f49b878eb4 Add system requirements section to installation guide 2025-03-16 19:26:42 +00:00
UmputunandGitHub 5fe843de71 Merge pull request #1912 from umputun/dependabot/go_modules/backend/golang.org/x/net-0.36.0
Bump golang.org/x/net from 0.33.0 to 0.36.0 in /backend
2025-03-12 20:30:34 -05:00
dependabot[bot]andDmitry Verkhoturov 6c9ade9062 Bump golang.org/x/net from 0.33.0 to 0.36.0 in /backend
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.33.0 to 0.36.0.
- [Commits](https://github.com/golang/net/compare/v0.33.0...v0.36.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-12 22:58:15 +00:00
dependabot[bot]andGitHub ed3c104ba4 Bump prismjs from 1.29.0 to 1.30.0 in /site
Bumps [prismjs](https://github.com/PrismJS/prism) from 1.29.0 to 1.30.0.
- [Release notes](https://github.com/PrismJS/prism/releases)
- [Changelog](https://github.com/PrismJS/prism/blob/master/CHANGELOG.md)
- [Commits](https://github.com/PrismJS/prism/compare/v1.29.0...v1.30.0)

---
updated-dependencies:
- dependency-name: prismjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 22:24:16 +00:00
dependabot[bot]andGitHub a90b4296c6 Bump pnpm/action-setup in the github-actions-updates group
Bumps the github-actions-updates group with 1 update: [pnpm/action-setup](https://github.com/pnpm/action-setup).


Updates `pnpm/action-setup` from 4.0.0 to 4.1.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v4.0.0...v4.1.0)

---
updated-dependencies:
- dependency-name: pnpm/action-setup
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-09 18:01:26 +00:00
UmputunandGitHub 0ebe893125 Merge pull request #1908 from umputun/paskal/disable_npm_updates
Disable dependabot automatic PRs for NPM modules
2025-03-09 13:00:21 -05:00
Dmitry Verkhoturov e32ca020c0 Disable dependabot automatic PRs for NPM modules
These have to be updated manually
because there are always breaking changes.
2025-03-09 17:17:20 +01:00
UmputunandGitHub 558350c546 Merge pull request #1897 from umputun/paskal/claude
Add first version of the CLAUDE.md
2025-03-01 17:19:34 -06:00
UmputunandGitHub c50a5d6245 Merge pull request #1907 from umputun/paskal/get_rid_of_dockerhub
Migrate Docker images from Docker Hub to GitHub Container Registry
2025-03-01 17:19:01 -06:00
Dmitry Verkhoturov bb650d7526 Migrate Docker images from Docker Hub to GitHub Container Registry
This commit replaces all references to `umputun/remark42` Docker images
on Docker Hub with `ghcr.io/umputun/remark42` from the GitHub
Container Registry. It updates various Docker Compose files,
documentation, and the Makefile to use the new image location.
It also updates the kubernetes example to use the latest version.

Docker Hub is going to kill free pulls for too long by now.
2025-03-01 23:15:09 +00:00
Dmitry Verkhoturov 551212db82 Add first version of the CLAUDE.md 2025-02-25 22:05:25 +00:00
dependabot[bot]andUmputun 2e00002413 Bump the go-modules-updates group in /backend with 5 updates
Bumps the go-modules-updates group in /backend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.0` | `1.10.1` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.1.0` | `5.2.0` |
| [github.com/go-pkgz/rest](https://github.com/go-pkgz/rest) | `1.19.0` | `1.20.2` |
| [golang.org/x/image](https://github.com/golang/image) | `0.22.0` | `0.23.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.31.0` | `0.33.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.10.0 to 1.10.1
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.0...v1.10.1)

Updates `github.com/go-chi/chi/v5` from 5.1.0 to 5.2.0
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.1.0...v5.2.0)

Updates `github.com/go-pkgz/rest` from 1.19.0 to 1.20.2
- [Release notes](https://github.com/go-pkgz/rest/releases)
- [Commits](https://github.com/go-pkgz/rest/compare/v1.19.0...v1.20.2)

Updates `golang.org/x/image` from 0.22.0 to 0.23.0
- [Commits](https://github.com/golang/image/compare/v0.22.0...v0.23.0)

Updates `golang.org/x/net` from 0.31.0 to 0.33.0
- [Commits](https://github.com/golang/net/compare/v0.31.0...v0.33.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/rest
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-02 14:02:42 -06:00
UmputunandGitHub 81f70aa287 Merge pull request #1875 from codiflow/codiflow-patch-1 2024-12-28 20:44:53 -06:00
Christian 🦄andDmitry Verkhoturov f104e3c775 Update de.json
Updated Telegram bot description
2024-12-29 03:43:57 +01:00
UmputunandGitHub be076d03e9 Merge pull request #1878 from cubismod/webhook-params
Docs update describing available variables for webhook templating
2024-12-21 11:26:57 -06:00
Ryan e1166a48cf add a table describing additional variables available for use with webhook templating 2024-12-20 21:08:15 -06:00
Umputun 3f0789fd90 mod tidy for memory store example 2024-12-17 23:39:29 -06:00
UmputunandGitHub 518ae79556 Merge pull request #1869 from aliksend/aliksend/fix-importing-anonymous-comments-from-commento
Straightforward fix for importing anonymous comments from commento
2024-12-17 16:04:38 -06:00
UmputunandGitHub 3c55238bdd Merge pull request #1874 from umputun/dependabot/go_modules/backend/golang.org/x/crypto-0.31.0
Bump golang.org/x/crypto from 0.29.0 to 0.31.0 in /backend
2024-12-17 13:55:45 -06:00
Alik Send 556b95a655 Fix tests. Add test for anonymous comment 2024-12-16 23:16:26 -06:00
Alik Send bfef15f05f Straightforward fix for importing anonymous comments from commento
Fixes #1821
2024-12-16 23:16:26 -06:00
UmputunandGitHub c3ba55ba43 Merge pull request #1699 from umputun/paskal/comments_pagination
add pagination to GET /api/v1/find endpoint
2024-12-11 20:17:23 -06:00
dependabot[bot]andGitHub 0af8b2eab0 Bump golang.org/x/crypto from 0.29.0 to 0.31.0 in /backend
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.29.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.29.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-11 22:23:01 +00:00
Dmitry Verkhoturov 82a0888c42 add pagination to GET /api/v1/find endpoint
`format=tree` pagination provides top-level comments with all replies
and returns the last top-level comment as `last_comment` to be used
as `offset` for the next page. If comments and replies overflow
the limit, the one stepping out of the limit will not be returned.
 If the first comment and its replies after the given offset overflow
 the limit, it will be returned with all the replies.

`format=plain` pagination works by providing all comments and returning
the last comment as `last_comment` to be used as `offset`
for the next page.
2024-12-10 14:26:55 +00:00
UmputunandGitHub d5162d3fe6 Merge pull request #1758 from umputun/paskal/jwt_v5
Update to go-pkgz/auth/v2 and golang-jwt/jwt/v5
2024-12-09 19:53:00 -06:00
Dmitry Verkhoturov e61a46efff Improve error message for checking claims.Audience 2024-12-10 00:45:04 +00:00
Dmitry Verkhoturov f473105c52 add tests for jwt5 multiple auds and improve existing tests 2024-12-09 01:54:04 +00:00
Dmitry Verkhoturov c2d386230c vendor new modules 2024-12-09 01:54:03 +00:00
Dmitry Verkhoturov e3ad01b555 migrate to go-pkgz/auth/v2 2024-12-09 01:54:03 +00:00
Dmitry Verkhoturov 8d9e55c33c update github.com/golang-jwt/jwt to v5 2024-12-09 01:54:03 +00:00
UmputunandGitHub 6402ef9cae Merge pull request #1871 from umputun/paskal/improve_tests
Improve tests
2024-12-08 19:53:15 -06:00
Dmitry Verkhoturov 4ed48dd85c Improve tests 2024-12-09 01:45:57 +00:00
UmputunandGitHub 9628312b5d Merge pull request #1865 from umputun/dependabot/go_modules/backend/go-modules-updates-81f599025a 2024-12-01 22:32:10 -06:00
UmputunandGitHub c65c2b395d Merge branch 'master' into dependabot/go_modules/backend/go-modules-updates-81f599025a 2024-12-01 22:31:58 -06:00
UmputunandGitHub 27671c50c4 Merge pull request #1866 from umputun/dependabot/npm_and_yarn/site/npm-modules-updates-for-tests-87ff55c30d 2024-12-01 22:31:40 -06:00
UmputunandGitHub 56ac1bb841 Merge branch 'master' into dependabot/npm_and_yarn/site/npm-modules-updates-for-tests-87ff55c30d 2024-12-01 22:31:28 -06:00
dependabot[bot]andDmitry Verkhoturov 0aadf0ba86 Bump the go-modules-updates group in /backend with 3 updates
Bumps the go-modules-updates group in /backend with 3 updates: [github.com/stretchr/testify](https://github.com/stretchr/testify), [golang.org/x/crypto](https://github.com/golang/crypto) and [golang.org/x/net](https://github.com/golang/net).

Updates `github.com/stretchr/testify` from 1.9.0 to 1.10.0
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.9.0...v1.10.0)

Updates `golang.org/x/crypto` from 0.27.0 to 0.29.0
- [Commits](https://github.com/golang/crypto/compare/v0.27.0...v0.29.0)

Updates `golang.org/x/net` from 0.29.0 to 0.31.0
- [Commits](https://github.com/golang/net/compare/v0.29.0...v0.31.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-02 04:24:48 +00:00
UmputunandGitHub 34c667b83a Merge pull request #1862 from umputun/paskal/improve_server_test 2024-12-01 22:24:11 -06:00
dependabot[bot]andDmitry Verkhoturov f2e5758b6c Bump the npm-modules-updates-for-tests group in /site with 2 updates
Bumps the npm-modules-updates-for-tests group in /site with 2 updates: [prettier](https://github.com/prettier/prettier) and [tailwindcss](https://github.com/tailwindlabs/tailwindcss).

Updates `prettier` from 3.3.3 to 3.4.1
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.3.3...3.4.1)

Updates `tailwindcss` from 3.4.14 to 3.4.15
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/v3.4.15/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/compare/v3.4.14...v3.4.15)

---
updated-dependencies:
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-modules-updates-for-tests
- dependency-name: tailwindcss
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-02 04:03:21 +00:00
Dmitry Verkhoturov cd7f616596 Fix server test configuration
Previously, tmp boltdb was always created in directory 8080.
2024-12-02 03:36:32 +00:00
UmputunandGitHub a0c412ee1e Merge pull request #1861 from umputun/paskal/example_dependabot 2024-12-01 21:33:33 -06:00
Dmitry VerkhoturovandGitHub 15f5b7dde5 Stop updating examples via dependabot
Examples should be updated alongside the backend directory and it makes no sense to have separate update PRs for it.
2024-12-02 03:04:46 +00:00
UmputunandGitHub a561588117 Merge pull request #1859 from umputun/dependabot/github_actions/github-actions-updates-d109cd9e8b
Bump codecov/codecov-action from 4 to 5 in the github-actions-updates group
2024-12-01 21:01:52 -06:00
dependabot[bot]andGitHub ac36dccd19 Bump codecov/codecov-action in the github-actions-updates group
Bumps the github-actions-updates group with 1 update: [codecov/codecov-action](https://github.com/codecov/codecov-action).


Updates `codecov/codecov-action` from 4 to 5
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-01 11:33:06 +00:00
UmputunandGitHub afdac27651 Merge pull request #1853 from umputun/paskal/docker-compose
Update docker-compose.yml with example for local reverse proxy
2024-11-19 18:59:10 -06:00
Dmitry VerkhoturovandGitHub 995c4963fb Update docker-compose.yml with example for local reverse proxy
In discussion #1852 I realised that we don't have example with reverse proxy running outside of docker, this commit fixes that.
2024-11-20 00:57:46 +00:00
UmputunandGitHub 16ff2690f0 Merge pull request #1824 from mgkbadola/discord-oauth
feat: Discord OAuth support
2024-11-18 14:57:31 -06:00
Mrigank BadolaandGitHub d0dc1131ea Merge branch 'master' into discord-oauth 2024-11-12 21:24:24 +05:30
Dmitry VerkhoturovandUmputun d5e3602e54 Fix fetch module import so that latest release version would be shown on website 2024-11-10 18:30:48 -06:00
Dmitry VerkhoturovandUmputun a4772bd9de Do not fail if frontend pre-commit hook fails 2024-11-10 18:12:43 -06:00
Dmitry VerkhoturovandUmputun d23b7003c6 Downgrade eleventy to fix the website 2024-11-10 18:05:51 -06:00
Dmitry VerkhoturovandUmputun 3646c4a871 Drop linux/arm/v7 docker image from site as it doesn't build well 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun 29fc63f116 Clarify steps names for site deployment 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun 8c59bab921 Add nvmrc to /site, update yarn.lock 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun c42511d5a1 Fix cache-dependency-path for actions/setup-go caching to work, fix CI files path 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun 9b5f6ee2c5 Remove deprecated set-output function usage in GitHub actions
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2024-11-10 17:25:16 -06:00
dependabot[bot]andUmputun b5579152cb Bump the npm-modules-updates-for-tests group across 1 directory with 12 updates
Bumps the npm-modules-updates-for-tests group with 12 updates in the /frontend/packages/api directory:

| Package | From | To |
| --- | --- | --- |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `18.19.64` | `18.19.64` |
| [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) | `5.62.0` | `8.13.0` |
| [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) | `5.62.0` | `8.13.0` |
| [@vitest/coverage-c8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-c8) | `0.22.1` | `0.33.0` |
| [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) | `8.10.0` | `9.1.0` |
| [eslint-plugin-prettier](https://github.com/prettier/eslint-plugin-prettier) | `4.2.1` | `5.2.1` |
| [jsdom](https://github.com/jsdom/jsdom) | `20.0.3` | `25.0.1` |
| [msw](https://github.com/mswjs/msw) | `0.44.2` | `1.3.5` |
| [prettier](https://github.com/prettier/prettier) | `2.8.8` | `3.3.3` |
| [typescript](https://github.com/microsoft/TypeScript) | `4.7.4` | `4.9.5` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `0.22.1` | `2.1.4` |

Updates `@types/node` from 18.19.64 to 22.9.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@typescript-eslint/eslint-plugin` from 5.62.0 to 8.13.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.13.0/packages/eslint-plugin)

Updates `@typescript-eslint/parser` from 5.62.0 to 8.13.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.13.0/packages/parser)

Updates `@vitest/coverage-c8` from 0.22.1 to 0.33.0
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v0.33.0/packages/coverage-c8)

Updates `eslint-config-prettier` from 8.10.0 to 9.1.0
- [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/eslint-config-prettier/compare/v8.10.0...v9.1.0)

Updates `eslint-plugin-prettier` from 4.2.1 to 5.2.1
- [Release notes](https://github.com/prettier/eslint-plugin-prettier/releases)
- [Changelog](https://github.com/prettier/eslint-plugin-prettier/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prettier/eslint-plugin-prettier/compare/v4.2.1...v5.2.1)

Updates `jsdom` from 20.0.3 to 25.0.1
- [Release notes](https://github.com/jsdom/jsdom/releases)
- [Changelog](https://github.com/jsdom/jsdom/blob/main/Changelog.md)
- [Commits](https://github.com/jsdom/jsdom/compare/20.0.3...25.0.1)

Updates `msw` from 0.44.2 to 2.6.3
- [Release notes](https://github.com/mswjs/msw/releases)
- [Changelog](https://github.com/mswjs/msw/blob/main/CHANGELOG.md)
- [Commits](https://github.com/mswjs/msw/compare/v0.44.2...v2.6.3)

Updates `prettier` from 2.8.8 to 3.3.3
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/2.8.8...3.3.3)

Updates `typescript` from 4.9.5 to 5.6.3
- [Release notes](https://github.com/microsoft/TypeScript/releases)
- [Changelog](https://github.com/microsoft/TypeScript/blob/main/azure-pipelines.release.yml)
- [Commits](https://github.com/microsoft/TypeScript/compare/v4.9.5...v5.6.3)

Updates `vitest` from 0.22.1 to 2.1.4
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v2.1.4/packages/vitest)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@typescript-eslint/eslint-plugin"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@typescript-eslint/parser"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@vitest/coverage-c8"
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-modules-updates-for-tests
- dependency-name: eslint
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: eslint-config-prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: eslint-plugin-prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: jsdom
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: msw
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: typescript
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: vitest
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-10 14:00:52 -06:00
dependabot[bot]andUmputun 3c78a54be6 Bump the npm-modules-updates-for-tests group
Bumps the npm-modules-updates-for-tests group in /frontend/e2e with 3 updates: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node), [nanoid](https://github.com/ai/nanoid) and [typescript](https://github.com/microsoft/TypeScript).

Updates `@types/node` from 18.19.64 to 22.9.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `nanoid` from 4.0.2 to 5.0.8
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/4.0.2...5.0.8)

Updates `typescript` from 4.9.5 to 5.6.3
- [Release notes](https://github.com/microsoft/TypeScript/releases)
- [Changelog](https://github.com/microsoft/TypeScript/blob/main/azure-pipelines.release.yml)
- [Commits](https://github.com/microsoft/TypeScript/compare/v4.9.5...v5.6.3)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: nanoid
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: typescript
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-10 13:55:30 -06:00
Dmitry VerkhoturovandUmputun 3001b37812 Update pnpm from v7 to v8
That suppose to help with Dependabot security updates not updating lockfile.
2024-11-10 13:31:03 -06:00
Dmitry VerkhoturovandUmputun 63048cc798 Specify package manager version in frontend project configuration
This is necessary for Dependabot updates to work properly.
2024-11-10 12:42:07 -06:00
Mrigank Badola 1a27913404 chore: update go-pkgz/auth package 2024-11-10 12:40:07 +05:30
Mrigank Badola 4a39ceee8d sync: Merge branch 'master' of https://github.com/mgkbadola/remark42 into discord-oauth 2024-11-10 10:43:10 +05:30
dependabot[bot]andUmputun 1e659917d2 Bump the npm-modules-updates-for-tests group in /site with 7 updates
Bumps the npm-modules-updates-for-tests group in /site with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [@11ty/eleventy](https://github.com/11ty/eleventy) | `2.0.1` | `3.0.0` |
| [@tailwindcss/typography](https://github.com/tailwindlabs/tailwindcss-typography) | `0.5.10` | `0.5.15` |
| [date-fns](https://github.com/date-fns/date-fns) | `3.3.1` | `4.1.0` |
| [markdown-it](https://github.com/markdown-it/markdown-it) | `14.0.0` | `14.1.0` |
| [markdown-it-anchor](https://github.com/valeriangalliat/markdown-it-anchor) | `8.6.7` | `9.2.0` |
| [prettier](https://github.com/prettier/prettier) | `2.8.8` | `3.3.3` |
| [tailwindcss](https://github.com/tailwindlabs/tailwindcss) | `3.4.1` | `3.4.14` |


Updates `@11ty/eleventy` from 2.0.1 to 3.0.0
- [Release notes](https://github.com/11ty/eleventy/releases)
- [Changelog](https://github.com/11ty/eleventy/blob/main/docs/release-instructions.md)
- [Commits](https://github.com/11ty/eleventy/compare/v2.0.1...v3.0.0)

Updates `@tailwindcss/typography` from 0.5.10 to 0.5.15
- [Release notes](https://github.com/tailwindlabs/tailwindcss-typography/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss-typography/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss-typography/compare/v0.5.10...v0.5.15)

Updates `date-fns` from 3.3.1 to 4.1.0
- [Release notes](https://github.com/date-fns/date-fns/releases)
- [Changelog](https://github.com/date-fns/date-fns/blob/main/CHANGELOG.md)
- [Commits](https://github.com/date-fns/date-fns/compare/v3.3.1...v4.1.0)

Updates `markdown-it` from 14.0.0 to 14.1.0
- [Changelog](https://github.com/markdown-it/markdown-it/blob/master/CHANGELOG.md)
- [Commits](https://github.com/markdown-it/markdown-it/compare/14.0.0...14.1.0)

Updates `markdown-it-anchor` from 8.6.7 to 9.2.0
- [Release notes](https://github.com/valeriangalliat/markdown-it-anchor/releases)
- [Changelog](https://github.com/valeriangalliat/markdown-it-anchor/blob/master/CHANGELOG.md)
- [Commits](https://github.com/valeriangalliat/markdown-it-anchor/compare/v8.6.7...v9.2.0)

Updates `prettier` from 2.8.8 to 3.3.3
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/2.8.8...3.3.3)

Updates `tailwindcss` from 3.4.1 to 3.4.14
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/v3.4.14/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/compare/v3.4.1...v3.4.14)

---
updated-dependencies:
- dependency-name: "@11ty/eleventy"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@tailwindcss/typography"
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-modules-updates-for-tests
- dependency-name: date-fns
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: markdown-it
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-modules-updates-for-tests
- dependency-name: markdown-it-anchor
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: tailwindcss
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-08 17:54:33 -06:00
daniyalandUmputun 5c586e12bd feat: add "fa" locale 2024-11-08 16:38:09 -06:00
Dmitry VerkhoturovandUmputun 73ca4a1b6d Capitalize "AS" keyword in Dockerfile FROM instructions 2024-11-08 16:27:10 -06:00
Dmitry VerkhoturovandUmputun 7604548035 Update go to 1.23, golangci-lint to 1.61, baseimage to 1.14 2024-11-08 16:19:58 -06:00
dependabot[bot]andUmputun 7529aa7e17 Bump the go-modules-updates group in /backend with 9 updates
Bumps the go-modules-updates group in /backend with 9 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.9.2` | `1.10.0` |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.13.0` | `2.14.0` |
| [github.com/go-pkgz/notify](https://github.com/go-pkgz/notify) | `1.1.1` | `1.2.0` |
| [github.com/go-pkgz/repeater](https://github.com/go-pkgz/repeater) | `1.1.3` | `1.2.0` |
| [github.com/rs/xid](https://github.com/rs/xid) | `1.5.0` | `1.6.0` |
| [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt) | `1.3.10` | `1.3.11` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.25.0` | `0.27.0` |
| [golang.org/x/image](https://github.com/golang/image) | `0.18.0` | `0.22.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.27.0` | `0.29.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.9.2 to 1.10.0
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.9.2...v1.10.0)

Updates `github.com/alecthomas/chroma/v2` from 2.13.0 to 2.14.0
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Changelog](https://github.com/alecthomas/chroma/blob/master/.goreleaser.yml)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.13.0...v2.14.0)

Updates `github.com/go-pkgz/notify` from 1.1.1 to 1.2.0
- [Release notes](https://github.com/go-pkgz/notify/releases)
- [Commits](https://github.com/go-pkgz/notify/compare/v1.1.1...v1.2.0)

Updates `github.com/go-pkgz/repeater` from 1.1.3 to 1.2.0
- [Release notes](https://github.com/go-pkgz/repeater/releases)
- [Commits](https://github.com/go-pkgz/repeater/compare/v1.1.3...v1.2.0)

Updates `github.com/rs/xid` from 1.5.0 to 1.6.0
- [Release notes](https://github.com/rs/xid/releases)
- [Commits](https://github.com/rs/xid/compare/v1.5.0...v1.6.0)

Updates `go.etcd.io/bbolt` from 1.3.10 to 1.3.11
- [Release notes](https://github.com/etcd-io/bbolt/releases)
- [Commits](https://github.com/etcd-io/bbolt/compare/v1.3.10...v1.3.11)

Updates `golang.org/x/crypto` from 0.25.0 to 0.27.0
- [Commits](https://github.com/golang/crypto/compare/v0.25.0...v0.27.0)

Updates `golang.org/x/image` from 0.18.0 to 0.22.0
- [Commits](https://github.com/golang/image/compare/v0.18.0...v0.22.0)

Updates `golang.org/x/net` from 0.27.0 to 0.29.0
- [Commits](https://github.com/golang/net/compare/v0.27.0...v0.29.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/notify
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/repeater
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/rs/xid
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: go.etcd.io/bbolt
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-08 16:19:58 -06:00
Dmitry VerkhoturovandUmputun c1a447b873 unpin bugfix js version, use codecov github action 2024-11-08 15:56:59 -06:00
dependabot[bot]andUmputun 3c110eb0ec Bump the github-actions-updates group with 5 updates
Bumps the github-actions-updates group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) | `3` | `6` |
| [pnpm/action-setup](https://github.com/pnpm/action-setup) | `2.4.0` | `4.0.0` |
| [actions/cache](https://github.com/actions/cache) | `3` | `4` |
| [andresz1/size-limit-action](https://github.com/andresz1/size-limit-action) | `7313b26c76b3666c1dc41e2ca05370e201a9b7de` | `94bc357df29c36c8f8d50ea497c3e225c3c95d1d` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `2` | `4` |


Updates `golangci/golangci-lint-action` from 3 to 6
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3...v6)

Updates `pnpm/action-setup` from 2.4.0 to 4.0.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v2.4.0...v4.0.0)

Updates `actions/cache` from 3 to 4
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

Updates `andresz1/size-limit-action` from 7313b26c76b3666c1dc41e2ca05370e201a9b7de to 94bc357df29c36c8f8d50ea497c3e225c3c95d1d
- [Release notes](https://github.com/andresz1/size-limit-action/releases)
- [Commits](https://github.com/andresz1/size-limit-action/compare/7313b26c76b3666c1dc41e2ca05370e201a9b7de...94bc357df29c36c8f8d50ea497c3e225c3c95d1d)

Updates `actions/upload-artifact` from 2 to 4
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2...v4)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: andresz1/size-limit-action
  dependency-type: direct:production
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-08 15:56:59 -06:00
Dmitry VerkhoturovandUmputun d7494d5392 add dependabot updates for Go, TypeScript, GitHub Actions 2024-11-07 19:13:55 -06:00
W.T. ChangandUmputun c1048e95b3 fix: missing comma in example frontend configuration 2024-10-27 01:58:27 -05:00
Dmitry VerkhoturovandUmputun 5b6d8de807 Deprecate Twitter OAuth and remove from feature list 2024-10-20 17:18:14 -05:00
Dmitry VerkhoturovandUmputun dd2cff6a13 Clarify ALLOWED_HOSTS description and usage
This clarifies that the parameter sets CSP 'frame-ancestors'
to limit hosts allowed to embed comments. The commit also improves
the documentation on how to use ALLOWED_HOSTS with AUTH_SAME_SITE
for different setup scenarios.

We might want to change AUTH_SAME_SITE to `strong` in v2.0 as it works
on the subdomain of the same site as well as current Lax option.
2024-10-20 17:17:35 -05:00
Dmitry VerkhoturovandUmputun 68fe6eb55f Update docker-compose command to docker compose
Compose is part of docker command for years and during local development
 on Mac I have to change makefile locally for commands to work.
2024-10-20 17:17:10 -05:00
Dmitry VerkhoturovandUmputun 4d5f9f269b Add missing AUTH_APPLE_KID env variable cleanup 2024-10-20 16:06:50 -05:00
Dmitry VerkhoturovandUmputun 6140d82eb2 Fix CSP img-src directive to allow everything without proxy
Change the default img-src value to "*" and sets it to "'self'" when
image proxy is enabled. The previous state was inversion of this logic
which was wrong.
2024-10-20 15:55:51 -05:00
Mrigank Badola cff261c15b chore: add discord in server test file 2024-10-20 11:47:47 +05:30
Mrigank Badola 18ea40582a sync: Merge remote-tracking branch 'origin' into discord-oauth 2024-10-20 11:46:33 +05:30
Dmitry VerkhoturovandUmputun f9d4837567 Add Content-Security-Policy and Permissions-Policy headers
`Content-Security-Policy` now restricts resource loading and execution
to enhance security:
  - `default-src 'none'`: Disallow all resource loading by default.
  - `base-uri 'none'`: Prevents the use of `<base>` tag to change the
  base URL for relative URLs.
  - `form-action 'none'`: Disallows form submissions.
  - `connect-src 'self'`: Restricts the origins that can be connected to
   (via XHR, WebSockets, etc.) to the same origin.
  - `frame-src 'self'`: Restricts the origins that can be embedded using
   `<frame>` and `<iframe>` to the same origin (for `/web/` demo
    endpoint).
  - `frame-ancestors %s;`: Specifies the origins that are allowed to
  embed this content in a frame. If no specific origins are allowed, it
  defaults to `*` (any origin). This enhances security by controlling
  which sites can embed your content.
  - `img-src 'self'`: Allows images to be loaded only from the same
  origin. If `imageProxyEnabled` is true, allows images from any origin
  (`*`).
  - `script-src 'self' 'unsafe-inline'`: Allows scripts to be loaded and
   executed only from the same origin and allows inline scripts.
  - `style-src 'self' 'unsafe-inline'`: Allows styles to be loaded and
  applied only from the same origin and allows inline styles.
  - `font-src data:`: Allows fonts to be loaded from data URIs.
  - `object-src 'none'`: Disallows the use of `<object>`, `<embed>`, and
   `<applet>` tags.

`Permissions-Policy` now restricts the use of certain browser features
which we don't use to enhance user privacy and security:
  - `accelerometer=()`: Disables the use of the accelerometer sensor.
  - `autoplay=()`: Disables automatic playback of media.
  - `camera=()`: Disables the use of the camera.
  - `cross-origin-isolated=()`: Disallows the page from being treated as
   cross-origin isolated.
  - `display-capture=()`: Disables the ability to capture the display.
  - `encrypted-media=()`: Disables the use of Encrypted Media Extensions
  .
  - `fullscreen=()`: Disables the ability to use fullscreen mode.
  - `geolocation=()`: Disables the use of geolocation.
  - `gyroscope=()`: Disables the use of the gyroscope sensor.
  - `keyboard-map=()`: Disables the use of the keyboard map.
  - `magnetometer=()`: Disables the use of the magnetometer sensor.
  - `microphone=()`: Disables the use of the microphone.
  - `midi=()`: Disables the use of the MIDI API.
  - `payment=()`: Disables the Payment Request API.
  - `picture-in-picture=()`: Disables the use of Picture-in-Picture mode
  .
  - `publickey-credentials-get=()`: Disables the use of the Web
  Authentication API.
  - `screen-wake-lock=()`: Disables the ability to prevent the screen
  from dimming.
  - `sync-xhr=()`: Disables synchronous XMLHttpRequest.
  - `usb=()`: Disables the use of the USB API.
  - `xr-spatial-tracking=()`: Disables the use of spatial tracking in
  WebXR.
  - `clipboard-read=()`: Disables the ability to read from the clipboard
  .
  - `clipboard-write=()`: Disables the ability to write to the clipboard
  .
  - `gamepad=()`: Disables the use of the Gamepad API.
  - `hid=()`: Disables the use of the Human Interface Device API.
  - `idle-detection=()`: Disables the ability to detect idle state.
  - `interest-cohort=()`: Disables the use of interest cohort tracking.
  - `serial=()`: Disables the use of the Serial API.
  - `unload=()`: Disables the ability to use the `beforeunload` and
  `unload` events.
  - `window-management=()`: Disables the ability to use window
  management APIs.
2024-10-15 17:53:12 -05:00
Mrigank Badola e5ae07b1c2 chore: add cid and csec in readmes and backend docker compose yml 2024-10-09 09:45:10 +05:30
Mrigank Badola 4fbb3b59be chore: add changes in accordance to paskal/discord_poc 2024-10-09 08:57:47 +05:30
Dmitry VerkhoturovandUmputun 9fb3014229 Detect proper avatar type to return instead of returning image/* 2024-09-22 14:37:29 -05:00
dependabot[bot]andUmputun 2a9b29dd53 Bump micromatch from 4.0.5 to 4.0.8 in /site
Bumps [micromatch](https://github.com/micromatch/micromatch) from 4.0.5 to 4.0.8.
- [Release notes](https://github.com/micromatch/micromatch/releases)
- [Changelog](https://github.com/micromatch/micromatch/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/micromatch/compare/4.0.5...4.0.8)

---
updated-dependencies:
- dependency-name: micromatch
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-20 22:31:53 -05:00
Ivan BandUmputun 872b818323 docs: fix frontend paths in translation guide (frontend/{path} -> frontend/apps/remark42/{path}) 2024-09-19 19:06:06 -05:00
Dmitry VerkhoturovandUmputun 4a7bee1d98 Fix restrictions for anonymous usernames
Tested on remark42 demo to clarify what works and what doesn't.
2024-09-19 19:05:43 -05:00
dependabot[bot]andUmputun cbe793fb42 Bump path-to-regexp from 6.2.1 to 6.3.0 in /site
Bumps [path-to-regexp](https://github.com/pillarjs/path-to-regexp) from 6.2.1 to 6.3.0.
- [Release notes](https://github.com/pillarjs/path-to-regexp/releases)
- [Changelog](https://github.com/pillarjs/path-to-regexp/blob/master/History.md)
- [Commits](https://github.com/pillarjs/path-to-regexp/compare/v6.2.1...v6.3.0)

---
updated-dependencies:
- dependency-name: path-to-regexp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-19 19:05:22 -05:00
Dmitry VerkhoturovandUmputun cbf9a82a92 Move gcc installation before backend files addition
This allows to not reinstall apk files when backend files change and the
new image is built.
2024-09-19 19:05:01 -05:00
Dmitry VerkhoturovandUmputun 6cd5c45a6c Fix problem with logout button
The logout auth endpoint was returning no response body and type
application/json which is not valid, this commit changes it to return
plain/text instead which makes it valid.
2024-09-19 19:04:21 -05:00
adueckandUmputun 0bc85a6ff6 added RTL support 2024-08-24 18:08:09 -05:00
Tomy HsiehandUmputun 88bf4b7d70 feat: Update CLI help message 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun 26c5425646 📖 docs: restore numbering 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun 15d2ab9644 🕺🏻 style: Format 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun 50c56cb771 📖 docs: Update apple integration docs 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun e65f71b958 🛠 fix: Fix sign in with apple integration 2024-08-23 19:19:59 -05:00
Dmitry VerkhoturovandUmputun a9b439602b update go modules 2024-07-30 20:23:33 -05:00
Dmitry VerkhoturovandUmputun d2027f5241 switch playwright (e2e) to latest stable version 2024-07-30 20:23:06 -05:00
Dmitry VerkhoturovandUmputun 95966f6407 add escaping of comment text in webhook default JSON template 2024-07-01 23:41:50 -05:00
Umputun 8df986e70a Add content type check for images endpoint
A check in image proxy for validating content type of requested images added. Modified the related tests to accommodate these changes.
2024-07-01 14:47:25 -05:00
dependabot[bot]andUmputun 71a6d0b385 Bump pug from 3.0.2 to 3.0.3 in /site
Bumps [pug](https://github.com/pugjs/pug) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/pugjs/pug/releases)
- [Commits](https://github.com/pugjs/pug/compare/pug@3.0.2...pug@3.0.3)

---
updated-dependencies:
- dependency-name: pug
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:12:14 -05:00
dependabot[bot]andUmputun 974d4aaf55 Bump ejs from 3.1.9 to 3.1.10 in /site
Bumps [ejs](https://github.com/mde/ejs) from 3.1.9 to 3.1.10.
- [Release notes](https://github.com/mde/ejs/releases)
- [Commits](https://github.com/mde/ejs/compare/v3.1.9...v3.1.10)

---
updated-dependencies:
- dependency-name: ejs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:11:10 -05:00
dependabot[bot]andUmputun dc8d7d46cb Bump braces from 3.0.2 to 3.0.3 in /site
Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3.
- [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/braces/compare/3.0.2...3.0.3)

---
updated-dependencies:
- dependency-name: braces
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:10:47 -05:00
dependabot[bot]andUmputun c4ace9fc0c Bump ws from 8.16.0 to 8.17.1 in /site
Bumps [ws](https://github.com/websockets/ws) from 8.16.0 to 8.17.1.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/8.16.0...8.17.1)

---
updated-dependencies:
- dependency-name: ws
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:10:26 -05:00
Dmitry VerkhoturovandUmputun 16b07ded66 fix golangci-lint reported unused parameters in example module 2024-06-30 12:07:06 -05:00
Tomy HsiehandUmputun c04705947a 📖 docs: Update README 2024-06-22 12:04:49 -05:00
Dmitry VerkhoturovandUmputun eadd65e247 update docker images, clarify comments 2024-06-05 13:05:42 -05:00
Xin LiandUmputun 4428f79046 SubscribeByEmailForm: Ensure onInput and onClick props are typed correctly.
Resolves #1771
2024-06-04 11:06:14 -05:00
Armen MkrtchyanandUmputun bad6af87f7 Update .golangci.yml
Sorted linters alphabetically, removed duplicates
2024-05-30 11:30:19 -05:00
Edward NavarroandUmputun f7ba43e5f1 Complete and update Spanish translations 2024-05-27 15:22:09 -05:00
Dmitry VerkhoturovandUmputun 661f042cb4 pin golangci-lint version to latest available, fix reported errors 2024-05-09 22:32:00 -05:00
Pavel FrancírekandUmputun 877765cda2 Update cs.json - typos
Only typo corrections.
2024-04-11 11:37:58 -05:00
Dmitry VerkhoturovandUmputun 4bb0017060 update go modules 2024-04-10 19:33:20 -05:00
Dmitry VerkhoturovandUmputun e0423b8683 fix type for value for refresh token cache
It was set to string by mistake, proper type is token.Claims.
2024-03-22 04:16:29 -05:00
Dmitry VerkhoturovandUmputun 5a781693aa hide delete button for non-admin users after edit period expires 2024-03-17 16:47:28 -05:00
Dmitry VerkhoturovandUmputun e5743185b0 collect /find Info for tree and plain types consistently
MakeTree calculated Info locally for historical reasons,
and the results were consistent with the dataService.Info call
but calculated differently.

That change fixes that, ensuring that Info is requested
in the same manner.
2024-03-16 12:49:45 -05:00
dependabot[bot]andUmputun 1510aec17c Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /backend
Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-13 18:33:05 -05:00
Dmitry VerkhoturovandUmputun 01837b69e5 fix readonly status, deleted count for plain /find request 2024-03-04 15:44:19 -06:00
Dmitry VerkhoturovandUmputun d02099844e apply prettier to site files 2024-02-25 18:27:06 -06:00
Dmitry VerkhoturovandUmputun 6fcfaa12b7 update site dependencies 2024-02-25 18:27:06 -06:00
Dmitry VerkhoturovandUmputun 6269c19881 add more tests for GET /find endpoint 2024-02-21 10:43:28 -06:00
Dmitry VerkhoturovandUmputun 1313dee829 update to lcw v2 with generic types 2024-02-20 14:15:35 -06:00
Dmitry VerkhoturovandUmputun 3210de8f7b update go modules 2024-02-20 12:23:16 -06:00
Dmitry VerkhoturovandUmputun 532573fb34 fix problems reported by golangci-lint 2024-02-20 12:10:39 -06:00
Konstantsin KryuleniaandUmputun e1173bbcad use specific pnpm version 2024-02-03 12:09:57 -06:00
Konstantsin KryuleniaandUmputun e748951182 skip lint jpeg files 2024-02-03 12:09:57 -06:00
Konstantsin KryuleniaandUmputun df8670752a copy example image to public folder 2024-02-03 12:09:57 -06:00
Dmitry VerkhoturovandUmputun 654250f033 don't load kitten picture from third party site 2024-02-03 12:09:57 -06:00
Dmitry VerkhoturovandUmputun 0050c65596 add meaningful error for lack of auth on import, remap and backup
Previously, the error printed was just the following:

error response "401 Unauthorized", Unauthorized"

New error:

error response "401 Unauthorized", ensure you have set ADMIN_PASSWD
and provided it to the command you're running: Unauthorized
2024-01-28 12:16:24 -06:00
Dmitry VerkhoturovandUmputun 02db7a917d fix lack of error on file site export
Previously, status 200 was set for file export, which is used
for backup, which resulted in an inability to set an error status code
in case of a problem with file generation.

After this change, status code 200 would be written automatically by Go
before we start writing the response's body.
2024-01-28 12:16:24 -06:00
Dmitry VerkhoturovandUmputun 81c30e01f8 cleanup images from deleted comments
Previously, images were deleted only from comments deleted
before EditDuration expiration. After this change, any deletion
of the comment deletes images if they are not used elsewhere
in comments under the same page.
2024-01-20 13:29:06 -06:00
Paul MineevandUmputun 82c617806d chore: remove theme from comment content 2024-01-18 02:56:10 -06:00
Paul MineevandUmputun e043dc2ac3 fix: break long words in code tag, combine all styles in one file, move styles to correct place 2024-01-18 02:56:10 -06:00
Dmitry VerkhoturovandUmputun cbd73865bd update go modules, update go-pkgz/auth to latest commit 2024-01-11 15:57:39 -06:00
Dmitry VerkhoturovandUmputun 884b5685eb update docker images and github CI actions 2024-01-11 01:26:30 -06:00
NavyStackandUmputun 3f14651653 fix: add missing Japanese translation 2024-01-08 11:23:03 -06:00
NavyStackandUmputun 310b797679 fix: add missing Japanese translation 2024-01-08 11:23:03 -06:00
NavyStackandUmputun 0594565143 FIX: Character escaping 2024-01-08 11:23:03 -06:00
NavyStackandUmputun d4c153662b update: Korean translation 2024-01-08 11:23:03 -06:00
NavyStackandUmputun f64b0b8831 fix: Korean missing strings 2024-01-08 11:23:03 -06:00
Vladimir DandUmputun 94893b77dc bump deps 2023-12-26 11:47:09 -06:00
Vladimir DandUmputun 30f46efa5b TLS InsecureSkipVerify option 2023-12-26 11:47:09 -06:00
Vladimir DandUmputun e0904603c6 go-pkgz/auth and go-pkgz/email modules updated, bump deps 2023-12-26 11:47:09 -06:00
Dmitry VerkhoturovandUmputun d143932924 add MIN_COMMENT_SIZE parameter 2023-12-02 12:16:26 -06:00
Dmitry Verkhoturov dcc7613409 allow disabling fancy HTML formatting
It might be necessary if the comments should preserve
original quotes instead of replacing them with angled ones.
2023-11-26 09:13:53 +01:00
Dmitry VerkhoturovandUmputun d04d2097f8 fix Commento import URL
Previously, it was not using the domain
and relying on another export format.
2023-11-20 10:59:05 -06:00
Dmitry Verkhoturov ce678bf967 fix Commento top-level comments import
Previously, top-level comments were incorrectly assigned
parent comment id "root", which made them non-root,
so they are not returned when requested
in the `/find?format=tree` API call.

To fix the previously imported comments, please export all your comments
and replace `"pid":"root"` with `"pid":""` and then re-import them.
2023-11-18 20:30:18 +01:00
Dmitry VerkhoturovandUmputun cd481d401d add tests for admin Store and DataService 2023-11-04 12:49:40 -05:00
Dmitry VerkhoturovandUmputun 618c267370 combine multiple post info in DataStore.Info instead of returning first
Previously, only the first one was returned for site-wide requests,
and now all returned information will be correctly aggregated,
and the PostInfo.URL and PostInfo.ReadOnly parameters will be dropped.
2023-11-04 12:49:40 -05:00
Dmitry VerkhoturovandUmputun 307866f7f5 simplify BoltDB.Info code
The new code does the same as the old one but doesn't call the checkFlag
in case ReadOnly is already set based on age.
2023-11-04 12:46:38 -05:00
Dmitry VerkhoturovandUmputun 19e1616129 allow title extraction only from full match of AllowedHosts
Previously, we extracted the second-level domain,
but it doesn't make sense for a list of domains defined explicitly
to display the comments.
2023-11-04 12:45:19 -05:00
dependabot[bot]andUmputun c6506b8905 Bump luxon from 2.3.0 to 2.5.2 in /site
Bumps [luxon](https://github.com/moment/luxon) from 2.3.0 to 2.5.2.
- [Release notes](https://github.com/moment/luxon/releases)
- [Changelog](https://github.com/moment/luxon/blob/master/CHANGELOG.md)
- [Commits](https://github.com/moment/luxon/compare/2.3.0...2.5.2)

---
updated-dependencies:
- dependency-name: luxon
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-28 03:15:10 -05:00
Paul MineevandUmputun 676ae77456 update links styles 2023-10-28 01:56:04 -05:00
Paul MineevandUmputun b93fc48b73 fix types on error message extraction 2023-10-28 01:45:03 -05:00
Paul MineevandUmputun 4be664e78d fix styles around admin controls 2023-10-28 01:36:17 -05:00
Paul MineevandUmputun 62aaa35287 fix types on error message extraction 2023-10-28 01:35:58 -05:00
Dmitry VerkhoturovandUmputun 69b18d3536 fix wrapped errors checks
As errors can be wrapped in recent versions of Go, the proper way
to check the error types are `errors.As` and `errors.Is`.
2023-10-10 23:36:04 -05:00
Dmitry VerkhoturovandUmputun efceed6f68 limit TitleExtractor to allow only Remark42 whitelisted domains
Allowed domains consist of `REMARK_URL` second-level domain (or whole IP in case it's IP like `127.0.0.1`) and `ALLOWED_HOSTS`. That is needed to prevent Remark42 from asking arbitrary servers and storing the page title as the comment.PostTitle.

Previous behaviour allowed the caller of the API to create a comment
with an arbitrary URL and learn the title of the page, which might be
accessible to the server Remark42 is installed on but not to the user
outside that network (CWE-918).
2023-10-10 23:34:35 -05:00
Dmitry VerkhoturovandUmputun f4358173c7 limit TitleExtractor to allow only Remark42 whitelisted domains
Allowed domains consist of `REMARK_URL` second-level domain (or whole IP in case it's IP like `127.0.0.1`) and `ALLOWED_HOSTS`. That is needed to prevent Remark42 from asking arbitrary servers and storing the page title as the comment.PostTitle.

Previous behaviour allowed the caller of the API to create a comment
with an arbitrary URL and learn the title of the page, which might be
accessible to the server Remark42 is installed on but not to the user
outside that network (CWE-918).
2023-10-10 23:34:35 -05:00
Dmitry VerkhoturovandUmputun 7a71d47556 remove all HTML tags from comment title and username
Previously, we stripped unsafe HTML tags but left some,
but it's not expected to have a link in a title or username,
so the new behaviour is stripping everything.
2023-10-10 12:41:26 -05:00
Dmitry VerkhoturovandUmputun 41d27e2a7f clarify titles for frontend and backend configuration doc 2023-10-10 11:51:26 -05:00
Dmitry VerkhoturovandUmputun 10e4686f1a reproduce report of CWE-918 from #1677 2023-10-10 11:43:56 -05:00
sharief007andUmputun eba447319d Extract error msg from API response for preview. 2023-10-08 12:53:04 -05:00
Dmitry VerkhoturovandUmputun 40a0d7ca62 update Telegram notifications instructions
Remove outdated information about functionality which was already implemented.
2023-09-12 20:45:25 -05:00
Dzung DoandUmputun c9b6f9272f Update vi.json 2023-09-04 01:35:57 -05:00
Umputun 1f2500f16f switch to stable, updated auth 2023-08-21 11:20:24 -05:00
2757 changed files with 246486 additions and 218698 deletions
+3 -5
View File
@@ -10,10 +10,6 @@
/frontend/node_modules/
/frontend/apps/remark42/node_modules/
/frontend/apps/remark42/public/
# e2e tests arficats
/frontend/e2e/playwright-report/
/frontend/e2e/playwright/.cache/
/frontend/e2e/test-results/
# source files
docker-compose.yml
@@ -36,4 +32,6 @@ debug.test
*.test
remark42
/backend/var/
/playwright-report/
# go e2e suite, never built into the image
/e2e/
+1 -1
View File
@@ -3,4 +3,4 @@
# review when someone opens a pull request.
* @umputun
frontend/* @akellbl4 @Mavrin
frontend/* @umputun @akellbl4 @Mavrin
+66
View File
@@ -0,0 +1,66 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
version: 2
# npm updates are switched off entirely. open-pull-requests-limit bounds version
# updates only, so the ignore entries below are what also stops security updates;
# removing the npm entries would not work, as security updates come from alerts
# rather than from this file.
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
groups:
"GitHub Actions updates":
patterns:
- "*"
- package-ecosystem: "gomod"
directory: "/backend"
schedule:
interval: "monthly"
groups:
"Go modules updates":
dependency-type: "production"
- package-ecosystem: "gomod"
directory: "/e2e"
schedule:
interval: "monthly"
groups:
"Go modules updates":
dependency-type: "production"
- package-ecosystem: "npm"
directory: "/frontend"
open-pull-requests-limit: 0
ignore:
- dependency-name: "*"
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
- package-ecosystem: "npm"
directory: "/frontend/apps/remark42"
open-pull-requests-limit: 0
ignore:
- dependency-name: "*"
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
- package-ecosystem: "docker"
directory: "/site"
schedule:
interval: "monthly"
groups:
"Site image updates":
patterns:
- "*"
+55 -11
View File
@@ -5,24 +5,34 @@ on:
branches:
tags:
paths:
- ".github/workflows/ci-test-backend.yml"
- ".github/workflows/ci-backend.yml"
- "backend/**"
- "Dockerfile"
- "docker-init.sh"
- ".dockerignore"
- "!backend/scripts/**"
- "!**.md"
pull_request:
types: [opened, reopened]
paths:
- ".github/workflows/ci-test-backend.yml"
- ".github/workflows/ci-backend.yml"
- "backend/**"
- "Dockerfile"
- "docker-init.sh"
- ".dockerignore"
- "!backend/scripts/**"
- "!**.md"
jobs:
test:
name: Test & Coverage
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: debug if needed
run: if [[ "$DEBUG" == "true" ]]; then env; fi
@@ -30,13 +40,15 @@ jobs:
DEBUG: ${{secrets.DEBUG}}
- name: install go
uses: actions/setup-go@v3
uses: actions/setup-go@v7
with:
go-version: "1.20"
go-version: "1.25"
check-latest: true
cache-dependency-path: backend
- name: test and build backend
run: |
go test -race -timeout=60s -covermode=atomic -coverprofile=$GITHUB_WORKSPACE/profile.cov_tmp ./...
go test -race -timeout=300s -covermode=atomic -coverprofile=$GITHUB_WORKSPACE/profile.cov_tmp ./...
cat $GITHUB_WORKSPACE/profile.cov_tmp | grep -v "_mock.go" > $GITHUB_WORKSPACE/profile.cov
go build -race ./...
working-directory: backend/app
@@ -52,15 +64,15 @@ jobs:
TZ: "America/Chicago"
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
uses: golangci/golangci-lint-action@v9
with:
version: latest
version: "v2.13.1"
working-directory: backend/app
- name: golangci-lint on example directory
uses: golangci/golangci-lint-action@v3
uses: golangci/golangci-lint-action@v9
with:
version: latest
version: "v2.13.1"
args: --config ../../.golangci.yml
working-directory: backend/_example/memory_store
@@ -71,3 +83,35 @@ jobs:
working-directory: backend
env:
COVERALLS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
vulncheck:
name: Vulnerability scan
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: install go
uses: actions/setup-go@v7
with:
go-version: "1.25"
check-latest: true
# both go.sum files so the cache key covers the main and example modules scanned below
cache-dependency-path: |
backend/go.sum
backend/_example/memory_store/go.sum
- name: govulncheck
run: |
go install golang.org/x/vuln/cmd/govulncheck@v1.5.0
govulncheck ./...
(cd _example/memory_store && govulncheck ./...)
working-directory: backend
env:
# ignore the committed vendor dirs and resolve modules from the cache so
# both the main module and the nested example module scan consistently
GOFLAGS: "-mod=readonly"
+39 -77
View File
@@ -1,18 +1,6 @@
name: build
on:
push:
branches:
tags:
paths:
- ".github/workflows/ci-build.yml"
- "backend/**"
- "frontend/apps/**"
- ".dockerignore"
- "docker-init.sh"
- "Dockerfile"
- "!**.md"
- "!frontend/packages/**"
pull_request:
paths:
- ".github/workflows/ci-build.yml"
@@ -23,79 +11,53 @@ on:
- "Dockerfile"
- "!**.md"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build-images:
name: Build Docker images
name: Validate Docker build
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v3
- name: set up QEMU
uses: docker/setup-qemu-action@v2
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v4
- name: available platforms
run: echo ${{ steps.buildx.outputs.platforms }}
- name: build docker image without pushing (only outside master)
if: ${{ github.ref != 'refs/heads/master' }}
- name: free disk space
run: |
docker buildx build \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true \
--platform linux/amd64 .
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
docker system prune -af
- name: build example docker image without pushing (only outside master)
if: ${{ github.ref != 'refs/heads/master' }}
run: |
docker buildx build \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true \
--platform linux/amd64 -f backend/_example/memory_store/Dockerfile .
- name: build docker image without pushing
uses: docker/build-push-action@v7
with:
context: .
platforms: linux/amd64
load: true
cache-from: type=gha,scope=main
cache-to: type=gha,scope=main,mode=max,ignore-error=true
build-args: |
SKIP_BACKEND_TEST=true
SKIP_FRONTEND_TEST=true
- name: build and deploy master image to ghcr.io and dockerhub
if: ${{ github.ref == 'refs/heads/master' }}
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo "GITHUB_REF=${GITHUB_REF}, GITHUB_SHA=${GITHUB_SHA}, GIT_BRANCH=${ref}"
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
echo ${DOCKER_HUB_TOKEN} | docker login -u umputun --password-stdin
docker buildx build --push \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true --build-arg CI=github \
--build-arg GITHUB_SHA=${GITHUB_SHA} --build-arg GIT_BRANCH=${ref} --build-arg GITHUB_REF=${GITHUB_REF} \
--platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42:${ref} -t umputun/remark42:${ref} .
- name: deploy tagged (latest) to ghcr.io and dockerhub
if: ${{ startsWith(github.ref, 'refs/tags/') }}
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo "GITHUB_REF=${GITHUB_REF}, GITHUB_SHA=${GITHUB_SHA}, GIT_BRANCH=${ref}"
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
echo ${DOCKER_HUB_TOKEN} | docker login -u umputun --password-stdin
docker buildx build --push \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true --build-arg CI=github \
--build-arg GITHUB_SHA=${GITHUB_SHA} --build-arg GIT_BRANCH=${ref} --build-arg GITHUB_REF=${GITHUB_REF} \
--platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42:${ref} -t ghcr.io/umputun/remark42:latest \
-t umputun/remark42:${ref} -t umputun/remark42:latest .
- name: remote deployment to remark42.com from master
if: ${{ github.ref == 'refs/heads/master' }}
env:
UPDATER_KEY: ${{ secrets.UPDATER_KEY }}
run: curl -s https://jess.umputun.com/update/remark42-core/${UPDATER_KEY}
- name: build example docker image without pushing
uses: docker/build-push-action@v7
with:
context: .
file: backend/_example/memory_store/Dockerfile
platforms: linux/amd64
load: true
cache-from: type=gha,scope=example
cache-to: type=gha,scope=example,mode=max,ignore-error=true
build-args: |
SKIP_BACKEND_TEST=true
SKIP_FRONTEND_TEST=true
+44
View File
@@ -0,0 +1,44 @@
name: compose
on:
push:
branches:
- master
paths:
- ".github/workflows/ci-compose.yml"
- "**compose*.yml"
- "**compose*.yaml"
pull_request:
paths:
- ".github/workflows/ci-compose.yml"
- "**compose*.yml"
- "**compose*.yaml"
jobs:
validate:
name: Validate compose files
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: validate tracked compose files
run: |
set -euo pipefail
n=0
# null-delimited to stay safe with unusual filenames; exclude this
# workflow (its name contains "compose") and vendored compose files.
# filenames are not echoed as workflow commands to avoid log-command injection
while IFS= read -r -d '' f; do
docker compose -f "$f" config --quiet
n=$((n + 1))
done < <(git ls-files -z '*compose*.yml' '*compose*.yaml' ':!:*/vendor/*' ':!:.github/*')
if [ "$n" -eq 0 ]; then
echo "no compose files found" >&2
exit 1
fi
echo "validated $n compose file(s)"
+37
View File
@@ -0,0 +1,37 @@
name: docs versions
on:
push:
branches:
- master
paths:
- ".github/workflows/ci-docs-versions.yml"
- "scripts/check-documented-versions.sh"
- "site/content/docs/getting-started/installation/index.md"
- "backend/go.mod"
- "frontend/apps/remark42/package.json"
- "frontend/.nvmrc"
pull_request:
paths:
- ".github/workflows/ci-docs-versions.yml"
- "scripts/check-documented-versions.sh"
- "site/content/docs/getting-started/installation/index.md"
- "backend/go.mod"
- "frontend/apps/remark42/package.json"
- "frontend/.nvmrc"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
check:
name: Documented versions
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: Check documented versions against the repository
run: ./scripts/check-documented-versions.sh
-153
View File
@@ -1,153 +0,0 @@
name: "@remark42/api"
on:
push:
branches:
- master
paths:
- ".github/workflows/ci-frontend-api.yml"
- "frontend/packages/**"
- "!**.md"
pull_request:
paths:
- ".github/workflows/ci-frontend-api.yml"
- "frontend/packages/**"
- "!**.md"
jobs:
type-check:
name: Type check
runs-on: ubuntu-latest
strategy:
matrix:
node: [16.15.1]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
with:
version: 7
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
- name: Run type check
run: pnpm type-check:api
working-directory: ./frontend
lint:
name: Lint
runs-on: ubuntu-latest
strategy:
matrix:
node: [16.15.1]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
with:
version: 7
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
- name: Run linters
run: pnpm lint:api
working-directory: ./frontend/
test:
name: Tests & Coverage
runs-on: ubuntu-latest
strategy:
matrix:
node: [16.15.1]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
with:
version: 7
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
- name: Test & Coverage
run: pnpm coverage:api
working-directory: ./frontend
- name: Submit coverage
run: ${{ github.workspace }}/frontend/apps/remark42/node_modules/.bin/codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+73 -99
View File
@@ -6,54 +6,46 @@ on:
- master
paths:
- ".github/workflows/ci-frontend.yml"
- "frontend/apps/remark42/**"
- "frontend/**"
- "!**.md"
pull_request:
paths:
- ".github/workflows/ci-frontend.yml"
- "frontend/apps/remark42/**"
- "frontend/**"
- "!**.md"
jobs:
translations-check:
name: Translations check
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [24]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
uses: actions/checkout@v7
with:
node-version: ${{ matrix.node }}
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
uses: pnpm/action-setup@v6.0.10
with:
version: 7
version: 10.10.0
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
- name: Install node
uses: actions/setup-node@v7
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
node-version: ${{ matrix.node }}
cache: "pnpm"
cache-dependency-path: frontend/apps/remark42/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
run: pnpm install --frozen-lockfile
working-directory: ./frontend/apps/remark42
- name: Translations check
run: pnpm translation-check
@@ -62,42 +54,34 @@ jobs:
type-check:
name: Type check
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [24]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
uses: actions/checkout@v7
with:
node-version: ${{ matrix.node }}
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
uses: pnpm/action-setup@v6.0.10
with:
version: 7
version: 10.10.0
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
- name: Install node
uses: actions/setup-node@v7
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
node-version: ${{ matrix.node }}
cache: "pnpm"
cache-dependency-path: frontend/apps/remark42/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
run: pnpm install --frozen-lockfile
working-directory: ./frontend/apps/remark42
- name: Run type check
run: pnpm type-check
@@ -106,42 +90,34 @@ jobs:
lint:
name: Eslint & Stylelint
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [24]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
uses: actions/checkout@v7
with:
node-version: ${{ matrix.node }}
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
uses: pnpm/action-setup@v6.0.10
with:
version: 7
version: 10.10.0
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
- name: Install node
uses: actions/setup-node@v7
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
node-version: ${{ matrix.node }}
cache: "pnpm"
cache-dependency-path: frontend/apps/remark42/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
run: pnpm install --frozen-lockfile
working-directory: ./frontend/apps/remark42
- name: Run linters
run: pnpm lint
@@ -151,21 +127,25 @@ jobs:
name: Size limit
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
permissions:
contents: read
pull-requests: write
env:
CI_JOB_NUMBER: 1
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
uses: pnpm/action-setup@v6.0.10
with:
version: 7
version: 10.10.0
run_install: false
- name: Check bundle size
uses: andresz1/size-limit-action@dd31dce7dcc72a041fd3e49abf0502b13fc4ce05
uses: andresz1/size-limit-action@94bc357df29c36c8f8d50ea497c3e225c3c95d1d
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
directory: ./frontend/apps/remark42
@@ -174,48 +154,42 @@ jobs:
test:
name: Tests & Coverage
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [24]
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install node
uses: actions/setup-node@v3
uses: actions/checkout@v7
with:
node-version: ${{ matrix.node }}
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
id: pnpm-install
uses: pnpm/action-setup@v6.0.10
with:
version: 7
version: 10.10.0
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
- name: Setup pnpm cache
uses: actions/cache@v3
- name: Install node
uses: actions/setup-node@v7
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
node-version: ${{ matrix.node }}
cache: "pnpm"
cache-dependency-path: frontend/apps/remark42/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
working-directory: ./frontend
run: pnpm install --frozen-lockfile
working-directory: ./frontend/apps/remark42
- name: Test & Coverage
run: pnpm coverage
working-directory: ./frontend/apps/remark42
- name: Submit coverage
run: ${{ github.workspace }}/frontend/apps/remark42/node_modules/.bin/codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v7
with:
token: ${{ secrets.CODECOV_TOKEN }}
working-directory: ./frontend/apps/remark42
codecov_yml_path: ./frontend/apps/remark42/codecov.yml
+144 -38
View File
@@ -1,69 +1,175 @@
name: site
on:
release:
types: [published]
push:
branches:
- master
tags:
paths:
- ".github/workflows/ci-site.yml"
- "site/**"
- "!**/CLAUDE.md"
- "!site/README.md"
pull_request:
paths:
- ".github/workflows/ci-site.yml"
- "site/**"
- "!**/CLAUDE.md"
- "!site/README.md"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
name: Build
validate:
name: Build site image (pull request)
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: checkout
uses: actions/checkout@v3
uses: actions/checkout@v7
with:
persist-credentials: false
- name: set up QEMU
uses: docker/setup-qemu-action@v2
- name: set up docker buildx
uses: docker/setup-buildx-action@v4
- name: build image without pushing
uses: docker/build-push-action@v7
with:
context: ./site
load: true
push: false
cache-from: |
type=gha,scope=site-pr
type=gha,scope=site-linux/amd64
cache-to: type=gha,scope=site-pr,mode=max,ignore-error=true
build:
name: Build site image (${{ matrix.platform }})
if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/')
runs-on: ${{ matrix.runner }}
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
artifact: linux-amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
artifact: linux-arm64
steps:
- name: checkout
uses: actions/checkout@v7
with:
persist-credentials: false
- name: set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v4
- name: available platforms
run: echo ${{ steps.buildx.outputs.platforms }}
- name: login to ghcr.io
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: build and deploy master image to ghcr.io and dockerhub
if: ${{ github.ref == 'refs/heads/master' }}
- name: build and push by digest
id: build
uses: docker/build-push-action@v7
with:
context: ./site
platforms: ${{ matrix.platform }}
cache-from: type=gha,scope=site-${{ matrix.platform }}
cache-to: type=gha,scope=site-${{ matrix.platform }},mode=max,ignore-error=true
outputs: type=image,name=ghcr.io/umputun/remark42-site,push-by-digest=true,name-canonical=true,push=true
- name: export digest
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: upload digest
uses: actions/upload-artifact@v7
with:
name: site-digests-${{ matrix.artifact }}
path: /tmp/digests/*
retention-days: 1
merge:
name: Create site multi-arch manifest
runs-on: ubuntu-latest
needs: build
permissions:
contents: read
packages: write
steps:
- name: download digests
uses: actions/download-artifact@v8
with:
path: /tmp/digests
pattern: site-digests-*
merge-multiple: true
- name: verify all digests present
run: |
expected=2
actual=$(find /tmp/digests -maxdepth 1 -type f | wc -l)
if [ "$actual" -ne "$expected" ]; then
echo "Expected $expected digests, found $actual"
ls -la /tmp/digests
exit 1
fi
echo "All $expected digests present"
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: create manifest and push
working-directory: /tmp/digests
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
working-directory: ./site
GITHUB_REF: ${{ github.ref }}
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo GITHUB_REF - $ref
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
docker buildx build --push --no-cache --platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42-site:${ref} .
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
docker buildx imagetools create \
-t ghcr.io/umputun/remark42-site:${ref} \
-t ghcr.io/umputun/remark42-site:latest \
$(printf 'ghcr.io/umputun/remark42-site@sha256:%s ' *)
else
docker buildx imagetools create \
-t ghcr.io/umputun/remark42-site:${ref} \
$(printf 'ghcr.io/umputun/remark42-site@sha256:%s ' *)
fi
- name: deploy tagged (latest) to ghcr.io and dockerhub
if: ${{ startsWith(github.ref, 'refs/tags/') }}
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
working-directory: ./site
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo "GITHUB_REF=$ref, GITHUB_SHA=${GITHUB_SHA}"
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
docker buildx build --push --no-cache --platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42-site:${ref} -t ghcr.io/umputun/remark42-site:latest .
deploy:
name: Deploy site
runs-on: ubuntu-latest
needs: merge
if: github.ref == 'refs/heads/master' || github.event_name == 'release'
permissions: {} # only calls an external URL via curl, no GitHub API access needed
- name: remote site deployment from master
if: ${{ github.ref == 'refs/heads/master' }}
steps:
- name: trigger deployment
env:
UPDATER_KEY: ${{ secrets.UPDATER_KEY }}
run: curl https://jess.umputun.com/update/remark42-site/${UPDATER_KEY}
run: curl -sf https://jess.umputun.com/update/remark42-site/${UPDATER_KEY}
+215
View File
@@ -0,0 +1,215 @@
name: docker
on:
workflow_run:
workflows: [backend, frontend]
types: [completed]
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: true
jobs:
build:
name: Build Docker image (${{ matrix.platform }})
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event != 'pull_request' &&
(github.event.workflow_run.head_branch == 'master' ||
startsWith(github.event.workflow_run.head_branch, 'v'))
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
artifact: linux-amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
artifact: linux-arm64
runs-on: ${{ matrix.runner }}
steps:
- name: checkout
uses: actions/checkout@v7
with:
ref: ${{ github.event.workflow_run.head_sha }}
persist-credentials: false
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: login to DockerHub
uses: docker/login-action@v4
with:
username: umputun
password: ${{ secrets.DOCKER_HUB_TOKEN }}
- name: free disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
docker system prune -af
- name: build and push to ghcr.io by digest
id: build-ghcr
uses: docker/build-push-action@v7
with:
context: .
platforms: ${{ matrix.platform }}
cache-from: type=gha,scope=${{ matrix.platform }}
cache-to: type=gha,scope=${{ matrix.platform }},mode=max
build-args: |
SKIP_BACKEND_TEST=true
SKIP_FRONTEND_TEST=true
CI=github
GITHUB_SHA=${{ github.event.workflow_run.head_sha }}
GIT_BRANCH=${{ github.event.workflow_run.head_branch }}
GITHUB_REF=refs/heads/${{ github.event.workflow_run.head_branch }}
outputs: type=image,name=ghcr.io/umputun/remark42,push-by-digest=true,name-canonical=true,push=true
- name: build and push to DockerHub by digest
id: build-dockerhub
uses: docker/build-push-action@v7
with:
context: .
platforms: ${{ matrix.platform }}
cache-from: type=gha,scope=${{ matrix.platform }}
build-args: |
SKIP_BACKEND_TEST=true
SKIP_FRONTEND_TEST=true
CI=github
GITHUB_SHA=${{ github.event.workflow_run.head_sha }}
GIT_BRANCH=${{ github.event.workflow_run.head_branch }}
GITHUB_REF=refs/heads/${{ github.event.workflow_run.head_branch }}
outputs: type=image,name=umputun/remark42,push-by-digest=true,name-canonical=true,push=true
- name: export digests
run: |
mkdir -p /tmp/digests/ghcr /tmp/digests/dockerhub
digest_ghcr="${{ steps.build-ghcr.outputs.digest }}"
digest_dockerhub="${{ steps.build-dockerhub.outputs.digest }}"
touch "/tmp/digests/ghcr/${digest_ghcr#sha256:}"
touch "/tmp/digests/dockerhub/${digest_dockerhub#sha256:}"
- name: upload ghcr digest
uses: actions/upload-artifact@v7
with:
name: digests-ghcr-${{ matrix.artifact }}
path: /tmp/digests/ghcr/*
retention-days: 1
- name: upload dockerhub digest
uses: actions/upload-artifact@v7
with:
name: digests-dockerhub-${{ matrix.artifact }}
path: /tmp/digests/dockerhub/*
retention-days: 1
merge:
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: build
permissions:
contents: read
packages: write
steps:
- name: download ghcr digests
uses: actions/download-artifact@v8
with:
path: /tmp/digests/ghcr
pattern: digests-ghcr-*
merge-multiple: true
- name: download dockerhub digests
uses: actions/download-artifact@v8
with:
path: /tmp/digests/dockerhub
pattern: digests-dockerhub-*
merge-multiple: true
- name: verify all digests present
run: |
expected=2
for registry in ghcr dockerhub; do
actual=$(find /tmp/digests/$registry -maxdepth 1 -type f | wc -l)
if [ "$actual" -ne "$expected" ]; then
echo "Expected $expected digests for $registry, found $actual"
ls -la /tmp/digests/$registry
exit 1
fi
done
echo "All digests present for both registries"
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: login to DockerHub
uses: docker/login-action@v4
with:
username: umputun
password: ${{ secrets.DOCKER_HUB_TOKEN }}
- name: create ghcr.io manifest and push
working-directory: /tmp/digests/ghcr
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
if [[ "$HEAD_BRANCH" == v* ]]; then
docker buildx imagetools create \
-t ghcr.io/umputun/remark42:${HEAD_BRANCH} \
-t ghcr.io/umputun/remark42:latest \
$(printf 'ghcr.io/umputun/remark42@sha256:%s ' *)
else
docker buildx imagetools create \
-t ghcr.io/umputun/remark42:${HEAD_BRANCH} \
$(printf 'ghcr.io/umputun/remark42@sha256:%s ' *)
fi
- name: create DockerHub manifest and push
working-directory: /tmp/digests/dockerhub
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
if [[ "$HEAD_BRANCH" == v* ]]; then
docker buildx imagetools create \
-t umputun/remark42:${HEAD_BRANCH} \
-t umputun/remark42:latest \
$(printf 'umputun/remark42@sha256:%s ' *)
else
docker buildx imagetools create \
-t umputun/remark42:${HEAD_BRANCH} \
$(printf 'umputun/remark42@sha256:%s ' *)
fi
deploy:
name: Deploy to remark42.com
runs-on: ubuntu-latest
needs: merge
if: github.event.workflow_run.head_branch == 'master'
permissions: {} # only calls an external URL via curl, no GitHub API access needed
steps:
- name: trigger deployment
env:
UPDATER_KEY: ${{ secrets.UPDATER_KEY }}
run: curl -sf https://jess.umputun.com/update/remark42-core/${UPDATER_KEY}
+108 -17
View File
@@ -4,31 +4,122 @@ on:
push:
branches: [master]
paths:
- "frontend/apps/remark42/**"
- "frontend/e2e/**"
- ".github/workflows/e2e-tests.yml"
- "backend/**"
- "frontend/**"
- "e2e/**"
- "compose-e2e-test.yml"
- "Dockerfile"
- "!**.md"
pull_request:
branches: [master]
paths:
- "frontend/apps/remark42/**"
- "frontend/e2e/**"
- ".github/workflows/e2e-tests.yml"
- "backend/**"
- "frontend/**"
- "e2e/**"
- "compose-e2e-test.yml"
- "Dockerfile"
- "!**.md"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
tests:
name: Tests
timeout-minutes: 60
# cheap gate: catches a compile break or a lint regression in the build-tagged suite
# without paying for the docker build and the browser download
vet:
name: Vet
timeout-minutes: 10
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Build & run containers
id: tests
run: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -f compose-e2e-test.yml up --build --quiet-pull --exit-code-from tests
- uses: actions/upload-artifact@v2
if: always()
uses: actions/checkout@v7
with:
name: playwright-report
path: ./playwright-report/
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: e2e/go.mod
cache-dependency-path: e2e/go.sum
- name: Vet
run: cd e2e && go vet -tags=e2e ./...
- name: Lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.13.1
working-directory: e2e
args: --build-tags=e2e --config ../backend/.golangci.yml
tests:
name: Tests
needs: vet
# generous against the docker build plus one 8m go test: a job cancelled on timeout skips
# its own failure steps, so the run would end with neither logs nor traces
timeout-minutes: 45
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Set up Go
uses: actions/setup-go@v7
with:
go-version-file: e2e/go.mod
cache-dependency-path: e2e/go.sum
# two directories: the driver (node plus the npm package) and the browser builds,
# which include firefox and webkit for the rendering tests
- name: Cache playwright driver and browsers
uses: actions/cache@v6
with:
path: |
~/.cache/ms-playwright
~/.cache/ms-playwright-go
key: playwright-${{ hashFiles('e2e/go.sum') }}
restore-keys: playwright-
# E2E_STAMP is what the suite compares the running stack against, so a stack started here
# has to carry the same value `make e2e-up` and the suite itself would give it
- name: Build & start the stack
run: |
./e2e/tls/generate.sh
COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 E2E_STAMP=$(./e2e/stamp.sh) \
docker compose -f compose-e2e-test.yml up -d --build --quiet-pull --wait
# no retry: a failure here is evidence about a suite too young to have a flake rate,
# and a rerun is how an intermittent regression becomes invisible. revisit when there
# are failures on record to look at
- name: Run e2e
# stamps this run's comment threads with the CI run, so a thread url in a trace or a
# log names the run it came from
env:
E2E_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
# 20m, matching the Makefile. the suite runs about four minutes on a laptop and a runner
# is slower, so a tighter budget turns a loaded runner into a timeout panic instead of a
# readable failure. the job's own timeout above is what bounds a wedged run
run: cd e2e && go test -tags=e2e -count 1 -timeout 20m -v ./...
- name: Server logs on failure
if: failure()
run: docker compose -f compose-e2e-test.yml logs --tail=200
- name: Upload browser traces
if: always()
uses: actions/upload-artifact@v7
with:
name: playwright-traces
path: e2e/traces/
retention-days: 30
if-no-files-found: ignore
+145
View File
@@ -0,0 +1,145 @@
name: release
on:
push:
tags:
- "v*"
pull_request:
paths:
- ".github/workflows/release.yml"
- ".goreleaser.yml"
- "Makefile"
- "scripts/**"
- "backend/**"
- "frontend/**"
- "!backend/**.md"
- "!frontend/**.md"
- "README.md"
- "LICENSE"
permissions:
contents: read
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- name: install go
uses: actions/setup-go@v7
with:
go-version: "1.25"
check-latest: true
cache-dependency-path: backend/go.sum
- name: install pnpm
uses: pnpm/action-setup@v6.0.10
with:
version: 10.10.0
run_install: false
- name: install node
uses: actions/setup-node@v7
with:
node-version: 24
cache: "pnpm"
cache-dependency-path: frontend/apps/remark42/pnpm-lock.yaml
- name: test and build backend
run: |
go test -race -timeout=300s ./...
go build -race ./...
working-directory: backend/app
env:
TZ: "America/Chicago"
- name: test examples
run: |
go test -race ./...
go build -race ./...
working-directory: backend/_example/memory_store
env:
TZ: "America/Chicago"
- name: install frontend dependencies
run: pnpm install --frozen-lockfile
working-directory: frontend/apps/remark42
env:
CI: "true"
- name: check frontend
run: |
pnpm lint
pnpm type-check
pnpm test --runInBand
working-directory: frontend/apps/remark42
env:
CI: "true"
- name: check goreleaser snapshot
if: github.event_name == 'pull_request'
uses: goreleaser/goreleaser-action@v7
with:
version: latest
args: release --snapshot --clean --skip=publish
env:
SKIP_PNPM_INSTALL: "true"
- name: clean generated release assets
if: always()
run: ./scripts/cleanup-release-assets.sh
release:
if: github.event_name == 'push'
needs: validate
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- name: install go
uses: actions/setup-go@v7
with:
go-version: "1.25"
check-latest: true
cache-dependency-path: backend/go.sum
- name: install pnpm
uses: pnpm/action-setup@v6.0.10
with:
version: 10.10.0
run_install: false
- name: install node
uses: actions/setup-node@v7
with:
node-version: 24
cache: "pnpm"
cache-dependency-path: frontend/apps/remark42/pnpm-lock.yaml
- name: install frontend dependencies
run: pnpm install --frozen-lockfile
working-directory: frontend/apps/remark42
env:
CI: "true"
- name: run goreleaser
uses: goreleaser/goreleaser-action@v7
with:
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SKIP_PNPM_INSTALL: "true"
- name: clean generated release assets
if: always()
run: ./scripts/cleanup-release-assets.sh
+10 -1
View File
@@ -15,6 +15,7 @@ debug.test
.mongo
remark42
/bin/
/dist/
/backend/var/
/backend/app/var/
/backend/app/cmd/web/
@@ -24,5 +25,13 @@ compose-private-frontend.yml
compose-private.yml
/backend/_example/*/vendor
http-client.env.json
/playwright-report/
/backend/app/cmd/var
# ralphex progress logs
.ralphex/progress/
# traces from failed e2e runs
/e2e/traces/
# self-signed certificate for the e2e https services, made by e2e/tls/generate.sh
/e2e/tls/*.pem
+60
View File
@@ -0,0 +1,60 @@
version: 2
project_name: remark42
git:
ignore_tags:
- backend/*
before:
hooks:
- ./scripts/prepare-release-assets.sh
builds:
- id: remark42
dir: backend
main: ./app
binary: "remark42.{{ .Os }}-{{ .Arch }}"
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
- freebsd
- windows
goarch:
- amd64
- arm64
- "386"
ignore:
- goos: darwin
goarch: "386"
- goos: freebsd
goarch: arm64
- goos: freebsd
goarch: "386"
- goos: windows
goarch: arm64
- goos: windows
goarch: "386"
ldflags:
- -s -w -X main.revision={{ .Tag }}-{{ .ShortCommit }}-{{ trimsuffix (replace (replace .CommitDate "-" "") ":" "") "Z" }}
archives:
- id: remark42
ids:
- remark42
name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}"
formats:
- tar.gz
format_overrides:
- goos: windows
formats:
- zip
files:
- LICENSE
- README.md
release:
name_template: "Version {{ .Version }}"
mode: keep-existing
+100
View File
@@ -0,0 +1,100 @@
# Remark42 Development Guidelines
## Build/Test/Lint Commands
- **Backend**:
- Run server: `make rundev`
- Build: `make backend`
- Race test: `make race_test`
- **Backend Testing**:
- Run all tests: `cd backend/app && go test -timeout=300s -count 1 ./...`
- Run single test: `cd backend/app && go test -run TestName ./path/to/package`
- **IMPORTANT**: Run example tests: `cd backend/_example/memory_store && go test -race ./... && go build -race ./...`
- **Frontend**:
- Development: `cd frontend/apps/remark42 && pnpm dev`
- Tests: `cd frontend/apps/remark42 && pnpm test`
- **End-to-end**: `make e2e` drives the widget in a real browser; see `e2e/README.md`. Build-tagged, so `go test ./...` never runs it.
- **Lint**:
- Backend: `cd backend && golangci-lint run`
- **IMPORTANT**: Example lint: `cd backend/_example/memory_store && golangci-lint run --config ../../.golangci.yml`
- Frontend: `cd frontend/apps/remark42 && pnpm lint`
- **Before committing**: Always run tests and linter on both main backend AND examples
- **Go module changes**:
- **Any** change to `backend/go.mod` or `backend/go.sum` requires `go mod tidy` in `backend/_example/memory_store` in the same commit. That covers dependency bumps, adding or removing a dependency, and changing the `go` directive, not only version updates.
- Only `go mod tidy` there, not `go mod vendor`: the example's vendor directory is gitignored (`.gitignore:26`), so its output is never committed, while a stale local copy silently becomes what the example resolves against.
- The example module replaces `github.com/umputun/remark42/backend` with `../../`, so it carries the backend's dependencies as indirect entries. Leaving them stale fails the `test examples` CI step with `go: updates to go.mod needed; to update it: go mod tidy`.
- This applies to Dependabot pull requests too: the bot updates `backend/` only, so its Go module PRs need the example tidied before they can go green.
## Backend Test Determinism
Backend tests must never depend on how fast the machine is. CI runs them under `-race` with coverage on a shared runner, so any test that assumes an operation finishes within some duration eventually fails on a rerun-and-it-passes basis.
- **Wait on a condition, never on a duration.** Use `require.Eventually` / `require.EventuallyWithT` to poll for the state the assertion needs, and `require.Never` when the point is that something did *not* happen. A bare `time.Sleep` before an assertion is a defect; sleeping until a deadline you computed, as `waitPastMillisecond` does, is not.
- **Polling closures must not touch `*testing.T`.** testify runs them on a separate goroutine, where `t.FailNow` is undefined behaviour. Assert on the `*assert.CollectT` that `EventuallyWithT` hands the closure, so the real error also lands in the failure message.
- **Mind the rate limiter when polling over HTTP.** Route groups are capped independently and most of the caps are hard-coded in `rest.go`, out of reach of a test: `/auth/` at 2 req/s and the admin, protected and image routes at 10 req/s. Only the open-route group is settable, via `openRouteLimiter` (100 in `startupT`). Poll with the existing constants rather than a new number, `httpPoll` for anything issuing an HTTP request and `pollInterval` only for in-process or filesystem checks, or the poll manufactures the 429s it then has to interpret.
- **When a test needs time to have passed, pin the clock input rather than waiting for it:** `os.Chtimes` for file ages, an explicit `store.Comment.Timestamp` for anything that formats a timestamp.
- **Prefer a `testing/synctest` bubble** where the code under test has no real I/O. Inside one the clock is fake, so `time.Sleep` is instant and deterministic. `app/notify`, `app/store/service`, `app/store/image`, `app/store/engine`, `app/providers`, `app/migrator` and `_example/memory_store/accessor` already use it, and most surviving `time.Sleep` calls live in them.
- **Helpers fail loudly.** A wait that gives up must call `t.Fatal`/`require` naming what it was waiting for, never return silently and leave the next assertion to fail with something unrelated. Because these packages run `goleak.VerifyTestMain`, a failing helper also exits the test goroutine, so anything that started a server in a goroutine must `defer cancel()` or `defer srv.Shutdown()` right after launching it; otherwise a failed readiness wait is reported as a goroutine leak rather than the failure that caused it.
- **Take ports and paths from outside the test.** Ports come from the kernel with `net.Listen("tcp", ":0")`, files from `t.TempDir()`. `go test ./...` runs package binaries concurrently, so a number out of a fixed range or a fixed name under `/tmp` lets two of them collide.
- **Close idle connections before shutting a test server down.** Clients built as `http.Client{Timeout: x}` share `http.DefaultTransport`, and `Shutdown` waits on their keep-alive connections until its own deadline expires.
- **Keep the test timeout budgets aligned.** `Makefile`, `ci-backend.yml`, `release.yml` and the command above all use `-timeout=300s`; the wait helpers allow 30s per condition, so a shorter per-package budget turns a slow runner into a timeout panic instead of a readable failure.
`chooseUnusedPort` and the server-start wait helpers are duplicated in `app`, `app/cmd`, `app/rest/api` and `_example/memory_store/server`. Nothing shares them today; keep the copies in step when changing one.
## Release Procedure
Remark42 uses two tags for each release:
- `vX.Y.Z` - product release tag used by GitHub releases, GoReleaser binary artifacts, and Docker image publishing.
- `backend/vX.Y.Z` - nested Go module tag for `github.com/umputun/remark42/backend`.
Release flow:
1. Create the GitHub release for `vX.Y.Z` with title `Version X.Y.Z`. The GitHub release must exist before the `vX.Y.Z` tag reaches the remote; `gh release create vX.Y.Z` satisfies this because it creates and pushes the tag.
2. The `vX.Y.Z` tag triggers GoReleaser, which builds and uploads binary artifacts to the existing release.
3. Create and push the matching backend module tag pointing at the same commit:
```bash
git fetch origin --tags
git tag backend/vX.Y.Z vX.Y.Z
git push origin backend/vX.Y.Z
```
GoReleaser must ignore `backend/*` tags in `.goreleaser.yml` so release notes and current-tag detection use only product tags. Docker image publishing stays separate and is handled by the existing Docker workflow.
For local artifact runs, install GoReleaser, Go 1.25, Node 24+ and PNPM 10, then use `make release`. The target runs a snapshot/no-publish GoReleaser build, leaves local artifacts and metadata in `dist/`, and cleans generated frontend embed files after GoReleaser exits. Do not run raw `goreleaser release` for local artifacts unless you also run `./scripts/cleanup-release-assets.sh` afterward.
## Milestones and Issue Labels
**Milestones** — one `vX.Y.Z` milestone per release. Assign every merged PR, and every issue closed by a code change, to the milestone of the release it shipped in.
- Decide which release a PR belongs to by whether its merge commit is **contained in a release tag** — not by comparing dates (a tag can be cut from an earlier commit, or moved). `git fetch --tags`, then `git tag --contains <merge_sha> | grep '^v' | sort -V | head -1` is its release. If no release tag contains it yet, it belongs to the next (unreleased) version's milestone — create it if missing (`gh api repos/umputun/remark42/milestones -f title="vX.Y.Z"`).
- An **issue gets a milestone only when it was closed by a code change** (a linked closing PR/commit); take the milestone from that PR/commit (via the commit-in-tag rule). Issues closed as `duplicate`/`invalid`/`wontfix`/answered get no milestone.
- Find unassigned: `gh pr list --state merged --search "no:milestone"`, `gh issue list --state closed --search "no:milestone"`. Assign with `gh pr edit N --milestone "vX.Y.Z"` / `gh issue edit N --milestone "vX.Y.Z"`.
**Issue labels** — classify each issue with a type and an area (add priority when relevant):
- Type: `bug`, `enhancement`, `question`, `documentation`, `discussion`
- Area: `backend`, `frontend`, `site`, `CI`, `design`, `localization`
- Priority: `important`, `minor`, `some day`
- Contribution: `help wanted`, `good-first-issue`
- Resolution (on close, when applicable): `duplicate`, `invalid`, `wontfix`, `no-action-needed`
- PR auto-labels (applied by Dependabot/Actions, not manual PRs): `dependencies`, `go`, `javascript`, `github_actions`
## Code Style
- **Backend**: Formatting with golangci-lint, strict error handling
- **Frontend**: TypeScript with ESLint, Stylelint and Prettier
- **Imports**: Group stdlib, external packages, then internal packages
- **CSS**: All components use CSS Modules (`component.module.css`). Class naming: BEM block = `.root`, elements = camelCase, modifiers = camelCase. Use `clsx` for conditional class composition. `raw-content.css` is the only global CSS file (syntax highlighting utility). Root wrapper keeps bare `.dark`/`.light` theme class — 8+ module CSS files depend on `:global(.dark)` ancestor. `comment_highlighting` uses `:global()` for imperative `classList` usage in root.tsx
## Key Backend Packages
- **Web/API**: `github.com/go-pkgz/routegroup`, `github.com/go-pkgz/rest`
- **Auth**: `github.com/go-pkgz/auth/v2`
- **Logging**: `github.com/go-pkgz/lgr`
- **Testing**: `github.com/stretchr/testify`
- **Notifications**: `github.com/go-pkgz/notify`
## Repository Structure
- Backend: Go server using BoltDB for storage
- Frontend: Preact/Redux-based UI with iframe embedding
- `/web` is served from two sources, in lookup order: the frontend build output
(`frontend/apps/remark42/public`, embedded at `backend/app/cmd/web` or read from `--web-root`),
then `backend/app/webassets/assets`, embedded in the binary. A plain page or image the bundler
does not process belongs in `webassets`; anything needing templating or the widget's CSS/JS goes
through webpack. A name present in both is served from the frontend build.
+13 -12
View File
@@ -1,17 +1,19 @@
FROM --platform=$BUILDPLATFORM node:16.15.1-alpine AS frontend-deps
FROM --platform=$BUILDPLATFORM node:24-alpine AS frontend-deps
ARG SKIP_FRONTEND_TEST
ARG SKIP_FRONTEND_BUILD
# the manifest's prepare script installs husky hooks, which needs a git repository the build
# context does not have. husky itself skips on CI, and this is the same flag the build stage sets
ENV CI=true
WORKDIR /srv/frontend/
WORKDIR /srv/frontend/apps/remark42/
COPY ./frontend/package.json ./frontend/pnpm-lock.yaml ./frontend/pnpm-workspace.yaml /srv/frontend/
COPY ./frontend/apps/remark42/package.json /srv/frontend/apps/remark42/
COPY ./frontend/apps/remark42/package.json ./frontend/apps/remark42/pnpm-lock.yaml /srv/frontend/apps/remark42/
RUN \
if [[ -z "$SKIP_FRONTEND_BUILD" || -z "$SKIP_FRONTEND_TEST" ]]; then \
apk add --no-cache --update git && \
npm i -g pnpm@7; \
npm i -g pnpm@10.10.0; \
fi
RUN --mount=type=cache,id=pnpm,target=/root/.pnpm-store/v3 \
@@ -45,7 +47,7 @@ RUN \
echo 'Skip frontend build'; \
fi
FROM umputun/baseimage:buildgo-v1.11.0 as build-backend
FROM umputun/baseimage:buildgo-v1.17.0 AS build-backend
ARG CI
ARG GITHUB_REF
@@ -54,15 +56,14 @@ ARG GIT_BRANCH
ARG SKIP_BACKEND_TEST
ARG BACKEND_TEST_TIMEOUT
# install gcc in order to be able to go test package with -race
RUN apk --no-cache add gcc libc-dev
ADD backend /build/backend
# to embed the frontend files statically into Remark42 binary
COPY --from=build-frontend /srv/frontend/apps/remark42/public/ /build/backend/app/cmd/web/
RUN find /build/backend/app/cmd/web/ -regex '.*\.\(html\|js\|mjs\)$' -print -exec sed -i "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
WORKDIR /build/backend
# install gcc in order to be able to go test package with -race
RUN apk --no-cache add gcc libc-dev
RUN echo go version: `go version`
# run tests
@@ -81,7 +82,7 @@ RUN \
echo "version=$version" && \
go build -o remark42 -ldflags "-X main.revision=${version} -s -w" ./app
FROM umputun/baseimage:app-v1.11.0
FROM umputun/baseimage:app-v1.17.0
ARG GITHUB_SHA
@@ -89,7 +90,7 @@ LABEL org.opencontainers.image.authors="Umputun <umputun@gmail.com>" \
org.opencontainers.image.description="Remark42 comment engine" \
org.opencontainers.image.documentation="https://remark42.com/docs/getting-started/" \
org.opencontainers.image.licenses="MIT" \
org.opencontainers.image.source="https://github.com/umputun/remark42.git" \
org.opencontainers.image.source="https://github.com/umputun/remark42" \
org.opencontainers.image.title="Remark42" \
org.opencontainers.image.url="https://remark42.com/" \
org.opencontainers.image.revision="${GITHUB_SHA}"
-64
View File
@@ -1,64 +0,0 @@
FROM node:16.15.1-alpine AS frontend-deps
ENV CI=true
WORKDIR /srv/frontend
COPY ./frontend/package.json ./frontend/pnpm-lock.yaml ./frontend/pnpm-workspace.yaml /srv/frontend/
COPY ./frontend/apps/remark42/package.json /srv/frontend/apps/remark42/package.json
RUN apk add --no-cache --update git && npm i -g pnpm@7
RUN --mount=type=cache,id=pnpm,target=/root/.pnpm-store/v3 pnpm i
FROM frontend-deps AS build-frontend
ENV NODE_ENV=production
ENV CI=true
WORKDIR /srv/frontend/apps/remark42/
COPY ./frontend/apps/remark42/ /srv/frontend/apps/remark42/
RUN pnpm build
FROM umputun/baseimage:buildgo-v1.9.2 as build-backend
ARG GITHUB_TOKEN
ARG GITHUB_REF
ARG GITHUB_SHA
WORKDIR /build/backend
ADD backend /build/backend
ADD README.md /build/
ADD LICENSE /build/
COPY --from=build-frontend /srv/frontend/apps/remark42/public/ /build/backend/app/cmd/web/
RUN find /build/backend/app/cmd/web/ -regex '.*\.\(html\|js\|mjs\)$' -print -exec sed -i "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
RUN \
version=$("/script/version.sh") && echo "version=${version}" && \
GOOS=linux GOARCH=amd64 go build -o remark42.linux-amd64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=linux GOARCH=386 go build -o remark42.linux-386 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=linux GOARCH=arm go build -o remark42.linux-arm -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=linux GOARCH=arm64 go build -o remark42.linux-arm64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=windows GOARCH=amd64 go build -o remark42.windows-amd64.exe -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=darwin GOARCH=amd64 go build -o remark42.darwin-amd64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=darwin GOARCH=arm64 go build -o remark42.darwin-arm64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=freebsd GOARCH=amd64 go build -o remark42.freebsd-amd64 -ldflags "-X main.revision=${version} -s -w" ./app
RUN \
apk add --no-cache --update zip && \
cp ../LICENSE ./LICENSE && cp ../README.md ./README.md && \
tar cvzf remark42.linux-amd64.tar.gz remark42.linux-amd64 LICENSE README.md && \
tar cvzf remark42.linux-386.tar.gz remark42.linux-386 LICENSE README.md && \
tar cvzf remark42.linux-arm.tar.gz remark42.linux-arm LICENSE README.md && \
tar cvzf remark42.linux-arm64.tar.gz remark42.linux-arm64 LICENSE README.md && \
tar cvzf remark42.darwin-amd64.tar.gz remark42.darwin-amd64 LICENSE README.md && \
tar cvzf remark42.darwin-arm64.tar.gz remark42.darwin-arm64 LICENSE README.md && \
tar cvzf remark42.freebsd-amd64.tar.gz remark42.freebsd-amd64 LICENSE README.md && \
zip remark42.windows-amd64.zip remark42.windows-amd64.exe LICENSE README.md
FROM alpine
COPY --from=build-backend /build/backend/remark42.* /artifacts/
RUN ls -la /artifacts/*
CMD ["sleep", "100"]
+31 -27
View File
@@ -2,54 +2,58 @@ OS=linux
ARCH=amd64
GITHUB_REF=$(shell git rev-parse --symbolic-full-name HEAD)
GITHUB_SHA=$(shell git rev-parse --short HEAD)
CLEANUP_RELEASE_ASSETS=$(CURDIR)/scripts/cleanup-release-assets.sh
bin:
docker build -f Dockerfile.artifacts -t remark42.bin .
- @docker rm -f remark42.bin 2>/dev/null || exit 0
docker run -d --name=remark42.bin remark42.bin
docker cp remark42.bin:/artifacts/remark42.$(OS)-$(ARCH) remark42
docker rm -f remark42.bin
@set -e; \
./scripts/prepare-release-assets.sh; \
trap '$(CLEANUP_RELEASE_ASSETS)' EXIT; \
cd backend && CGO_ENABLED=0 GOOS=$(OS) GOARCH=$(ARCH) go build -o ../remark42 -ldflags "-X main.revision=$(GITHUB_REF)-$(GITHUB_SHA) -s -w" ./app
docker:
DOCKER_BUILDKIT=1 docker build -t umputun/remark42 --build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) \
DOCKER_BUILDKIT=1 docker build -t umputun/remark42 -t ghcr.io/umputun/remark42 --build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) \
--build-arg CI=true --build-arg SKIP_FRONTEND_TEST=true --build-arg SKIP_BACKEND_TEST=true .
dockerx:
docker buildx build --build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) --build-arg CI=true \
--build-arg SKIP_FRONTEND_TEST=true --build-arg SKIP_BACKEND_TEST=true \
--progress=plain --platform linux/amd64,linux/arm/v7,linux/arm64 \
--progress=plain --platform linux/amd64,linux/arm64 \
-t ghcr.io/umputun/remark42:master -t umputun/remark42:master .
release:
docker build -f Dockerfile.artifacts --no-cache --pull --build-arg CI=true \
--build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) -t remark42.bin .
- @docker rm -f remark42.bin 2>/dev/null || exit 0
- @mkdir -p bin
docker run -d --name=remark42.bin remark42.bin
docker cp remark42.bin:/artifacts/remark42.linux-amd64.tar.gz bin/remark42.linux-amd64.tar.gz
docker cp remark42.bin:/artifacts/remark42.linux-386.tar.gz bin/remark42.linux-386.tar.gz
docker cp remark42.bin:/artifacts/remark42.linux-arm64.tar.gz bin/remark42.linux-arm64.tar.gz
docker cp remark42.bin:/artifacts/remark42.darwin-amd64.tar.gz bin/remark42.darwin-amd64.tar.gz
docker cp remark42.bin:/artifacts/remark42.darwin-arm64.tar.gz bin/remark42.darwin-arm64.tar.gz
docker cp remark42.bin:/artifacts/remark42.freebsd-amd64.tar.gz bin/remark42.freebsd-amd64.tar.gz
docker cp remark42.bin:/artifacts/remark42.windows-amd64.zip bin/remark42.windows-amd64.zip
docker rm -f remark42.bin
@set -e; \
trap '$(CLEANUP_RELEASE_ASSETS)' EXIT; \
goreleaser release --snapshot --clean --skip=publish
race_test:
cd backend/app && go test -race -timeout=60s -count 1 ./...
cd backend/app && go test -race -timeout=300s -count 1 ./...
backend:
docker-compose -f compose-dev-backend.yml build
docker compose -f compose-dev-backend.yml build
frontend:
docker-compose -f compose-dev-frontend.yml build
docker compose -f compose-dev-frontend.yml build
rundev:
SKIP_BACKEND_TEST=true SKIP_FRONTEND_TEST=true GITHUB_REF=$(GITHUB_REF) GITHUB_SHA=$(GITHUB_SHA) CI=true \
docker-compose -f compose-private.yml build
docker-compose -f compose-private.yml up
docker compose -f compose-private.yml build
docker compose -f compose-private.yml up
# stamped the same way the suite stamps a stack it starts itself, so one brought up here is
# accepted instead of rejected as belonging to another checkout
e2e-up:
./e2e/tls/generate.sh
E2E_STAMP=$$(./e2e/stamp.sh) docker compose -f compose-e2e-test.yml up -d --build --quiet-pull --wait
e2e-down:
docker compose -f compose-e2e-test.yml down -v
# the suite brings the stack up itself when it finds none, so e2e-up is only worth running
# to keep the containers between invocations
e2e:
docker compose -f compose-e2e-test.yml up --build --quiet-pull --exit-code-from tests
cd e2e && go test -tags=e2e -count 1 -timeout 20m ./...
.PHONY: bin backend
e2e-ui:
cd e2e && E2E_HEADLESS=false E2E_KEEP=1 go test -tags=e2e -count 1 -v -timeout 20m ./...
.PHONY: bin docker dockerx release race_test backend frontend rundev e2e e2e-up e2e-down e2e-ui
+1 -1
View File
@@ -2,7 +2,7 @@
Remark42 is a self-hosted, lightweight and simple (yet functional) comment engine, which doesn't spy on users. It can be embedded into blogs, articles, or any other place where readers add comments.
* Social login via Google, Twitter, Facebook, Microsoft, GitHub, Yandex, Patreon and Telegram
* Social login via Google, Facebook, Microsoft, GitHub, Apple, Yandex, Patreon, Discord, Telegram and custom OAuth2 providers
* Login via email
* Optional anonymous access
* Multi-level nested comments with both tree and plain presentations
+1 -1
View File
@@ -12,4 +12,4 @@ We release patches for security vulnerabilities.
## Reporting a Vulnerability
Please report (suspected) security vulnerabilities to umputun@gmail.com. You will receive a response from us within 48 hours. If the issue is confirmed, we will release a patch as soon as possible depending on complexity but historically within a few days.
Please report (suspected) security vulnerabilities either by using GitHub's [private vulnerability reporting](https://github.com/umputun/remark42/security/advisories/new) (click the "Report a vulnerability" button on the [Security tab](https://github.com/umputun/remark42/security)) or by emailing umputun@gmail.com. You will receive a response within 48 hours. If the issue is confirmed, we will release a patch as soon as possible depending on complexity but historically within a few days.
+63 -61
View File
@@ -1,68 +1,70 @@
run:
timeout: 5m
output:
format: tab
skip-dirs:
- vendor
linters-settings:
govet:
check-shadowing: true
maligned:
suggest-new: true
goconst:
min-len: 2
min-occurrences: 2
misspell:
locale: US
lll:
line-length: 140
gocritic:
enabled-tags:
- performance
- style
- experimental
disabled-checks:
- wrapperFunc
# TODO: feel free to remove these excludes and fix the code
- hugeParam
- rangeValCopy
version: "2"
linters:
default: none
enable:
- bodyclose
- megacheck
- revive
- govet
- unconvert
- gas
- gocyclo
- copyloopvar
- dupl
- gochecknoinits
- gocritic
- gocyclo
- gosec
- govet
- ineffassign
- misspell
- nakedret
- prealloc
- revive
- staticcheck
- unconvert
- unparam
- unused
- typecheck
- ineffassign
- stylecheck
- gochecknoinits
- exportloopref
- gocritic
- nakedret
- gosimple
- prealloc
fast: false
disable-all: true
issues:
exclude-rules:
- text: "at least one file in a package should have a package comment"
linters:
- stylecheck
- text: "package-comments: should have a package comment"
linters:
- revive
- path: _test\.go
linters:
- gosec
- dupl
exclude-use-default: false
settings:
gosec:
excludes:
- G117 # false positive: struct field name matches "secret" pattern
gocritic:
disabled-checks:
- wrapperFunc
- hugeParam
- rangeValCopy
enabled-tags:
- performance
- style
- experimental
govet:
enable:
- shadow
misspell:
locale: US
exclusions:
generated: lax
rules:
- linters:
- staticcheck
text: at least one file in a package should have a package comment
- linters:
- revive
text: 'package-comments: should have a package comment'
- linters:
- revive
text: 'var-naming: avoid meaningless package names'
- linters:
- revive
text: 'var-naming: avoid package names that conflict with Go standard library package names'
- linters:
- dupl
- gosec
path: _test\.go
paths:
- vendor
- third_party$
- builtin$
- examples$
formatters:
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
+1 -1
View File
@@ -1 +1 @@
../site/src/docs/contributing/backend/index.md
../site/content/docs/contributing/backend/index.md
+3 -3
View File
@@ -1,11 +1,11 @@
FROM umputun/baseimage:buildgo-v1.9.2 as build-backend
FROM umputun/baseimage:buildgo-v1.17.0 AS build-backend
ADD backend /build/backend
WORKDIR /build/backend/_example/memory_store
RUN go build -o /build/bin/memory_store -ldflags "-X main.revision=0.0.0 -s -w"
FROM umputun/baseimage:app-v1.9.2
FROM umputun/baseimage:app-v1.17.0
ARG GITHUB_SHA
@@ -13,7 +13,7 @@ LABEL org.opencontainers.image.authors="Umputun <umputun@gmail.com>" \
org.opencontainers.image.description="Remark42 comment engine example JRPC memory store" \
org.opencontainers.image.documentation="https://github.com/umputun/remark42/tree/master/backend/_example/memory_store" \
org.opencontainers.image.licenses="MIT" \
org.opencontainers.image.source="https://github.com/umputun/remark42.git" \
org.opencontainers.image.source="https://github.com/umputun/remark42" \
org.opencontainers.image.title="Remark42 JRPC example memory store" \
org.opencontainers.image.url="https://remark42.com/" \
org.opencontainers.image.revision="${GITHUB_SHA}"
+3 -3
View File
@@ -4,9 +4,9 @@
In order to run remark42 with memory_store copy provided `compose-dev-memstore.yml` to the root directory and run:
1. `docker-compose -f compose-dev-memstore.yml build`
1. `docker-compose -f compose-dev-memstore.yml up`
1. `docker compose -f compose-dev-memstore.yml build`
1. `docker compose -f compose-dev-memstore.yml up`
As usual, demo site will run on http://127.0.0.1:8080/web/
note: in order to work with the latest (current) version of master `go.mod` uses replacement directive for the backend package. In real-life usage `replace github.com/umputun/remark42/backend => ../../` should not be used.
note: in order to work with the latest (current) version of master `go.mod` uses replacement directive for the backend package. In real-life usage `replace github.com/umputun/remark42/backend => ../../` should not be used.
+25 -27
View File
@@ -251,11 +251,11 @@ func (m *MemData) Flag(req engine.FlagRequest) (val bool, err error) {
// ListFlags get list of flagged keys, like blocked & verified user
// works for full locator (post flags) or with userID
func (m *MemData) ListFlags(req engine.FlagRequest) (res []interface{}, err error) {
func (m *MemData) ListFlags(req engine.FlagRequest) (res []any, err error) {
m.mu.RLock()
defer m.mu.RUnlock()
res = []interface{}{}
res = []any{}
switch req.Flag {
case engine.Verified:
@@ -293,17 +293,17 @@ func (m *MemData) UserDetail(req engine.UserDetailRequest) ([]engine.UserDetailE
defer m.mu.Unlock()
if req.Update == "" { // read detail value, no update requested
return m.getUserDetail(req)
return m.getUserDetail(req), nil
}
return m.setUserDetail(req)
return m.setUserDetail(req), nil
case engine.AllUserDetails:
// list of all details returned in case request is a read request
// (Update is not set) and does not have UserID or Detail set
if req.Update == "" && req.UserID == "" { // read list of all details
m.mu.Lock()
defer m.mu.Unlock()
return m.listDetails(req.Locator)
return m.listDetails(req.Locator), nil
}
return nil, fmt.Errorf("unsupported request with userdetail all")
default:
@@ -319,7 +319,8 @@ func (m *MemData) Delete(req engine.DeleteRequest) error {
switch {
case req.UserDetail != "": // delete user detail
return m.deleteUserDetail(req.Locator, req.UserID, req.UserDetail)
m.deleteUserDetail(req.Locator, req.UserID, req.UserDetail)
return nil
case req.Locator.URL != "" && req.CommentID != "" && req.UserDetail == "": // delete comment
return m.deleteComment(req.Locator, req.CommentID, req.DeleteMode)
@@ -332,7 +333,8 @@ func (m *MemData) Delete(req engine.DeleteRequest) error {
return e
}
}
return m.deleteUserDetail(req.Locator, req.UserID, engine.AllUserDetails)
m.deleteUserDetail(req.Locator, req.UserID, engine.AllUserDetails)
return nil
case req.Locator.SiteID != "" && req.Locator.URL == "" && req.CommentID == "" && req.UserID == "" && req.UserDetail == "": // delete site
if _, ok := m.posts[req.Locator.SiteID]; !ok {
@@ -389,10 +391,7 @@ func (m *MemData) checkFlag(req engine.FlagRequest) (val bool) {
func (m *MemData) setFlag(req engine.FlagRequest) (res bool, err error) {
status := false
if req.Update == engine.FlagTrue {
status = true
}
status := req.Update == engine.FlagTrue
switch req.Flag {
@@ -437,29 +436,29 @@ func (m *MemData) setFlag(req engine.FlagRequest) (res bool, err error) {
// getUserDetail returns UserDetailEntry with requested userDetail (omitting other details)
// as an only element of the slice.
func (m *MemData) getUserDetail(req engine.UserDetailRequest) ([]engine.UserDetailEntry, error) {
func (m *MemData) getUserDetail(req engine.UserDetailRequest) []engine.UserDetailEntry {
if meta, ok := m.metaUsers[req.UserID]; ok {
if meta.SiteID != req.Locator.SiteID {
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
switch req.Detail {
case engine.UserEmail:
return []engine.UserDetailEntry{{UserID: req.UserID, Email: meta.Details.Email}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Email: meta.Details.Email}}
case engine.UserTelegram:
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: meta.Details.Telegram}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: meta.Details.Telegram}}
}
}
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
// setUserDetail sets requested userDetail, returning complete updated UserDetailEntry as an onlyIps
// element of the slice in case of success
func (m *MemData) setUserDetail(req engine.UserDetailRequest) ([]engine.UserDetailEntry, error) {
func (m *MemData) setUserDetail(req engine.UserDetailRequest) []engine.UserDetailEntry {
var entry metaUser
if meta, ok := m.metaUsers[req.UserID]; ok {
if meta.SiteID != req.Locator.SiteID {
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
entry = meta
}
@@ -476,42 +475,42 @@ func (m *MemData) setUserDetail(req engine.UserDetailRequest) ([]engine.UserDeta
case engine.UserEmail:
entry.Details.Email = req.Update
m.metaUsers[req.UserID] = entry
return []engine.UserDetailEntry{{UserID: req.UserID, Email: req.Update}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Email: req.Update}}
case engine.UserTelegram:
entry.Details.Telegram = req.Update
m.metaUsers[req.UserID] = entry
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: req.Update}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: req.Update}}
}
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
// listDetails lists all available users details for given siteID
func (m *MemData) listDetails(loc store.Locator) ([]engine.UserDetailEntry, error) {
func (m *MemData) listDetails(loc store.Locator) []engine.UserDetailEntry {
var res []engine.UserDetailEntry
for _, u := range m.metaUsers {
if u.SiteID == loc.SiteID {
res = append(res, u.Details)
}
}
return res, nil
return res
}
// deleteUserDetail deletes requested UserDetail or whole UserDetailEntry,
// deletion of the absent entry doesn't produce error.
// Trying to delete user with wrong siteID doesn't to anything and doesn't produce error.
func (m *MemData) deleteUserDetail(locator store.Locator, userID string, userDetail engine.UserDetail) error {
func (m *MemData) deleteUserDetail(locator store.Locator, userID string, userDetail engine.UserDetail) {
var entry metaUser
if meta, ok := m.metaUsers[userID]; ok {
if meta.SiteID != locator.SiteID {
return nil
return
}
entry = meta
}
if entry == (metaUser{}) || entry.Details == (engine.UserDetailEntry{}) {
// absent entry means that we should not do anything
return nil
return
}
switch userDetail {
@@ -529,7 +528,6 @@ func (m *MemData) deleteUserDetail(locator store.Locator, userID string, userDet
}
m.metaUsers[userID] = entry
return nil
}
func (m *MemData) get(loc store.Locator, commentID string) (store.Comment, error) {
@@ -10,6 +10,7 @@ import (
"fmt"
"sort"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -198,7 +199,7 @@ func TestMemData_FindForUserPagination(t *testing.T) {
}
// write 200 comments
for i := 0; i < 200; i++ {
for i := range 200 {
c.ID = fmt.Sprintf("idd-%d", i)
c.Text = fmt.Sprintf("text #%d", i)
c.Timestamp = time.Date(2017, 12, 20, 15, 18, i, 0, time.Local)
@@ -286,7 +287,7 @@ func TestMemData_CountUser(t *testing.T) {
func TestMemData_InfoPost(t *testing.T) {
b := prepMem(t)
ts := func(min int) time.Time { return time.Date(2017, 12, 20, 15, 18, min, 0, time.Local).In(time.UTC) }
ts := func(minute int) time.Time { return time.Date(2017, 12, 20, 15, 18, minute, 0, time.Local).In(time.UTC) }
// add one more for https://radio-t.com/2
comment := store.Comment{
@@ -484,7 +485,7 @@ func TestMemData_FlagVerified(t *testing.T) {
func TestMemData_FlagListVerified(t *testing.T) {
b := prepMem(t)
toIDs := func(inp []interface{}) (res []string) {
toIDs := func(inp []any) (res []string) {
res = make([]string, len(inp))
for i, v := range inp {
vv, ok := v.(string)
@@ -521,51 +522,52 @@ func TestMemData_FlagListVerified(t *testing.T) {
}
func TestMemData_FlagListBlocked(t *testing.T) {
b := prepMem(t)
setBlocked := func(site, user string, status engine.FlagStatus, ttl time.Duration) error {
req := engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: site}, UserID: user, Update: status,
TTL: ttl}
_, err := b.Flag(req)
return err
}
toBlocked := func(inp []interface{}) (res []store.BlockedUser) {
res = make([]store.BlockedUser, len(inp))
for i, v := range inp {
vv, ok := v.(store.BlockedUser)
require.True(t, ok)
res[i] = vv
synctest.Test(t, func(t *testing.T) {
b := prepMem(t)
setBlocked := func(site, user string, status engine.FlagStatus, ttl time.Duration) error {
req := engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: site}, UserID: user, Update: status,
TTL: ttl}
_, err := b.Flag(req)
return err
}
return res
}
assert.NoError(t, setBlocked("radio-t", "user1", engine.FlagTrue, 0))
assert.NoError(t, setBlocked("radio-t", "user2", engine.FlagTrue, 50*time.Millisecond))
assert.NoError(t, setBlocked("radio-t", "user3", engine.FlagFalse, 0))
vv, err := b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
toBlocked := func(inp []any) (res []store.BlockedUser) {
res = make([]store.BlockedUser, len(inp))
for i, v := range inp {
vv, ok := v.(store.BlockedUser)
require.True(t, ok)
res[i] = vv
}
return res
}
assert.NoError(t, setBlocked("radio-t", "user1", engine.FlagTrue, 0))
assert.NoError(t, setBlocked("radio-t", "user2", engine.FlagTrue, 50*time.Millisecond))
assert.NoError(t, setBlocked("radio-t", "user3", engine.FlagFalse, 0))
blockedList := toBlocked(vv)
var blockedIds = make([]string, len(blockedList))
for i, x := range blockedList {
blockedIds[i] = x.ID
}
require.Equal(t, 2, len(blockedList), b.metaUsers)
assert.ElementsMatch(t, []string{"user1", "user2"}, blockedIds)
t.Logf("%+v", blockedList)
vv, err := b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
// check block expiration
time.Sleep(50 * time.Millisecond)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
blockedList = toBlocked(vv)
require.Equal(t, 1, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
blockedList := toBlocked(vv)
var blockedIDs = make([]string, len(blockedList))
for i, x := range blockedList {
blockedIDs[i] = x.ID
}
require.Equal(t, 2, len(blockedList), b.metaUsers)
assert.ElementsMatch(t, []string{"user1", "user2"}, blockedIDs)
t.Logf("%+v", blockedList)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "bad"}})
assert.NoError(t, err)
assert.Equal(t, 0, len(vv))
// check block expiration
time.Sleep(50 * time.Millisecond)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
blockedList = toBlocked(vv)
require.Equal(t, 1, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "bad"}})
assert.NoError(t, err)
assert.Equal(t, 0, len(vv))
})
}
func TestMemData_DeleteComment(t *testing.T) {
@@ -70,6 +70,17 @@ func (m *MemImage) Load(id string) ([]byte, error) {
return img, nil
}
// Delete image by ID
func (m *MemImage) Delete(id string) error {
m.mu.Lock()
// delete key from permanent and staging storage
delete(m.images, id)
delete(m.insertTime, id)
delete(m.imagesStaging, id)
m.mu.Unlock()
return nil
}
// Commit moves image from staging to permanent
func (m *MemImage) Commit(id string) error {
m.mu.RLock()
@@ -18,7 +18,7 @@ import (
)
// gopher png for test, from https://golang.org/src/image/png/example_test.go
const gopher = "iVBORw0KGgoAAAANSUhEUgAAAEsAAAA8CAAAAAALAhhPAAAFfUlEQVRYw62XeWwUVRzHf2" +
const rawGopher = "iVBORw0KGgoAAAANSUhEUgAAAEsAAAA8CAAAAAALAhhPAAAFfUlEQVRYw62XeWwUVRzHf2" +
"+OPbo9d7tsWyiyaZti6eWGAhISoIGKECEKCAiJJkYTiUgTMYSIosYYBBIUIxoSPIINEBDi2VhwkQrVsj1ESgu9doHWdrul7ba" +
"73WNm3vOPtsseM9MdwvvrzTs+8/t95ze/33sI5BqiabU6m9En8oNjduLnAEDLUsQXFF8tQ5oxK3vmnNmDSMtrncks9Hhtt" +
"/qeWZapHb1ha3UqYSWVl2ZmpWgaXMXGohQAvmeop3bjTRtv6SgaK/Pb9/bFzUrYslbFAmHPp+3WhAYdr+7GN/YnpN46Opv55VDs" +
@@ -38,7 +38,9 @@ const gopher = "iVBORw0KGgoAAAANSUhEUgAAAEsAAAA8CAAAAAALAhhPAAAFfUlEQVRYw62XeWwU
"1y98c3D27eppUjsZ6fql3jcd5rUe7+ZIlLNQny3Rd+E5Tct3WVhTM5RBCEdiEK0b6B+/ca2gYU393nFj/n1AygRQxPIUA043M42u85+z2S" +
"nssKrPl8Mx76NL3E6eXc3be7OD+H4WHbJkKI8AU8irbITQjZ+0hQcPEgId/Fn/pl9crKH02+5o2b9T/eMx7pKoskYgAAAABJRU5ErkJggg=="
func gopherPNG() io.Reader { return base64.NewDecoder(base64.StdEncoding, strings.NewReader(gopher)) }
func gopherPNG() io.Reader {
return base64.NewDecoder(base64.StdEncoding, strings.NewReader(rawGopher))
}
func TestMemImage_LoadAfterSave(t *testing.T) {
svc := NewMemImageStore()
@@ -57,7 +59,8 @@ func TestMemImage_LoadAfterSave(t *testing.T) {
assert.NoError(t, err)
assert.Equal(t, gopher, img)
svc.ResetCleanupTimer(id)
err = svc.ResetCleanupTimer(id)
assert.NoError(t, err)
err = svc.Commit(id)
assert.NoError(t, err)
@@ -70,6 +73,26 @@ func TestMemImage_LoadAfterSave(t *testing.T) {
assert.Equal(t, gopher, img)
}
func TestMemImage_LoadAfterDelete(t *testing.T) {
svc := NewMemImageStore()
gopher, err := io.ReadAll(gopherPNG())
assert.NoError(t, err)
id := "test_img"
err = svc.Save(id, gopher)
assert.NoError(t, err)
err = svc.Delete(id)
assert.NoError(t, err)
img, err := svc.Load(id)
assert.EqualError(t, err, "image test_img not found")
assert.Empty(t, img)
err = svc.ResetCleanupTimer(id)
assert.EqualError(t, err, "image test_img not found")
}
func TestMemImage_CommitFail(t *testing.T) {
svc := NewMemImageStore()
err := svc.Commit("test_id")
@@ -11,7 +11,7 @@ services:
args:
- SKIP_BACKEND_TEST=true
- SKIP_FRONTEND_TEST=true
image: umputun/remark42:dev
image: ghcr.io/umputun/remark42:dev
container_name: "remark42-dev"
hostname: "remark42-dev"
restart: always
+20 -28
View File
@@ -1,42 +1,34 @@
module github.com/umputun/remark42/memory_store
go 1.20
go 1.25.0
require (
github.com/go-pkgz/jrpc v0.3.0
github.com/go-pkgz/lgr v0.11.0
github.com/jessevdk/go-flags v1.5.0
github.com/stretchr/testify v1.8.4
github.com/go-pkgz/jrpc v0.4.2
github.com/go-pkgz/lgr v0.12.4
github.com/jessevdk/go-flags v1.6.1
github.com/stretchr/testify v1.12.1
github.com/umputun/remark42/backend v1.1000.0
)
require (
github.com/Depado/bfchroma/v2 v2.0.0 // indirect
github.com/PuerkitoBio/goquery v1.8.1 // indirect
github.com/ajg/form v1.5.1 // indirect
github.com/alecthomas/chroma/v2 v2.8.0 // indirect
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/alecthomas/chroma/v2 v2.27.0 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/didip/tollbooth/v7 v7.0.1 // indirect
github.com/didip/tollbooth_chi v0.0.0-20220719025231-d662a7f6928f // indirect
github.com/dlclark/regexp2 v1.10.0 // indirect
github.com/go-chi/chi/v5 v5.0.10 // indirect
github.com/go-chi/render v1.0.3 // indirect
github.com/go-pkgz/expirable-cache v1.0.0 // indirect
github.com/go-pkgz/rest v1.17.0 // indirect
github.com/gorilla/css v1.0.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/microcosm-cc/bluemonday v1.0.25 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rs/xid v1.5.0 // indirect
github.com/dlclark/regexp2/v2 v2.7.1 // indirect
github.com/go-pkgz/rest v1.24.0 // indirect
github.com/go-pkgz/routegroup v1.6.1 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/microcosm-cc/bluemonday v1.0.27 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
go.etcd.io/bbolt v1.3.7 // indirect
golang.org/x/image v0.11.0 // indirect
golang.org/x/net v0.14.0 // indirect
golang.org/x/sys v0.11.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
go.etcd.io/bbolt v1.5.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/image v0.45.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/sys v0.47.0 // indirect
)
replace github.com/umputun/remark42/backend v1.1000.0 => ../../
+45 -119
View File
@@ -1,126 +1,52 @@
github.com/Depado/bfchroma/v2 v2.0.0 h1:IRpN9BPkNwEpR6w1ectIcNWOuhDSLx+8f1pn83fzxx8=
github.com/Depado/bfchroma/v2 v2.0.0/go.mod h1:wFwW/Pw8Tnd0irzgO9Zxtxgzp3aPS8qBWlyadxujxmw=
github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
github.com/ajg/form v1.5.1 h1:t9c7v8JUKu/XxOGBU0yjNpaMloxGEJhUkqFRq0ibGeU=
github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY=
github.com/alecthomas/assert/v2 v2.2.1 h1:XivOgYcduV98QCahG8T5XTezV5bylXe+lBxLG2K2ink=
github.com/alecthomas/chroma/v2 v2.8.0 h1:w9WJUjFFmHHB2e8mRpL9jjy3alYDlU0QLDezj1xE264=
github.com/alecthomas/chroma/v2 v2.8.0/go.mod h1:yrkMI9807G1ROx13fhe1v6PN2DDeaR73L3d+1nmYQtw=
github.com/alecthomas/repr v0.2.0 h1:HAzS41CIzNW5syS8Mf9UwXhNH1J9aix/BvDRf1Ml2Yk=
github.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA=
github.com/andybalholm/cascadia v1.3.2 h1:3Xi6Dw5lHF15JtdcmAHD3i1+T8plmv7BQ/nsViSLyss=
github.com/andybalholm/cascadia v1.3.2/go.mod h1:7gtRlve5FxPPgIgX36uWBX58OdBsSS6lUvCFb+h7KvU=
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/chroma/v2 v2.27.0 h1:FodwmyOBgJULFYmDqibcp9pvfDLWdtPRh9v/r5BXYZs=
github.com/alecthomas/chroma/v2 v2.27.0/go.mod h1:NjJ3ciIgrqBNeIkWZ4e46nseoLDslxU1LmfCoL+wcY8=
github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/didip/tollbooth/v7 v7.0.0/go.mod h1:VZhDSGl5bDSPj4wPsih3PFa4Uh9Ghv8hgacaTm5PRT4=
github.com/didip/tollbooth/v7 v7.0.1 h1:TkT4sBKoQoHQFPf7blQ54iHrZiTDnr8TceU+MulVAog=
github.com/didip/tollbooth/v7 v7.0.1/go.mod h1:VZhDSGl5bDSPj4wPsih3PFa4Uh9Ghv8hgacaTm5PRT4=
github.com/didip/tollbooth_chi v0.0.0-20220719025231-d662a7f6928f h1:jtKwihcLmUC9BAhoJ9adCUqdSSZcOdH2KL7mPTUm2aw=
github.com/didip/tollbooth_chi v0.0.0-20220719025231-d662a7f6928f/go.mod h1:q9C80dnsuVRP2dAskjnXRNWdUJqtGgwG9wNrzt0019s=
github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0=
github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/go-chi/chi/v5 v5.0.7/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
github.com/go-chi/chi/v5 v5.0.10 h1:rLz5avzKpjqxrYwXNfmjkrYYXOyLJd37pz53UFHC6vk=
github.com/go-chi/chi/v5 v5.0.10/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
github.com/go-chi/render v1.0.1/go.mod h1:pq4Rr7HbnsdaeHagklXub+p6Wd16Af5l9koip1OvJns=
github.com/go-chi/render v1.0.3 h1:AsXqd2a1/INaIfUSKq3G5uA8weYx20FOsM7uSoCyyt4=
github.com/go-chi/render v1.0.3/go.mod h1:/gr3hVkmYR0YlEy3LxCuVRFzEu9Ruok+gFqbIofjao0=
github.com/go-pkgz/expirable-cache v0.1.0/go.mod h1:GTrEl0X+q0mPNqN6dtcQXksACnzCBQ5k/k1SwXJsZKs=
github.com/go-pkgz/expirable-cache v1.0.0 h1:ns5+1hjY8hntGv8bPaQd9Gr7Jyo+Uw5SLyII40aQdtA=
github.com/go-pkgz/expirable-cache v1.0.0/go.mod h1:GTrEl0X+q0mPNqN6dtcQXksACnzCBQ5k/k1SwXJsZKs=
github.com/go-pkgz/jrpc v0.3.0 h1:Fls38KqPsHzvp0FWfivr6cGnncC+iFBodHBqvUPY+0U=
github.com/go-pkgz/jrpc v0.3.0/go.mod h1:MFtKs75JESiSqVicsQkgN2iDFFuCd3gVT1/vKiwRi00=
github.com/go-pkgz/lgr v0.11.0 h1:9XH5o+vj09L0sRWEswIGK1lJ6g07xVB4/Z28RV9Z+qM=
github.com/go-pkgz/lgr v0.11.0/go.mod h1:4rdRmMSs4yGFjnUg0rSDbKx21LmFNZoH4y8OLl3qDnU=
github.com/go-pkgz/rest v1.15.6/go.mod h1:KUWAqbDteYGS/CiXftomQsKjtEOifXsJ36Ka0skYbmk=
github.com/go-pkgz/rest v1.17.0 h1:LoBI/lDBMuqwWhOOkc6thM9NnwJO+K9nWvCOjZ7BAgE=
github.com/go-pkgz/rest v1.17.0/go.mod h1:HHlLOt02NJc2sgffXBF6hYVMcRo4Gz3vjg43zTzN7VM=
github.com/gorilla/css v1.0.0 h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/dlclark/regexp2/v2 v2.7.1 h1:yqDtwI1ptXXvEUNpYTk2lad4jLtAcKqkzepn4savSk4=
github.com/dlclark/regexp2/v2 v2.7.1/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU=
github.com/go-pkgz/jrpc v0.4.2 h1:gY5mmxp9/dFd1WsHybVZILQpF11YNWWS3Ga+Pc5aIAU=
github.com/go-pkgz/jrpc v0.4.2/go.mod h1:ZtnMpIXYmwXh6W44XO2lE5Lh5J+6KeeMIvw+vF9xXRQ=
github.com/go-pkgz/lgr v0.12.4 h1:lDeQ4BR28ldXrKau6BOjq7A8nHzcXz+MF4xUfV4l1Ok=
github.com/go-pkgz/lgr v0.12.4/go.mod h1:Lw6DkNRnCPyX07mqkiUK/p+eA1opq4GKkWfWia64RA8=
github.com/go-pkgz/rest v1.24.0 h1:GAUCgx7U8xCOC2OynLjhCRMhtnMQH4d1mTdKpQyX2yI=
github.com/go-pkgz/rest v1.24.0/go.mod h1:dl3EWiuFB4hRTo2Sknj6UrQGFRAYvANK6/NyW8qQPxc=
github.com/go-pkgz/routegroup v1.6.1 h1:6I/0LabazpZsHAI+jYPeyH/KU2cvZF0bFylUScMNi+Q=
github.com/go-pkgz/routegroup v1.6.1/go.mod h1:Pmu04fhgWhRtBMIJ8HXppnnzOPjnL/IEPBIdO2zmeqg=
github.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=
github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/jessevdk/go-flags v1.5.0 h1:1jKYvbxEjfUl0fmqTCOfonvskHHXMjBySTLW4y9LFvc=
github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/microcosm-cc/bluemonday v1.0.25 h1:4NEwSfiJ+Wva0VxN5B8OwMicaJvD8r9tlJWm9rtloEg=
github.com/microcosm-cc/bluemonday v1.0.25/go.mod h1:ZIOjCQp1OrzBBPIJmfX4qDYFuhU02nx4bn030ixfHLE=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/jessevdk/go-flags v1.6.1 h1:Cvu5U8UGrLay1rZfv/zP7iLpSHGUZ/Ou68T0iX1bBK4=
github.com/jessevdk/go-flags v1.6.1/go.mod h1:Mk8T1hIAWpOiJiHa9rJASDK2UGWji0EuPGBnNLMooyc=
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
github.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.etcd.io/bbolt v1.3.7 h1:j+zJOnnEjF/kyHlDDgGnVL/AIqIJPq8UoB2GSNfkUfQ=
go.etcd.io/bbolt v1.3.7/go.mod h1:N9Mkw9X8x5fupy0IKsmuqVtoGDyxsaDlbk4Rd05IAQw=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/image v0.11.0 h1:ds2RoQvBvYTiJkwpSFDwCcDFNX7DqjL2WsUgTNk0Ooo=
golang.org/x/image v0.11.0/go.mod h1:bglhjqbqVuEb9e9+eNR45Jfu7D+T4Qan+NhQk8Ck2P8=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns=
golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14=
golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/image v0.45.0 h1:FMb1nTbH5H9vF55SriQHgFw5GnNL9Jg6L25BwXKzhB0=
golang.org/x/image v0.45.0/go.mod h1:n62x/7RqlwXDvGsSU4u6IUTUf6KghUZ9Bt7cG/T9Fx4=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
@@ -73,7 +73,7 @@ func (s *RPC) admEnabledHndl(id uint64, params json.RawMessage) (rr jrpc.Respons
// onEvent returns nothing, callback to OnEvent
func (s *RPC) admEventHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
var siteID string
var ps []interface{}
var ps []any
if err := json.Unmarshal(params, &ps); err != nil {
return jrpc.Response{Error: err.Error()}
}
@@ -217,7 +217,7 @@ func TestRPC_listFlagsHndl(t *testing.T) {
flags, err = re.ListFlags(verifyFlagReq)
require.NoError(t, err)
assert.Equal(t, []interface{}{"u1"}, flags)
assert.Equal(t, []any{"u1"}, flags)
verifiedUsers := make([]string, 0, len(flags))
for _, v := range flags {
verifiedUsers = append(verifiedUsers, v.(string))
+10 -1
View File
@@ -35,7 +35,6 @@ func (s *RPC) imgResetClnTimerHndl(id uint64, params json.RawMessage) (rr jrpc.R
}
err := s.img.ResetCleanupTimer(fileID)
return jrpc.EncodeResponse(id, nil, err)
}
func (s *RPC) imgLoadHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
@@ -47,6 +46,16 @@ func (s *RPC) imgLoadHndl(id uint64, params json.RawMessage) (rr jrpc.Response)
return jrpc.EncodeResponse(id, value, err)
}
func (s *RPC) imgDeleteHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
var fileID string
if err := json.Unmarshal(params, &fileID); err != nil {
return jrpc.Response{Error: err.Error()}
}
err := s.img.Delete(fileID)
return jrpc.EncodeResponse(id, nil, err)
}
func (s *RPC) imgCommitHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
var fileID string
if err := json.Unmarshal(params, &fileID); err != nil {
@@ -115,14 +115,17 @@ func TestRPC_imgCleanupHndl(t *testing.T) {
assert.Equal(t, 1462, len(img))
assert.Equal(t, gopherPNGBytes(), img)
// wait for image to expire
time.Sleep(time.Millisecond * 50)
// reset the time to cleanup
// age the image past the ttl used below, so the reset that follows is what keeps it on
// staging rather than the image simply being young
const stagingTTL = 500 * time.Millisecond
time.Sleep(stagingTTL + 100*time.Millisecond)
// reset the time to cleanup, which leaves a full ttl before it could be collected again
err = ri.ResetCleanupTimer(id)
assert.NoError(t, err)
// cleanup, should not affect the new image
err = ri.Cleanup(context.TODO(), time.Millisecond*45)
err = ri.Cleanup(context.TODO(), stagingTTL)
assert.NoError(t, err)
// load after cleanup should succeed
@@ -158,4 +161,9 @@ func TestRPC_imgInfoHndl(t *testing.T) {
info, err = ri.Info()
assert.NoError(t, err)
assert.False(t, info.FirstStagingImageTS.IsZero())
err = ri.Delete("test_img")
assert.NoError(t, err)
_, err = ri.Load("test_img")
assert.EqualError(t, err, "image test_img not found")
}
@@ -60,6 +60,7 @@ func (s *RPC) addHandlers() {
"save_with_id": s.imgSaveWithIDHndl,
"reset_cleanup_timer": s.imgResetClnTimerHndl,
"load": s.imgLoadHndl,
"delete": s.imgDeleteHndl,
"commit": s.imgCommitHndl,
"cleanup": s.imgCleanupHndl,
"info": s.imgInfoHndl,
@@ -8,7 +8,6 @@ package server
import (
"fmt"
"math/rand"
"net"
"net/http"
"testing"
@@ -20,27 +19,31 @@ import (
"github.com/umputun/remark42/memory_store/accessor"
)
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
break
}
}
// chooseUnusedPort asks the kernel for a free port from the ephemeral range, which makes a
// collision between concurrently running package test binaries very unlikely
func chooseUnusedPort(t *testing.T) int {
t.Helper()
ln, err := net.Listen("tcp", ":0")
require.NoError(t, err, "no free port available")
port := ln.Addr().(*net.TCPAddr).Port
require.NoError(t, ln.Close())
return port
}
func waitForHTTPServerStart(port int) {
// wait for up to 3 seconds for server to start before returning it
// waitForHTTPServerStart blocks until the server on port answers, failing the test naming the
// port if it never does
func waitForHTTPServerStart(t *testing.T, port int) {
t.Helper()
client := http.Client{Timeout: time.Second}
for i := 0; i < 300; i++ {
time.Sleep(time.Millisecond * 10)
if resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port)); err == nil {
_ = resp.Body.Close()
return
defer client.CloseIdleConnections()
require.Eventually(t, func() bool {
resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port))
if err != nil {
return false
}
}
_ = resp.Body.Close()
return true
}, 30*time.Second, 10*time.Millisecond, "http server on port %d didn't start", port)
}
func prepTestStore(t *testing.T) (port int, teardown func()) {
@@ -61,14 +64,17 @@ func prepTestStore(t *testing.T) (port int, teardown func()) {
admRecDisabled.Enabled = false
adm.Set("test-site-disabled", admRecDisabled)
port = chooseRandomUnusedPort()
port = chooseUnusedPort(t)
go func() {
_ = s.Run(port)
}()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
return port, func() {
// every test client here uses http.DefaultTransport, so their keep-alive connections
// sit in one shared pool; Shutdown waits on them and hits its own 5s deadline otherwise
http.DefaultTransport.(*http.Transport).CloseIdleConnections()
require.NoError(t, s.Shutdown())
}
}
+1 -1
View File
@@ -7,7 +7,7 @@ import (
log "github.com/go-pkgz/lgr"
bolt "go.etcd.io/bbolt"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/v2/avatar"
)
// AvatarCommand set of flags and command for avatar migration
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"os"
"testing"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/v2/avatar"
"github.com/jessevdk/go-flags"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
+1 -1
View File
@@ -47,7 +47,7 @@ func (ec *BackupCommand) Execute(_ []string) error {
req.SetBasicAuth("admin", ec.AdminPasswd)
// get with timeout
resp, err := client.Do(req.WithContext(ctx))
resp, err := client.Do(req.WithContext(ctx)) //nolint:gosec // exportURL is built from operator-supplied CLI flags, not user input
if err != nil {
return fmt.Errorf("request failed for %s: %w", exportURL, err)
}
+28
View File
@@ -1,10 +1,12 @@
package cmd
import (
"encoding/base64"
"fmt"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/jessevdk/go-flags"
@@ -16,6 +18,10 @@ func TestBackup_Execute(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/export")
assert.Equal(t, "GET", r.Method)
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:secret", string(auth))
fmt.Fprint(w, "blah\nblah2\n12345678\n")
}))
defer ts.Close()
@@ -34,6 +40,28 @@ func TestBackup_Execute(t *testing.T) {
assert.Equal(t, "blah\nblah2\n12345678\n", string(data))
}
func TestBackup_ExecuteNoPassword(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/export")
assert.Equal(t, "GET", r.Method)
t.Logf("Authorization: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
require.Equal(t, "admin:", string(auth))
w.WriteHeader(http.StatusUnauthorized)
fmt.Fprint(w, "Unauthorized")
}))
defer ts.Close()
cmd := BackupCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p := flags.NewParser(&cmd, flags.Default)
_, err := p.ParseArgs([]string{"--site=remark", "--path=/tmp", "--file={{.SITE}}-test.export"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
}
func TestBackup_ExecuteFailedStatus(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/export")
+4 -4
View File
@@ -77,7 +77,7 @@ func (cc *CleanupCommand) procSpam(comments []store.Comment) int {
log.Printf("[WARN] can't remove comment, %v", err)
}
}
comment.Text = strings.Replace(comment.Text, "\n", " ", -1)
comment.Text = strings.ReplaceAll(comment.Text, "\n", " ")
log.Printf("[SPAM] %+v [%.0f%%]", comment, score)
}
}
@@ -179,7 +179,7 @@ func (cc *CleanupCommand) listComments(postURL string) ([]store.Comment, error)
commentsWithInfo := struct {
Comments []store.Comment `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}{}
if err = json.NewDecoder(r.Body).Decode(&commentsWithInfo); err != nil {
@@ -199,7 +199,7 @@ func (cc *CleanupCommand) deleteComment(c store.Comment) error { //nolint:dupl /
client := http.Client{}
defer client.CloseIdleConnections()
r, err := client.Do(req)
r, err := client.Do(req) //nolint:gosec // RemarkURL comes from operator CLI flag, not user input
if err != nil {
return fmt.Errorf("delete request failed for comment %s, %s: %w", c.ID, c.Locator.URL, err)
}
@@ -221,7 +221,7 @@ func (cc *CleanupCommand) setTitle(c store.Comment) error { //nolint:dupl // not
client := http.Client{}
defer client.CloseIdleConnections()
r, err := client.Do(req)
r, err := client.Do(req) //nolint:gosec // RemarkURL comes from operator CLI flag, not user input
if err != nil {
return fmt.Errorf("title request failed for comment %s, %s: %w", c.ID, c.Locator.URL, err)
}
+8 -10
View File
@@ -9,7 +9,6 @@ import (
"testing"
"time"
"github.com/go-chi/chi/v5"
"github.com/jessevdk/go-flags"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -46,7 +45,6 @@ func TestCleanup_IsSpam(t *testing.T) {
}
for n, tt := range tbl {
tt := tt
checkName := fmt.Sprintf("check-%d-%s", n, tt.name)
t.Run(checkName, func(t *testing.T) {
c := store.Comment{ID: checkName, Text: tt.text, Score: tt.score}
@@ -59,7 +57,7 @@ func TestCleanup_IsSpam(t *testing.T) {
}
func TestCleanup_postsInRange(t *testing.T) {
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, nil)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -82,7 +80,7 @@ func TestCleanup_postsInRange(t *testing.T) {
}
func TestCleanup_listComments(t *testing.T) {
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, nil)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -108,7 +106,7 @@ func TestCleanup_listComments(t *testing.T) {
func TestCleanup_ExecuteSpam(t *testing.T) {
cleaned := cleanedComments{}
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, &cleaned)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -127,7 +125,7 @@ func TestCleanup_ExecuteSpam(t *testing.T) {
func TestCleanup_ExecuteTitle(t *testing.T) {
titledComments := cleanedComments{}
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, &titledComments)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -143,7 +141,7 @@ func TestCleanup_ExecuteTitle(t *testing.T) {
assert.Equal(t, []string{"/api/v1/admin/title/1", "/api/v1/admin/title/2", "/api/v1/admin/title/3", "/api/v1/admin/title/11"}, titledComments.ids)
}
func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
func cleanupRoutes(t *testing.T, r *http.ServeMux, c *cleanedComments) {
r.HandleFunc("/api/v1/list", func(w http.ResponseWriter, r *http.Request) {
require.Equal(t, "GET", r.Method)
require.Equal(t, "site=remark&limit=10000", r.URL.RawQuery)
@@ -174,7 +172,7 @@ func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
commentsWithInfo := struct {
Comments []store.Comment `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}{}
switch r.URL.Query().Get("url") {
@@ -195,7 +193,7 @@ func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
require.NoError(t, json.NewEncoder(w).Encode(commentsWithInfo))
})
r.HandleFunc("/api/v1/admin/comment/{id}", func(w http.ResponseWriter, r *http.Request) {
r.HandleFunc("/api/v1/admin/comment/{id}", func(_ http.ResponseWriter, r *http.Request) {
require.Equal(t, "DELETE", r.Method)
t.Log("delete ", r.URL.Path)
c.lock.Lock()
@@ -203,7 +201,7 @@ func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
c.lock.Unlock()
})
r.HandleFunc("/api/v1/admin/title/{id}", func(w http.ResponseWriter, r *http.Request) {
r.HandleFunc("/api/v1/admin/title/{id}", func(_ http.ResponseWriter, r *http.Request) {
require.Equal(t, "PUT", r.Method)
t.Log("title for ", r.URL.Path)
c.lock.Lock()
+4 -1
View File
@@ -115,13 +115,16 @@ func responseError(resp *http.Response) error {
if e != nil {
body = []byte("")
}
if resp.StatusCode == http.StatusUnauthorized {
return fmt.Errorf("error response %q, ensure you have set ADMIN_PASSWD and provided it to the command you're running: %s", resp.Status, body)
}
return fmt.Errorf("error response %q, %s", resp.Status, body)
}
// mkdir -p for all dirs
func makeDirs(dirs ...string) error {
for _, dir := range dirs {
if err := os.MkdirAll(dir, 0o700); err != nil { // If path is already a directory, MkdirAll does nothing
if err := os.MkdirAll(dir, 0o700); err != nil { // if path is already a directory, MkdirAll does nothing
return fmt.Errorf("can't make directory %s: %w", dir, err)
}
}
+1 -1
View File
@@ -42,7 +42,7 @@ func (ic *ImportCommand) Execute(_ []string) error {
}
req.SetBasicAuth("admin", ic.AdminPasswd)
resp, err := client.Do(req.WithContext(ctx)) // closes request's reader
resp, err := client.Do(req.WithContext(ctx)) //nolint:gosec // importURL built from operator CLI flags, not user input; closes request's reader
if err != nil {
return fmt.Errorf("request failed for %s: %w", importURL, err)
}
+45 -5
View File
@@ -1,12 +1,13 @@
package cmd
import (
"encoding/base64"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
log "github.com/go-pkgz/lgr"
"github.com/jessevdk/go-flags"
@@ -18,6 +19,10 @@ func TestImport_Execute(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
assert.Equal(t, "POST", r.Method)
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:secret", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "blah\nblah2\n12345678\n", string(body))
@@ -46,6 +51,42 @@ func TestImport_Execute(t *testing.T) {
assert.NoError(t, err)
}
func TestImport_ExecuteNoPassword(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
assert.Equal(t, "POST", r.Method)
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "blah\nblah2\n12345678\n", string(body))
w.WriteHeader(401)
fmt.Fprint(w, "Unauthorized")
}))
defer ts.Close()
cmd := ImportCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p := flags.NewParser(&cmd, flags.Default)
_, err := p.ParseArgs([]string{"--site=remark", "--file=testdata/import.txt"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
cmd = ImportCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p = flags.NewParser(&cmd, flags.Default)
_, err = p.ParseArgs([]string{"--site=remark", "--file=testdata/import.txt.gz"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
}
func TestImport_ExecuteFailed(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
@@ -91,15 +132,14 @@ func TestImport_ExecuteFailed(t *testing.T) {
}
func TestImport_ExecuteTimeout(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ts := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
assert.Equal(t, "POST", r.Method)
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "blah\nblah2\n12345678\n", string(body))
time.Sleep(500 * time.Millisecond)
fmt.Fprintln(w, "some response")
fmt.Fprintln(w, string(body))
// hold the response until the client gives up on its own timeout
<-r.Context().Done()
}))
defer ts.Close()
+2 -2
View File
@@ -34,13 +34,13 @@ func (rc *RemapCommand) Execute(_ []string) error {
ctx, cancel := context.WithTimeout(context.Background(), rc.Timeout)
defer cancel()
remapURL := fmt.Sprintf("%s/api/v1/admin/remap?site=%s", rc.RemarkURL, rc.Site)
req, err := http.NewRequest(http.MethodPost, remapURL, rulesReader)
req, err := http.NewRequest(http.MethodPost, remapURL, rulesReader) //nolint:gosec // RemarkURL is operator CLI flag, not user input
if err != nil {
return fmt.Errorf("can't make remap request for %s: %w", remapURL, err)
}
req.SetBasicAuth("admin", rc.AdminPasswd)
resp, err := client.Do(req.WithContext(ctx))
resp, err := client.Do(req.WithContext(ctx)) //nolint:gosec // see above
if err != nil {
return fmt.Errorf("request failed for %s: %w", remapURL, err)
}
+35
View File
@@ -1,9 +1,12 @@
package cmd
import (
"encoding/base64"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/jessevdk/go-flags"
@@ -16,6 +19,10 @@ func TestRemap_Execute(t *testing.T) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/remap")
assert.Equal(t, "POST", r.Method)
assert.Equal(t, "remark", r.URL.Query().Get("site"))
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:secret", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "http://oldsite.com* https://newsite.com*\nhttp://oldsite.com/from-old-page/1 https://newsite.com/to-new-page/1", string(body))
@@ -33,3 +40,31 @@ func TestRemap_Execute(t *testing.T) {
err = cmd.Execute(nil)
assert.NoError(t, err)
}
func TestRemap_ExecuteNoPassword(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/remap")
assert.Equal(t, "POST", r.Method)
assert.Equal(t, "remark", r.URL.Query().Get("site"))
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "http://oldsite.com* https://newsite.com*\nhttp://oldsite.com/from-old-page/1 https://newsite.com/to-new-page/1", string(body))
w.WriteHeader(401)
fmt.Fprint(w, "Unauthorized")
}))
defer ts.Close()
cmd := RemapCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p := flags.NewParser(&cmd, flags.Default)
_, err := p.ParseArgs([]string{"--site=remark", "--file=testdata/remap_urls.txt"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
}
+418 -146
View File
@@ -2,38 +2,44 @@ package cmd
import (
"context"
"crypto/sha1" //nolint:gosec // used only for stable ID hashing, not for security
"embed"
"encoding/json"
"fmt"
"net"
"net/http"
"net/url"
"os"
"os/signal"
"path"
"regexp"
"slices"
"strings"
"syscall"
"time"
"github.com/go-pkgz/jrpc"
"github.com/go-pkgz/lcw/eventbus"
"github.com/go-pkgz/lcw/v2/eventbus"
log "github.com/go-pkgz/lgr"
ntf "github.com/go-pkgz/notify"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/kyokomi/emoji/v2"
bolt "go.etcd.io/bbolt"
"golang.org/x/oauth2"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/provider"
"github.com/go-pkgz/auth/provider/sender"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
"github.com/go-pkgz/auth/v2/provider"
"github.com/go-pkgz/auth/v2/provider/sender"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
"github.com/umputun/remark42/backend/app/migrator"
"github.com/umputun/remark42/backend/app/notify"
"github.com/umputun/remark42/backend/app/providers"
"github.com/umputun/remark42/backend/app/rest/api"
"github.com/umputun/remark42/backend/app/rest/proxy"
"github.com/umputun/remark42/backend/app/safehttp"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/admin"
"github.com/umputun/remark42/backend/app/store/engine"
@@ -58,34 +64,37 @@ type ServerCommand struct {
SSL SSLGroup `group:"ssl" namespace:"ssl" env-namespace:"SSL"`
ImageProxy ImageProxyGroup `group:"image-proxy" namespace:"image-proxy" env-namespace:"IMAGE_PROXY"`
Sites []string `long:"site" env:"SITE" default:"remark" description:"site names" env-delim:","`
AnonymousVote bool `long:"anon-vote" env:"ANON_VOTE" description:"enable anonymous votes (works only with VOTES_IP enabled)"`
AdminPasswd string `long:"admin-passwd" env:"ADMIN_PASSWD" default:"" description:"admin basic auth password"`
BackupLocation string `long:"backup" env:"BACKUP_PATH" default:"./var/backup" description:"backups location"`
MaxBackupFiles int `long:"max-back" env:"MAX_BACKUP_FILES" default:"10" description:"max backups to keep"`
LegacyImageProxy bool `long:"img-proxy" env:"IMG_PROXY" description:"[deprecated, use image-proxy.http2https] enable image proxy"`
MaxCommentSize int `long:"max-comment" env:"MAX_COMMENT_SIZE" default:"2048" description:"max comment size"`
MaxVotes int `long:"max-votes" env:"MAX_VOTES" default:"-1" description:"maximum number of votes per comment"`
RestrictVoteIP bool `long:"votes-ip" env:"VOTES_IP" description:"restrict votes from the same ip"`
DurationVoteIP time.Duration `long:"votes-ip-time" env:"VOTES_IP_TIME" default:"5m" description:"same ip vote duration"`
LowScore int `long:"low-score" env:"LOW_SCORE" default:"-5" description:"low score threshold"`
CriticalScore int `long:"critical-score" env:"CRITICAL_SCORE" default:"-10" description:"critical score threshold"`
PositiveScore bool `long:"positive-score" env:"POSITIVE_SCORE" description:"enable positive score only"`
ReadOnlyAge int `long:"read-age" env:"READONLY_AGE" default:"0" description:"read-only age of comments, days"`
EditDuration time.Duration `long:"edit-time" env:"EDIT_TIME" default:"5m" description:"edit window"`
AdminEdit bool `long:"admin-edit" env:"ADMIN_EDIT" description:"unlimited edit for admins"`
Port int `long:"port" env:"REMARK_PORT" default:"8080" description:"port"`
Address string `long:"address" env:"REMARK_ADDRESS" default:"" description:"listening address"`
WebRoot string `long:"web-root" env:"REMARK_WEB_ROOT" default:"./web" description:"web root directory"`
UpdateLimit float64 `long:"update-limit" env:"UPDATE_LIMIT" default:"0.5" description:"updates/sec limit"`
RestrictedWords []string `long:"restricted-words" env:"RESTRICTED_WORDS" description:"words prohibited to use in comments" env-delim:","`
RestrictedNames []string `long:"restricted-names" env:"RESTRICTED_NAMES" description:"names prohibited to use by user" env-delim:","`
EnableEmoji bool `long:"emoji" env:"EMOJI" description:"enable emoji"`
SimpleView bool `long:"simple-view" env:"SIMPLE_VIEW" description:"minimal comment editor mode"`
ProxyCORS bool `long:"proxy-cors" env:"PROXY_CORS" description:"disable internal CORS and delegate it to proxy"`
AllowedHosts []string `long:"allowed-hosts" env:"ALLOWED_HOSTS" description:"limit hosts/sources allowed to embed comments" env-delim:","`
SubscribersOnly bool `long:"subscribers-only" env:"SUBSCRIBERS_ONLY" description:"enable commenting only for Patreon subscribers"`
DisableSignature bool `long:"disable-signature" env:"DISABLE_SIGNATURE" description:"disable server signature in headers"`
Sites []string `long:"site" env:"SITE" default:"remark" description:"site names" env-delim:","`
AnonymousVote bool `long:"anon-vote" env:"ANON_VOTE" description:"enable anonymous votes (works only with VOTES_IP enabled)"`
AdminPasswd string `long:"admin-passwd" env:"ADMIN_PASSWD" default:"" description:"admin basic auth password"`
BackupLocation string `long:"backup" env:"BACKUP_PATH" default:"./var/backup" description:"backups location"`
MaxBackupFiles int `long:"max-back" env:"MAX_BACKUP_FILES" default:"10" description:"max backups to keep"`
LegacyImageProxy bool `long:"img-proxy" env:"IMG_PROXY" description:"[deprecated, use image-proxy.http2https] enable image proxy"`
MinCommentSize int `long:"min-comment" env:"MIN_COMMENT_SIZE" default:"0" description:"min comment size"`
MaxCommentSize int `long:"max-comment" env:"MAX_COMMENT_SIZE" default:"2048" description:"max comment size"`
MaxVotes int `long:"max-votes" env:"MAX_VOTES" default:"-1" description:"maximum number of votes per comment"`
RestrictVoteIP bool `long:"votes-ip" env:"VOTES_IP" description:"restrict votes from the same ip"`
DurationVoteIP time.Duration `long:"votes-ip-time" env:"VOTES_IP_TIME" default:"5m" description:"same ip vote duration"`
LowScore int `long:"low-score" env:"LOW_SCORE" default:"-5" description:"low score threshold"`
CriticalScore int `long:"critical-score" env:"CRITICAL_SCORE" default:"-10" description:"critical score threshold"`
PositiveScore bool `long:"positive-score" env:"POSITIVE_SCORE" description:"enable positive score only"`
ReadOnlyAge int `long:"read-age" env:"READONLY_AGE" default:"0" description:"read-only age of comments, days"`
EditDuration time.Duration `long:"edit-time" env:"EDIT_TIME" default:"5m" description:"edit window; set to 0 to disable comment editing and staged image cleanup"`
AdminEdit bool `long:"admin-edit" env:"ADMIN_EDIT" description:"unlimited edit for admins"`
Port int `long:"port" env:"REMARK_PORT" default:"8080" description:"port"`
Address string `long:"address" env:"REMARK_ADDRESS" default:"" description:"listening address"`
WebRoot string `long:"web-root" env:"REMARK_WEB_ROOT" default:"./web" description:"web root directory"`
UpdateLimit float64 `long:"update-limit" env:"UPDATE_LIMIT" default:"0.5" description:"updates/sec limit"`
TrustedProxies []string `long:"trusted-proxy" env:"TRUSTED_PROXY" description:"reverse-proxy networks (CIDR or IP) trusted to set the client IP; if unset, trusted from any client (see docs)" env-delim:","`
RestrictedWords []string `long:"restricted-words" env:"RESTRICTED_WORDS" description:"words prohibited to use in comments" env-delim:","`
RestrictedNames []string `long:"restricted-names" env:"RESTRICTED_NAMES" description:"names prohibited to use by user" env-delim:","`
EnableEmoji bool `long:"emoji" env:"EMOJI" description:"enable emoji"`
SimpleView bool `long:"simple-view" env:"SIMPLE_VIEW" description:"minimal comment editor mode"`
ProxyCORS bool `long:"proxy-cors" env:"PROXY_CORS" description:"disable internal CORS and delegate it to proxy"`
AllowedHosts []string `long:"allowed-hosts" env:"ALLOWED_HOSTS" description:"limit hosts/sources allowed to embed comments via CSP 'frame-ancestors'" env-delim:","`
SubscribersOnly bool `long:"subscribers-only" env:"SUBSCRIBERS_ONLY" description:"enable commenting only for Patreon subscribers"`
DisableSignature bool `long:"disable-signature" env:"DISABLE_SIGNATURE" description:"disable server signature in headers"`
DisableFancyTextFormatting bool `long:"disable-fancy-text-formatting" env:"DISABLE_FANCY_TEXT_FORMATTING" description:"disable fancy comments text formatting (replacement of quotes, dashes, fractions, etc)"`
Auth struct {
TTL struct {
@@ -93,30 +102,32 @@ type ServerCommand struct {
Cookie time.Duration `long:"cookie" env:"COOKIE" default:"200h" description:"auth cookie TTL"`
} `group:"ttl" namespace:"ttl" env-namespace:"TTL"`
SendJWTHeader bool `long:"send-jwt-header" env:"SEND_JWT_HEADER" description:"send JWT as a header instead of cookie"`
SendJWTHeader bool `long:"send-jwt-header" env:"SEND_JWT_HEADER" description:"also send JWT as a header, so the frontend can store it in a client-side cookie that survives third-party cookie blocking; server-set cookies are still sent (note: increases vulnerability to XSS attacks)"`
SameSite string `long:"same-site" env:"SAME_SITE" description:"set same site policy for cookies" choice:"default" choice:"none" choice:"lax" choice:"strict" default:"default"` // nolint
Apple AppleGroup `group:"apple" namespace:"apple" env-namespace:"APPLE" description:"Apple OAuth"`
Google AuthGroup `group:"google" namespace:"google" env-namespace:"GOOGLE" description:"Google OAuth"`
Github AuthGroup `group:"github" namespace:"github" env-namespace:"GITHUB" description:"Github OAuth"`
Facebook AuthGroup `group:"facebook" namespace:"facebook" env-namespace:"FACEBOOK" description:"Facebook OAuth"`
Microsoft AuthGroup `group:"microsoft" namespace:"microsoft" env-namespace:"MICROSOFT" description:"Microsoft OAuth"`
Yandex AuthGroup `group:"yandex" namespace:"yandex" env-namespace:"YANDEX" description:"Yandex OAuth"`
Twitter AuthGroup `group:"twitter" namespace:"twitter" env-namespace:"TWITTER" description:"Twitter OAuth"`
Patreon AuthGroup `group:"patreon" namespace:"patreon" env-namespace:"PATREON" description:"Patreon OAuth"`
Telegram bool `long:"telegram" env:"TELEGRAM" description:"Enable Telegram auth (using token from telegram.token)"`
Dev bool `long:"dev" env:"DEV" description:"enable dev (local) oauth2"`
Anonymous bool `long:"anon" env:"ANON" description:"enable anonymous login"`
Apple AppleGroup `group:"apple" namespace:"apple" env-namespace:"APPLE" description:"Apple OAuth"`
Google AuthGroup `group:"google" namespace:"google" env-namespace:"GOOGLE" description:"Google OAuth"`
Github AuthGroup `group:"github" namespace:"github" env-namespace:"GITHUB" description:"Github OAuth"`
Facebook AuthGroup `group:"facebook" namespace:"facebook" env-namespace:"FACEBOOK" description:"Facebook OAuth"`
Microsoft MicrosoftAuthGroup `group:"microsoft" namespace:"microsoft" env-namespace:"MICROSOFT" description:"Microsoft OAuth"`
Yandex AuthGroup `group:"yandex" namespace:"yandex" env-namespace:"YANDEX" description:"Yandex OAuth"`
Twitter AuthGroup `group:"twitter" namespace:"twitter" env-namespace:"TWITTER" description:"[deprecated, doesn't work] Twitter OAuth"`
Patreon AuthGroup `group:"patreon" namespace:"patreon" env-namespace:"PATREON" description:"Patreon OAuth"`
Discord AuthGroup `group:"discord" namespace:"discord" env-namespace:"DISCORD" description:"Discord OAuth"`
Custom CustomAuthGroup `group:"custom" namespace:"custom" env-namespace:"CUSTOM" description:"Custom OAuth2 provider"`
Telegram bool `long:"telegram" env:"TELEGRAM" description:"Enable Telegram auth (using token from telegram.token)"`
Dev bool `long:"dev" env:"DEV" description:"enable dev (local) oauth2"`
Anonymous bool `long:"anon" env:"ANON" description:"enable anonymous login"`
Email struct {
Enable bool `long:"enable" env:"ENABLE" description:"enable auth via email"`
From string `long:"from" env:"FROM" description:"from email address"`
Subject string `long:"subj" env:"SUBJ" default:"remark42 confirmation" description:"email's subject"`
ContentType string `long:"content-type" env:"CONTENT_TYPE" default:"text/html" description:"content type"`
Host string `long:"host" env:"HOST" description:"[deprecated, use --smtp.host] SMTP host"`
Port int `long:"port" env:"PORT" description:"[deprecated, use --smtp.port] SMTP password"`
SMTPPassword string `long:"passwd" env:"PASSWD" description:"[deprecated, use --smtp.password] SMTP port"`
SMTPUserName string `long:"user" env:"USER" description:"[deprecated, use --smtp.username] enable TLS"`
TLS bool `long:"tls" env:"TLS" description:"[deprecated, use --smtp.tls] SMTP TCP connection timeout"`
Port int `long:"port" env:"PORT" description:"[deprecated, use --smtp.port] SMTP port"`
SMTPPassword string `long:"passwd" env:"PASSWD" description:"[deprecated, use --smtp.password] SMTP password"`
SMTPUserName string `long:"user" env:"USER" description:"[deprecated, use --smtp.username] SMTP user name"`
TLS bool `long:"tls" env:"TLS" description:"[deprecated, use --smtp.tls] enable TLS"`
TimeOut time.Duration `long:"timeout" env:"TIMEOUT" default:"10s" description:"[deprecated, use --smtp.timeout] SMTP TCP connection timeout"`
MsgTemplate string `long:"template" env:"TEMPLATE" description:"[deprecated] message template file" default:"email_confirmation_login.html.tmpl"`
} `group:"email" namespace:"email" env-namespace:"EMAIL"`
@@ -136,7 +147,7 @@ type ImageProxyGroup struct {
// AppleGroup defines options for Apple auth params
type AppleGroup struct {
CID string `long:"cid" env:"CID" description:"Apple client ID"`
CID string `long:"cid" env:"CID" description:"Apple client ID (App ID or Services ID)"`
TID string `long:"tid" env:"TID" description:"Apple service ID"`
KID string `long:"kid" env:"KID" description:"Private key ID"`
PrivateKeyFilePath string `long:"private-key-filepath" env:"PRIVATE_KEY_FILEPATH" description:"Private key file location" default:"/srv/var/apple.p8"`
@@ -148,6 +159,28 @@ type AuthGroup struct {
CSEC string `long:"csec" env:"CSEC" description:"OAuth client secret"`
}
// MicrosoftAuthGroup defines options group for Microsoft auth params
type MicrosoftAuthGroup struct {
CID string `long:"cid" env:"CID" description:"OAuth client ID"`
CSEC string `long:"csec" env:"CSEC" description:"OAuth client secret"`
Tenant string `long:"tenant" env:"TENANT" description:"Azure AD tenant ID, domain, or 'common' (default)" default:"common"`
}
// CustomAuthGroup defines options group for custom OAuth2 provider params
type CustomAuthGroup struct {
Name string `long:"name" env:"NAME" description:"custom provider name used in auth route"`
CID string `long:"cid" env:"CID" description:"OAuth client ID"`
CSEC string `long:"csec" env:"CSEC" description:"OAuth client secret"`
AuthURL string `long:"auth-url" env:"AUTH_URL" description:"OAuth authorization endpoint"`
TokenURL string `long:"token-url" env:"TOKEN_URL" description:"OAuth token endpoint"`
InfoURL string `long:"info-url" env:"INFO_URL" description:"OAuth user info endpoint"`
Scopes []string `long:"scopes" env:"SCOPES" env-delim:"," description:"OAuth scopes"`
IDField string `long:"id-field" env:"ID_FIELD" default:"sub" description:"user info field used as unique id"`
NameField string `long:"name-field" env:"NAME_FIELD" default:"name" description:"user info field used as display name"`
PictureField string `long:"picture-field" env:"PICTURE_FIELD" default:"picture" description:"user info field used as avatar url"`
EmailField string `long:"email-field" env:"EMAIL_FIELD" default:"email" description:"user info field used as email"`
}
// StoreGroup defines options group for store params
type StoreGroup struct {
Type string `long:"type" env:"TYPE" description:"type of storage" choice:"bolt" choice:"rpc" default:"bolt"` // nolint
@@ -217,14 +250,16 @@ type TelegramGroup struct {
// SMTPGroup defines options for SMTP server connection, used in auth and notify modules
type SMTPGroup struct {
Host string `long:"host" env:"HOST" description:"SMTP host"`
Port int `long:"port" env:"PORT" description:"SMTP port"`
Username string `long:"username" env:"USERNAME" description:"SMTP user name"`
Password string `long:"password" env:"PASSWORD" description:"SMTP password"`
TLS bool `long:"tls" env:"TLS" description:"enable TLS"`
LoginAuth bool `long:"login_auth" env:"LOGIN_AUTH" description:"enable LOGIN auth instead of PLAIN"`
StartTLS bool `long:"starttls" env:"STARTTLS" description:"enable StartTLS"`
TimeOut time.Duration `long:"timeout" env:"TIMEOUT" default:"10s" description:"SMTP TCP connection timeout"`
Host string `long:"host" env:"HOST" description:"SMTP host"`
Port int `long:"port" env:"PORT" description:"SMTP port"`
HELOHost string `long:"helo_host" env:"HELO_HOST" description:"SMTP HELO/EHLO hostname"`
Username string `long:"username" env:"USERNAME" description:"SMTP user name"`
Password string `long:"password" env:"PASSWORD" description:"SMTP password"`
TLS bool `long:"tls" env:"TLS" description:"enable TLS"`
InsecureSkipVerify bool `long:"insecure_skip_verify" env:"INSECURE_SKIP_VERIFY" description:"skip certificate verification"`
LoginAuth bool `long:"login_auth" env:"LOGIN_AUTH" description:"enable LOGIN auth instead of PLAIN"`
StartTLS bool `long:"starttls" env:"STARTTLS" description:"enable StartTLS"`
TimeOut time.Duration `long:"timeout" env:"TIMEOUT" default:"10s" description:"SMTP TCP connection timeout"`
}
// NotifyGroup defines options for notification
@@ -250,8 +285,8 @@ type NotifyGroup struct {
} `group:"slack" namespace:"slack" env-namespace:"SLACK"`
Webhook struct {
URL string `long:"url" env:"URL" description:"webhook URL for admin notifications"`
Template string `long:"template" env:"TEMPLATE" description:"webhook authentication template" default:"{\"text\": \"{{.Text}}\"}"`
Headers []string `long:"headers" description:"webhook authentication headers in format --notify.webhook.headers=Header1:Value1,Value2,... [$NOTIFY_WEBHOOK_HEADERS]"` // env NOTIFY_WEBHOOK_HEADERS split in code bellow to allow , inside ""
Template string `long:"template" env:"TEMPLATE" description:"webhook payload template (Go text/template); falls back to {\"text\": {{.Text | escapeJSONString}}} when empty"`
Headers []string `long:"headers" description:"webhook headers in format --notify.webhook.headers=Header1:Value1,Value2,... [$NOTIFY_WEBHOOK_HEADERS]"` // env NOTIFY_WEBHOOK_HEADERS split in code below to allow , inside ""
Timeout time.Duration `long:"timeout" env:"TIMEOUT" description:"webhook timeout" default:"5s"`
} `group:"webhook" namespace:"webhook" env-namespace:"WEBHOOK"`
}
@@ -309,6 +344,7 @@ func (s *ServerCommand) Execute(_ []string) error {
log.Printf("[INFO] start server on port %s:%d", s.Address, s.Port)
resetEnv(
"SECRET",
"AUTH_APPLE_KID",
"AUTH_GOOGLE_CSEC",
"AUTH_GITHUB_CSEC",
"AUTH_FACEBOOK_CSEC",
@@ -316,6 +352,8 @@ func (s *ServerCommand) Execute(_ []string) error {
"AUTH_TWITTER_CSEC",
"AUTH_YANDEX_CSEC",
"AUTH_PATREON_CSEC",
"AUTH_DISCORD_CSEC",
"AUTH_CUSTOM_CSEC",
"TELEGRAM_TOKEN",
"SMTP_PASSWORD",
"ADMIN_PASSWD",
@@ -402,6 +440,12 @@ func (s *ServerCommand) HandleDeprecatedFlags() (result []DeprecatedFlag) {
if s.Notify.Telegram.API != "https://api.telegram.org/bot" {
result = append(result, DeprecatedFlag{Old: "notify.telegram.api", Version: "1.9"})
}
if s.Auth.Twitter.CID != "" {
result = append(result, DeprecatedFlag{Old: "auth.twitter.cid", Version: "1.14"})
}
if s.Auth.Twitter.CSEC != "" {
result = append(result, DeprecatedFlag{Old: "auth.twitter.csec", Version: "1.14"})
}
return append(result, s.findDeprecatedFlagsCollisions()...)
}
@@ -459,12 +503,87 @@ func stringsSetAndDifferent(s1, s2 string) bool {
}
func contains(s string, a []string) bool {
for _, t := range a {
if t == s {
return true
return slices.Contains(a, s)
}
var reservedCustomProviderNames = map[string]struct{}{
"email": {},
"anonymous": {},
"google": {},
"github": {},
"facebook": {},
"yandex": {},
"twitter": {},
"microsoft": {},
"patreon": {},
"discord": {},
"telegram": {},
"dev": {},
"apple": {},
}
var validCustomProviderName = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]*$`)
func isReservedCustomProviderName(name string) bool {
_, ok := reservedCustomProviderNames[name]
return ok
}
func isValidCustomProviderName(name string) bool {
return validCustomProviderName.MatchString(name)
}
func customProviderSourceID(data provider.UserData, cfg CustomAuthGroup) string {
sourceID := data.Value(cfg.IDField)
if sourceID == "" {
sourceID = data.Value(cfg.EmailField)
}
if sourceID == "" {
sourceID = data.Value(cfg.NameField)
}
if sourceID == "" {
sourceID = data.Value(cfg.PictureField)
}
if sourceID == "" {
payload, err := json.Marshal(data)
if err != nil {
log.Printf("[WARN] failed to serialize custom oauth user data for ID fallback: %v", err)
} else {
sourceID = string(payload)
}
}
return false
if sourceID == "" || sourceID == "{}" {
log.Printf("[WARN] custom oauth provider returned no stable user identifier fields, falling back to hashed payload")
}
return sourceID
}
func (c CustomAuthGroup) isConfigured() bool {
return c.Name != "" || c.CID != "" || c.CSEC != "" || c.AuthURL != "" || c.TokenURL != "" || c.InfoURL != "" ||
len(c.Scopes) > 0 || c.IDField != "sub" || c.NameField != "name" || c.PictureField != "picture" || c.EmailField != "email"
}
func (c CustomAuthGroup) missingRequired() []string {
missing := []string{}
if c.Name == "" {
missing = append(missing, "AUTH_CUSTOM_NAME")
}
if c.CID == "" {
missing = append(missing, "AUTH_CUSTOM_CID")
}
if c.CSEC == "" {
missing = append(missing, "AUTH_CUSTOM_CSEC")
}
if c.AuthURL == "" {
missing = append(missing, "AUTH_CUSTOM_AUTH_URL")
}
if c.TokenURL == "" {
missing = append(missing, "AUTH_CUSTOM_TOKEN_URL")
}
if c.InfoURL == "" {
missing = append(missing, "AUTH_CUSTOM_INFO_URL")
}
return missing
}
// newServerApp prepares application and return it with all active parts
@@ -479,6 +598,18 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
}
log.Printf("[INFO] root url=%s", s.RemarkURL)
// parse trusted proxies up front so a bad CIDR fails before any resource is allocated
trustedProxies, err := api.ParseTrustedProxies(s.TrustedProxies)
if err != nil {
return nil, fmt.Errorf("invalid --trusted-proxy: %w", err)
}
switch {
case len(trustedProxies) == 0:
log.Printf("[WARN] --trusted-proxy not set: forwarding headers are trusted from any client and can be spoofed to bypass rate limiting / vote dedup; set it behind a reverse proxy (see docs)")
case api.TrustsAnyPeer(trustedProxies):
log.Printf("[WARN] --trusted-proxy has a catch-all (0.0.0.0/0 or ::/0): forwarding headers are trusted from any client, re-opening the spoofing bypass; scope it to your proxy network")
}
storeEngine, err := s.makeDataStore()
if err != nil {
return nil, fmt.Errorf("failed to make data store engine: %w", err)
@@ -500,11 +631,12 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
EditDuration: s.EditDuration,
AdminEdits: s.AdminEdit,
AdminStore: adminStore,
MinCommentSize: s.MinCommentSize,
MaxCommentSize: s.MaxCommentSize,
MaxVotes: s.MaxVotes,
PositiveScore: s.PositiveScore,
ImageService: imageService,
TitleExtractor: service.NewTitleExtractor(http.Client{Timeout: time.Second * 5}),
TitleExtractor: service.NewTitleExtractor(http.Client{Timeout: time.Second * 5, Transport: safehttp.Transport()}, s.getAllowedDomains()),
RestrictedWordsMatcher: service.NewRestrictedWordsMatcher(service.StaticRestrictedWordsLister{Words: s.RestrictedWords}),
}
dataService.RestrictSameIPVotes.Enabled = s.RestrictVoteIP
@@ -540,7 +672,7 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
Cache: loadingCache,
NativeImporter: &migrator.Native{DataStore: dataService},
DisqusImporter: &migrator.Disqus{DataStore: dataService},
WordPressImporter: &migrator.WordPress{DataStore: dataService},
WordPressImporter: &migrator.WordPress{DataStore: dataService, DisableFancyTextFormatting: s.DisableFancyTextFormatting},
CommentoImporter: &migrator.Commento{DataStore: dataService},
NativeExporter: &migrator.Native{DataStore: dataService},
URLMapperMaker: migrator.NewURLMapper,
@@ -575,33 +707,36 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
}
srv := &api.Rest{
Version: s.Revision,
DataService: dataService,
WebRoot: s.WebRoot,
WebFS: webFS,
RemarkURL: s.RemarkURL,
ImageProxy: imgProxy,
CommentFormatter: commentFormatter,
Migrator: migr,
ReadOnlyAge: s.ReadOnlyAge,
SharedSecret: s.SharedSecret,
Authenticator: authenticator,
Cache: loadingCache,
NotifyService: notifyService,
TelegramService: telegramService,
SSLConfig: sslConfig,
UpdateLimiter: s.UpdateLimit,
ImageService: imageService,
EmailNotifications: contains("email", s.Notify.Users),
TelegramNotifications: contains("telegram", s.Notify.Users) && telegramService != nil,
EmojiEnabled: s.EnableEmoji,
AnonVote: s.AnonymousVote && s.RestrictVoteIP,
SimpleView: s.SimpleView,
ProxyCORS: s.ProxyCORS,
AllowedAncestors: s.AllowedHosts,
SendJWTHeader: s.Auth.SendJWTHeader,
SubscribersOnly: s.SubscribersOnly,
DisableSignature: s.DisableSignature,
Version: s.Revision,
DataService: dataService,
WebRoot: s.WebRoot,
WebFS: webFS,
RemarkURL: s.RemarkURL,
ImageProxy: imgProxy,
CommentFormatter: commentFormatter,
Migrator: migr,
ReadOnlyAge: s.ReadOnlyAge,
SharedSecret: s.SharedSecret,
TrustedProxies: trustedProxies,
Authenticator: authenticator,
Cache: loadingCache,
NotifyService: notifyService,
TelegramService: telegramService,
SSLConfig: sslConfig,
UpdateLimiter: s.UpdateLimit,
ImageService: imageService,
EmailNotifications: contains("email", s.Notify.Users),
TelegramNotifications: contains("telegram", s.Notify.Users) && telegramService != nil,
EmojiEnabled: s.EnableEmoji,
AnonVote: s.AnonymousVote && s.RestrictVoteIP,
SimpleView: s.SimpleView,
ProxyCORS: s.ProxyCORS,
AllowedAncestors: s.AllowedHosts,
SendJWTHeader: s.Auth.SendJWTHeader,
SubscribersOnly: s.SubscribersOnly,
DisableSignature: s.DisableSignature,
DisableFancyTextFormatting: s.DisableFancyTextFormatting,
ExternalImageProxy: s.ImageProxy.CacheExternal,
}
srv.ScoreThresholds.Low, srv.ScoreThresholds.Critical = s.LowScore, s.CriticalScore
@@ -633,6 +768,83 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
}, nil
}
// Extract domains from s.AllowedHosts and second level domain from s.RemarkURL.
// It can be and IP like http://127.0.0.1 in which case we need to use whole IP as domain
// Beware, if s.RemarkURL is in third-level domain like https://example.co.uk, co.uk will be returned.
func (s *ServerCommand) getAllowedDomains() []string {
rawDomains := s.AllowedHosts
rawDomains = append(rawDomains, s.RemarkURL)
allowedDomains := []string{}
for _, rawURL := range rawDomains {
// case of 'self' AllowedHosts, which is not a valid rawURL name
if rawURL == "self" || rawURL == "'self'" || rawURL == "\"self\"" {
continue
}
// AllowedHosts usually don't have https:// prefix, so we're adding it just to make parsing below work the same way as for RemarkURL
if !strings.HasPrefix(rawURL, "http://") && !strings.HasPrefix(rawURL, "https://") {
rawURL = "https://" + rawURL
}
parsedURL, err := url.Parse(rawURL)
if err != nil {
log.Printf("[WARN] failed to parse URL %s for TitleExtract whitelist: %v", rawURL, err)
continue
}
domain := parsedURL.Hostname()
if domain == "" || // don't add empty domain as it will allow everything to be extracted
(len(strings.Split(domain, ".")) < 2 && // don't allow single-word domains like "com"
domain != "localhost") { // localhost is an exceptional single-word domain which is allowed
continue
}
// only for RemarkURL if domain is not IP and has more than two levels, extract second level domain.
// for AllowedHosts we don't do this as they are exact list of domains which can host comments, but
// remarkURL might be on a subdomain and we must allow parent domain to be used for TitleExtract.
if rawURL == s.RemarkURL && net.ParseIP(domain) == nil && len(strings.Split(domain, ".")) > 2 {
domain = strings.Join(strings.Split(domain, ".")[len(strings.Split(domain, "."))-2:], ".")
}
allowedDomains = append(allowedDomains, domain)
}
return allowedDomains
}
// getAllowedRedirectHosts normalises s.AllowedHosts into the form that
// go-pkgz/auth's redirect validator expects. Strips http(s) schemes and
// paths; preserves explicit ports (the validator matches both host-only
// and host:port, so an entry without a port accepts any port while an
// entry with a port restricts to that port). Skips CSP sentinels
// ('self' / "self") and wildcard entries (*, *.example.com) that are
// valid CSP source expressions but not valid hostnames.
func (s *ServerCommand) getAllowedRedirectHosts() []string {
out := make([]string, 0, len(s.AllowedHosts))
for _, raw := range s.AllowedHosts {
raw = strings.TrimSpace(raw)
if raw == "" || raw == "self" || raw == "'self'" || raw == `"self"` {
continue
}
if strings.ContainsRune(raw, '*') { // CSP wildcard, not a host
continue
}
// add scheme so url.Parse populates Hostname()/Host consistently for bare hosts
toParse := raw
if !strings.HasPrefix(toParse, "http://") && !strings.HasPrefix(toParse, "https://") {
toParse = "https://" + toParse
}
u, err := url.Parse(toParse)
if err != nil || u.Hostname() == "" {
log.Printf("[WARN] skipping invalid AllowedHosts entry %q for redirect allowlist: %v", raw, err)
continue
}
if u.Port() != "" {
out = append(out, u.Host) // preserve explicit host:port so allowlist is port-specific
continue
}
out = append(out, u.Hostname())
}
return out
}
// Run all application objects
func (a *serverApp) run(ctx context.Context) error {
if a.AdminPasswd != "" {
@@ -822,27 +1034,28 @@ func (s *ServerCommand) makeAdminStore() (admin.Store, error) {
func (s *ServerCommand) makeCache() (LoadingCache, error) {
log.Printf("[INFO] make cache, type=%s", s.Cache.Type)
o := cache.NewOpts[[]byte]()
switch s.Cache.Type {
case "redis_pub_sub":
redisPubSub, err := eventbus.NewRedisPubSub(s.Cache.RedisAddr, "remark42-cache")
if err != nil {
return nil, fmt.Errorf("cache backend initialization, redis PubSub initialisation: %w", err)
}
backend, err := cache.NewLruCache(cache.MaxCacheSize(s.Cache.Max.Size), cache.MaxValSize(s.Cache.Max.Value),
cache.MaxKeys(s.Cache.Max.Items), cache.EventBus(redisPubSub))
backend, err := cache.NewLruCache(o.MaxCacheSize(s.Cache.Max.Size), o.MaxValSize(s.Cache.Max.Value),
o.MaxKeys(s.Cache.Max.Items), o.EventBus(redisPubSub))
if err != nil {
return nil, fmt.Errorf("cache backend initialization: %w", err)
}
return cache.NewScache(backend), nil
return cache.NewScache[[]byte](backend), nil
case "mem":
backend, err := cache.NewLruCache(cache.MaxCacheSize(s.Cache.Max.Size), cache.MaxValSize(s.Cache.Max.Value),
cache.MaxKeys(s.Cache.Max.Items))
backend, err := cache.NewLruCache(o.MaxCacheSize(s.Cache.Max.Size), o.MaxValSize(s.Cache.Max.Value),
o.MaxKeys(s.Cache.Max.Items))
if err != nil {
return nil, fmt.Errorf("cache backend initialization: %w", err)
}
return cache.NewScache(backend), nil
return cache.NewScache[[]byte](backend), nil
case "none":
return cache.NewScache(&cache.Nop{}), nil
return cache.NewScache[[]byte](&cache.Nop[[]byte]{}), nil
}
return nil, fmt.Errorf("unsupported cache type %s", s.Cache.Type)
}
@@ -857,10 +1070,9 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
if s.Auth.Apple.CID != "" && s.Auth.Apple.TID != "" && s.Auth.Apple.KID != "" {
err := authenticator.AddAppleProvider(
provider.AppleConfig{
ClientID: s.Auth.Apple.CID,
TeamID: s.Auth.Apple.TID,
KeyID: s.Auth.Apple.KID,
ResponseMode: "query", // default is form_post which wouldn't work here
ClientID: s.Auth.Apple.CID,
TeamID: s.Auth.Apple.TID,
KeyID: s.Auth.Apple.KID,
},
provider.LoadApplePrivateKeyFromFile(s.Auth.Apple.PrivateKeyFilePath),
)
@@ -882,7 +1094,7 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
providersCount++
}
if s.Auth.Microsoft.CID != "" && s.Auth.Microsoft.CSEC != "" {
authenticator.AddProvider("microsoft", s.Auth.Microsoft.CID, s.Auth.Microsoft.CSEC)
authenticator.AddMicrosoftProvider(s.Auth.Microsoft.CID, s.Auth.Microsoft.CSEC, s.Auth.Microsoft.Tenant)
providersCount++
}
if s.Auth.Yandex.CID != "" && s.Auth.Yandex.CSEC != "" {
@@ -897,6 +1109,49 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
authenticator.AddProvider("patreon", s.Auth.Patreon.CID, s.Auth.Patreon.CSEC)
providersCount++
}
if s.Auth.Discord.CID != "" && s.Auth.Discord.CSEC != "" {
authenticator.AddProvider("discord", s.Auth.Discord.CID, s.Auth.Discord.CSEC)
providersCount++
}
if s.Auth.Custom.isConfigured() {
missing := s.Auth.Custom.missingRequired()
if len(missing) > 0 {
return fmt.Errorf("custom oauth provider configuration is incomplete, missing: %s", strings.Join(missing, ", "))
}
customName := strings.ToLower(strings.TrimSpace(s.Auth.Custom.Name))
if !isValidCustomProviderName(customName) {
return fmt.Errorf("custom oauth provider name %q is invalid, expected pattern %q", customName, validCustomProviderName.String())
}
if isReservedCustomProviderName(customName) {
return fmt.Errorf("custom oauth provider name %q is reserved", customName)
}
authenticator.AddCustomProvider(customName, auth.Client{Cid: s.Auth.Custom.CID, Csecret: s.Auth.Custom.CSEC}, provider.CustomHandlerOpt{
Endpoint: oauth2.Endpoint{
AuthURL: s.Auth.Custom.AuthURL,
TokenURL: s.Auth.Custom.TokenURL,
},
InfoURL: s.Auth.Custom.InfoURL,
Scopes: s.Auth.Custom.Scopes,
MapUserFn: func(data provider.UserData, _ []byte) token.User {
sourceID := customProviderSourceID(data, s.Auth.Custom)
hashID := token.HashID(sha1.New(), sourceID) //nolint:gosec // stable provider user id hash
user := token.User{
ID: customName + "_" + hashID,
Name: data.Value(s.Auth.Custom.NameField),
Picture: data.Value(s.Auth.Custom.PictureField),
Email: data.Value(s.Auth.Custom.EmailField),
}
if user.Name == "" {
user.Name = "noname_" + hashID[:4]
}
return user
},
})
providersCount++
}
if s.Auth.Dev {
log.Print("[INFO] dev access enabled")
@@ -910,18 +1165,20 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
if s.Auth.Email.Enable {
params := sender.EmailParams{
Host: s.SMTP.Host,
Port: s.SMTP.Port,
SMTPUserName: s.SMTP.Username,
SMTPPassword: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
StartTLS: s.SMTP.StartTLS,
LoginAuth: s.SMTP.LoginAuth,
TLS: s.SMTP.TLS,
Charset: "UTF-8",
From: s.Auth.Email.From,
Subject: s.Auth.Email.Subject,
ContentType: s.Auth.Email.ContentType,
Host: s.SMTP.Host,
Port: s.SMTP.Port,
HELOHost: s.SMTP.HELOHost,
SMTPUserName: s.SMTP.Username,
SMTPPassword: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
StartTLS: s.SMTP.StartTLS,
LoginAuth: s.SMTP.LoginAuth,
TLS: s.SMTP.TLS,
InsecureSkipVerify: s.SMTP.InsecureSkipVerify,
Charset: "UTF-8",
From: s.Auth.Email.From,
Subject: s.Auth.Email.Subject,
ContentType: s.Auth.Email.ContentType,
}
sndr := sender.NewEmailClient(params, log.Default())
tmpl, err := templates.Read(s.Auth.Email.MsgTemplate)
@@ -958,7 +1215,7 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
}
return true, nil
}),
// Custom user ID generator, used to distinguish anonymous users with the same login
// custom user ID generator, used to distinguish anonymous users with the same login
// coming from different IPs
func(user string, r *http.Request) string {
return user + r.RemoteAddr
@@ -1043,14 +1300,14 @@ func (s *ServerCommand) makeNotifyDestinations(authenticator *auth.Service) ([]n
VerificationSubject: s.Notify.Email.VerificationSubject,
UnsubscribeURL: s.RemarkURL + "/email/unsubscribe.html",
// TODO: uncomment after #560 frontend part is ready and URL is known
// SubscribeURL: s.RemarkURL + "/subscribe.html?token=",
// subscribeURL: s.RemarkURL + "/subscribe.html?token=",
TokenGenFn: func(userID, email, site string) (string, error) {
claims := token.Claims{
Handshake: &token.Handshake{ID: userID + "::" + email},
StandardClaims: jwt.StandardClaims{
Audience: site,
ExpiresAt: time.Now().Add(100 * 365 * 24 * time.Hour).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{site},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(100 * 365 * 24 * time.Hour)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
Issuer: "remark42",
},
}
@@ -1065,16 +1322,18 @@ func (s *ServerCommand) makeNotifyDestinations(authenticator *auth.Service) ([]n
emailParams.AdminEmails = s.Admin.Shared.Email
}
smtpParams := ntf.SMTPParams{
Host: s.SMTP.Host,
Port: s.SMTP.Port,
TLS: s.SMTP.TLS,
StartTLS: s.SMTP.StartTLS,
LoginAuth: s.SMTP.LoginAuth,
Username: s.SMTP.Username,
Password: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
ContentType: "text/html",
Charset: "UTF-8",
Host: s.SMTP.Host,
Port: s.SMTP.Port,
HELOHost: s.SMTP.HELOHost,
TLS: s.SMTP.TLS,
StartTLS: s.SMTP.StartTLS,
InsecureSkipVerify: s.SMTP.InsecureSkipVerify,
LoginAuth: s.SMTP.LoginAuth,
Username: s.SMTP.Username,
Password: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
ContentType: "text/html",
Charset: "UTF-8",
}
emailService, err := notify.NewEmail(emailParams, smtpParams)
if err != nil {
@@ -1145,6 +1404,12 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
SendJWTHeader: s.Auth.SendJWTHeader,
SameSiteCookie: s.parseSameSite(s.Auth.SameSite),
SecureCookies: strings.HasPrefix(s.RemarkURL, "https://"),
// enable the `from` redirect allowlist in go-pkgz/auth v2.1.2+ — limits
// post-auth redirects to RemarkURL's own host plus any configured
// AllowedHosts. Prevents the OAuth open-redirect / phishing vector.
AllowedRedirectHosts: token.AllowedHostsFunc(func() ([]string, error) {
return s.getAllowedRedirectHosts(), nil
}),
SecretReader: token.SecretFunc(func(aud string) (string, error) { // get secret per site
return admns.Key(aud)
}),
@@ -1152,10 +1417,16 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
if c.User == nil {
return c
}
c.User.SetAdmin(ds.IsAdmin(c.Audience, c.User.ID))
c.User.SetBoolAttr("blocked", ds.IsBlocked(c.Audience, c.User.ID))
// audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
if len(c.Audience) != 1 {
return c
}
audience := c.Audience[0]
c.User.SetAdmin(ds.IsAdmin(audience, c.User.ID))
c.User.SetBoolAttr("blocked", ds.IsBlocked(audience, c.User.ID))
var err error
c.User.Email, err = ds.GetUserEmail(c.Audience, c.User.ID)
c.User.Email, err = ds.GetUserEmail(audience, c.User.ID)
if err != nil {
log.Printf("[WARN] can't read email for %s, %v", c.User.ID, err)
}
@@ -1175,7 +1446,7 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
return c
}),
AdminPasswd: s.AdminPasswd,
Validator: token.ValidatorFunc(func(token string, claims token.Claims) bool { // check on each auth call (in middleware)
Validator: token.ValidatorFunc(func(_ string, claims token.Claims) bool { // check on each auth call (in middleware)
if claims.User == nil {
return false
}
@@ -1277,20 +1548,21 @@ func splitAtCommas(s string) []string {
// authRefreshCache used by authenticator to minimize repeatable token refreshes
type authRefreshCache struct {
cache.LoadingCache
cache.LoadingCache[token.Claims]
}
func newAuthRefreshCache() *authRefreshCache {
expirableCache, _ := cache.NewExpirableCache(cache.TTL(5 * time.Minute))
o := cache.NewOpts[token.Claims]()
expirableCache, _ := cache.NewExpirableCache(o.TTL(5 * time.Minute))
return &authRefreshCache{LoadingCache: expirableCache}
}
// Get implements cache getter with key converted to string
func (c *authRefreshCache) Get(key interface{}) (interface{}, bool) {
return c.LoadingCache.Peek(key.(string))
func (c *authRefreshCache) Get(key string) (token.Claims, bool) {
return c.Peek(key)
}
// Set implements cache setter with key converted to string
func (c *authRefreshCache) Set(key, value interface{}) {
_, _ = c.LoadingCache.Get(key.(string), func() (interface{}, error) { return value, nil })
func (c *authRefreshCache) Set(key string, value token.Claims) {
_, _ = c.LoadingCache.Get(key, func() (token.Claims, error) { return value, nil })
}
+503 -105
View File
@@ -5,7 +5,6 @@ import (
"crypto/tls"
"fmt"
"io"
"math/rand"
"net"
"net/http"
"os"
@@ -15,8 +14,9 @@ import (
"testing"
"time"
"github.com/go-pkgz/auth/token"
"github.com/golang-jwt/jwt"
"github.com/go-pkgz/auth/v2/provider"
"github.com/go-pkgz/auth/v2/token"
"github.com/golang-jwt/jwt/v5"
"github.com/jessevdk/go-flags"
"go.uber.org/goleak"
@@ -24,15 +24,33 @@ import (
"github.com/stretchr/testify/require"
)
const (
// budget for a server to bind and answer, generous enough for a loaded CI runner
serverStartTimeout = 30 * time.Second
serverStartPoll = 10 * time.Millisecond
// budget for a server to stop once asked. tight enough to catch a shutdown that hangs,
// loose enough not to depend on how loaded the runner is
serverStopTimeout = 10 * time.Second
// connect budget for a single probe. kept off the poll interval so a slow loopback connect
// on a loaded runner does not look like a server that is not listening
probeDialTimeout = time.Second
// the /auth/ group is limited to 2 req/s, so retries sit at its refill interval rather than
// above it, which would only manufacture more 429s
authRetryPoll = 500 * time.Millisecond
)
func TestServerApp(t *testing.T) {
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
return o
})
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
// send ping
resp, err := http.Get(fmt.Sprintf("http://localhost:%d/api/v1/ping", port))
@@ -47,7 +65,7 @@ func TestServerApp(t *testing.T) {
// add comment
client := http.Client{Timeout: 10 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
@@ -67,7 +85,7 @@ func TestServerApp(t *testing.T) {
}
func TestServerApp_DevMode(t *testing.T) {
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
o.AdminPasswd = "password"
@@ -76,10 +94,10 @@ func TestServerApp_DevMode(t *testing.T) {
})
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
providers := app.restSrv.Authenticator.Providers()
require.Equal(t, 10+1, len(providers), "extra auth provider")
require.Equal(t, 11+1, len(providers), "extra auth provider")
assert.Equal(t, "dev", providers[len(providers)-2].Name(), "dev auth provider")
// send ping
resp, err := http.Get(fmt.Sprintf("http://localhost:%d/api/v1/ping", port))
@@ -95,8 +113,32 @@ func TestServerApp_DevMode(t *testing.T) {
app.Wait()
}
func TestServerApp_CustomOAuthProvider(t *testing.T) {
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
o.Auth.Custom.Name = "oidc"
o.Auth.Custom.CID = "cid"
o.Auth.Custom.CSEC = "csec"
o.Auth.Custom.AuthURL = "https://example.com/oauth2/authorize"
o.Auth.Custom.TokenURL = "https://example.com/oauth2/token"
o.Auth.Custom.InfoURL = "https://example.com/oauth2/userinfo"
return o
})
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(t, port)
providers := app.restSrv.Authenticator.Providers()
require.Equal(t, 11+1, len(providers), "extra auth provider")
assert.Equal(t, "oidc", providers[len(providers)-2].Name(), "custom auth provider")
cancel()
app.Wait()
}
func TestServerApp_AnonMode(t *testing.T) {
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
o.Auth.Anonymous = true
@@ -104,10 +146,10 @@ func TestServerApp_AnonMode(t *testing.T) {
})
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
providers := app.restSrv.Authenticator.Providers()
require.Equal(t, 10+1, len(providers), "extra auth provider for anon")
require.Equal(t, 11+1, len(providers), "extra auth provider for anon")
assert.Equal(t, "anonymous", providers[len(providers)-1].Name(), "anon auth provider")
client := http.Client{Timeout: 10 * time.Second}
@@ -123,13 +165,12 @@ func TestServerApp_AnonMode(t *testing.T) {
assert.Equal(t, "pong", string(body))
// try to login with good name
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=blah123&aud=remark", port))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=blah123&aud=remark", port))
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
// try to add a comment as good anonymous
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
@@ -143,58 +184,44 @@ func TestServerApp_AnonMode(t *testing.T) {
assert.Equal(t, http.StatusCreated, resp.StatusCode)
// try to login with non-latin name
time.Sleep(time.Second)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=Раз_Два%20%20Три_34567&aud=remark", port))
require.NoError(t, err)
nonLatin := fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=Раз_Два%20%20Три_34567&aud=remark", port)
resp = getRetryThrottled(t, &client, nonLatin)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
// try to login with bad name
time.Sleep(time.Second)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=**blah123&aud=remark", port))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=**blah123&aud=remark", port))
defer resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try to login with short name
time.Sleep(time.Second)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=bl%%20%%20&aud=remark", port))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=bl%%20%%20&aud=remark", port))
defer resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try to login with name what have space in prefix
time.Sleep(time.Second)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=%%20somebody&aud=remark", port))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=%%20somebody&aud=remark", port))
defer resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try to login with name what have space in suffix
time.Sleep(time.Second)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=somebody%%20&aud=remark", port))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=somebody%%20&aud=remark", port))
defer resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try to login with long name
time.Sleep(time.Second)
ln := strings.Repeat("x", 65)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=%s&aud=remark", port, ln))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=%s&aud=remark", port, ln))
defer resp.Body.Close()
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try to login with admin name
time.Sleep(time.Second)
resp, err = client.Get(fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=umpUtun&aud=remark", port))
require.NoError(t, err)
resp = getRetryThrottled(t, &client, fmt.Sprintf("http://localhost:%d/auth/anonymous/login?user=umpUtun&aud=remark", port))
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
// try to add a comment as anonymous with admin name
time.Sleep(time.Second)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
@@ -225,12 +252,12 @@ func getAuthFromCookie(t *testing.T, app *serverApp, resp *http.Response) (tkn s
func TestServerApp_WithSSL(t *testing.T) {
opts := ServerCommand{}
sslPort := chooseRandomUnusedPort()
sslPort := chooseUnusedPort(t)
opts.SetCommon(CommonOpts{RemarkURL: fmt.Sprintf("https://localhost:%d", sslPort), SharedSecret: "123456"})
// prepare options
p := flags.NewParser(&opts, flags.Default)
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
_, err := p.ParseArgs([]string{"--admin-passwd=password", "--port=" + strconv.Itoa(port), "--store.bolt.path=/tmp/xyz", "--backup=/tmp",
"--avatar.type=bolt", "--avatar.bolt.file=/tmp/ava-test.db",
"--ssl.type=static", "--ssl.cert=testdata/cert.pem", "--ssl.key=testdata/key.pem",
@@ -245,12 +272,13 @@ func TestServerApp_WithSSL(t *testing.T) {
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
defer cancel() // this context is not the one createAppFromCmd registers for cleanup
go func() { _ = app.run(ctx) }()
waitForHTTPSServerStart(sslPort)
waitForServerStart(t, sslPort, port) // the redirect check below uses the plain http port
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
@@ -287,7 +315,7 @@ func TestServerApp_WithRemote(t *testing.T) {
// prepare options
p := flags.NewParser(&opts, flags.Default)
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
_, err := p.ParseArgs([]string{"--admin-passwd=password", "--cache.type=none",
"--store.type=rpc", "--store.rpc.api=http://127.0.0.1",
"--port=" + strconv.Itoa(port), "--avatar.fs.path=/tmp",
@@ -301,8 +329,9 @@ func TestServerApp_WithRemote(t *testing.T) {
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
defer cancel() // this context is not the one createAppFromCmd registers for cleanup
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
// send ping
resp, err := http.Get(fmt.Sprintf("http://localhost:%d/api/v1/ping", port))
@@ -352,6 +381,16 @@ func TestServerApp_Failed(t *testing.T) {
assert.EqualError(t, err, "invalid remark42 url demo.remark42.com")
t.Log(err)
// invalid trusted proxy CIDR fails fast, before any resource is created
opts = ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p = flags.NewParser(&opts, flags.Default)
_, err = p.ParseArgs([]string{"--backup=/tmp", "--trusted-proxy=nonsense"})
assert.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err, `invalid --trusted-proxy: invalid trusted proxy "nonsense"`)
t.Log(err)
// wrong store type
opts = ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
@@ -389,37 +428,209 @@ func TestServerApp_Failed(t *testing.T) {
"failed to make authenticator: an AppleProvider creating failed: "+
"provided private key is not ECDSA")
t.Log(err)
// incomplete custom oauth config
opts = ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p = flags.NewParser(&opts, flags.Default)
_, err = p.ParseArgs([]string{"--store.bolt.path=/tmp", "--backup=/tmp", "--image.fs.path=/tmp", "--auth.custom.name=oidc", "--auth.custom.cid=123"})
assert.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err,
"failed to make authenticator: custom oauth provider configuration is incomplete, missing: "+
"AUTH_CUSTOM_CSEC, AUTH_CUSTOM_AUTH_URL, AUTH_CUSTOM_TOKEN_URL, AUTH_CUSTOM_INFO_URL")
t.Log(err)
}
func TestIsReservedCustomProviderName(t *testing.T) {
reserved := []string{
"email", "anonymous", "google", "github", "facebook", "yandex", "twitter",
"microsoft", "patreon", "discord", "telegram", "dev", "apple",
}
for _, name := range reserved {
t.Run(name, func(t *testing.T) {
assert.True(t, isReservedCustomProviderName(name))
})
}
assert.False(t, isReservedCustomProviderName("oidc"))
}
func TestIsValidCustomProviderName(t *testing.T) {
valid := []string{"oidc", "codeberg", "provider_1", "provider-1", "a1"}
for _, name := range valid {
t.Run("valid_"+name, func(t *testing.T) {
assert.True(t, isValidCustomProviderName(name))
})
}
invalid := []string{"", " has-space", "has space", "Uppercase", "provider!", "-provider", "_provider"}
for _, name := range invalid {
t.Run("invalid_"+strings.ReplaceAll(name, " ", "_"), func(t *testing.T) {
assert.False(t, isValidCustomProviderName(name))
})
}
}
func TestCustomProviderSourceID(t *testing.T) {
cfg := CustomAuthGroup{IDField: "sub", EmailField: "email", NameField: "name", PictureField: "picture"}
assert.Equal(t, "user-1", customProviderSourceID(provider.UserData{"sub": "user-1", "email": "a@example.com"}, cfg))
assert.Equal(t, "a@example.com", customProviderSourceID(provider.UserData{"email": "a@example.com"}, cfg))
assert.Equal(t, "alice", customProviderSourceID(provider.UserData{"name": "alice"}, cfg))
assert.Equal(t, "https://example.com/avatar.png", customProviderSourceID(provider.UserData{"picture": "https://example.com/avatar.png"}, cfg))
assert.Equal(t, `{"login":"alice"}`, customProviderSourceID(provider.UserData{"login": "alice"}, cfg))
assert.Equal(t, "{}", customProviderSourceID(provider.UserData{}, cfg))
}
func TestServerApp_InvalidCustomOAuthProviderName(t *testing.T) {
baseArgs := []string{
"--store.bolt.path=/tmp",
"--backup=/tmp",
"--image.fs.path=/tmp",
"--auth.custom.cid=123",
"--auth.custom.csec=456",
"--auth.custom.auth-url=https://example.com/oauth2/authorize",
"--auth.custom.token-url=https://example.com/oauth2/token",
"--auth.custom.info-url=https://example.com/oauth2/userinfo",
}
t.Run("reserved", func(t *testing.T) {
opts := ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p := flags.NewParser(&opts, flags.Default)
_, err := p.ParseArgs(append(baseArgs, "--auth.custom.name=twitter"))
require.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err, `failed to make authenticator: custom oauth provider name "twitter" is reserved`)
})
t.Run("not_url_safe", func(t *testing.T) {
opts := ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p := flags.NewParser(&opts, flags.Default)
_, err := p.ParseArgs(append(baseArgs, "--auth.custom.name=bad name"))
require.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err, `failed to make authenticator: custom oauth provider name "bad name" is invalid, expected pattern "^[a-z0-9][a-z0-9_-]*$"`)
})
}
func TestServerApp_Shutdown(t *testing.T) {
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = chooseRandomUnusedPort()
o.Port = port
return o
})
time.AfterFunc(100*time.Millisecond, func() {
cancel()
})
st := time.Now()
err := app.run(ctx)
assert.NoError(t, err)
assert.True(t, time.Since(st).Seconds() < 1, "should take about 100msec")
// cancel once the server actually answers, so the test measures shutdown and not startup.
// the deferred cancel also covers a failed wait, keeping app.run from racing the next test
errCh := make(chan error, 1)
go func() { errCh <- app.run(ctx) }()
defer cancel()
waitForHTTPServerStart(t, port)
cancel()
select {
case err := <-errCh:
assert.NoError(t, err)
case <-time.After(serverStopTimeout):
t.Fatal("server app did not stop after context cancel")
}
app.Wait()
}
func TestServerApp_MainSignal(t *testing.T) {
done := make(chan struct{})
go func() {
<-done
time.Sleep(250 * time.Millisecond)
err := syscall.Kill(syscall.Getpid(), syscall.SIGTERM)
// TestServerApp_ClaimsUpd covers the hook the authenticator runs on every token mint, refresh
// included: it stamps admin, blocked and email onto the claims and blocks impersonation of a
// restricted name. Calling the updater directly keeps it independent of when a token expires.
func TestServerApp_ClaimsUpd(t *testing.T) {
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
return o
})
// the app owns stores and services that only run closes, so it goes through the usual
// lifecycle here rather than being built and abandoned
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(t, port)
defer app.Wait()
defer cancel()
upd := app.restSrv.Authenticator.TokenService().ClaimsUpd
require.NotNil(t, upd, "claims updater wired into the token service")
claimsFor := func(id, name string) token.Claims {
return token.Claims{
RegisteredClaims: jwt.RegisteredClaims{Audience: jwt.ClaimStrings{"remark"}},
User: &token.User{ID: id, Name: name},
}
}
t.Run("plain user gets no attributes", func(t *testing.T) {
res := upd.Update(claimsFor("provider1_dev", "developer"))
assert.False(t, res.User.IsAdmin(), "not an admin")
assert.False(t, res.User.BoolAttr("blocked"), "not blocked")
assert.Empty(t, res.User.Email, "no email on file")
})
t.Run("admin from the admin store", func(t *testing.T) {
res := upd.Update(claimsFor("id1", "admin one"))
assert.True(t, res.User.IsAdmin(), "id1 is listed as admin")
})
t.Run("blocked user carries the blocked attribute", func(t *testing.T) {
require.NoError(t, app.restSrv.DataService.SetBlock("remark", "blocked_user", true, time.Hour))
res := upd.Update(claimsFor("blocked_user", "blocked"))
assert.True(t, res.User.BoolAttr("blocked"), "block is reflected on refresh")
})
t.Run("email is read from the store", func(t *testing.T) {
_, err := app.restSrv.DataService.SetUserEmail("remark", "with_email", "user@example.com")
require.NoError(t, err)
}()
res := upd.Update(claimsFor("with_email", "someone"))
assert.Equal(t, "user@example.com", res.User.Email)
})
t.Run("anonymous impersonating a restricted name is blocked", func(t *testing.T) {
res := upd.Update(claimsFor("anonymous_x", " UmpUtun "))
assert.True(t, res.User.BoolAttr("blocked"), "restricted name matched case and space insensitively")
})
t.Run("email user impersonating a restricted name is blocked", func(t *testing.T) {
res := upd.Update(claimsFor("email_x", "bobuk"))
assert.True(t, res.User.BoolAttr("blocked"))
})
t.Run("regular user may carry a restricted name", func(t *testing.T) {
res := upd.Update(claimsFor("provider1_someone", "umputun"))
assert.False(t, res.User.BoolAttr("blocked"), "only anonymous and email logins are checked")
})
t.Run("claims without a user pass through", func(t *testing.T) {
res := upd.Update(token.Claims{RegisteredClaims: jwt.RegisteredClaims{Audience: jwt.ClaimStrings{"remark"}}})
assert.Nil(t, res.User)
})
t.Run("claims without exactly one audience pass through", func(t *testing.T) {
c := claimsFor("id1", "admin one")
c.Audience = jwt.ClaimStrings{"remark", "second"}
res := upd.Update(c)
assert.False(t, res.User.IsAdmin(), "attributes need a single audience to resolve the site")
})
}
func TestServerApp_MainSignal(t *testing.T) {
sigErr := make(chan error, 1)
s := ServerCommand{}
s.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p := flags.NewParser(&s, flags.Default)
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
args := []string{"test", "--store.bolt.path=/tmp/xyz", "--backup=/tmp", "--avatar.type=bolt",
"--avatar.bolt.file=/tmp/ava-test.db", "--port=" + strconv.Itoa(port), "--image.fs.path=/tmp"}
defer os.Remove("/tmp/xyz")
@@ -427,11 +638,52 @@ func TestServerApp_MainSignal(t *testing.T) {
defer os.Remove("/tmp/ava-test.db")
_, err := p.ParseArgs(args)
require.NoError(t, err)
st := time.Now()
close(done)
// the signal goes out only once the server answers: SIGTERM landing before the handler is
// installed kills the test process, so a wait that timed out reports instead of sending it
go func() {
started := waitForServerPort(port, serverStartTimeout)
// signal either way: Execute blocks until it gets one, so bailing out here would hang
// the test until the package timeout instead of failing with the reason
killErr := syscall.Kill(syscall.Getpid(), syscall.SIGTERM)
if !started {
killErr = fmt.Errorf("server on port %d didn't start", port)
}
sigErr <- killErr
}()
err = s.Execute(args)
assert.NoError(t, err, "execute should be without errors")
assert.True(t, time.Since(st).Seconds() < 5, "should take under five sec", time.Since(st).Seconds())
require.NoError(t, <-sigErr, "SIGTERM not delivered")
}
func TestServerApp_RunCanceledBeforeRESTStart(t *testing.T) {
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
return o
})
cancel()
errCh := make(chan error, 1)
go func() { errCh <- app.run(ctx) }()
// the budget is generous on purpose: the assertion is that run exits rather than hangs, and
// store construction can take a while on a loaded runner
select {
case err := <-errCh:
require.NoError(t, err)
app.Wait()
case <-time.After(serverStartTimeout):
waitForHTTPServerStart(t, port)
app.restSrv.Shutdown()
select {
case <-errCh:
app.Wait()
case <-time.After(serverStartTimeout):
t.Fatal("server app did not stop after forced REST shutdown")
}
t.Fatal("server app should exit when context is canceled before REST server starts")
}
}
func TestServerApp_DeprecatedArgs(t *testing.T) {
@@ -457,6 +709,8 @@ func TestServerApp_DeprecatedArgs(t *testing.T) {
"--notify.telegram.token=abcd",
"--notify.telegram.timeout=3m",
"--notify.telegram.api=http://example.org",
"--auth.twitter.cid=123",
"--auth.twitter.csec=456",
}
assert.Empty(t, s.SMTP.Host)
assert.Empty(t, s.SMTP.Port)
@@ -482,6 +736,8 @@ func TestServerApp_DeprecatedArgs(t *testing.T) {
{Old: "notify.telegram.token", New: "telegram.token", Version: "1.9"},
{Old: "notify.telegram.timeout", New: "telegram.timeout", Version: "1.9"},
{Old: "notify.telegram.api", Version: "1.9"},
{Old: "auth.twitter.cid", Version: "1.14"},
{Old: "auth.twitter.csec", Version: "1.14"},
},
deprecatedFlags)
assert.Equal(t, "smtp.example.org", s.SMTP.Host)
@@ -591,25 +847,26 @@ func Test_ACMEEmail(t *testing.T) {
}
func TestServerAuthHooks(t *testing.T) {
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
return o
})
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
// make a token for user dev
// make a token for user dev. nothing here checks expiry, so the lifetime only has to
// outlast the whole test
tkService := app.restSrv.Authenticator.TokenService()
tkService.TokenDuration = time.Second
tkService.TokenDuration = time.Hour
claims := token.Claims{
StandardClaims: jwt.StandardClaims{
Audience: "remark",
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark"},
Issuer: "remark",
ExpiresAt: time.Now().Add(time.Second).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
},
User: &token.User{
ID: "github_dev",
@@ -624,7 +881,7 @@ func TestServerAuthHooks(t *testing.T) {
defer client.CloseIdleConnections()
// add comment
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-630/", "site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tk)
@@ -633,13 +890,13 @@ func TestServerAuthHooks(t *testing.T) {
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusCreated, resp.StatusCode, "non-blocked user able to post")
// add comment with no-aud claim
claimsNoAud := claims
claimsNoAud.Audience = ""
tkNoAud, err := tkService.Token(claimsNoAud)
// try to add comment with no-aud claim
badClaimsNoAud := claims
badClaimsNoAud.Audience = jwt.ClaimStrings{""}
tkNoAud, err := tkService.Token(badClaimsNoAud)
require.NoError(t, err)
t.Logf("no-aud claims: %s", tkNoAud)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
@@ -651,6 +908,43 @@ func TestServerAuthHooks(t *testing.T) {
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "user without aud claim rejected, \n"+tkNoAud+"\n"+string(body))
// try to add comment with multiple auds
badClaimsMultipleAud := claims
badClaimsMultipleAud.Audience = jwt.ClaimStrings{"remark", "second_aud"}
tkMultipleAuds, err := tkService.Token(badClaimsMultipleAud)
require.NoError(t, err)
t.Logf("multiple aud claims: %s", tkMultipleAuds)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tkMultipleAuds)
resp, err = client.Do(req)
require.NoError(t, err)
body, err = io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "user with multiple auds claim rejected, \n"+tkMultipleAuds+"\n"+string(body))
// try to add comment without user set
badClaimsNoUser := claims
badClaimsNoUser.Audience = jwt.ClaimStrings{"remark"}
badClaimsNoUser.User = nil
tkNoUser, err := tkService.Token(badClaimsNoUser)
require.NoError(t, err)
t.Logf("no user claims: %s", tkNoUser)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tkNoUser)
resp, err = client.Do(req)
require.NoError(t, err)
body, err = io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "user without user information rejected, \n"+tkNoUser+"\n"+string(body))
// block user github_dev as admin
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("http://localhost:%d/api/v1/admin/user/github_dev?site=remark&block=1&ttl=10d", port), http.NoBody)
@@ -665,7 +959,7 @@ func TestServerAuthHooks(t *testing.T) {
t.Log(string(b))
// try add a comment with blocked user
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123 blah", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tk)
@@ -674,8 +968,7 @@ func TestServerAuthHooks(t *testing.T) {
body, err = io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.True(t, resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusUnauthorized,
"blocked user can't post, \n"+tk+"\n"+string(body))
assert.Equal(t, http.StatusForbidden, resp.StatusCode, "blocked user can't post, \n"+tk+"\n"+string(body))
cancel()
app.Wait()
@@ -697,7 +990,6 @@ func TestServerCommand_parseSameSite(t *testing.T) {
cmd := ServerCommand{}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.res, cmd.parseSameSite(tt.inp))
})
@@ -726,40 +1018,129 @@ func Test_splitAtCommas(t *testing.T) {
}
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
break
}
func Test_getAllowedDomains(t *testing.T) {
tbl := []struct {
s ServerCommand
allowedDomains []string
}{
// correct example, parsed and returned as allowed domain
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "https://remark42.example.org"}}, []string{"example.org"}},
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "http://remark42.example.org"}}, []string{"example.org"}},
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "http://localhost"}}, []string{"localhost"}},
// incorrect URLs, so Hostname is empty but returned list doesn't include empty string as it would allow any domain
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "bad hostname"}}, []string{}},
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "not_a_hostname"}}, []string{}},
// test removal of 'self', multiple AllowedHosts. No deduplication is expected
{ServerCommand{AllowedHosts: []string{"'self'", "example.org", "test.example.org", "remark42.com"}, CommonOpts: CommonOpts{RemarkURL: "https://example.org"}}, []string{"example.org", "test.example.org", "remark42.com", "example.org"}},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.allowedDomains, tt.s.getAllowedDomains())
})
}
}
func Test_getAllowedRedirectHosts(t *testing.T) {
tbl := []struct {
name string
hosts []string
want []string
}{
{name: "empty", hosts: nil, want: []string{}},
{name: "bare hostnames pass through", hosts: []string{"example.com", "admin.example.com"}, want: []string{"example.com", "admin.example.com"}},
{name: "https scheme stripped", hosts: []string{"https://example.com"}, want: []string{"example.com"}},
{name: "http scheme stripped", hosts: []string{"http://example.com"}, want: []string{"example.com"}},
{name: "scheme with path strips path", hosts: []string{"https://example.com/embed"}, want: []string{"example.com"}},
{name: "explicit port preserved as host:port", hosts: []string{"example.com:8080"}, want: []string{"example.com:8080"}},
{name: "scheme with explicit port preserved", hosts: []string{"https://example.com:8443"}, want: []string{"example.com:8443"}},
{name: "scheme without port stays bare host", hosts: []string{"https://example.com"}, want: []string{"example.com"}},
{name: "self sentinel filtered", hosts: []string{"'self'", "self", `"self"`, "example.com"}, want: []string{"example.com"}},
{name: "wildcards filtered", hosts: []string{"*", "*.example.com", "https://*.example.com", "example.com"}, want: []string{"example.com"}},
{name: "empty entries filtered", hosts: []string{"", " ", "example.com"}, want: []string{"example.com"}},
{name: "mixed real-world", hosts: []string{"'self'", "https://blog.example.com", "admin.example.com:8443", "*.cdn.example.com"},
want: []string{"blog.example.com", "admin.example.com:8443"}},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
s := ServerCommand{AllowedHosts: tt.hosts}
assert.Equal(t, tt.want, s.getAllowedRedirectHosts())
})
}
}
// chooseUnusedPort asks the kernel for a free port from the ephemeral range, which makes a
// collision between concurrently running package test binaries very unlikely
func chooseUnusedPort(t *testing.T) int {
t.Helper()
ln, err := net.Listen("tcp", ":0")
require.NoError(t, err, "no free port available")
port := ln.Addr().(*net.TCPAddr).Port
require.NoError(t, ln.Close())
return port
}
func waitForHTTPServerStart(port int) {
// wait for up to 3 seconds for server to start before returning it
// waitForHTTPServerStart blocks until the server on port answers, failing the test naming the
// port if it never does
func waitForHTTPServerStart(t *testing.T, port int) {
t.Helper()
client := http.Client{Timeout: time.Second}
defer client.CloseIdleConnections()
for i := 0; i < 300; i++ {
time.Sleep(time.Millisecond * 10)
if resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port)); err == nil {
_ = resp.Body.Close()
return
require.Eventually(t, func() bool {
resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port))
if err != nil {
return false
}
_ = resp.Body.Close()
return true
}, serverStartTimeout, serverStartPoll, "http server on port %d didn't start", port)
}
// waitForServerStart blocks until something accepts on every listed port, failing the test
// naming the port that never came up
func waitForServerStart(t *testing.T, ports ...int) {
t.Helper()
for _, port := range ports {
require.True(t, waitForServerPort(port, serverStartTimeout), "server on port %d didn't start", port)
}
}
func waitForHTTPSServerStart(port int) {
// wait for up to 3 seconds for HTTPS server to start
for i := 0; i < 300; i++ {
time.Sleep(time.Millisecond * 10)
conn, _ := net.DialTimeout("tcp", fmt.Sprintf("localhost:%d", port), time.Millisecond*10)
if conn != nil {
_ = conn.Close()
break
// getRetryThrottled issues a GET and retries while the auth routes answer 429, since the /auth/
// group is limited to 2 req/s and this test logs in more often than that. a transport error is
// retried a couple of times and then reported as itself, so a dead server is not read as throttling
func getRetryThrottled(t *testing.T, client *http.Client, url string) *http.Response {
t.Helper()
const transportRetries = 2
errCount := 0
for deadline := time.Now().Add(serverStartTimeout); time.Now().Before(deadline); time.Sleep(authRetryPoll) {
r, err := client.Get(url)
if err != nil {
errCount++
require.LessOrEqual(t, errCount, transportRetries, "request to %s failed: %v", url, err)
continue
}
if r.StatusCode == http.StatusTooManyRequests {
_ = r.Body.Close()
continue
}
return r
}
t.Fatalf("request to %s kept being rate limited", url)
return nil
}
// waitForServerPort blocks until something accepts on port, reporting whether it came up.
// unlike the require-based helpers it is safe to call off the test goroutine.
func waitForServerPort(port int, timeout time.Duration) bool {
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
conn, err := net.DialTimeout("tcp", fmt.Sprintf("localhost:%d", port), probeDialTimeout)
if err == nil {
_ = conn.Close()
return true
}
time.Sleep(serverStartPoll)
}
return false
}
func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serverApp, context.Context, context.CancelFunc) {
@@ -771,7 +1152,6 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
_, err := p.ParseArgs([]string{"--admin-passwd=password", "--site=remark"})
require.NoError(t, err)
cmd.Avatar.FS.Path, cmd.Avatar.Type, cmd.BackupLocation, cmd.Image.FS.Path = "/tmp/remark42_test", "fs", "/tmp/remark42_test", "/tmp/remark42_test"
cmd.Store.Bolt.Path = fmt.Sprintf("/tmp/%d", cmd.Port)
cmd.Store.Bolt.Timeout = 10 * time.Second
cmd.Auth.Apple.CID, cmd.Auth.Apple.KID, cmd.Auth.Apple.TID = "cid", "kid", "tid"
cmd.Auth.Apple.PrivateKeyFilePath = "testdata/apple.p8"
@@ -782,6 +1162,7 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
cmd.Auth.Microsoft.CSEC, cmd.Auth.Microsoft.CID = "csec", "cid"
cmd.Auth.Twitter.CSEC, cmd.Auth.Twitter.CID = "csec", "cid"
cmd.Auth.Patreon.CSEC, cmd.Auth.Patreon.CID = "csec", "cid"
cmd.Auth.Discord.CSEC, cmd.Auth.Discord.CID = "csec", "cid"
cmd.Auth.Telegram = true
cmd.Telegram.Token = "token"
cmd.Auth.Email.Enable = true
@@ -802,7 +1183,10 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
cmd.RestrictedNames = []string{"umputun", "bobuk"}
cmd.emailMsgTemplatePath = "../../templates/email_reply.html.tmpl"
cmd.emailVerificationTemplatePath = "../../templates/email_confirmation_subscription.html.tmpl"
cmd = fn(cmd)
// as is uses port, call it after fn which could set it
cmd.Store.Bolt.Path = fmt.Sprintf("/tmp/%d", cmd.Port)
app, ctx, cancel := createAppFromCmd(t, cmd)
@@ -819,6 +1203,9 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
func createAppFromCmd(t *testing.T, cmd ServerCommand) (*serverApp, context.Context, context.CancelFunc) {
ctx, cancel := context.WithCancel(context.Background())
// a require in a readiness wait exits the test goroutine, so without this an app started in
// a goroutine would never be stopped and goleak would report it instead of the failure
t.Cleanup(cancel)
app, err := cmd.newServerApp(ctx)
require.NoError(t, err)
return app, ctx, cancel
@@ -826,5 +1213,16 @@ func createAppFromCmd(t *testing.T, cmd ServerCommand) (*serverApp, context.Cont
func TestMain(m *testing.M) {
// ignore is added only for GitHub Actions, can't reproduce locally
goleak.VerifyTestMain(m, goleak.IgnoreTopFunction("net/http.(*Server).Shutdown"))
goleak.VerifyTestMain(
m,
// the shutdown goroutine in serverApp.run is not joined by Wait, and Rest.Shutdown gives
// httpServer.Shutdown a second, which can outlast goleak's retry budget on a loaded runner
goleak.IgnoreTopFunction("net/http.(*Server).Shutdown"),
// this will be fixed in https://github.com/hashicorp/golang-lru/issues/159
goleak.IgnoreTopFunction("github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1"),
// regexp2, pulled in by chroma for syntax highlighting, keeps one shared clock goroutine
// alive for up to a second after the last match with a timeout, sleeping in 100ms ticks.
// it ends on its own, but a binary that finishes inside that window is reported as leaking
goleak.IgnoreAnyFunction("github.com/dlclark/regexp2/v2.runClock"),
)
}
+5 -6
View File
@@ -1,6 +1,7 @@
package main
import (
"errors"
"fmt"
"os"
"os/signal"
@@ -55,7 +56,8 @@ func main() {
}
if _, err := p.Parse(); err != nil {
if flagsErr, ok := err.(*flags.Error); ok && flagsErr.Type == flags.ErrHelp {
var flagsErr *flags.Error
if errors.As(err, &flagsErr) && flagsErr.Type == flags.ErrHelp {
os.Exit(0)
}
os.Exit(1)
@@ -90,14 +92,11 @@ func logDeprecatedParams(params []cmd.DeprecatedFlag) {
func getDump() string {
maxSize := 5 * 1024 * 1024
stacktrace := make([]byte, maxSize)
length := runtime.Stack(stacktrace, true)
if length > maxSize {
length = maxSize
}
length := min(runtime.Stack(stacktrace, true), maxSize)
return string(stacktrace[:length])
}
// nolint:gochecknoinits // can't avoid it in this place
//nolint:gochecknoinits // can't avoid it in this place
func init() {
// catch SIGQUIT and print stack traces
sigChan := make(chan os.Signal, 1)
+43 -31
View File
@@ -3,7 +3,6 @@ package main
import (
"fmt"
"io"
"math/rand"
"net"
"net/http"
"net/http/httptest"
@@ -25,7 +24,7 @@ func Test_Main(t *testing.T) {
require.NoError(t, err)
defer os.RemoveAll(dir)
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
os.Args = []string{"test", "server", "--secret=123456", "--store.bolt.path=" + dir, "--backup=/tmp",
"--avatar.fs.path=" + dir, "--port=" + strconv.Itoa(port), "--url=https://demo.remark42.com", "--dbg", "--notify.type=none"}
@@ -48,7 +47,7 @@ func Test_Main(t *testing.T) {
<-finished
}()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
resp, err := http.Get(fmt.Sprintf("http://localhost:%d/api/v1/ping", port))
require.NoError(t, err)
defer resp.Body.Close()
@@ -63,9 +62,9 @@ func TestMain_WithWebhook(t *testing.T) {
require.NoError(t, err)
defer os.RemoveAll(dir)
var webhookSent int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.StoreInt32(&webhookSent, 1)
var webhookSent atomic.Int32
ts := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
webhookSent.Store(1)
assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
b, e := io.ReadAll(r.Body)
@@ -76,7 +75,7 @@ func TestMain_WithWebhook(t *testing.T) {
}))
defer ts.Close()
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
os.Args = []string{"test", "server", "--secret=123456", "--store.bolt.path=" + dir, "--backup=/tmp",
"--avatar.fs.path=" + dir, "--port=" + strconv.Itoa(port), "--url=https://demo.remark42.com", "--dbg",
"--admin-passwd=password", "--site=remark", "--notify.admins=webhook"}
@@ -98,9 +97,6 @@ func TestMain_WithWebhook(t *testing.T) {
finished := make(chan struct{})
go func() {
main()
assert.Eventually(t, func() bool {
return atomic.LoadInt32(&webhookSent) == int32(1)
}, time.Second, 100*time.Millisecond, "webhook was not sent")
close(finished)
}()
@@ -110,52 +106,68 @@ func TestMain_WithWebhook(t *testing.T) {
<-finished
}()
waitForHTTPServerStart(port)
waitForHTTPServerStart(t, port)
resp, err := http.Post(fmt.Sprintf("http://admin:password@localhost:%d/api/v1/comment", port), "",
strings.NewReader(`{"text": "env test", "locator":{"url": "https://radio-t.com", "site": "remark"}}`))
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusCreated, resp.StatusCode)
// wait for webhook to be sent before shutting down
assert.Eventually(t, func() bool {
return webhookSent.Load() == int32(1)
}, 30*time.Second, 10*time.Millisecond, "webhook was not sent")
}
func TestGetDump(t *testing.T) {
dump := getDump()
assert.True(t, strings.Contains(dump, "goroutine"))
assert.True(t, strings.Contains(dump, "[running]"))
assert.True(t, strings.Contains(dump, "backend/app/main.go"))
assert.Contains(t, dump, "goroutine")
assert.Contains(t, dump, "[running]")
assert.Contains(t, dump, "backend/app/main.go")
t.Logf("\n dump: %s", dump)
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
break
}
}
// chooseUnusedPort asks the kernel for a free port from the ephemeral range, which makes a
// collision between concurrently running package test binaries very unlikely
func chooseUnusedPort(t *testing.T) int {
t.Helper()
ln, err := net.Listen("tcp", ":0")
require.NoError(t, err, "no free port available")
port := ln.Addr().(*net.TCPAddr).Port
require.NoError(t, ln.Close())
return port
}
func waitForHTTPServerStart(port int) {
// wait for up to 10 seconds for server to start before returning it
// waitForHTTPServerStart blocks until the server on port answers, failing the test naming the
// port if it never does
func waitForHTTPServerStart(t *testing.T, port int) {
t.Helper()
client := http.Client{Timeout: time.Second}
defer client.CloseIdleConnections()
for i := 0; i < 100; i++ {
time.Sleep(time.Millisecond * 100)
if resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port)); err == nil {
_ = resp.Body.Close()
return
require.Eventually(t, func() bool {
resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port))
if err != nil {
return false
}
}
_ = resp.Body.Close()
return true
}, 30*time.Second, 10*time.Millisecond, "http server on port %d didn't start", port)
}
func TestMain(m *testing.M) {
// both ignores are for leaks which are detected locally
goleak.VerifyTestMain(
m,
goleak.IgnoreTopFunction("github.com/umputun/remark42/backend/app.init.0.func1"),
// the shutdown goroutine in serverApp.run is not joined by Wait, and Rest.Shutdown gives
// httpServer.Shutdown a second, which can outlast goleak's retry budget on a loaded runner
goleak.IgnoreTopFunction("net/http.(*Server).Shutdown"),
goleak.IgnoreTopFunction("github.com/umputun/remark42/backend/app.init.0.func1"),
// this will be fixed in https://github.com/hashicorp/golang-lru/issues/159
goleak.IgnoreTopFunction("github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1"),
// regexp2, pulled in by chroma for syntax highlighting, keeps one shared clock goroutine
// alive for up to a second after the last match with a timeout, sleeping in 100ms ticks.
// it ends on its own, but a binary that finishes inside that window is reported as leaking
goleak.IgnoreAnyFunction("github.com/dlclark/regexp2/v2.runClock"),
)
}
+33 -15
View File
@@ -1,11 +1,13 @@
package migrator
import (
"compress/gzip"
"context"
"fmt"
"io"
"os"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -49,9 +51,7 @@ func TestBackup_MakeBackup(t *testing.T) {
expFile := fmt.Sprintf("/tmp/remark-backups.test/backup-site1-%s.gz", time.Now().Format("20060102"))
assert.Equal(t, expFile, fname)
fi, err := os.Lstat(expFile)
assert.NoError(t, err)
assert.Equal(t, int64(52), fi.Size())
assert.Equal(t, exportedPayload, gzContent(t, expFile))
}
func TestBackup_Do(t *testing.T) {
@@ -59,24 +59,42 @@ func TestBackup_Do(t *testing.T) {
defer os.RemoveAll(loc)
assert.NoError(t, os.MkdirAll(loc, 0o700))
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(time.Second)
cancel()
}()
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(time.Second)
cancel()
}()
bk := AutoBackup{BackupLocation: loc, SiteID: "site1", KeepMax: 3, Exporter: &mockExporter{}, Duration: 600 * time.Millisecond}
bk.Do(ctx)
bk := AutoBackup{BackupLocation: loc, SiteID: "site1", KeepMax: 3, Exporter: &mockExporter{}, Duration: 600 * time.Millisecond}
bk.Do(ctx)
expFile := fmt.Sprintf("/tmp/remark-backups.test/backup-site1-%s.gz", time.Now().Format("20060102"))
fi, err := os.Lstat(expFile)
assert.NoError(t, err)
assert.Equal(t, int64(52), fi.Size())
expFile := fmt.Sprintf("/tmp/remark-backups.test/backup-site1-%s.gz", time.Now().Format("20060102"))
assert.Equal(t, exportedPayload, gzContent(t, expFile))
})
}
const exportedPayload = "some export blah blah 1234567890"
// the compressed size is not assertable: it moves with the compress/flate version
func gzContent(t *testing.T, name string) string {
t.Helper()
fh, err := os.Open(name) //nolint:gosec // path is built by the test
require.NoError(t, err)
defer func() { assert.NoError(t, fh.Close()) }()
gz, err := gzip.NewReader(fh)
require.NoError(t, err)
defer func() { assert.NoError(t, gz.Close()) }()
b, err := io.ReadAll(gz)
require.NoError(t, err)
return string(b)
}
type mockExporter struct{}
func (mock *mockExporter) Export(w io.Writer, _ string) (int, error) {
_, err := w.Write([]byte("some export blah blah 1234567890"))
_, err := w.Write([]byte(exportedPayload))
return 1000, err
}
+21 -3
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"io"
"net/url"
"time"
"github.com/umputun/remark42/backend/app/store"
@@ -47,7 +48,7 @@ type commentoCommenter struct {
Link string `json:"link"`
Photo string `json:"photo"`
Provider string `json:"provider,omitempty"`
JoinDate time.Time `json:"joinDate,omitempty"`
JoinDate time.Time `json:"joinDate"`
IsModerator bool `json:"isModerator"`
}
@@ -100,6 +101,11 @@ func (d *Commento) convert(r io.Reader, siteID string) (ch chan store.Comment) {
}
}
usersMap["anonymous"] = store.User{
Name: "Anonymous",
ID: "commento_" + store.EncodeID("anonymous"),
}
for _, comment := range exportedData.Comments {
u, ok := usersMap[comment.CommenterHex]
if !ok {
@@ -110,16 +116,28 @@ func (d *Commento) convert(r io.Reader, siteID string) (ch chan store.Comment) {
continue
}
parentID := comment.ParentHex
// comments with ParentHex == "root" are top-level comments
if parentID == "root" {
parentID = ""
}
commentURL, e := url.JoinPath("https://", comment.Domain, comment.Path)
if e != nil {
log.Printf("[WARN] can't construct comment URL in commento import, %s", err.Error())
}
log.Printf("[ERROR] commentoURL: %s", commentURL)
c := store.Comment{
ID: comment.CommentHex,
Locator: store.Locator{
URL: comment.Path,
URL: commentURL,
SiteID: siteID,
},
User: u,
Text: comment.Markdown,
Timestamp: comment.CreationDate,
ParentID: comment.ParentHex,
ParentID: parentID,
Imported: true,
}
+16 -3
View File
@@ -27,11 +27,11 @@ func TestCommento_Import(t *testing.T) {
require.NoError(t, err)
size, err := d.Import(fh, "test")
assert.NoError(t, err)
assert.Equal(t, 2, size)
assert.Equal(t, 3, size)
last, err := dataStore.Last("test", 10, time.Time{}, adminUser)
assert.NoError(t, err)
require.Equal(t, 2, len(last), "2 comments imported")
require.Equal(t, 3, len(last), "3 comments imported")
t.Log(last[0])
@@ -44,11 +44,24 @@ func TestCommento_Import(t *testing.T) {
assert.Equal(t, "commento_35369aeb6ac5255de30410a0f86dc71eb9c6d0ca", c.User.ID)
assert.True(t, c.Imported)
c = last[2] // anonymous comment
assert.Equal(t, "Example comment created by user.", c.Text)
assert.Equal(t, "e7069a7dfcfaed43caf62300a9b0edb1c124ad79d0f5887c93649c15d7f69945", c.ID)
assert.Equal(t, "", c.ParentID)
assert.Equal(t, store.Locator{SiteID: "test", URL: "https://example.com/blog/post/2"}, c.Locator)
assert.Equal(t, "Anonymous", c.User.Name)
assert.Equal(t, "commento_0a92fab3230134cca6eadd9898325b9b2ae67998", c.User.ID)
assert.True(t, c.Imported)
posts, err := dataStore.List("test", 0, 0)
assert.NoError(t, err)
assert.Equal(t, 1, len(posts), "1 post")
assert.Equal(t, 2, len(posts), "2 posts")
count, err := dataStore.Count(store.Locator{SiteID: "test", URL: "https://example.com/blog/post/1"})
assert.NoError(t, err)
assert.Equal(t, 2, count)
count, err = dataStore.Count(store.Locator{SiteID: "test", URL: "https://example.com/blog/post/2"})
assert.NoError(t, err)
assert.Equal(t, 1, count)
}
+2 -2
View File
@@ -175,7 +175,7 @@ func (d *Disqus) convert(r io.Reader, siteID string) (ch chan store.Comment) {
func (*Disqus) cleanText(text string) string {
text = strings.TrimSpace(text)
text = strings.Replace(text, "\n", "", -1)
text = strings.Replace(text, "\t", "", -1)
text = strings.ReplaceAll(text, "\n", "")
text = strings.ReplaceAll(text, "\t", "")
return text
}
+1 -1
View File
@@ -122,7 +122,7 @@ func TestDisqus_Convert(t *testing.T) {
require.NoError(t, err)
ch := d.convert(fh, "test")
res := []store.Comment{}
res := make([]store.Comment, 0, 4)
for comment := range ch {
res = append(res, comment)
}
+3 -3
View File
@@ -38,7 +38,7 @@ func (u *URLMapper) loadRules(reader io.Reader) error {
u.rules = make(map[string]string)
for _, row := range strings.Split(rulesText, "\n") {
for row := range strings.SplitSeq(rulesText, "\n") {
row = strings.TrimSpace(row)
urls := strings.Split(row, " ")
if len(urls) != 2 {
@@ -64,8 +64,8 @@ func (u *URLMapper) URL(url string) string {
}
oldURL = strings.TrimSuffix(oldURL, "*")
newURL = strings.TrimSuffix(newURL, "*")
if strings.HasPrefix(url, oldURL) {
return newURL + strings.TrimPrefix(url, oldURL)
if after, ok := strings.CutPrefix(url, oldURL); ok {
return newURL + after
}
}
// search failed, return given url
+2 -2
View File
@@ -77,11 +77,11 @@ func TestMigrator_ImportCommento(t *testing.T) {
Provider: "commento",
})
assert.NoError(t, err)
assert.Equal(t, 2, size)
assert.Equal(t, 3, size)
last, err := dataStore.Last("test", 10, time.Time{}, store.User{})
assert.NoError(t, err)
assert.Equal(t, 2, len(last), "2 comments imported")
assert.Equal(t, 3, len(last), "3 comments imported")
}
func TestMigrator_ImportNative(t *testing.T) {
+3 -2
View File
@@ -6,6 +6,7 @@ import (
"encoding/json"
"fmt"
"io"
"slices"
"sync/atomic"
log "github.com/go-pkgz/lgr"
@@ -46,8 +47,8 @@ func (n *Native) Export(w io.Writer, siteID string) (size int, err error) {
log.Printf("[DEBUG] exporting %d topics", len(topics))
commentsCount := 0
for i := len(topics) - 1; i >= 0; i-- { // topics from List sorted in opposite direction
topic := topics[i]
for _, topic := range slices.Backward(topics) { // topics from List sorted in opposite direction
comments, e := n.DataStore.Find(store.Locator{SiteID: siteID, URL: topic.URL}, "time", adminUser)
if e != nil {
return commentsCount, e
+1 -1
View File
@@ -162,7 +162,7 @@ func TestNative_ImportManyWithError(t *testing.T) {
buf := &bytes.Buffer{}
buf.WriteString(`{"version":1, "users":[], "posts":[]}` + "\n")
for i := 0; i < 100; i++ {
for i := range 100 {
fmt.Fprintf(buf, goodRec, i)
}
buf.WriteString("{}\n")
+3 -3
View File
@@ -4,7 +4,7 @@
{
"commentHex": "e7069a7dfcfaed43caf62300a9b0edb1c124ad79d0f5887c93649c15d7f69945",
"domain": "example.com",
"url": "https://example.com/blog/post/1",
"url": "/blog/post/2",
"commenterHex": "anonymous",
"markdown": "Example comment created by user.",
"html": "",
@@ -18,7 +18,7 @@
{
"commentHex": "7d77e39fcd813241d6281478cc8f21ab5f807d043c750bc1a936bc23b34fb854",
"domain": "example.com",
"url": "https://example.com/blog/post/1",
"url": "/blog/post/1",
"commenterHex": "a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"markdown": "Example 2 comment created by user.",
"html": "",
@@ -32,7 +32,7 @@
{
"commentHex": "ea5f7bcd6ac9bb7b657f7d0569831104e1bcf9c253d03c1e16bf9654c49a5ce9",
"domain": "example.com",
"url": "https://example.com/blog/post/1",
"url": "/blog/post/1",
"commenterHex": "bd1290ab5c858cf2a05903c2a9a61fd63399c6635db38cc6597002195e22e061",
"markdown": "Great reply!",
"html": "",
+3 -2
View File
@@ -16,7 +16,8 @@ const wpTimeLayout = "2006-01-02 15:04:05"
// WordPress implements Importer from WP xml
type WordPress struct {
DataStore Store
DataStore Store
DisableFancyTextFormatting bool
}
type wpItem struct {
@@ -138,7 +139,7 @@ func (w *WordPress) convert(r io.Reader, siteID string) chan store.Comment {
ParentID: comment.PID,
Imported: true,
}
commentsCh <- commentFormatter.Format(c)
commentsCh <- commentFormatter.Format(c, w.DisableFancyTextFormatting)
stats.inpComments++
if stats.inpComments%1000 == 0 {
log.Printf("[DEBUG] processed %d comments", stats.inpComments)
+17 -5
View File
@@ -24,7 +24,7 @@ func TestWordPress_Import(t *testing.T) {
dataStore := service.DataStore{Engine: b, AdminStore: admin.NewStaticStore("12345", nil, []string{}, "")}
defer dataStore.Close()
wp := WordPress{DataStore: &dataStore}
wp := WordPress{DataStore: &dataStore, DisableFancyTextFormatting: false}
size, err := wp.Import(strings.NewReader(xmlTestWP), siteID)
assert.NoError(t, err)
assert.Equal(t, 3, size)
@@ -41,7 +41,7 @@ func TestWordPress_Import(t *testing.T) {
assert.Equal(t, "e8b1e92bbcf5b9bb88472f9bdb82d1b8c7ed39d6", c.User.IP)
ts, _ := time.Parse(wpTimeLayout, "2010-08-18 15:19:14")
assert.Equal(t, ts, c.Timestamp)
assert.Equal(t, c.Text, "<p>Mekkatorque was over in that tent up to the right</p>\n")
assert.Equal(t, "<p>«Mekkatorque» was over in that tent up to the right</p>\n", c.Text)
assert.True(t, c.Imported)
posts, err := dataStore.List(siteID, 0, 0)
@@ -54,13 +54,25 @@ func TestWordPress_Import(t *testing.T) {
count, err := dataStore.Count(store.Locator{URL: "https://realmenweardress.es/2010/07/do-you-rp/", SiteID: siteID})
assert.NoError(t, err)
assert.Equal(t, 3, count)
// test with DisableFancyTextFormatting
wp = WordPress{DataStore: &dataStore, DisableFancyTextFormatting: true}
size, err = wp.Import(strings.NewReader(xmlTestWP), siteID)
assert.NoError(t, err)
assert.Equal(t, 3, size)
last, err = dataStore.Last(siteID, 10, time.Time{}, adminUser)
assert.NoError(t, err)
require.Equal(t, 3, len(last), "3 comments imported")
assert.Equal(t, "<p>&#34;Mekkatorque&#34; was over in that tent up to the right</p>\n", last[0].Text)
}
func TestWordPress_Convert(t *testing.T) {
wp := WordPress{}
ch := wp.convert(strings.NewReader(xmlTestWP), "testWP")
comments := []store.Comment{}
comments := make([]store.Comment, 0, 3)
for c := range ch {
comments = append(comments, c)
}
@@ -88,7 +100,7 @@ func TestWP_Convert_MD(t *testing.T) {
wp := WordPress{}
ch := wp.convert(strings.NewReader(xmlTestWPmd), "siteID")
comments := []store.Comment{}
comments := make([]store.Comment, 0, 3)
for c := range ch {
comments = append(comments, c)
}
@@ -247,7 +259,7 @@ var xmlTestWP = `
<wp:comment_author_IP><![CDATA[128.243.253.117]]></wp:comment_author_IP>
<wp:comment_date><![CDATA[2010-08-18 15:19:14]]></wp:comment_date>
<wp:comment_date_gmt><![CDATA[2010-08-18 15:19:14]]></wp:comment_date_gmt>
<wp:comment_content><![CDATA[Mekkatorque was over in that tent up to the right]]></wp:comment_content>
<wp:comment_content><![CDATA["Mekkatorque" was over in that tent up to the right]]></wp:comment_content>
<wp:comment_approved><![CDATA[1]]></wp:comment_approved>
<wp:comment_type><![CDATA[]]></wp:comment_type>
<wp:comment_parent>13</wp:comment_parent>
+41 -16
View File
@@ -3,15 +3,16 @@ package notify
import (
"bytes"
"context"
"errors"
"fmt"
"html/template"
"net/url"
"text/template"
"time"
log "github.com/go-pkgz/lgr"
ntf "github.com/go-pkgz/notify"
"github.com/go-pkgz/repeater"
"github.com/hashicorp/go-multierror"
"github.com/go-pkgz/repeater/v2"
"github.com/microcosm-cc/bluemonday"
"github.com/umputun/remark42/backend/app/templates"
)
@@ -26,7 +27,7 @@ type EmailParams struct {
SubscribeURL string // full subscribe handler URL
UnsubscribeURL string // full unsubscribe handler URL
TokenGenFn func(userID, email, site string) (string, error) // Unsubscribe token generation function
TokenGenFn func(userID, email, site string) (string, error) // unsubscribe token generation function
}
// Email implements notify.Destination for email
@@ -42,12 +43,12 @@ type Email struct {
type msgTmplData struct {
UserName string
UserPicture string
CommentText string
CommentText template.HTML
CommentLink string
CommentDate time.Time
ParentUserName string
ParentUserPicture string
ParentCommentText string
ParentCommentText template.HTML
ParentCommentLink string
ParentCommentDate time.Time
PostTitle string
@@ -56,6 +57,30 @@ type msgTmplData struct {
ForAdmin bool
}
// emailCommentPolicy sanitizes comment HTML for inclusion in notification emails.
// It is intentionally stricter than the store-level UGC policy used for web rendering:
// links (<a>) and images (<img>) are dropped so a comment can't smuggle phishing links
// or remote tracking pixels into an email sent from the legitimate remark42 address,
// while basic inline and block text formatting is preserved.
var emailCommentPolicy = func() *bluemonday.Policy {
p := bluemonday.NewPolicy()
p.AllowElements(
"p", "br", "hr", "div", "span",
"b", "strong", "i", "em", "u", "s", "strike", "del", "ins", "sub", "sup", "mark", "small",
"blockquote", "q", "cite",
"code", "pre", "kbd", "samp", "var",
"ul", "ol", "li", "dl", "dt", "dd",
"h1", "h2", "h3", "h4", "h5", "h6",
)
return p
}()
// emailSafeHTML strips links and images from pre-rendered comment HTML and returns
// it as template.HTML so html/template renders the remaining safe formatting as-is.
func emailSafeHTML(commentHTML string) template.HTML {
return template.HTML(emailCommentPolicy.Sanitize(commentHTML)) //nolint:gosec // sanitized above: <a>/<img> dropped, only formatting tags survive
}
// verifyTmplData store data for verification message template execution
type verifyTmplData struct {
User string
@@ -135,23 +160,23 @@ func (e *Email) Send(ctx context.Context, req Request) error {
default:
}
result := new(multierror.Error)
var errs []error
for _, email := range req.Emails {
err := e.buildAndSendMessage(ctx, req, email, false)
if err != nil {
result = multierror.Append(fmt.Errorf("problem sending user email notification to %q: %w", email, err))
errs = append(errs, fmt.Errorf("problem sending user email notification to %q: %w", email, err))
}
}
for _, email := range e.AdminEmails {
err := e.buildAndSendMessage(ctx, req, email, true)
if err != nil {
result = multierror.Append(fmt.Errorf("problem sending admin email notification to %q: %w", email, err))
errs = append(errs, fmt.Errorf("problem sending admin email notification to %q: %w", email, err))
}
}
return result.ErrorOrNil()
return errors.Join(errs...)
}
func (e *Email) buildAndSendMessage(ctx context.Context, req Request, email string, forAdmin bool) error {
@@ -161,14 +186,14 @@ func (e *Email) buildAndSendMessage(ctx context.Context, req Request, email stri
return err
}
return repeater.NewDefault(5, time.Millisecond*250).Do(
return repeater.NewFixed(5, time.Millisecond*250).Do(
ctx,
func() error {
return e.Email.Send(
ctx,
fmt.Sprintf("mailto:%s?from=%s&unsubscribeLink=%s&subject=%s",
email,
e.From,
url.QueryEscape(e.From),
url.QueryEscape(msg.unsubscribeLink),
url.QueryEscape(msg.subject),
),
@@ -196,14 +221,14 @@ func (e *Email) SendVerification(ctx context.Context, req VerificationRequest) e
return err
}
return repeater.NewDefault(5, time.Millisecond*250).Do(
return repeater.NewFixed(5, time.Millisecond*250).Do(
ctx,
func() error {
return e.Email.Send(
ctx,
fmt.Sprintf("mailto:%s?from=%s&subject=%s",
req.Email,
e.From,
url.QueryEscape(e.From),
url.QueryEscape(e.VerificationSubject),
),
msg,
@@ -257,7 +282,7 @@ func (e *Email) buildMessageFromRequest(req Request, email string, forAdmin bool
tmplData := msgTmplData{
UserName: req.Comment.User.Name,
UserPicture: req.Comment.User.Picture,
CommentText: req.Comment.Text,
CommentText: emailSafeHTML(req.Comment.Text),
CommentLink: commentURLPrefix + req.Comment.ID,
CommentDate: req.Comment.Timestamp,
PostTitle: req.Comment.PostTitle,
@@ -269,7 +294,7 @@ func (e *Email) buildMessageFromRequest(req Request, email string, forAdmin bool
if req.Comment.ParentID != "" {
tmplData.ParentUserName = req.parent.User.Name
tmplData.ParentUserPicture = req.parent.User.Picture
tmplData.ParentCommentText = req.parent.Text
tmplData.ParentCommentText = emailSafeHTML(req.parent.Text)
tmplData.ParentCommentLink = commentURLPrefix + req.parent.ID
tmplData.ParentCommentDate = req.parent.Timestamp
}
+57 -8
View File
@@ -3,8 +3,8 @@ package notify
import (
"context"
"fmt"
"html/template"
"testing"
"text/template"
ntf "github.com/go-pkgz/notify"
"github.com/stretchr/testify/assert"
@@ -34,7 +34,7 @@ func TestEmailNew(t *testing.T) {
assert.NotNil(t, email, "email returned")
assert.NotNil(t, email.msgTmpl, "e.template is set")
assert.Equal(t, emailParams.From, email.EmailParams.From, "emailParams.From unchanged after creation")
assert.Equal(t, emailParams.From, email.From, "emailParams.From unchanged after creation")
if smtpParams.TimeOut == 0 {
assert.Equal(t, defaultEmailTimeout, email.TimeOut, "empty emailParams.TimeOut changed to default")
} else {
@@ -88,7 +88,6 @@ func Test_initTemplatesErr(t *testing.T) {
}
for _, d := range testSet {
d := d
t.Run(d.name, func(t *testing.T) {
e, err := NewEmail(d.emailParams, ntf.SMTPParams{})
require.Error(t, err)
@@ -111,10 +110,10 @@ func TestEmailSendErrors(t *testing.T) {
e.msgTmpl, err = template.New("test").Parse("{{.Test}}")
assert.NoError(t, err)
assert.EqualError(t, e.Send(context.Background(), Request{Comment: store.Comment{ID: "999"}, parent: store.Comment{User: store.User{ID: "test"}}, Emails: []string{"bad@example.org"}}),
"1 error occurred:\n\t* problem sending user email notification to \"bad@example.org\": "+
"problem sending user email notification to \"bad@example.org\": "+
"error executing template to build comment reply message: "+
"template: test:1:2: executing \"test\" at <.Test>: "+
"can't evaluate field Test in type notify.msgTmplData\n\n")
"can't evaluate field Test in type notify.msgTmplData")
ctx, cancel := context.WithCancel(context.Background())
cancel()
@@ -122,8 +121,14 @@ func TestEmailSendErrors(t *testing.T) {
"sending email messages about comment \"999\" aborted due to canceled context")
assert.EqualError(t, e.Send(context.Background(), Request{Comment: store.Comment{ID: "999"}, parent: store.Comment{User: store.User{ID: "error"}}, Emails: []string{"bad@example.org"}}),
"1 error occurred:\n\t* problem sending user email notification to \"bad@example.org\":"+
" error creating token for unsubscribe link: token generation error\n\n")
"problem sending user email notification to \"bad@example.org\":"+
" error creating token for unsubscribe link: token generation error")
// errors for all failed recipients are reported, not just the last one
assert.EqualError(t, e.Send(context.Background(),
Request{Comment: store.Comment{ID: "999"}, parent: store.Comment{User: store.User{ID: "error"}}, Emails: []string{"bad1@example.org", "bad2@example.org"}}),
"problem sending user email notification to \"bad1@example.org\": error creating token for unsubscribe link: token generation error\n"+
"problem sending user email notification to \"bad2@example.org\": error creating token for unsubscribe link: token generation error")
}
func TestEmailSend_ExitConditions(t *testing.T) {
@@ -165,7 +170,7 @@ User: test_user
01.01.0001 at 00:00
Comment:
test@example.org for parent_user
Unsubscribe link: https://remark42.com/api/v1/email/unsubscribe?site=&tkn=token
Unsubscribe link: https://remark42.com/api/v1/email/unsubscribe?site=&amp;tkn=token
`, msg.body)
assert.Equal(t, "https://remark42.com/api/v1/email/unsubscribe?site=&tkn=token", msg.unsubscribeLink)
assert.Equal(t, `New reply to your comment for "test_title"`, msg.subject)
@@ -191,6 +196,50 @@ admin@example.org
assert.Empty(t, msg.unsubscribeLink)
}
func TestEmail_CommentTextSanitizedForEmail(t *testing.T) {
// comment HTML reaching the email path is sanitized by the store-level UGC policy,
// which permits <a> and <img>. The email must drop both so a comment can't inject
// phishing links or remote tracking pixels into a notification (GHSA-74pc-3r2m-ppx3).
email, err := NewEmail(EmailParams{
From: "from@example.org",
MsgTemplatePath: "testdata/msg.html.tmpl",
}, ntf.SMTPParams{})
require.NoError(t, err)
email.TokenGenFn = TokenGenFn
malicious := `hello <a href="https://phishing.example/verify">click to verify</a>` +
` <img src="https://attacker.example/track.png" width="1" height="1"> <b>kept</b>`
req := Request{
Comment: store.Comment{ID: "999", User: store.User{ID: "1", Name: "test_user"}, PostTitle: "test_title", Text: malicious},
Emails: []string{"test@example.org"},
}
msg, err := email.buildMessageFromRequest(req, req.Emails[0], false)
require.NoError(t, err)
assert.NotContains(t, msg.body, "phishing.example", "phishing link must be stripped")
assert.NotContains(t, msg.body, "attacker.example", "tracking pixel must be stripped")
assert.NotContains(t, msg.body, "<img", "no image tags in email body")
assert.NotContains(t, msg.body, "<a ", "no anchor tags in email body")
assert.Contains(t, msg.body, "click to verify", "anchor text is preserved, only the link is dropped")
assert.Contains(t, msg.body, "<b>kept</b>", "basic formatting is preserved")
}
// emailSafeHTML drops links/images while keeping inline/block formatting and escaping nothing extra.
func TestEmailSafeHTML(t *testing.T) {
tbl := []struct{ name, in, want string }{
{"strips anchor keeps text", `<a href="http://evil">x</a>`, "x"},
{"strips image entirely", `a<img src="http://evil/t.png">b`, "ab"},
{"keeps bold/italic/code", `<b>b</b><i>i</i><code>c</code>`, `<b>b</b><i>i</i><code>c</code>`},
{"keeps blockquote and lists", `<blockquote>q</blockquote><ul><li>x</li></ul>`, `<blockquote>q</blockquote><ul><li>x</li></ul>`},
{"drops onclick handlers", `<span onclick="alert(1)">s</span>`, `<span>s</span>`},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, string(emailSafeHTML(tt.in)))
})
}
}
func TestEmail_SendVerification(t *testing.T) {
email, err := NewEmail(EmailParams{
From: "from@example.org",
+4 -4
View File
@@ -19,7 +19,7 @@ type Service struct {
queue chan Request
verificationQueue chan VerificationRequest
closed uint32 // non-zero means closed. uses uint instead of bool for atomic
closed atomic.Uint32 // non-zero means closed. uses uint instead of bool for atomic
ctx context.Context
cancel context.CancelFunc
}
@@ -83,7 +83,7 @@ func NewService(dataService Store, size int, destinations ...Destination) *Servi
// Submit Request to internal channel if not busy, drop if can't send
func (s *Service) Submit(req Request) {
if len(s.destinations) == 0 || atomic.LoadUint32(&s.closed) != 0 {
if len(s.destinations) == 0 || s.closed.Load() != 0 {
return
}
if s.dataService != nil && req.Comment.ParentID != "" {
@@ -130,7 +130,7 @@ func (s *Service) getNotificationTargets(
// SubmitVerification to internal channel if not busy, drop if can't send
func (s *Service) SubmitVerification(req VerificationRequest) {
if len(s.destinations) == 0 || atomic.LoadUint32(&s.closed) != 0 {
if len(s.destinations) == 0 || s.closed.Load() != 0 {
return
}
select {
@@ -155,7 +155,7 @@ func (s *Service) Close() {
s.cancel()
<-s.ctx.Done()
}
atomic.StoreUint32(&s.closed, 1)
s.closed.Store(1)
}
func (s *Service) do() {
+27 -12
View File
@@ -4,7 +4,6 @@ import (
"context"
"fmt"
"sync"
"time"
log "github.com/go-pkgz/lgr"
)
@@ -16,35 +15,40 @@ type MockDest struct {
id int
closed bool
lock sync.Mutex
block chan struct{} // if non-nil, Send/SendVerification wait on it before recording, letting tests pin the consumer
}
// Send mock
func (m *MockDest) Send(ctx context.Context, r Request) error {
if m.block != nil {
<-m.block
}
m.lock.Lock()
defer m.lock.Unlock()
select {
case <-time.After(10 * time.Millisecond):
m.data = append(m.data, r)
log.Printf("sent %s -> %d", r.Comment.ID, m.id)
case <-ctx.Done():
if err := ctx.Err(); err != nil {
log.Printf("ctx closed %d", m.id)
m.closed = true
return nil
}
m.data = append(m.data, r)
log.Printf("sent %s -> %d", r.Comment.ID, m.id)
return nil
}
// SendVerification mock
func (m *MockDest) SendVerification(ctx context.Context, v VerificationRequest) error {
if m.block != nil {
<-m.block
}
m.lock.Lock()
defer m.lock.Unlock()
select {
case <-time.After(10 * time.Millisecond):
m.verificationData = append(m.verificationData, v)
log.Printf("sent verification %s -> %d", v.User, m.id)
case <-ctx.Done():
if err := ctx.Err(); err != nil {
log.Printf("verification ctx closed %d", m.id)
m.closed = true
return nil
}
m.verificationData = append(m.verificationData, v)
log.Printf("sent verification %s -> %d", v.User, m.id)
return nil
}
@@ -66,4 +70,15 @@ func (m *MockDest) GetVerify() []VerificationRequest {
return res
}
func (m *MockDest) String() string { return fmt.Sprintf("mock id=%d, closed=%v", m.id, m.closed) }
// IsClosed returns closed status safely
func (m *MockDest) IsClosed() bool {
m.lock.Lock()
defer m.lock.Unlock()
return m.closed
}
func (m *MockDest) String() string {
m.lock.Lock()
defer m.lock.Unlock()
return fmt.Sprintf("mock id=%d, closed=%v", m.id, m.closed)
}
+213 -190
View File
@@ -2,10 +2,8 @@ package notify
import (
"fmt"
"math/rand"
"sync/atomic"
"testing"
"time"
"testing/synctest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -26,255 +24,280 @@ func TestService_NoDestinations(t *testing.T) {
}
func TestService_WithDestinations(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
s.Submit(Request{Comment: store.Comment{ID: "100"}})
time.Sleep(time.Millisecond * 110)
s.Submit(Request{Comment: store.Comment{ID: "101"}})
time.Sleep(time.Millisecond * 110)
s.Submit(Request{Comment: store.Comment{ID: "102"}})
time.Sleep(time.Millisecond * 110)
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "100"}})
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "101"}})
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "102"}})
synctest.Wait()
s.Close()
require.Equal(t, 3, len(d1.Get()), "got all comments to d1")
require.Equal(t, 3, len(d2.Get()), "got all comments to d2")
require.Equal(t, 3, len(d1.Get()), "got all comments to d1")
require.Equal(t, 3, len(d2.Get()), "got all comments to d2")
assert.Equal(t, "100", d1.Get()[0].Comment.ID)
assert.Equal(t, "101", d1.Get()[1].Comment.ID)
assert.Equal(t, "102", d1.Get()[2].Comment.ID)
assert.Equal(t, "100", d1.Get()[0].Comment.ID)
assert.Equal(t, "101", d1.Get()[1].Comment.ID)
assert.Equal(t, "102", d1.Get()[2].Comment.ID)
})
}
func TestService_WithDrops(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
// gated destinations pin the consumer on the first item so the size-1 queue
// fills deterministically and the overflow is dropped regardless of scheduling
gate := make(chan struct{})
d1, d2 := &MockDest{id: 1, block: gate}, &MockDest{id: 2, block: gate}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
s.Submit(Request{Comment: store.Comment{ID: "100"}})
s.Submit(Request{Comment: store.Comment{ID: "101"}})
s.Submit(Request{Comment: store.Comment{ID: "102"}})
time.Sleep(time.Millisecond * 21)
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "100"}}) // consumed, consumer blocks in Send on the gate
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "101"}}) // fills the size-1 queue
s.Submit(Request{Comment: store.Comment{ID: "102"}}) // queue full, dropped
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "111"}}) // safe to send after close
close(gate) // release the consumer: it finishes 100 then processes 101
synctest.Wait()
s.Close()
assert.LessOrEqual(t, len(d1.Get()), 2, "at least one comment from three dropped from d1, got: %v", d1.Get())
assert.LessOrEqual(t, len(d2.Get()), 2, "at least one comment from three dropped from d2, got: %v", d2.Get())
s.Submit(Request{Comment: store.Comment{ID: "111"}}) // safe to send after close
require.Len(t, d1.Get(), 2, "one comment of three dropped from d1, got: %v", d1.Get())
require.Len(t, d2.Get(), 2, "one comment of three dropped from d2, got: %v", d2.Get())
assert.Equal(t, "100", d1.Get()[0].Comment.ID)
assert.Equal(t, "101", d1.Get()[1].Comment.ID)
})
}
func TestService_SubmitVerificationWithDrops(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
// gated destinations pin the consumer on the first item so the size-1 queue
// fills deterministically and the overflow is dropped regardless of scheduling
gate := make(chan struct{})
d1, d2 := &MockDest{id: 1, block: gate}, &MockDest{id: 2, block: gate}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
s.SubmitVerification(VerificationRequest{
SiteID: "remark",
User: "testUser",
Email: "test@example.org",
Token: "testToken",
s.SubmitVerification(VerificationRequest{
SiteID: "remark",
User: "testUser",
Email: "test@example.org",
Token: "testToken",
}) // consumed, consumer blocks in SendVerification on the gate
synctest.Wait()
s.SubmitVerification(VerificationRequest{User: "second"}) // fills the size-1 queue
s.SubmitVerification(VerificationRequest{User: "dropped"}) // queue full, dropped
synctest.Wait()
close(gate) // release the consumer: it finishes testUser then processes second
synctest.Wait()
s.Close()
s.SubmitVerification(VerificationRequest{}) // safe to send after close
require.Len(t, d2.GetVerify(), 2, "one request of three dropped from d2, got: %v", d2.GetVerify())
verifyDest := d1.GetVerify()
require.Len(t, verifyDest, 2, "one request of three dropped from d1, got: %v", verifyDest)
assert.Equal(t, "remark", verifyDest[0].SiteID)
assert.Equal(t, "testUser", verifyDest[0].User)
assert.Equal(t, "test@example.org", verifyDest[0].Email)
assert.Equal(t, "testToken", verifyDest[0].Token)
assert.Equal(t, "second", verifyDest[1].User)
})
s.SubmitVerification(VerificationRequest{})
s.SubmitVerification(VerificationRequest{})
time.Sleep(time.Millisecond * 21)
s.Close()
s.SubmitVerification(VerificationRequest{}) // safe to send after close
assert.LessOrEqual(t, len(d2.GetVerify()), 2, "one request from three dropped from d2, got: %v", d2.GetVerify())
verifyDest := d1.GetVerify()
require.LessOrEqual(t, len(verifyDest), 2, "one request from three dropped from d1, got: %v", verifyDest)
assert.Equal(t, "remark", verifyDest[0].SiteID)
assert.Equal(t, "testUser", verifyDest[0].User)
assert.Equal(t, "test@example.org", verifyDest[0].Email)
assert.Equal(t, "testToken", verifyDest[0].Token)
}
func TestService_Many(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 5, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 5, d1, d2)
assert.NotNil(t, s)
for i := 0; i < 10; i++ {
s.Submit(Request{Comment: store.Comment{ID: fmt.Sprintf("%d", 100+i)}})
s.SubmitVerification(VerificationRequest{User: fmt.Sprintf("%d", 100+i)})
time.Sleep(time.Millisecond * time.Duration(rand.Int31n(20)))
}
s.Close()
time.Sleep(time.Millisecond * 10)
for i := range 10 {
s.Submit(Request{Comment: store.Comment{ID: fmt.Sprintf("%d", 100+i)}})
s.SubmitVerification(VerificationRequest{User: fmt.Sprintf("%d", 100+i)})
}
s.Close()
assert.NotEqual(t, 10, len(d1.Get()), "some comments dropped from d1")
assert.NotEqual(t, 10, len(d1.GetVerify()), "some verifications dropped from d1")
assert.NotEqual(t, 10, len(d2.Get()), "some comments dropped from d2")
assert.NotEqual(t, 10, len(d2.GetVerify()), "some verifications dropped from d2")
assert.True(t, d1.closed)
assert.True(t, d2.closed)
assert.Equal(t, "mock id=1, closed=true", d1.String())
assert.NotEqual(t, 10, len(d1.Get()), "some comments dropped from d1")
assert.NotEqual(t, 10, len(d1.GetVerify()), "some verifications dropped from d1")
assert.NotEqual(t, 10, len(d2.Get()), "some comments dropped from d2")
assert.NotEqual(t, 10, len(d2.GetVerify()), "some verifications dropped from d2")
})
}
func TestService_WithParent(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}}
synctest.Test(t, func(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}}
dataStore.data["p1"] = store.Comment{ID: "p1"}
dataStore.data["p2"] = store.Comment{ID: "p2"}
dataStore.data["p1"] = store.Comment{ID: "p1"}
dataStore.data["p2"] = store.Comment{ID: "p2"}
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s.Submit(Request{Comment: store.Comment{ID: "c1", ParentID: "p1"}})
time.Sleep(time.Millisecond * 110)
s.Submit(Request{Comment: store.Comment{ID: "c11", ParentID: "p11"}})
time.Sleep(time.Millisecond * 110)
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "c1", ParentID: "p1"}})
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "c11", ParentID: "p11"}})
synctest.Wait()
s.Close()
destRes := dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ParentID)
assert.Equal(t, "p1", destRes[0].parent.ID)
assert.Equal(t, "p11", destRes[1].Comment.ParentID)
assert.Equal(t, "", destRes[1].parent.ID)
destRes := dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ParentID)
assert.Equal(t, "p1", destRes[0].parent.ID)
assert.Equal(t, "p11", destRes[1].Comment.ParentID)
assert.Equal(t, "", destRes[1].parent.ID)
})
}
func TestService_EmailRetrieval(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
synctest.Test(t, func(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.userDetails["u1"] = "u1@example.com"
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.userDetails["u1"] = "u1@example.com"
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
// one comment, one notification
s.Submit(Request{Comment: dataStore.data["p1"]})
time.Sleep(time.Millisecond * 110)
// one comment, one notification
s.Submit(Request{Comment: dataStore.data["p1"]})
synctest.Wait()
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
// reply to the first comment, same comment as one in original comment
s.Submit(Request{Comment: dataStore.data["p2"]})
time.Sleep(time.Millisecond * 110)
// reply to the first comment, same comment as one in original comment
s.Submit(Request{Comment: dataStore.data["p2"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.Empty(t, destRes[1].Emails, "u1 is not notified they are the one who left the comment")
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.Empty(t, destRes[1].Emails, "u1 is not notified they are the one who left the comment")
// another reply to the first comment, another user
s.Submit(Request{Comment: dataStore.data["p3"]})
time.Sleep(time.Millisecond * 110)
// another reply to the first comment, another user
s.Submit(Request{Comment: dataStore.data["p3"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p1", destRes[2].parent.ID)
assert.Equal(t, "u1", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p1", destRes[2].parent.ID)
assert.Equal(t, "u1", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
// reply to the last comment by another user, should trigger email retrieval error
s.Submit(Request{Comment: dataStore.data["p4"]})
time.Sleep(time.Millisecond * 110)
// reply to the last comment by another user, should trigger email retrieval error
s.Submit(Request{Comment: dataStore.data["p4"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u2", destRes[3].parent.User.ID)
assert.Empty(t, destRes[3].Emails, "no email can be retrieved for u2")
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u2", destRes[3].parent.User.ID)
assert.Empty(t, destRes[3].Emails, "no email can be retrieved for u2")
s.Close()
s.Close()
})
}
func TestService_Recursive(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
synctest.Test(t, func(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p2", User: store.User{ID: "u3"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.data["p5"] = store.Comment{ID: "p5", ParentID: "p4", User: store.User{ID: "u4"}}
dataStore.userDetails["u1"] = "u1@example.com"
// second comment goes without email address for notification
dataStore.userDetails["u3"] = "u3@example.com"
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p2", User: store.User{ID: "u3"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.data["p5"] = store.Comment{ID: "p5", ParentID: "p4", User: store.User{ID: "u4"}}
dataStore.userDetails["u1"] = "u1@example.com"
// second comment goes without email address for notification
dataStore.userDetails["u3"] = "u3@example.com"
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
// one comment from u1 with email set
s.Submit(Request{Comment: dataStore.data["p1"]})
time.Sleep(time.Millisecond * 110)
// one comment from u1 with email set
s.Submit(Request{Comment: dataStore.data["p1"]})
synctest.Wait()
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
// reply to the first comment from u2 without email set
s.Submit(Request{Comment: dataStore.data["p2"]})
time.Sleep(time.Millisecond * 110)
// reply to the first comment from u2 without email set
s.Submit(Request{Comment: dataStore.data["p2"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[1].Emails)
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[1].Emails)
// reply to the second comment from u3 with email set
s.Submit(Request{Comment: dataStore.data["p3"]})
time.Sleep(time.Millisecond * 110)
// reply to the second comment from u3 with email set
s.Submit(Request{Comment: dataStore.data["p3"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p2", destRes[2].parent.ID)
assert.Equal(t, "u2", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p2", destRes[2].parent.ID)
assert.Equal(t, "u2", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
// reply to the third comment from u1 (author of the first comment), only u3 should be notified
s.Submit(Request{Comment: dataStore.data["p4"]})
time.Sleep(time.Millisecond * 110)
// reply to the third comment from u1 (author of the first comment), only u3 should be notified
s.Submit(Request{Comment: dataStore.data["p4"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified once each")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u3", destRes[3].parent.User.ID)
assert.ElementsMatch(t, []string{"u3@example.com"}, destRes[3].Emails, "u1 is not notified they are the one who left the comment")
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified once each")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u3", destRes[3].parent.User.ID)
assert.ElementsMatch(t, []string{"u3@example.com"}, destRes[3].Emails, "u1 is not notified they are the one who left the comment")
// reply to the fourth comment from u4, u1 and u3 should be notified once as a result
s.Submit(Request{Comment: dataStore.data["p5"]})
time.Sleep(time.Millisecond * 110)
// reply to the fourth comment from u4, u1 and u3 should be notified once as a result
s.Submit(Request{Comment: dataStore.data["p5"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 5, len(destRes), "four comment notified once each")
assert.Equal(t, "p5", destRes[4].Comment.ID)
assert.Equal(t, "p4", destRes[4].parent.ID)
assert.Equal(t, "u1", destRes[4].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com", "u3@example.com"}, destRes[4].Emails, "u3 and u1 notified once")
destRes = dest.Get()
require.Equal(t, 5, len(destRes), "four comment notified once each")
assert.Equal(t, "p5", destRes[4].Comment.ID)
assert.Equal(t, "p4", destRes[4].parent.ID)
assert.Equal(t, "u1", destRes[4].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com", "u3@example.com"}, destRes[4].Emails, "u3 and u1 notified once")
s.Close()
s.Close()
})
}
func TestService_Nop(t *testing.T) {
s := NopService
s.Submit(Request{Comment: store.Comment{}})
s.Close()
assert.Equal(t, uint32(1), atomic.LoadUint32(&s.closed))
assert.Equal(t, uint32(1), s.closed.Load())
}
type mockStore struct {
+77
View File
@@ -0,0 +1,77 @@
package notify
import (
"fmt"
"strings"
"golang.org/x/net/html"
)
// pruneHTML prunes string keeping HTML closing tags.
// maxLength applies to visible text only, not HTML tags.
func pruneHTML(htmlText string, maxLength int) string {
var result strings.Builder
var endTokens []string
visibleLen := 0
suffix := "..."
suffixLen := len(suffix)
tokenizer := html.NewTokenizer(strings.NewReader(htmlText))
for {
if tokenizer.Next() == html.ErrorToken {
return result.String()
}
token := tokenizer.Token()
switch token.Type {
case html.CommentToken, html.DoctypeToken:
continue
case html.StartTagToken:
endTokens = append([]string{fmt.Sprintf("</%s>", token.Data)}, endTokens...)
result.WriteString(token.String())
case html.EndTagToken:
if len(endTokens) > 0 {
endTokens = endTokens[1:]
}
result.WriteString(token.String())
case html.SelfClosingTagToken:
result.WriteString(token.String())
case html.TextToken:
text := token.String()
if visibleLen+len(text)+suffixLen > maxLength {
remaining := maxLength - visibleLen - suffixLen
text = pruneStringToWord(text, remaining)
result.WriteString(text)
result.WriteString(suffix)
for _, endTag := range endTokens {
result.WriteString(endTag)
}
return result.String()
}
visibleLen += len(text)
result.WriteString(text)
}
}
}
// pruneStringToWord prunes string to specified length respecting word boundaries
func pruneStringToWord(text string, maxLength int) string {
if maxLength <= 0 {
return ""
}
if len(text) <= maxLength {
return text
}
// find last space at or before maxLength to cut at word boundary
lastSpace := strings.LastIndex(text[:maxLength+1], " ")
if lastSpace <= 0 {
return ""
}
return text[:lastSpace]
}
+47
View File
@@ -0,0 +1,47 @@
package notify
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestPruneHTML(t *testing.T) {
tests := []struct {
name string
html string
maxLength int
expected string
}{
{"within limit", "<p>Hello</p>", 20, "<p>Hello</p>"},
{"exceeds limit", "<p>Hello world, this is a long text</p>", 15, "<p>Hello world,...</p>"},
{"nested tags", "<div><p>Hello world</p><p>More text</p></div>", 20, "<div><p>Hello world</p><p>More...</p></div>"},
{"html comment stripped", "<!-- comment --><p>Hello</p>", 20, "<p>Hello</p>"},
{"self-closing tag", "<p>Hello<br/>World</p>", 8, "<p>Hello<br/>...</p>"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, pruneHTML(tt.html, tt.maxLength))
})
}
}
func TestPruneStringToWord(t *testing.T) {
tests := []struct {
name string
text string
maxLength int
expected string
}{
{"within limit", "hello world", 15, "hello world"},
{"cut at word boundary", "hello world and more", 11, "hello world"},
{"zero length", "hello", 0, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, pruneStringToWord(tt.text, tt.maxLength))
})
}
}
+9 -7
View File
@@ -2,14 +2,16 @@ package notify
import (
"context"
"errors"
"fmt"
"time"
log "github.com/go-pkgz/lgr"
ntf "github.com/go-pkgz/notify"
"github.com/hashicorp/go-multierror"
)
const commentTextLengthLimit = 100
// TelegramParams contain settings for telegram notifications
type TelegramParams struct {
AdminChannelID string // unique identifier for the target chat or username of the target channel (in the format @channelusername)
@@ -45,14 +47,14 @@ func NewTelegram(params TelegramParams) (*Telegram, error) {
// Send to telegram recipients
func (t *Telegram) Send(ctx context.Context, req Request) error {
log.Printf("[DEBUG] send telegram notification for comment ID %s", req.Comment.ID)
result := new(multierror.Error)
var errs []error
msg := t.buildMessage(req)
if t.AdminChannelID != "" {
err := t.Telegram.Send(ctx, fmt.Sprintf("telegram:%s?parseMode=HTML", t.AdminChannelID), msg)
if err != nil {
result = multierror.Append(result,
errs = append(errs,
fmt.Errorf("problem sending admin telegram notification about comment ID %s to %s: %w",
req.Comment.ID, t.AdminChannelID, err,
),
@@ -64,7 +66,7 @@ func (t *Telegram) Send(ctx context.Context, req Request) error {
for _, user := range req.Telegrams {
err := t.Telegram.Send(ctx, fmt.Sprintf("telegram:%s?parseMode=HTML", user), msg)
if err != nil {
result = multierror.Append(result,
errs = append(errs,
fmt.Errorf("problem sending user telegram notification about comment ID %s to %q: %w",
req.Comment.ID, user, err,
),
@@ -72,7 +74,7 @@ func (t *Telegram) Send(ctx context.Context, req Request) error {
}
}
}
return result.ErrorOrNil()
return errors.Join(errs...)
}
// buildMessage generates message for generic notification about new comment
@@ -85,10 +87,10 @@ func (t *Telegram) buildMessage(req Request) string {
msg += fmt.Sprintf(" -> <a href=%q>%s</a>", commentURLPrefix+req.parent.ID, ntf.EscapeTelegramText(req.parent.User.Name))
}
msg += fmt.Sprintf("\n\n%s", ntf.TelegramSupportedHTML(req.Comment.Text))
msg += fmt.Sprintf("\n\n%s", pruneHTML(ntf.TelegramSupportedHTML(req.Comment.Text), commentTextLengthLimit))
if req.Comment.ParentID != "" {
msg += fmt.Sprintf("\n\n\"<i>%s</i>\"", ntf.TelegramSupportedHTML(req.parent.Text))
msg += fmt.Sprintf("\n\n\"<i>%s</i>\"", pruneHTML(ntf.TelegramSupportedHTML(req.parent.Text), commentTextLengthLimit))
}
if req.Comment.PostTitle != "" {
+9 -1
View File
@@ -30,7 +30,6 @@ func TestTelegram_Send(t *testing.T) {
err := tb.Send(context.Background(), Request{Comment: c, parent: cp, Telegrams: []string{"test_user_channel"}})
assert.Error(t, err)
assert.Contains(t, err.Error(), "2 errors occurred")
assert.Contains(t, err.Error(), "problem sending user telegram notification about comment ID 999 to \"test_user_channel\"")
assert.Contains(t, err.Error(), "problem sending admin telegram notification about comment ID 999 to remark_test")
@@ -53,6 +52,15 @@ some text
<b>Hello</b><i><b>World</b></i>`,
res)
// prune string keeping HTML closing tags
c = store.Comment{
Text: "<b>Lorem ipsum <i>dolor sit amet</i>, consectetur adipiscing <code>elit, sed do eiusmod tempor incididunt</code> ut labore et dolore magna aliqua.</b>",
}
res = tb.buildMessage(Request{Comment: c})
assert.Equal(t, `<a href="#remark42__comment-"></a>
<b>Lorem ipsum <i>dolor sit amet</i>, consectetur adipiscing <code>elit, sed do eiusmod tempor incididunt</code> ut...</b>`, res)
}
func TestTelegram_SendVerification(t *testing.T) {
+12 -2
View File
@@ -3,6 +3,7 @@ package notify
import (
"bytes"
"context"
"encoding/json"
"fmt"
"text/template"
"time"
@@ -12,7 +13,7 @@ import (
)
const (
webhookDefaultTemplate = `{"text": "{{.Text}}"}`
webhookDefaultTemplate = `{"text": {{.Text | escapeJSONString}}}`
)
// WebhookParams contain settings for webhook notifications
@@ -49,7 +50,7 @@ func NewWebhook(params WebhookParams) (*Webhook, error) {
params.Template = webhookDefaultTemplate
}
payloadTmpl, err := template.New("webhook").Parse(params.Template)
payloadTmpl, err := template.New("webhook").Funcs(template.FuncMap{"escapeJSONString": escapeJSONString}).Parse(params.Template)
if err != nil {
return nil, fmt.Errorf("unable to parse webhook template: %w", err)
}
@@ -82,3 +83,12 @@ func (w *Webhook) SendVerification(_ context.Context, _ VerificationRequest) err
func (w *Webhook) String() string {
return fmt.Sprintf("%s to %s", w.Webhook.String(), w.url)
}
// escapeJSONString escapes string for JSON
func escapeJSONString(s string) (string, error) {
b, err := json.Marshal(s)
if err != nil {
return "", err
}
return string(b), nil
}
+29
View File
@@ -2,6 +2,9 @@ package notify
import (
"context"
"io"
"net/http"
"net/http/httptest"
"testing"
"time"
@@ -34,6 +37,32 @@ func TestWebhook_NewWebhook(t *testing.T) {
assert.Contains(t, err.Error(), "unable to parse webhook template")
}
// https://github.com/umputun/remark42/issues/1791
func TestWebhook_ReceiveValidJSON(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/webhook-notify")
assert.Equal(t, "POST", r.Method)
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
t.Log("received body", string(body))
assert.JSONEq(t, `{"text": "<p>testme</p>\n"}`, string(body))
}))
defer ts.Close()
wh, err := NewWebhook(WebhookParams{
URL: ts.URL + "/webhook-notify",
Headers: []string{"Content-Type:application/json,text/plain"},
})
assert.NoError(t, err)
assert.NotNil(t, wh)
f := store.NewCommentFormatter()
c := store.Comment{Text: f.FormatText("testme", false), ParentID: "1", ID: "999"}
err = wh.Send(context.Background(), Request{Comment: c})
assert.NoError(t, err)
}
func TestWebhook_Send(t *testing.T) {
wh, err := NewWebhook(WebhookParams{
URL: "bad-url",
+3 -3
View File
@@ -15,7 +15,7 @@ import (
)
type tgRequester interface {
Request(ctx context.Context, method string, b []byte, data interface{}) error
Request(ctx context.Context, method string, b []byte, data any) error
}
// TGUpdatesReceiver used to dispatch telegram updates to multiple receivers
@@ -27,8 +27,8 @@ type TGUpdatesReceiver interface {
// DispatchTelegramUpdates dispatches telegram updates to provided list of receivers
// Blocks caller
func DispatchTelegramUpdates(ctx context.Context, requester tgRequester, receivers []TGUpdatesReceiver, period time.Duration) {
// Identifier of the first update to be requested.
// Should be equal to LastSeenUpdateID + 1
// identifier of the first update to be requested.
// should be equal to LastSeenUpdateID + 1
// See https://core.telegram.org/bots/api#getupdates
var updateOffset int
+10 -7
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"testing"
"testing/synctest"
"time"
ntf "github.com/go-pkgz/notify"
@@ -12,12 +13,14 @@ import (
)
func TestDispatchTelegramUpdates(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
poolPeriod := time.Millisecond * 100
go DispatchTelegramUpdates(ctx, &mockTGRequester{t: t}, []TGUpdatesReceiver{&mockTGUpdatesReceiver{t: t}}, poolPeriod)
time.Sleep(poolPeriod * 3)
cancel()
time.Sleep(poolPeriod)
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
poolPeriod := time.Millisecond * 100
go DispatchTelegramUpdates(ctx, &mockTGRequester{t: t}, []TGUpdatesReceiver{&mockTGUpdatesReceiver{t: t}}, poolPeriod)
time.Sleep(poolPeriod * 3)
cancel()
synctest.Wait()
})
}
const getUpdatesResp = `{
@@ -39,7 +42,7 @@ type mockTGRequester struct {
t *testing.T
}
func (m *mockTGRequester) Request(_ context.Context, _ string, _ []byte, data interface{}) error {
func (m *mockTGRequester) Request(_ context.Context, _ string, _ []byte, data any) error {
if m.hit < 2 {
m.hit++
assert.NoError(m.t, json.Unmarshal([]byte(getUpdatesResp), data))
+54 -34
View File
@@ -1,15 +1,16 @@
package api
import (
"errors"
"fmt"
"net/http"
"path"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
"github.com/go-pkgz/auth"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
@@ -44,7 +45,7 @@ type adminStore interface {
// DELETE /comment/{id}?site=siteID&url=post-url - removes comment
func (a *admin) deleteCommentCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
id := r.PathValue("id")
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
log.Printf("[INFO] delete comment %s", id)
@@ -54,13 +55,12 @@ func (a *admin) deleteCommentCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.SiteID, locator.URL, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"id": id, "locator": locator})
R.RenderJSON(w, R.JSON{"id": id, "locator": locator})
}
// DELETE /user/{userid}?site=side-id - delete all user comments for requested userid
func (a *admin) deleteUserCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
log.Printf("[INFO] delete all user comments for %s, site %s", userID, siteID)
@@ -69,13 +69,12 @@ func (a *admin) deleteUserCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(siteID).Scopes(userID, siteID, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"user_id": userID, "site_id": siteID})
R.RenderJSON(w, R.JSON{"user_id": userID, "site_id": siteID})
}
// GET /user/{userid}?site=side-id - get user info for requested userid
func (a *admin) getUserInfoCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
log.Printf("[INFO] get user info for %s, site %s", userID, siteID)
@@ -84,8 +83,7 @@ func (a *admin) getUserInfoCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get user info", rest.ErrInternal)
return
}
render.Status(r, http.StatusOK)
render.JSON(w, r, ucomments[0].User)
R.RenderJSON(w, ucomments[0].User)
}
// GET /deleteme?token=jwt - delete all user comments and details by user's request. Gets info about deleted used from provided token
@@ -107,33 +105,56 @@ func (a *admin) deleteMeRequestCtrl(w http.ResponseWriter, r *http.Request) {
return
}
if err = a.dataService.DeleteUserDetail(claims.Audience, claims.User.ID, engine.AllUserDetails); err != nil {
// audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
if len(claims.Audience) != 1 {
rest.SendErrorJSON(w, r, http.StatusBadRequest, fmt.Errorf("bad request"), "can't process token, claims.Audience expected to be a single element but it's not", rest.ErrActionRejected)
return
}
audience := claims.Audience[0]
if err = a.dataService.DeleteUserDetail(audience, claims.User.ID, engine.AllUserDetails); err != nil {
code := parseError(err, rest.ErrInternal)
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete user details for user", code)
return
}
if err = a.dataService.DeleteUser(claims.Audience, claims.User.ID, store.HardDelete); err != nil {
if err = a.dataService.DeleteUser(audience, claims.User.ID, store.HardDelete); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete user", rest.ErrNoAccess)
return
}
if claims.User.Picture != "" && a.authenticator.AvatarProxy() != nil {
avatarStore := a.authenticator.AvatarProxy().Store
if err = avatarStore.Remove(path.Base(claims.User.Picture)); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete user's avatar", rest.ErrInternal)
return
if avatarID := avatarIDFromPicture(claims.User.Picture); avatarID != "" {
// an already-removed avatar is fine (a repeated request stays idempotent), but a genuine
// store failure is surfaced now that avatar.ErrNotFound lets us tell the two apart
if err = a.authenticator.AvatarProxy().Store.Remove(avatarID); err != nil && !errors.Is(err, avatar.ErrNotFound) {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't delete user's avatar", rest.ErrInternal)
return
}
} else {
log.Printf("[WARN] unexpected avatar picture %q for user %s on site %s, skipping removal", claims.User.Picture, claims.User.ID, audience)
}
}
a.cache.Flush(cache.Flusher(claims.Audience).Scopes(claims.Audience, claims.User.ID, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"user_id": claims.User.ID, "site_id": claims.Audience})
a.cache.Flush(cache.Flusher(audience).Scopes(audience, claims.User.ID, lastCommentsScope))
R.RenderJSON(w, R.JSON{"user_id": claims.User.ID, "site_id": claims.Audience})
}
// avatarIDFromPicture returns the avatar-store object id for a user picture, or "" if the picture
// does not resolve to a well-formed id (the store names its objects "<hash>.image"). Guarding on the
// id shape keeps a malformed picture, e.g. a path sentinel, from making a filesystem-backed store
// target an unexpected path.
func avatarIDFromPicture(picture string) string {
if id := path.Base(picture); strings.HasSuffix(id, ".image") {
return id
}
return ""
}
// PUT /user/{userid}?site=side-id&block=1&ttl=7d - block or unblock user
func (a *admin) setBlockCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
blockStatus := r.URL.Query().Get("block") == "1"
@@ -156,7 +177,7 @@ func (a *admin) setBlockCtrl(w http.ResponseWriter, r *http.Request) {
}
}
a.cache.Flush(cache.Flusher(siteID).Scopes(userID, siteID, lastCommentsScope))
render.JSON(w, r, R.JSON{"user_id": userID, "site_id": siteID, "block": blockStatus})
R.RenderJSON(w, R.JSON{"user_id": userID, "site_id": siteID, "block": blockStatus})
}
// GET /blocked?site=siteID - list blocked users
@@ -167,7 +188,7 @@ func (a *admin) blockedUsersCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get blocked users", rest.ErrSiteNotFound)
return
}
render.JSON(w, r, users)
R.RenderJSON(w, users)
}
// PUT /readonly?site=siteID&url=post-url&ro=1 - set or reset read-only status for the post
@@ -194,12 +215,12 @@ func (a *admin) setReadOnlyCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL, locator.SiteID))
render.JSON(w, r, R.JSON{"locator": locator, "read-only": roStatus})
R.RenderJSON(w, R.JSON{"locator": locator, "read-only": roStatus})
}
// PUT /title/{id}?site=siteID&url=post-url - set comment PostTitle to page's title
func (a *admin) setTitleCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
id := r.PathValue("id")
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
c, err := a.dataService.SetTitle(locator, id)
@@ -210,13 +231,12 @@ func (a *admin) setTitleCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[INFO] set comment's title %s to %q", id, c.PostTitle)
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"id": id, "locator": locator})
R.RenderJSON(w, R.JSON{"id": id, "locator": locator})
}
// PUT /verify?site=siteID&url=post-url&ro=1 - set or reset read-only status for the post
// PUT /verify/{userid}?site=siteID&verified=1 - set or reset verified status for the user
func (a *admin) setVerifyCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
verifyStatus := r.URL.Query().Get("verified") == "1"
@@ -225,13 +245,13 @@ func (a *admin) setVerifyCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(siteID).Scopes(siteID, userID))
render.JSON(w, r, R.JSON{"user": userID, "verified": verifyStatus})
R.RenderJSON(w, R.JSON{"user": userID, "verified": verifyStatus})
}
// PUT /pin/{id}?site=siteID&url=post-url&pin=1
// mark/unmark comment as a special
func (a *admin) setPinCtrl(w http.ResponseWriter, r *http.Request) {
commentID := chi.URLParam(r, "id")
commentID := r.PathValue("id")
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
pinStatus := r.URL.Query().Get("pin") == "1"
@@ -240,5 +260,5 @@ func (a *admin) setPinCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL))
render.JSON(w, r, R.JSON{"id": commentID, "locator": locator, "pin": pinStatus})
R.RenderJSON(w, R.JSON{"id": commentID, "locator": locator, "pin": pinStatus})
}
+249 -69
View File
@@ -13,10 +13,10 @@ import (
"testing"
"time"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -62,7 +62,7 @@ func TestAdmin_Delete(t *testing.T) {
fmt.Sprintf("%s/api/v1/admin/comment/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1), http.NoBody)
require.NoError(t, err)
requireAdminOnly(t, req)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -75,19 +75,27 @@ func TestAdmin_Delete(t *testing.T) {
assert.Equal(t, "", cr.Text)
assert.True(t, cr.Deleted)
time.Sleep(250 * time.Millisecond)
// check last comments updated
res, code = get(t, ts.URL+"/api/v1/last/2?site=remark42")
assert.Equal(t, http.StatusOK, code)
comments = []store.Comment{}
err = json.Unmarshal([]byte(res), &comments)
assert.NoError(t, err)
assert.Equal(t, 1, len(comments), "should have 1 comments")
// the last-comments list refreshes asynchronously after the delete. the polling closure runs
// off the test goroutine, so it asserts on the CollectT it is handed rather than on t, which
// also puts the real transport or decode error in the failure message
pollClient := http.Client{Timeout: waitTimeout}
defer pollClient.CloseIdleConnections()
require.EventuallyWithT(t, func(c *assert.CollectT) {
lastResp, gErr := pollClient.Get(ts.URL + "/api/v1/last/2?site=remark42")
if !assert.NoError(c, gErr) {
return
}
defer lastResp.Body.Close()
assert.Equal(c, http.StatusOK, lastResp.StatusCode)
last := []store.Comment{}
assert.NoError(c, json.NewDecoder(lastResp.Body).Decode(&last))
assert.Len(c, last, 1, "should have 1 comments")
}, waitTimeout, httpPoll)
// check count updated
res, code = get(t, ts.URL+"/api/v1/count?site=remark42&url=https://radio-t.com/blah")
assert.Equal(t, http.StatusOK, code)
b := map[string]interface{}{}
b := map[string]any{}
err = json.Unmarshal([]byte(res), &b)
assert.NoError(t, err)
t.Logf("%#v", b)
@@ -111,7 +119,7 @@ func TestAdmin_Title(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
srv.DataService.TitleExtractor = service.NewTitleExtractor(http.Client{Timeout: time.Second})
srv.DataService.TitleExtractor = service.NewTitleExtractor(http.Client{Timeout: time.Second}, []string{"127.0.0.1"})
tss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.String() == "/post1" {
_, err := w.Write([]byte("<html><title>post1 blah 123</title><body> 2222</body></html>"))
@@ -139,7 +147,7 @@ func TestAdmin_Title(t *testing.T) {
fmt.Sprintf("%s/api/v1/admin/title/%s?site=remark42&url=%s/post1", ts.URL, id1, tss.URL), http.NoBody)
assert.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -174,7 +182,7 @@ func TestAdmin_DeleteUser(t *testing.T) {
req, err := http.NewRequest(http.MethodDelete, fmt.Sprintf("%s/api/v1/admin/user/%s?site=remark42", ts.URL, "id2"), http.NoBody)
assert.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
assert.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -275,7 +283,7 @@ func TestAdmin_Block(t *testing.T) {
req, err := http.NewRequest(http.MethodPut, url, http.NoBody)
assert.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
body, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
@@ -333,10 +341,12 @@ func TestAdmin_Block(t *testing.T) {
assert.NoError(t, err)
assert.Equal(t, false, j["block"])
// block with ttl
// block with ttl, checked in place rather than through another admin request, which would
// push this test over the 10 req/s limit on that route
makeTwoComments()
code, _ = block(1, "50ms")
code, _ = block(1, "500ms")
require.Equal(t, http.StatusOK, code)
require.True(t, srv.adminRest.dataService.IsBlocked("remark42", "user1"), "user1 blocked with ttl")
// get as regular user
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah&sort=+time")
@@ -348,9 +358,15 @@ func TestAdmin_Block(t *testing.T) {
assert.Equal(t, "test test #1", comments.Comments[2].Text, "comment not removed and not cleared")
assert.False(t, comments.Comments[2].Deleted, "not deleted")
srv.pubRest.cache = cache.NewScache(cache.NewNopCache()) // TODO: with lru cache it won't be refreshed and invalidated for long
srv.pubRest.cache = cache.NewScache[[]byte](cache.NewNopCache[[]byte]()) // TODO: with lru cache it won't be refreshed and invalidated for long
// time
time.Sleep(50 * time.Millisecond)
// the ttl above is wide enough that the checks in between cannot outlast it, so reaching
// here still inside the block, and the wait below observes it lapse
require.Eventually(t, func() bool {
return !srv.adminRest.dataService.IsBlocked("remark42", "user1")
}, waitTimeout, pollInterval, "block with ttl did not expire")
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah&sort=+time")
assert.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
@@ -383,23 +399,23 @@ func TestAdmin_BlockedList(t *testing.T) {
req, err := http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/user/%s?site=remark42&block=%d", ts.URL, "user1", 1), http.NoBody)
assert.NoError(t, err)
res, err := sendReq(t, req, adminUmputunToken)
res, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, res.Body.Close())
assert.Equal(t, http.StatusOK, res.StatusCode)
// block user2
// block user2 for long enough that the "two users blocked" check below cannot race the ttl
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/user/%s?site=remark42&block=%d&ttl=150ms", ts.URL, "user2", 1), http.NoBody)
fmt.Sprintf("%s/api/v1/admin/user/%s?site=remark42&block=%d&ttl=1h", ts.URL, "user2", 1), http.NoBody)
assert.NoError(t, err)
res, err = sendReq(t, req, adminUmputunToken)
res, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, res.Body.Close())
assert.Equal(t, http.StatusOK, res.StatusCode)
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/blocked?site=remark42", http.NoBody)
require.NoError(t, err)
res, err = sendReq(t, req, adminUmputunToken)
res, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.Equal(t, http.StatusOK, res.StatusCode)
users := []store.BlockedUser{}
@@ -412,18 +428,33 @@ func TestAdmin_BlockedList(t *testing.T) {
assert.Equal(t, "user2", users[1].ID)
assert.Equal(t, "user2 name", users[1].Name)
t.Logf("%+v", users)
time.Sleep(150 * time.Millisecond)
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/blocked?site=remark42", http.NoBody)
// re-block user2 with a short ttl and wait for it to lapse, so the lapse is observed
// independently of the check above
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/user/%s?site=remark42&block=%d&ttl=150ms", ts.URL, "user2", 1), http.NoBody)
require.NoError(t, err)
res, err = sendReq(t, req, adminUmputunToken)
res, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.Equal(t, http.StatusOK, res.StatusCode)
users = []store.BlockedUser{}
err = json.NewDecoder(res.Body).Decode(&users)
assert.NoError(t, err)
require.NoError(t, res.Body.Close())
assert.Equal(t, 1, len(users), "one user left blocked")
require.Equal(t, http.StatusOK, res.StatusCode)
// the closure runs off the test goroutine and asserts on the CollectT it is handed, never on t
require.EventuallyWithT(t, func(c *assert.CollectT) {
blockedReq, reqErr := http.NewRequest("GET", ts.URL+"/api/v1/admin/blocked?site=remark42", http.NoBody)
if !assert.NoError(c, reqErr) {
return
}
blockedResp, sendErr := sendReq(blockedReq, adminUmputunToken)
if !assert.NoError(c, sendErr) {
return
}
defer blockedResp.Body.Close()
assert.Equal(c, http.StatusOK, blockedResp.StatusCode)
blocked := []store.BlockedUser{}
assert.NoError(c, json.NewDecoder(blockedResp.Body).Decode(&blocked))
assert.Len(c, blocked, 1, "one user left blocked")
}, waitTimeout, httpPoll)
}
func TestAdmin_ReadOnly(t *testing.T) {
@@ -448,11 +479,11 @@ func TestAdmin_ReadOnly(t *testing.T) {
req, err := http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/readonly?site=remark42&url=https://radio-t.com/blah&ro=1", ts.URL), http.NoBody)
assert.NoError(t, err)
resp, err := sendReq(t, req, "") // non-admin user
resp, err := sendReq(req, "") // non-admin user
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -465,9 +496,9 @@ func TestAdmin_ReadOnly(t *testing.T) {
Locator: store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah"}}
b, err := json.Marshal(c)
assert.NoError(t, err, "can't marshal comment %+v", c)
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", bytes.NewBuffer(b))
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", bytes.NewBuffer(b))
require.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
@@ -476,7 +507,7 @@ func TestAdmin_ReadOnly(t *testing.T) {
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/readonly?site=remark42&url=https://radio-t.com/blah&ro=0", ts.URL), http.NoBody)
assert.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -489,9 +520,9 @@ func TestAdmin_ReadOnly(t *testing.T) {
Locator: store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah"}}
b, err = json.Marshal(c)
assert.NoError(t, err, "can't marshal comment %+v", c)
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", bytes.NewBuffer(b))
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site="+c.Locator.SiteID, bytes.NewBuffer(b))
require.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusCreated, resp.StatusCode)
@@ -506,13 +537,14 @@ func TestAdmin_ReadOnlyNoComments(t *testing.T) {
fmt.Sprintf("%s/api/v1/admin/readonly?site=remark42&url=https://radio-t.com/blah&ro=1", ts.URL), http.NoBody)
assert.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
_, err = srv.DataService.Info(store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah"}, 0)
assert.Error(t, err)
// test format "tree"
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah&format=tree")
assert.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
@@ -521,6 +553,16 @@ func TestAdmin_ReadOnlyNoComments(t *testing.T) {
assert.Equal(t, 0, len(comments.Comments), "should have 0 comments")
assert.True(t, comments.Info.ReadOnly)
t.Logf("%+v", comments)
// test format "plain"
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah")
assert.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
assert.NoError(t, err)
assert.Equal(t, 0, len(comments.Comments), "should have 0 comments")
assert.True(t, comments.Info.ReadOnly)
t.Logf("%+v", comments)
}
func TestAdmin_ReadOnlyWithAge(t *testing.T) {
@@ -542,7 +584,7 @@ func TestAdmin_ReadOnlyWithAge(t *testing.T) {
fmt.Sprintf("%s/api/v1/admin/readonly?site=remark42&url=https://radio-t.com/blah&ro=1", ts.URL), http.NoBody)
assert.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -554,7 +596,7 @@ func TestAdmin_ReadOnlyWithAge(t *testing.T) {
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/readonly?site=remark42&url=https://radio-t.com/blah&ro=0", ts.URL), http.NoBody)
assert.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
@@ -583,7 +625,7 @@ func TestAdmin_Verify(t *testing.T) {
fmt.Sprintf("%s/api/v1/admin/verify/user1?site=remark42&verified=1", ts.URL), http.NoBody)
assert.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -602,7 +644,7 @@ func TestAdmin_Verify(t *testing.T) {
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/verify/user1?site=remark42&verified=0", ts.URL), http.NoBody)
assert.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -653,7 +695,7 @@ func TestAdmin_ExportFile(t *testing.T) {
req, err := http.NewRequest("GET", ts.URL+"/api/v1/admin/export?site=remark42&mode=file", http.NoBody)
require.NoError(t, err)
requireAdminOnly(t, req)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -697,17 +739,17 @@ func TestAdmin_DeleteMeRequest(t *testing.T) {
claims := token.Claims{
SessionOnly: true,
StandardClaims: jwt.StandardClaims{
Audience: "remark42",
Id: "1234567",
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "1234567",
Issuer: "remark42",
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: time.Now().Add(30 * time.Minute).Unix(),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user1",
Picture: "pic.image",
Attributes: map[string]interface{}{
Picture: "https://demo.remark42.com/api/v1/avatar/pic.image", // production-shaped URL: removal must path.Base it to the avatar id
Attributes: map[string]any{
"delete_me": true,
},
},
@@ -736,6 +778,124 @@ func TestAdmin_DeleteMeRequest(t *testing.T) {
email, err = srv.DataService.GetUserEmail("remark42", "user1")
assert.NoError(t, err)
assert.Empty(t, email, "user1 email was deleted")
assert.NoFileExists(t, os.TempDir()+"/ava-remark42/42/pic.image", "user's avatar should be removed on deleteme")
}
// a delete_me request whose token carries a picture must still succeed when the avatar is
// already gone from the store: the user data is deleted and a missing avatar is tolerated
func TestAdmin_DeleteMeRequestMissingAvatar(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
c1 := store.Comment{Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user3 name", ID: "user3"}}
_, err := srv.DataService.Create(c1)
require.NoError(t, err)
claims := token.Claims{
SessionOnly: true,
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "2345678",
Issuer: "remark42",
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user3",
Picture: "missing.image", // no avatar file exists for this picture in the store
Attributes: map[string]any{
"delete_me": true,
},
},
}
tkn, err := srv.Authenticator.TokenService().Token(claims)
require.NoError(t, err)
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "a missing avatar must not fail the deletion")
_, err = srv.DataService.User("remark42", "user3", 0, 0, store.User{})
assert.EqualError(t, err, "no comments for user user3 in store", "user3 comments should be deleted")
}
// a genuine (non not-found) avatar-store failure must now surface, not be silently swallowed:
// avatar.ErrNotFound lets deleteMeRequestCtrl tell an already-gone avatar from a real error
func TestAdmin_DeleteMeRequestAvatarRemoveError(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
c1 := store.Comment{Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user5 name", ID: "user5"}}
_, err := srv.DataService.Create(c1)
require.NoError(t, err)
// put a non-empty directory where the avatar file is expected, so Store.Remove fails with a real
// error (directory not empty), not os.ErrNotExist - "pic" hashes to partition 42
require.NoError(t, os.MkdirAll(os.TempDir()+"/ava-remark42/42/pic.image", 0o700))
require.NoError(t, os.WriteFile(os.TempDir()+"/ava-remark42/42/pic.image/child", []byte("x"), 0o600))
claims := token.Claims{
SessionOnly: true,
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "4567890",
Issuer: "remark42",
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user5",
Picture: "https://demo.remark42.com/api/v1/avatar/pic.image",
Attributes: map[string]any{
"delete_me": true,
},
},
}
tkn, err := srv.Authenticator.TokenService().Token(claims)
require.NoError(t, err)
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusInternalServerError, resp.StatusCode, "a real avatar-store failure must surface, not be swallowed")
}
func TestAvatarIDFromPicture(t *testing.T) {
tbl := []struct {
name string
picture string
want string
}{
{"local avatar url", "https://demo.remark42.com/api/v1/avatar/cb42ff493ade696d88a3a590f136ae9e34de7c1b.image", "cb42ff493ade696d88a3a590f136ae9e34de7c1b.image"},
{"bare avatar id", "pic.image", "pic.image"},
{"parent sentinel", "https://demo.remark42.com/api/v1/avatar/..", ""},
{"trailing slash", "https://demo.remark42.com/api/v1/avatar/", ""},
{"root", "/", ""},
{"dotdot", "..", ""},
{"empty", "", ""},
{"provider url without image suffix", "https://example.com/pic.png", ""},
}
for _, tc := range tbl {
t.Run(tc.name, func(t *testing.T) {
assert.Equal(t, tc.want, avatarIDFromPicture(tc.picture))
})
}
}
func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
@@ -766,16 +926,16 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
// try with bad auth
claims := token.Claims{
SessionOnly: true,
StandardClaims: jwt.StandardClaims{
Audience: "remark42",
Id: "provider1_1234567",
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "provider1_1234567",
Issuer: "remark42",
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: time.Now().Add(30 * time.Minute).Unix(),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "provider1_user1",
Attributes: map[string]interface{}{
Attributes: map[string]any{
"delete_me": true,
},
},
@@ -791,10 +951,11 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try bad user
badClaims := claims
badClaims.User.ID = "no-such-id"
tkn, err = srv.Authenticator.TokenService().Token(badClaims)
// unknown user: deletion is idempotent, so a valid (signed) delete_me token for a user with
// no stored data is a no-op success rather than an error
badClaimsUser := claims
badClaimsUser.User.ID = "no-such-id"
tkn, err = srv.Authenticator.TokenService().Token(badClaimsUser)
assert.NoError(t, err)
req, err = http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
assert.NoError(t, err)
@@ -802,12 +963,13 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
resp, err = client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusBadRequest, resp.StatusCode, resp.Status)
assert.Equal(t, http.StatusOK, resp.StatusCode, resp.Status)
badClaimsUser.User.ID = "provider1_user1"
// try without deleteme flag
badClaims2 := claims
badClaims2.User.SetBoolAttr("delete_me", false)
tkn, err = srv.Authenticator.TokenService().Token(badClaims2)
badClaimsWithoutDeleteMe := claims
badClaimsWithoutDeleteMe.User.SetBoolAttr("delete_me", false)
tkn, err = srv.Authenticator.TokenService().Token(badClaimsWithoutDeleteMe)
assert.NoError(t, err)
req, err = http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
assert.NoError(t, err)
@@ -818,7 +980,25 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.True(t, strings.Contains(string(b), "can't use provided token"))
assert.Contains(t, string(b), "can't use provided token")
badClaimsWithoutDeleteMe.User.SetBoolAttr("delete_me", true)
// try with wrong audience
badClaimsMultipleAudience := claims
badClaimsMultipleAudience.Audience = jwt.ClaimStrings{"remark42", "something else"}
tkn, err = srv.Authenticator.TokenService().Token(badClaimsMultipleAudience)
assert.NoError(t, err)
req, err = http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
assert.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Contains(t, string(b), "can't process token, claims.Audience expected to be a single element but it's not")
badClaimsMultipleAudience.Audience = jwt.ClaimStrings{"remark42"}
}
func TestAdmin_GetUserInfo(t *testing.T) {
+360
View File
@@ -0,0 +1,360 @@
// Package api middleware: request-scoped HTTP middlewares used by the REST router.
package api
import (
"fmt"
"net"
"net/http"
"net/mail"
"regexp"
"strings"
"time"
"github.com/didip/tollbooth/v8"
"github.com/didip/tollbooth/v8/limiter"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/store"
)
// ipForwardingHeaders are the request headers R.RealIP derives the client IP from.
var ipForwardingHeaders = []string{"X-Real-IP", "X-Forwarded-For", "CF-Connecting-IP"}
// realIPMiddleware derives the client IP from forwarding headers (X-Real-IP / X-Forwarded-For /
// CF-Connecting-IP) via R.RealIP, but honors those headers only for requests whose direct peer
// is one of the trusted proxies. For any other peer it drops those headers and pins RemoteAddr to
// the real socket IP, so an untrusted client can't spoof the IP that per-IP controls (rate limiting,
// vote dedup, comment IP, anonymous id) and the request log key on.
//
// With no trusted proxies configured it falls back to trusting the headers from any client (the
// historical behavior). That is spoofable by design, so operators running behind a reverse proxy
// should set --trusted-proxy to the proxy's network — see the "trusted proxy" docs.
func realIPMiddleware(trustedProxies []*net.IPNet) func(http.Handler) http.Handler {
if len(trustedProxies) == 0 {
return R.RealIP
}
return func(next http.Handler) http.Handler {
fromTrusted := R.RealIP(next) // rewrites RemoteAddr from the forwarding headers
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
peer := directPeerIP(r.RemoteAddr)
if peer != nil && cidrsContain(trustedProxies, peer) {
fromTrusted.ServeHTTP(w, r) // trusted proxy: honor the forwarding headers
return
}
// untrusted peer: drop the forwarding headers and pin RemoteAddr to the real socket IP,
// so nothing downstream can be fooled by a spoofed header (R.RealIP normalizes
// RemoteAddr to a bare IP for trusted peers; do the same here for consistency)
for _, h := range ipForwardingHeaders {
r.Header.Del(h)
}
if peer != nil {
r.RemoteAddr = peer.String()
}
next.ServeHTTP(w, r)
})
}
}
// directPeerIP extracts the IP from a "host:port" (or bare host) RemoteAddr, or nil if unparseable.
func directPeerIP(remoteAddr string) net.IP {
host, _, err := net.SplitHostPort(remoteAddr)
if err != nil {
host = remoteAddr // may already be a bare IP with no port
}
return net.ParseIP(host)
}
// TrustsAnyPeer reports whether the trusted-proxy list contains a catch-all (0.0.0.0/0 or ::/0),
// which trusts forwarding headers from every client and re-opens the IP-spoofing bypass.
func TrustsAnyPeer(cidrs []*net.IPNet) bool {
for _, c := range cidrs {
if ones, _ := c.Mask.Size(); ones == 0 {
return true
}
}
return false
}
// cidrsContain reports whether ip falls within any of the CIDRs.
func cidrsContain(cidrs []*net.IPNet, ip net.IP) bool {
for _, c := range cidrs {
if c.Contains(ip) {
return true
}
}
return false
}
// ParseTrustedProxies parses a list of trusted-proxy entries into CIDRs. Each entry may be a CIDR
// (e.g. 172.16.0.0/12) or a bare IP (treated as a single host). Blank entries are skipped; a
// malformed entry is a hard error so a typo can't silently disable proxy trust.
func ParseTrustedProxies(entries []string) ([]*net.IPNet, error) {
var out []*net.IPNet
for _, e := range entries {
e = strings.TrimSpace(e)
if e == "" {
continue
}
if !strings.Contains(e, "/") { // bare IP -> single-host CIDR
ip := net.ParseIP(e)
if ip == nil {
return nil, fmt.Errorf("invalid trusted proxy %q", e)
}
// build the network from the normalized IP so a v4-mapped IPv6 (e.g. ::ffff:10.0.0.1)
// yields the intended /32 host, not a huge ::/32 range
bits := 128
if v4 := ip.To4(); v4 != nil {
ip, bits = v4, 32
}
out = append(out, &net.IPNet{IP: ip, Mask: net.CIDRMask(bits, bits)})
continue
}
_, network, err := net.ParseCIDR(e)
if err != nil {
return nil, fmt.Errorf("invalid trusted proxy CIDR %q: %w", e, err)
}
out = append(out, network)
}
return out, nil
}
// corsMiddleware builds the CORS middleware for the public API. With AllowedOrigins
// "*" and credentials enabled, rest.CORS reflects the request Origin into
// Access-Control-Allow-Origin (rather than a literal "*"), which browsers require
// for credentialed cross-origin requests.
//
// That combination is refused by default upstream, so it has to be asked for by name with
// CorsUnsafeAnyOriginWithCredentials. The wildcard stays because the comment widget is embedded on
// arbitrary third-party sites, which makes the set of origins unknowable. The consequence it carries
// is that any site a signed-in user visits can read authenticated responses, so state-changing
// requests have to keep being protected by something other than the origin, X-XSRF-Token today.
func corsMiddleware() func(http.Handler) http.Handler {
return R.CORS(
R.CorsAllowedOrigins("*"),
R.CorsAllowedMethods("GET", "POST", "PUT", "DELETE", "OPTIONS"),
R.CorsAllowedHeaders("Accept", "Authorization", "Content-Type", "X-XSRF-Token", "X-JWT"),
R.CorsExposedHeaders("Authorization"),
R.CorsAllowCredentials(true),
R.CorsUnsafeAnyOriginWithCredentials(true),
R.CorsMaxAge(300),
)
}
// rejectHead rejects HEAD requests with 405, advertising the given allowed methods in
// the Allow header. net/http.ServeMux routes HEAD to a "GET ..." handler, but per RFC
// 9110 GET/HEAD are safe methods; this guard is applied to the few GET routes whose
// handlers mutate state so they cannot be triggered by a (nominally side-effect-free)
// HEAD, preserving the pre-routegroup behavior. allow lists every method the resource
// supports (e.g. "GET" or "GET, POST") so the 405 Allow header is accurate.
func rejectHead(allow string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead {
w.Header().Set("Allow", allow)
http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
return
}
next.ServeHTTP(w, r)
})
}
}
// rejectAnonUser is a middleware rejecting anonymous users
func rejectAnonUser(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if strings.HasPrefix(user.ID, "anonymous_") {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// matchSiteID is a middleware rejecting users with mismatch between site param and and User.SiteID
func matchSiteID(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
// skip for basic auth user
if user.Name == "admin" && user.ID == "admin" {
next.ServeHTTP(w, r)
return
}
siteID := r.URL.Query().Get("site")
// require an explicit site so the user.SiteID check below cannot be bypassed
// by simply omitting the query parameter
if siteID == "" || user.SiteID != siteID {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// cacheControl is a middleware setting cache expiration. Using url+version as etag
func cacheControl(expiration time.Duration, version string) func(http.Handler) http.Handler {
etag := func(r *http.Request, version string) string {
s := version + ":" + r.URL.String()
return store.EncodeID(s)
}
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
e := `"` + etag(r, version) + `"`
w.Header().Set("Etag", e)
w.Header().Set("Cache-Control", fmt.Sprintf("max-age=%d, no-cache", int(expiration.Seconds())))
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, e) {
w.WriteHeader(http.StatusNotModified)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// apiCSPMiddleware overrides the global Content-Security-Policy on /api/v1 routes
// with a strict, default-deny policy. The global CSP (securityHeadersMiddleware) keeps
// 'self' 'unsafe-inline' for script-src/style-src because the widget HTML pages
// (/web/*.html) need inline bootstrap blocks. API responses serve JSON, XML/RSS, or
// images — none of those should ever execute scripts when rendered, so they get the
// strictest policy available as defense-in-depth against future trust-boundary bugs.
//
// Image-serving handlers (/api/v1/img, /api/v1/picture/{user}/{id}) re-apply the same
// rest.StrictImageCSP value at the handler level and additionally set Content-Disposition:
// inline; filename="image" (framing the response as a file rather than a renderable
// document) and X-Content-Type-Options: nosniff. The CSP re-apply is intentional belt-and-
// braces: if a future route refactor bypasses this middleware, the image handlers still
// emit the policy.
func apiCSPMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Security-Policy", rest.StrictImageCSP)
next.ServeHTTP(w, r)
})
}
// securityHeadersMiddleware sets security-related headers:
// - Content-Security-Policy: controls which resources the browser is allowed to load
// - Permissions-Policy: disables browser features (camera, mic, etc.) not needed by a comment widget
// - X-Content-Type-Options: prevents browsers from MIME-sniffing responses away from the declared type,
// stopping e.g. a user-uploaded image from being reinterpreted as executable HTML/JS
// - Referrer-Policy: controls how much URL information leaks in the Referer header on cross-origin
// requests; "strict-origin-when-cross-origin" sends only the origin (no path) to other domains
// and nothing at all on HTTPS→HTTP downgrades
func securityHeadersMiddleware(imageProxyEnabled bool, allowedAncestors []string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
imgSrc := "*"
if imageProxyEnabled {
imgSrc = "'self'"
}
frameAncestors := "*"
if len(allowedAncestors) > 0 {
frameAncestors = strings.Join(allowedAncestors, " ")
}
// font-src is set to 'none' (no @font-face / no base64 fonts in the bundle).
w.Header().Set("Content-Security-Policy", fmt.Sprintf("default-src 'none'; base-uri 'none'; form-action 'none'; connect-src 'self'; frame-src 'self' mailto:; img-src %s; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; font-src 'none'; object-src 'none'; frame-ancestors %s;", imgSrc, frameAncestors))
w.Header().Set("Permissions-Policy", "accelerometer=(), autoplay=(), camera=(), cross-origin-isolated=(), display-capture=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), unload=(), window-management=()")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Referrer-Policy", "strict-origin-when-cross-origin")
next.ServeHTTP(w, r)
})
}
}
// subscribersOnly is a middleware rejecting non-paid_sub users
func subscribersOnly(enable bool) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if enable {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if !user.PaidSub {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// validEmailAuth is a middleware for auth endpoints for email method.
// it rejects login request if user, site or email are suspicious
func validEmailAuth() func(http.Handler) http.Handler {
reUser := regexp.MustCompile(`^[\p{L}\d\s_]{4,64}$`) // matches ui side validation, adding min/max limitation
reSite := regexp.MustCompile(`^[a-zA-Z\d\s_.-]{1,64}$`)
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/auth/email/login" {
// not email login, skip the check
h.ServeHTTP(w, r)
return
}
if u := r.URL.Query().Get("user"); u != "" {
if !reUser.MatchString(u) {
log.Printf("[WARN] suspicious user rejected: %s", u)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if a := r.URL.Query().Get("address"); a != "" {
if _, err := mail.ParseAddress(a); err != nil {
log.Printf("[WARN] suspicious address rejected: %s", a)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if s := r.URL.Query().Get("site"); s != "" {
if !reSite.MatchString(s) {
log.Printf("[WARN] suspicious site rejected: %s", s)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// rateLimiter creates a rate limiting middleware with proper IP lookup configuration.
// tollbooth v8 requires explicit IP lookup method to be set.
// keys on RemoteAddr, which realIPMiddleware sets to the client IP (from the forwarding
// headers for trusted proxies, otherwise the real socket IP).
func rateLimiter(maxReq float64) func(http.Handler) http.Handler {
lmt := tollbooth.NewLimiter(maxReq, nil)
lmt.SetIPLookup(limiter.IPLookup{
Name: "RemoteAddr",
IndexFromRight: 0,
})
return tollbooth.HTTPMiddleware(lmt)
}
+442
View File
@@ -0,0 +1,442 @@
package api
import (
"fmt"
"net"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"
"github.com/go-pkgz/auth/v2/token"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/store"
)
// routes() wraps bounded routes with the enforcing rest.Timeout and deliberately leaves the
// streaming/long-polling routes (GET /export, /userdata, /wait) without it. This checks that
// contract holds against the vendored middleware: a slow handler under R.Timeout is aborted with
// 504 at the deadline, while a route left without it runs to completion.
func TestRouteTimeout(t *testing.T) {
slow := func(d time.Duration) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
select {
case <-r.Context().Done(): // return promptly once the enforcing timeout cancels the context
case <-time.After(d):
}
w.WriteHeader(http.StatusOK)
}
}
router := routegroup.New(http.NewServeMux())
router.With(R.Timeout(20*time.Millisecond)).HandleFunc("GET /bounded", slow(time.Second))
router.HandleFunc("GET /streaming", slow(30*time.Millisecond)) // no timeout, like /export and /wait
ts := httptest.NewServer(router)
defer ts.Close()
resp, err := http.Get(ts.URL + "/bounded")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusGatewayTimeout, resp.StatusCode, "route under R.Timeout is aborted at the deadline")
resp, err = http.Get(ts.URL + "/streaming")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "route without R.Timeout runs to completion")
}
// TestRateLimiter covers the middleware guarding every route group: a burst past the per-second
// allowance is refused with 429, and a client under the allowance is not. The limiter keys on
// RemoteAddr, so the two cases use different ones rather than waiting for a bucket to refill.
func TestRateLimiter(t *testing.T) {
router := routegroup.New(http.NewServeMux())
router.With(rateLimiter(1)).HandleFunc("GET /limited", func(http.ResponseWriter, *http.Request) {})
ts := httptest.NewServer(router)
defer ts.Close()
call := func(remoteAddr string) int {
req := httptest.NewRequest("GET", "http://example.com/limited", http.NoBody)
req.RemoteAddr = remoteAddr
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
resp := w.Result()
assert.NoError(t, resp.Body.Close())
return resp.StatusCode
}
// one request a second is allowed, so the first of a burst passes and the rest are refused
assert.Equal(t, http.StatusOK, call("1.2.3.4:1000"), "first request within the allowance")
refused := 0
for range 5 {
if call("1.2.3.4:1000") == http.StatusTooManyRequests {
refused++
}
}
assert.Equal(t, 5, refused, "burst past the allowance is refused")
// a different client has its own bucket and is unaffected
assert.Equal(t, http.StatusOK, call("5.6.7.8:1000"), "limit is per client, not global")
}
func TestRealIPMiddleware(t *testing.T) {
// call runs mw with the given peer and (optional) X-Real-IP header and returns what the
// downstream handler observes; state is per-call, so subtests don't share closure locals.
call := func(mw func(http.Handler) http.Handler, remoteAddr, xRealIP string) (addr, hdr string) {
next := http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
addr, hdr = r.RemoteAddr, r.Header.Get("X-Real-IP")
})
req := httptest.NewRequest(http.MethodGet, "/", http.NoBody)
req.RemoteAddr = remoteAddr
if xRealIP != "" {
req.Header.Set("X-Real-IP", xRealIP)
}
mw(next).ServeHTTP(httptest.NewRecorder(), req)
return addr, hdr
}
trusted, err := ParseTrustedProxies([]string{"172.16.0.0/12", "2001:db8::/32"})
require.NoError(t, err)
t.Run("no trusted proxies trusts the header from anyone (legacy)", func(t *testing.T) {
addr, _ := call(realIPMiddleware(nil), "203.0.113.9:1234", "8.8.8.8")
assert.Equal(t, "8.8.8.8", addr)
})
t.Run("trusted v4 peer: forwarding header sets the client IP", func(t *testing.T) {
addr, _ := call(realIPMiddleware(trusted), "172.18.0.5:5555", "8.8.8.8")
assert.Equal(t, "8.8.8.8", addr)
})
t.Run("trusted v6 peer: forwarding header honored", func(t *testing.T) {
addr, _ := call(realIPMiddleware(trusted), "[2001:db8::5]:5555", "8.8.8.8")
assert.Equal(t, "8.8.8.8", addr)
})
t.Run("trusted peer without a forwarding header falls back to the socket IP", func(t *testing.T) {
addr, _ := call(realIPMiddleware(trusted), "172.18.0.5:5555", "")
assert.Equal(t, "172.18.0.5", addr, "no header to honor, so the bare socket IP is used")
})
t.Run("untrusted peer: header stripped, RemoteAddr pinned to bare socket IP", func(t *testing.T) {
addr, hdr := call(realIPMiddleware(trusted), "203.0.113.9:1234", "8.8.8.8")
assert.Equal(t, "203.0.113.9", addr, "real socket IP with the port stripped")
assert.Empty(t, hdr, "spoofed forwarding header removed so nothing downstream can read it")
})
t.Run("unparseable RemoteAddr is treated as untrusted, header stripped", func(t *testing.T) {
addr, hdr := call(realIPMiddleware(trusted), "garbage", "8.8.8.8")
assert.Equal(t, "garbage", addr, "unparseable peer left as-is, not overwritten")
assert.Empty(t, hdr, "forwarding header still stripped for a non-trusted peer")
})
}
func TestParseTrustedProxies(t *testing.T) {
t.Run("cidr, bare v4, bare v6, blanks", func(t *testing.T) {
got, err := ParseTrustedProxies([]string{"172.16.0.0/12", " 10.0.0.1 ", "", "2001:db8::/32"})
require.NoError(t, err)
require.Len(t, got, 3)
assert.True(t, got[0].Contains(net.ParseIP("172.18.0.5")))
assert.True(t, got[1].Contains(net.ParseIP("10.0.0.1")))
assert.False(t, got[1].Contains(net.ParseIP("10.0.0.2")), "a bare IP is a single host")
assert.True(t, got[2].Contains(net.ParseIP("2001:db8::1")))
})
t.Run("v4-mapped IPv6 bare entry resolves to the v4 host", func(t *testing.T) {
got, err := ParseTrustedProxies([]string{"::ffff:10.0.0.1"})
require.NoError(t, err)
require.Len(t, got, 1)
assert.True(t, got[0].Contains(net.ParseIP("10.0.0.1")), "the intended /32 host")
assert.False(t, got[0].Contains(net.ParseIP("10.0.0.2")), "not a wider range")
})
t.Run("malformed entry is a hard error", func(t *testing.T) {
_, err := ParseTrustedProxies([]string{"172.16.0.0/12", "nonsense"})
require.Error(t, err)
_, err = ParseTrustedProxies([]string{"10.0.0.0/999"})
require.Error(t, err)
})
t.Run("all blank yields nil", func(t *testing.T) {
got, err := ParseTrustedProxies([]string{"", " "})
require.NoError(t, err)
assert.Empty(t, got)
})
}
func TestTrustsAnyPeer(t *testing.T) {
catchAll := func(entries ...string) bool {
cidrs, err := ParseTrustedProxies(entries)
require.NoError(t, err)
return TrustsAnyPeer(cidrs)
}
assert.True(t, catchAll("10.0.0.0/8", "0.0.0.0/0"), "v4 catch-all")
assert.True(t, catchAll("::/0"), "v6 catch-all")
assert.False(t, catchAll("172.16.0.0/12", "10.0.0.5"), "scoped ranges are not catch-all")
assert.False(t, catchAll(), "empty is not catch-all")
}
func TestRest_rejectAnonUser(t *testing.T) {
ts := httptest.NewServer(fakeAuth(rejectAnonUser(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprintln(w, "Hello")
}))))
defer ts.Close()
resp, err := http.Get(ts.URL)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "use not logged in")
resp, err = http.Get(ts.URL + "?fake_id=anonymous_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode, "anon rejected")
resp, err = http.Get(ts.URL + "?fake_id=real_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "real user")
}
func TestRest_cacheControl(t *testing.T) {
tbl := []struct {
url string
version string
exp time.Duration
etag string
maxAge int
}{
{"http://example.com/foo", "v1", time.Hour, "b433be1ea19edaee9dc92ca4b895b6bdf3c058cb", 3600},
{"http://example.com/foo2", "v1", 10 * time.Hour, "6d8466aef3246c1057452561acddf7ad9d0d99e0", 36000},
{"http://example.com/foo", "v2", time.Hour, "481700c52aab0dfbca99f3ffc2a4fbb27884c114", 3600},
{"https://example.com/foo", "v2", time.Hour, "bebd4f1b87f474792c4e75e5affe31fbf67f5778", 3600},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", tt.url, http.NoBody)
w := httptest.NewRecorder()
h := cacheControl(tt.exp, tt.version)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
t.Logf("%+v", resp.Header)
assert.Equal(t, `"`+tt.etag+`"`, resp.Header.Get("Etag"))
assert.Equal(t, `max-age=`+strconv.Itoa(int(tt.exp.Seconds()))+", no-cache", resp.Header.Get("Cache-Control"))
})
}
}
// TestRest_apiCSP locks in that /api/v1/* responses get a strict default-src 'none'
// override regardless of what the global CSP allows. The widget HTML pages
// (/web/*.html) still get the global CSP (with 'unsafe-inline' for bootstrap),
// so the test asserts the two policies diverge across origins.
func TestRest_apiCSP(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
client := http.Client{}
// JSON API endpoint — must carry the strict policy
resp, err := client.Get(ts.URL + "/api/v1/config")
require.NoError(t, err)
defer resp.Body.Close()
csp := resp.Header.Get("Content-Security-Policy")
assert.Contains(t, csp, "default-src 'none'",
"API responses must override the global CSP with default-src 'none'; got %q", csp)
assert.Contains(t, csp, "sandbox", "API CSP must include sandbox; got %q", csp)
assert.NotContains(t, csp, "'unsafe-inline'",
"API CSP must not allow inline scripts/styles; got %q", csp)
// RSS/XML endpoint — same strict policy, and the XML response itself must still be served
respRSS, err := client.Get(ts.URL + "/api/v1/rss/site?site=remark42")
require.NoError(t, err)
defer respRSS.Body.Close()
assert.Equal(t, http.StatusOK, respRSS.StatusCode, "RSS must still respond OK under strict CSP")
cspRSS := respRSS.Header.Get("Content-Security-Policy")
assert.Contains(t, cspRSS, "default-src 'none'", "RSS responses must carry the strict API CSP")
assert.Contains(t, cspRSS, "sandbox", "RSS CSP must include sandbox")
// widget HTML — must keep the global CSP (unchanged, lax to support inline bootstrap)
resp2, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp2.Body.Close()
csp2 := resp2.Header.Get("Content-Security-Policy")
assert.Contains(t, csp2, "'unsafe-inline'",
"widget HTML CSP must keep unsafe-inline for bootstrap; got %q", csp2)
}
// check CSP, img-src should be 'self' with proxy enabled and * without it
func TestRest_securityHeaders(t *testing.T) {
ts, _, teardown := startupT(t)
// with proxy disabled
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src *;")
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, "strict-origin-when-cross-origin", resp.Header.Get("Referrer-Policy"))
// httptest.Server.Close waits on connections still in use, and a deferred close does not run
// until the test ends, so the body has to be released before the server is torn down here
require.NoError(t, resp.Body.Close())
client.CloseIdleConnections()
teardown()
// check CSP with proxy enabled
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.ExternalImageProxy = true
})
defer teardown()
resp, err = client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src 'self';")
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, "strict-origin-when-cross-origin", resp.Header.Get("Referrer-Policy"))
}
func TestRest_subscribersOnly(t *testing.T) {
paidSubUser := &token.User{}
paidSubUser.SetPaidSub(true)
tbl := []struct {
subsOnly bool
user token.User
setUser bool
status int
}{
{true, token.User{}, false, http.StatusUnauthorized},
{true, token.User{}, true, http.StatusForbidden},
{false, token.User{}, false, http.StatusOK},
{false, token.User{}, true, http.StatusOK},
{true, *paidSubUser, true, http.StatusOK},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", http.NoBody)
if tt.setUser {
req = token.SetUserInfo(req, tt.user)
}
w := httptest.NewRecorder()
h := subscribersOnly(tt.subsOnly)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
func Test_validEmailAuth(t *testing.T) {
tbl := []struct {
req string
status int
}{
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=site-with-dash_and_underscore-and.dot&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=Евгений+Умпутун", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=12", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someonelooong+loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong", http.StatusForbidden},
{"/auth/twitter/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun+example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=bad!site&address=umputun%example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooongsite&address=umputun%example.com&user=someone", http.StatusForbidden},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com"+tt.req, http.NoBody)
w := httptest.NewRecorder()
h := validEmailAuth()(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
// TestRest_matchSiteID reproduces the multi-tenant isolation gap in the matchSiteID
// middleware. Before the fix, the check `if siteID != "" && user.SiteID != siteID`
// silently allowed any authenticated request that omitted the ?site= query param.
// On admin and user-mutation routes this meant the cross-site check was bypassable
// just by dropping the parameter. The fix requires ?site= to be present and to match
// the user's bound site.
func TestRest_matchSiteID(t *testing.T) {
wrapped := matchSiteID(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
}))
cases := []struct {
name string
userSite string
query string
want int
}{
{name: "matching site allowed", userSite: "site-a", query: "?site=site-a", want: http.StatusOK},
{name: "mismatched site forbidden", userSite: "site-a", query: "?site=site-b", want: http.StatusForbidden},
{name: "missing site param rejected", userSite: "site-a", query: "", want: http.StatusForbidden},
{name: "empty site param rejected", userSite: "site-a", query: "?site=", want: http.StatusForbidden},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r = rest.SetUserInfo(r, store.User{ID: "u", Name: "u", SiteID: c.userSite})
wrapped.ServeHTTP(w, r)
})
ts := httptest.NewServer(h)
defer ts.Close()
resp, err := http.Get(ts.URL + c.query)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, c.want, resp.StatusCode)
})
}
}
func TestCorsMiddleware(t *testing.T) {
h := corsMiddleware()(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
}))
t.Run("credentialed cross-origin reflects the request origin", func(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/", http.NoBody)
req.Header.Set("Origin", "https://example.com")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
// AllowedOrigins "*" with credentials must reflect the origin, never a literal "*"
assert.Equal(t, "https://example.com", rec.Header().Get("Access-Control-Allow-Origin"))
assert.Equal(t, "true", rec.Header().Get("Access-Control-Allow-Credentials"))
assert.Equal(t, "Authorization", rec.Header().Get("Access-Control-Expose-Headers"))
})
t.Run("preflight advertises configured methods, headers and max-age", func(t *testing.T) {
req := httptest.NewRequest(http.MethodOptions, "/", http.NoBody)
req.Header.Set("Origin", "https://example.com")
req.Header.Set("Access-Control-Request-Method", "POST")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusNoContent, rec.Code)
assert.Contains(t, rec.Header().Get("Access-Control-Allow-Methods"), "POST")
assert.Contains(t, rec.Header().Get("Access-Control-Allow-Headers"), "X-JWT")
assert.Equal(t, "300", rec.Header().Get("Access-Control-Max-Age"))
// preflight responses must vary on origin and the request method/headers so caches
// don't reuse one preflight across different requests
vary := rec.Header().Values("Vary")
assert.Contains(t, vary, "Origin")
assert.Contains(t, vary, "Access-Control-Request-Method")
assert.Contains(t, vary, "Access-Control-Request-Headers")
})
t.Run("same-origin request (no Origin) gets no CORS headers", func(t *testing.T) {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", http.NoBody))
assert.Empty(t, rec.Header().Get("Access-Control-Allow-Origin"))
})
}
+79 -40
View File
@@ -1,22 +1,26 @@
package api
import (
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"sync"
"time"
"github.com/go-chi/render"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/umputun/remark42/backend/app/migrator"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/store/engine"
)
// Migrator rest with import and export controllers
@@ -58,8 +62,7 @@ func (m *Migrator) importCtrl(w http.ResponseWriter, r *http.Request) {
go m.runImport(siteID, r.URL.Query().Get("provider"), tmpfile) // import runs in background and sets busy flag for site
render.Status(r, http.StatusAccepted)
render.JSON(w, r, R.JSON{"status": "import request accepted"})
_ = R.EncodeJSON(w, http.StatusAccepted, R.JSON{"status": "import request accepted"})
}
// POST /import/form?secret=key&site=site-id&provider=disqus|remark|wordpress
@@ -73,28 +76,47 @@ func (m *Migrator) importFormCtrl(w http.ResponseWriter, r *http.Request) {
return
}
if err := r.ParseMultipartForm(20 * 1024 * 1024); err != nil { // 20M max memory, if bigger will make a file
r.Body = http.MaxBytesReader(w, r.Body, 256*1024*1024) // hard cap on upload to prevent memory exhaustion
reader, err := r.MultipartReader()
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't parse multipart form", rest.ErrDecode)
return
}
file, _, err := r.FormFile("file")
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't get import file from the request", rest.ErrInternal)
return
}
defer func() { _ = file.Close() }()
tmpfile := ""
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't parse multipart form", rest.ErrDecode)
return
}
if part.FormName() != "file" {
_ = part.Close()
continue
}
tmpfile, err := m.saveTemp(file)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't save request to temp file", rest.ErrInternal)
tmpfile, err = m.saveTemp(part)
if closeErr := part.Close(); err == nil && closeErr != nil {
err = closeErr
}
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't save request to temp file", rest.ErrInternal)
return
}
break
}
if tmpfile == "" {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, fmt.Errorf("file field missing"),
"can't get import file from the request", rest.ErrInternal)
return
}
go m.runImport(siteID, r.URL.Query().Get("provider"), tmpfile) // import runs in background and sets busy flag for site
render.Status(r, http.StatusAccepted)
render.JSON(w, r, R.JSON{"status": "import request accepted"})
_ = R.EncodeJSON(w, http.StatusAccepted, R.JSON{"status": "import request accepted"})
}
// GET /wait?site=site-id
@@ -110,20 +132,16 @@ func (m *Migrator) waitCtrl(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(context.Background(), timeOut)
defer cancel()
for {
if !m.isBusy(siteID) {
break
}
for m.isBusy(siteID) {
select {
case <-ctx.Done():
render.Status(r, http.StatusGatewayTimeout)
render.JSON(w, r, R.JSON{"status": "timeout expired", "site_id": siteID})
_ = R.EncodeJSON(w, http.StatusGatewayTimeout, R.JSON{"status": "timeout expired", "site_id": siteID})
return
case <-time.After(100 * time.Millisecond):
}
}
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"status": "completed", "site_id": siteID})
R.RenderJSON(w, R.JSON{"status": "completed", "site_id": siteID})
}
// GET /export?site=site-id&secret=12345&?mode=file|stream
@@ -131,25 +149,47 @@ func (m *Migrator) waitCtrl(w http.ResponseWriter, r *http.Request) {
func (m *Migrator) exportCtrl(w http.ResponseWriter, r *http.Request) {
siteID := r.URL.Query().Get("site")
var writer io.Writer = w
if r.URL.Query().Get("mode") == "file" {
// buffer to memory to handle errors before committing to response
var buf bytes.Buffer
gzWriter := gzip.NewWriter(&buf)
if _, err := m.NativeExporter.Export(gzWriter, siteID); err != nil {
code, errCode := exportErrStatus(err)
rest.SendErrorJSON(w, r, code, err, "export failed", errCode)
return
}
if err := gzWriter.Close(); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "export failed", rest.ErrInternal)
return
}
exportFile := fmt.Sprintf("%s-%s.json.gz", siteID, time.Now().Format("20060102"))
w.Header().Set("Content-Type", "application/gzip")
w.Header().Set("Content-Disposition", "attachment;filename="+exportFile)
w.WriteHeader(http.StatusOK)
gzWriter := gzip.NewWriter(w)
defer func() {
if e := gzWriter.Close(); e != nil {
log.Printf("[WARN] can't close gzip writer, %s", e)
}
}()
writer = gzWriter
}
if _, err := m.NativeExporter.Export(writer, siteID); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "export failed", rest.ErrInternal)
w.Header().Set("Content-Length", strconv.Itoa(buf.Len()))
if _, err := io.Copy(w, &buf); err != nil {
log.Printf("[WARN] failed to write export response: %v", err)
}
return
}
// stream mode - write directly to response
if _, err := m.NativeExporter.Export(w, siteID); err != nil {
code, errCode := exportErrStatus(err)
rest.SendErrorJSON(w, r, code, err, "export failed", errCode)
}
}
// exportErrStatus maps an export failure to an HTTP status and error code: an unknown
// site is a client error (400), anything else is treated as internal (500).
// The bolt store returns the engine.ErrSiteNotFound sentinel; the rpc store loses typed
// errors over jrpc, so the "not found" message is matched as a fallback (export only ever
// hits a site-level lookup, so a "not found" here can only mean the site).
func exportErrStatus(err error) (status, errCode int) {
if errors.Is(err, engine.ErrSiteNotFound) || strings.Contains(err.Error(), "not found") {
return http.StatusBadRequest, rest.ErrSiteNotFound
}
return http.StatusInternalServerError, rest.ErrInternal
}
// POST /remap?site=site-id
@@ -177,7 +217,7 @@ func (m *Migrator) remapCtrl(w http.ResponseWriter, r *http.Request) {
return
}
defer func() {
if e = os.Remove(fh.Name()); e != nil {
if e = os.Remove(fh.Name()); e != nil { //nolint:gosec // fh.Name() is from os.CreateTemp, server-controlled
log.Printf("[WARN] failed to remove temp file %+v", e)
}
}()
@@ -204,8 +244,7 @@ func (m *Migrator) remapCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[DEBUG] convert request completed. site=%s, comments=%d", siteID, size)
}()
render.Status(r, http.StatusAccepted)
render.JSON(w, r, R.JSON{"status": "convert request accepted"})
_ = R.EncodeJSON(w, http.StatusAccepted, R.JSON{"status": "convert request accepted"})
}
// runImport reads from tmpfile and import for given siteID and provider
+197 -15
View File
@@ -9,6 +9,7 @@ import (
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
@@ -33,7 +34,7 @@ func TestMigrator_Import(t *testing.T) {
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah2"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
@@ -49,6 +50,22 @@ func TestMigrator_Import(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportForm(t *testing.T) {
@@ -84,15 +101,31 @@ func TestMigrator_ImportForm(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportFromWP(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(strings.Replace(xmlTestWP, "'", "`", -1))
r := strings.NewReader(strings.ReplaceAll(xmlTestWP, "'", "`"))
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=wordpress", r)
assert.NoError(t, err)
@@ -108,6 +141,22 @@ func TestMigrator_ImportFromWP(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://realmenweardress.es/2010/07/do-you-rp/")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 3, comments.Info.Count)
require.Equal(t, 3, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://realmenweardress.es/2010/07/do-you-rp/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 3, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments), "2 comments with 1 reply")
}
func TestMigrator_ImportFromCommento(t *testing.T) {
@@ -115,13 +164,13 @@ func TestMigrator_ImportFromCommento(t *testing.T) {
defer teardown()
r := strings.NewReader(`{"version":1,"comments":[{"commentHex":"7d77e39fcd813241d6281478cc8f21ab5f807d043c750bc1a936bc23b34fb854",
"domain":"example.com","url":"https://example.com/blog/post/1","commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"domain":"example.com","url":"/blog/post/1","commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"markdown":"Example content","html":"","parentHex":"root","score":0,"state":"approved","creationDate":"2021-03-17T12:09:47.722181Z",
"direction":0,"deleted":false}],"commenters":[{"commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"email":"somegreatmail@gmail.com","name":"User5276","link":"https://example.com/profile/257","photo":"https://secure.gravatar.com/avatar/8f279626d26175134b0d5c88648172f7",
"provider":"sso:example.com","joinDate":"2021-03-19T19:27:25.954285Z","isModerator":false}]}`)
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=commento", r)
assert.NoError(t, err)
@@ -137,6 +186,63 @@ func TestMigrator_ImportFromCommento(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/blog/post/1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://example.com/blog/post/1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportFromCommentoJSON(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r, err := os.Open("testdata/commento.json")
require.NoError(t, err)
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=commento", r)
assert.NoError(t, err)
req.Header.Add("Content-Type", "application/json; charset=utf-8")
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 7, comments.Info.Count)
require.Equal(t, 7, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 7, comments.Info.Count)
require.Equal(t, 5, len(comments.Comments), "five comments with two replies")
}
func TestMigrator_ImportRejected(t *testing.T) {
@@ -152,7 +258,7 @@ func TestMigrator_ImportRejected(t *testing.T) {
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah2"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native&secret=XYZ", r)
assert.NoError(t, err)
@@ -170,14 +276,18 @@ func TestMigrator_ImportDouble(t *testing.T) {
"picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}`
recs := []string{}
for i := 0; i < 50; i++ {
recs := make([]string, 0, 50)
for i := range 50 {
recs = append(recs, fmt.Sprintf(tmpl, i))
}
r := strings.NewReader(`{"version":1}` + strings.Join(recs, "\n")) // reader with 10k records
client := &http.Client{Timeout: 1 * time.Second}
// each request needs its own reader. client.Do returns once the response headers are in, which
// for an accepted import is before the transport's writeLoop has finished copying the body, so
// handing the same strings.Reader to the second NewRequest races that copy: NewRequest reads
// Len() to set ContentLength while WriteTo is still advancing it
body := `{"version":1}` + strings.Join(recs, "\n")
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", strings.NewReader(body))
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
assert.NoError(t, err)
@@ -188,7 +298,7 @@ func TestMigrator_ImportDouble(t *testing.T) {
client = &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err = http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
req, err = http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", strings.NewReader(body))
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
assert.NoError(t, err)
@@ -197,6 +307,20 @@ func TestMigrator_ImportDouble(t *testing.T) {
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusConflict, resp.StatusCode)
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 50, comments.Info.Count)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 50, comments.Info.Count)
}
func TestMigrator_ImportWaitExpired(t *testing.T) {
@@ -209,7 +333,7 @@ func TestMigrator_ImportWaitExpired(t *testing.T) {
"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}`
nRecs := 50
recs := make([]string, 0, nRecs)
for i := 0; i < nRecs; i++ {
for i := range nRecs {
recs = append(recs, fmt.Sprintf(tmpl, i))
}
r := strings.NewReader(`{"version":1}` + strings.Join(recs, "\n")) // reader with `nRecs` records
@@ -236,6 +360,14 @@ func TestMigrator_ImportWaitExpired(t *testing.T) {
assert.Equal(t, http.StatusGatewayTimeout, resp.StatusCode)
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
}
func TestMigrator_Export(t *testing.T) {
@@ -252,7 +384,7 @@ func TestMigrator_Export(t *testing.T) {
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
// import comments first
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
@@ -263,6 +395,32 @@ func TestMigrator_Export(t *testing.T) {
require.Equal(t, http.StatusAccepted, resp.StatusCode)
waitForMigrationCompletion(t, ts)
// export unknown site is a client error, not internal
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=file&site=test", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
errBody, err := io.ReadAll(resp.Body)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusBadRequest, resp.StatusCode)
require.Equal(t, "application/json", resp.Header.Get("Content-Type"))
assert.Contains(t, string(errBody), `"code":6`) // rest.ErrSiteNotFound, not ErrInternal
assert.Contains(t, string(errBody), `not found`) // error detail names the missing site
// unknown site in stream mode is also a client error
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=stream&site=test", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
errBody, err = io.ReadAll(resp.Body)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusBadRequest, resp.StatusCode)
assert.Contains(t, string(errBody), `"code":6`)
// check file mode
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=file&site=remark42", http.NoBody)
require.NoError(t, err)
@@ -339,6 +497,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo-another/")
@@ -347,6 +506,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
// we want remap urls to another domain - www.remark42.com
@@ -364,6 +524,16 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://www.remark42.com/demo/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://www.remark42.com/demo-another/")
@@ -372,6 +542,16 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://www.remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
// should find nothing from previous url
@@ -381,6 +561,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
@@ -388,6 +569,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
}
func TestMigrator_RemapReject(t *testing.T) {
@@ -395,7 +577,7 @@ func TestMigrator_RemapReject(t *testing.T) {
defer teardown()
// without admin credentials
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
rules := strings.NewReader(`https://remark42.com/* https://www.remark42.com/*`)
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/remap?site=remark42", rules)
+238 -338
View File
@@ -3,29 +3,22 @@ package api
import (
"bytes"
"context"
"embed"
"encoding/json"
"fmt"
"io/fs"
"net"
"net/http"
"net/mail"
"os"
"regexp"
"strings"
"sync"
"time"
"github.com/didip/tollbooth/v7"
"github.com/didip/tollbooth_chi"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/go-chi/cors"
"github.com/go-chi/render"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/rest/logger"
"github.com/go-pkgz/routegroup"
"github.com/umputun/remark42/backend/app/notify"
"github.com/umputun/remark42/backend/app/rest"
@@ -33,6 +26,7 @@ import (
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/image"
"github.com/umputun/remark42/backend/app/store/service"
"github.com/umputun/remark42/backend/app/webassets"
)
// Rest is a rest access server
@@ -51,34 +45,39 @@ type Rest struct {
AnonVote bool
WebRoot string
WebFS embed.FS
WebFS fs.FS
RemarkURL string
ReadOnlyAge int
SharedSecret string
TrustedProxies []*net.IPNet // reverse-proxy networks whose forwarding headers (X-Real-IP, X-Forwarded-For, ...) are trusted
ScoreThresholds struct {
Low int
Critical int
}
UpdateLimiter float64
EmailNotifications bool
TelegramNotifications bool
EmojiEnabled bool
SimpleView bool
ProxyCORS bool
SendJWTHeader bool
AllowedAncestors []string // sets Content-Security-Policy "frame-ancestors ..."
SubscribersOnly bool
DisableSignature bool // prevent signature from being added to headers
UpdateLimiter float64
EmailNotifications bool
TelegramNotifications bool
EmojiEnabled bool
SimpleView bool
ProxyCORS bool
SendJWTHeader bool
AllowedAncestors []string // sets Content-Security-Policy "frame-ancestors ..."
SubscribersOnly bool
DisableSignature bool // prevent signature from being added to headers
DisableFancyTextFormatting bool // disables SmartyPants in the comment text rendering of the posted comments
ExternalImageProxy bool
SSLConfig SSLConfig
httpsServer *http.Server
httpServer *http.Server
lock sync.Mutex
SSLConfig SSLConfig
httpsServer *http.Server
httpServer *http.Server
shutdownRequested bool
lock sync.Mutex
pubRest public
privRest private
adminRest admin
rssRest rss
pubRest public
privRest private
adminRest admin
rssRest rss
openRouteLimiter float64
}
// LoadingCache defines interface for caching
@@ -89,12 +88,17 @@ type LoadingCache interface {
}
const hardBodyLimit = 1024 * 64 // limit size of body
const openRouteLimiter = 10 // limit for open routes
const lastCommentsScope = "last"
type commentsWithInfo struct {
Comments []store.Comment `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}
type treeWithInfo struct {
*service.Tree
Info store.PostInfo `json:"info"`
}
// Run the lister and request's router, activate rest server
@@ -110,6 +114,11 @@ func (s *Rest) Run(address string, port int) {
s.lock.Lock()
s.httpServer = s.makeHTTPServer(address, port, s.routes())
s.httpServer.ErrorLog = log.ToStdLogger(log.Default(), "WARN")
if s.shutdownRequested {
s.lock.Unlock()
log.Print("[WARN] rest server start canceled")
return
}
s.lock.Unlock()
err := s.httpServer.ListenAndServe()
@@ -123,6 +132,11 @@ func (s *Rest) Run(address string, port int) {
s.httpServer = s.makeHTTPServer(address, port, s.httpToHTTPSRouter())
s.httpServer.ErrorLog = log.ToStdLogger(log.Default(), "WARN")
if s.shutdownRequested {
s.lock.Unlock()
log.Print("[WARN] rest server start canceled")
return
}
s.lock.Unlock()
go func() {
@@ -143,6 +157,11 @@ func (s *Rest) Run(address string, port int) {
s.httpServer = s.makeHTTPServer(address, port, s.httpChallengeRouter(m))
s.httpServer.ErrorLog = log.ToStdLogger(log.Default(), "WARN")
if s.shutdownRequested {
s.lock.Unlock()
log.Print("[WARN] rest server start canceled")
return
}
s.lock.Unlock()
@@ -164,6 +183,7 @@ func (s *Rest) Shutdown() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
s.lock.Lock()
s.shutdownRequested = true
if s.httpServer != nil {
if err := s.httpServer.Shutdown(ctx); err != nil {
log.Printf("[DEBUG] http shutdown error, %s", err)
@@ -191,9 +211,14 @@ func (s *Rest) makeHTTPServer(address string, port int, router http.Handler) *ht
}
}
func (s *Rest) routes() chi.Router {
router := chi.NewRouter()
router.Use(middleware.Throttle(1000), middleware.RealIP, R.Recoverer(log.Default()))
func (s *Rest) routes() http.Handler {
if s.openRouteLimiter == 0 {
// set the default open route limiter. Just a safety measure as it should be set by Run method anyway
s.openRouteLimiter = openRouteLimiter
}
router := routegroup.New(http.NewServeMux())
router.Use(R.Throttle(1000), realIPMiddleware(s.TrustedProxies), R.Recoverer(log.Default()))
router.Use(securityHeadersMiddleware(s.ExternalImageProxy, s.AllowedAncestors))
if !s.DisableSignature {
router.Use(R.AppInfo("remark42", "umputun", s.Version))
}
@@ -204,20 +229,7 @@ func (s *Rest) routes() chi.Router {
if s.ProxyCORS {
log.Printf("[WARN] internal CORS disabled")
} else {
corsMiddleware := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowedMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS"},
AllowedHeaders: []string{"Accept", "Authorization", "Content-Type", "X-XSRF-Token", "X-JWT"},
ExposedHeaders: []string{"Authorization"},
AllowCredentials: true,
MaxAge: 300,
})
router.Use(corsMiddleware.Handler)
}
if len(s.AllowedAncestors) > 0 {
log.Printf("[INFO] allowed from %+v only", s.AllowedAncestors)
router.Use(frameAncestors(s.AllowedAncestors))
router.Use(corsMiddleware())
}
ipFn := func(ip string) string { return store.HashValue(ip, s.SharedSecret)[:12] } // logger uses it for anonymization
@@ -225,137 +237,164 @@ func (s *Rest) routes() chi.Router {
authHandler, avatarHandler := s.Authenticator.Handlers()
router.Group(func(r chi.Router) {
r.Use(middleware.Timeout(5 * time.Second))
r.Use(logInfoWithBody, tollbooth_chi.LimitHandler(tollbooth.NewLimiter(2, nil)), middleware.NoCache)
router.Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(5 * time.Second))
r.Use(logInfoWithBody, rateLimiter(2), R.NoCache)
r.Use(validEmailAuth()) // reject suspicious email logins
r.Mount("/auth", authHandler)
r.Handle("/auth/", authHandler)
})
router.Group(func(r chi.Router) {
r.Use(middleware.Timeout(5 * time.Second))
r.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(100, nil)))
r.Mount("/avatar", avatarHandler)
router.Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(5 * time.Second))
r.Use(rateLimiter(100))
r.Handle("/avatar/", avatarHandler)
})
authMiddleware := s.Authenticator.Middleware()
// api routes
router.Route("/api/v1", func(rapi chi.Router) {
rapi.Group(func(rava chi.Router) {
rava.Use(middleware.Timeout(5 * time.Second))
rava.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(100, nil)))
rava.Mount("/avatar", avatarHandler)
rapi := router.Mount("/api/v1")
rapi.Use(apiCSPMiddleware)
rapi.Group().Route(func(rava *routegroup.Bundle) {
rava.Use(R.Timeout(5 * time.Second))
rava.Use(rateLimiter(100))
rava.Handle("/avatar/", avatarHandler)
})
// open routes
rapi.Group().Route(func(ropen *routegroup.Bundle) {
ropen.Use(R.Timeout(30 * time.Second))
ropen.Use(rateLimiter(s.openRouteLimiter))
ropen.Use(authMiddleware.Trace, R.NoCache, logInfoWithBody)
ropen.HandleFunc("GET /config", s.configCtrl)
ropen.HandleFunc("GET /find", s.pubRest.findCommentsCtrl)
ropen.HandleFunc("GET /id/{id}", s.pubRest.commentByIDCtrl)
ropen.HandleFunc("GET /comments", s.pubRest.findUserCommentsCtrl)
ropen.HandleFunc("GET /last/{limit}", s.pubRest.lastCommentsCtrl)
ropen.HandleFunc("GET /count", s.pubRest.countCtrl)
ropen.HandleFunc("POST /counts", s.pubRest.countMultiCtrl)
ropen.HandleFunc("GET /list", s.pubRest.listCtrl)
ropen.HandleFunc("GET /info", s.pubRest.infoCtrl)
ropen.Mount("/rss").Route(func(rrss *routegroup.Bundle) {
rrss.HandleFunc("GET /post", s.rssRest.postCommentsCtrl)
rrss.HandleFunc("GET /site", s.rssRest.siteCommentsCtrl)
rrss.HandleFunc("GET /reply", s.rssRest.repliesCtrl)
})
})
// open routes, cached. /img lives here (not in the NoCache group above) because
// R.NoCache strips If-None-Match from incoming requests, which would
// defeat the proxy handler's 304 short-circuit. The handler sets a 30-day
// max-age on validated success responses (with a versioned etag for cache
// invalidation on revalidation); error responses get Cache-Control: no-store
// so transient failures aren't pinned in the cache.
rapi.Group().Route(func(ropen *routegroup.Bundle) {
ropen.Use(R.Timeout(30 * time.Second))
ropen.Use(rateLimiter(10))
ropen.Use(authMiddleware.Trace, logInfoWithBody)
ropen.HandleFunc("GET /img", s.ImageProxy.Handler)
ropen.HandleFunc("GET /picture/{user}/{id}", s.pubRest.loadPictureCtrl)
ropen.HandleFunc("GET /qr/telegram", s.pubRest.telegramQrCtrl)
})
// protected routes, require auth
rapi.Group().Route(func(rauth *routegroup.Bundle) {
rauth.Use(rateLimiter(10))
rauth.Use(authMiddleware.Auth, matchSiteID, R.NoCache, logInfoWithBody)
// GET /userdata streams a gzipped export of the user's data straight to the client, so it
// deliberately runs without R.Timeout: that middleware buffers the whole response in memory
// before sending and aborts at the deadline, which would hold a full export in RAM and truncate it.
rauth.HandleFunc("GET /userdata", s.privRest.userAllDataCtrl)
rauth.Group().Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(30 * time.Second))
r.HandleFunc("GET /user", s.privRest.userInfoCtrl)
})
})
// admin routes, require auth and admin users only
rapi.Mount("/admin").Route(func(radmin *routegroup.Bundle) {
radmin.Use(rateLimiter(10))
radmin.Use(authMiddleware.Auth, authMiddleware.AdminOnly, matchSiteID)
radmin.Use(R.NoCache, logInfoWithBody)
// bounded admin operations return small responses and get the enforcing request timeout
radmin.Group().Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(30 * time.Second))
r.HandleFunc("DELETE /comment/{id}", s.adminRest.deleteCommentCtrl)
r.HandleFunc("PUT /user/{userid}", s.adminRest.setBlockCtrl)
r.HandleFunc("DELETE /user/{userid}", s.adminRest.deleteUserCtrl)
r.HandleFunc("GET /user/{userid}", s.adminRest.getUserInfoCtrl)
r.With(rejectHead("GET")).HandleFunc("GET /deleteme", s.adminRest.deleteMeRequestCtrl)
r.HandleFunc("PUT /verify/{userid}", s.adminRest.setVerifyCtrl)
r.HandleFunc("PUT /pin/{id}", s.adminRest.setPinCtrl)
r.HandleFunc("GET /blocked", s.adminRest.blockedUsersCtrl)
r.HandleFunc("PUT /readonly", s.adminRest.setReadOnlyCtrl)
r.HandleFunc("PUT /title/{id}", s.adminRest.setTitleCtrl)
})
// open routes
rapi.Group(func(ropen chi.Router) {
ropen.Use(middleware.Timeout(30 * time.Second))
ropen.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
ropen.Use(authMiddleware.Trace, middleware.NoCache, logInfoWithBody)
ropen.Get("/config", s.configCtrl)
ropen.Get("/find", s.pubRest.findCommentsCtrl)
ropen.Get("/id/{id}", s.pubRest.commentByIDCtrl)
ropen.Get("/comments", s.pubRest.findUserCommentsCtrl)
ropen.Get("/last/{limit}", s.pubRest.lastCommentsCtrl)
ropen.Get("/count", s.pubRest.countCtrl)
ropen.Post("/counts", s.pubRest.countMultiCtrl)
ropen.Get("/list", s.pubRest.listCtrl)
ropen.Get("/info", s.pubRest.infoCtrl)
ropen.Get("/img", s.ImageProxy.Handler)
// migrator routes deliberately run without R.Timeout: GET /export streams a full-site
// backup, GET /wait long-polls for up to 15m, and import/remap ingest large uploads. The
// enforcing timeout buffers the whole response and aborts at the deadline, which would
// truncate backups, break waiting, and reject large imports.
radmin.HandleFunc("GET /export", s.adminRest.migrator.exportCtrl)
radmin.HandleFunc("POST /import", s.adminRest.migrator.importCtrl)
radmin.HandleFunc("POST /import/form", s.adminRest.migrator.importFormCtrl)
radmin.HandleFunc("POST /remap", s.adminRest.migrator.remapCtrl)
radmin.HandleFunc("GET /wait", s.adminRest.migrator.waitCtrl)
})
ropen.Route("/rss", func(rrss chi.Router) {
rrss.Get("/post", s.rssRest.postCommentsCtrl)
rrss.Get("/site", s.rssRest.siteCommentsCtrl)
rrss.Get("/reply", s.rssRest.repliesCtrl)
})
})
// protected routes, throttled to 10/s by default, controlled by external UpdateLimiter param
rapi.Group().Route(func(rauth *routegroup.Bundle) {
rauth.Use(R.Timeout(10 * time.Second))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, matchSiteID, subscribersOnly(s.SubscribersOnly))
rauth.Use(R.NoCache, logInfoWithBody)
// open routes, cached
rapi.Group(func(ropen chi.Router) {
ropen.Use(middleware.Timeout(30 * time.Second))
ropen.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
ropen.Use(authMiddleware.Trace, logInfoWithBody)
ropen.Get("/picture/{user}/{id}", s.pubRest.loadPictureCtrl)
ropen.Get("/qr/telegram", s.pubRest.telegramQrCtrl)
})
rauth.HandleFunc("PUT /comment/{id}", s.privRest.updateCommentCtrl)
rauth.HandleFunc("POST /preview", s.privRest.previewCommentCtrl)
rauth.HandleFunc("POST /comment", s.privRest.createCommentCtrl)
rauth.HandleFunc("PUT /vote/{id}", s.privRest.voteCtrl)
rauth.With(rejectAnonUser).HandleFunc("POST /deleteme", s.privRest.deleteMeCtrl)
rauth.With(rejectAnonUser).HandleFunc("GET /email", s.privRest.getEmailCtrl)
rauth.With(rejectAnonUser).HandleFunc("POST /email/subscribe", s.privRest.sendEmailConfirmationCtrl)
rauth.With(rejectAnonUser).HandleFunc("POST /email/confirm", s.privRest.setConfirmedEmailCtrl)
rauth.With(rejectAnonUser).HandleFunc("DELETE /email", s.privRest.deleteEmailCtrl)
rauth.With(rejectAnonUser, rejectHead("GET")).HandleFunc("GET /telegram/subscribe", s.privRest.telegramSubscribeCtrl)
rauth.With(rejectAnonUser).HandleFunc("DELETE /telegram", s.privRest.deleteTelegramCtrl)
})
// protected routes, require auth
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(30 * time.Second))
rauth.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
rauth.Use(authMiddleware.Auth, matchSiteID, middleware.NoCache, logInfoWithBody)
rauth.Get("/user", s.privRest.userInfoCtrl)
rauth.Get("/userdata", s.privRest.userAllDataCtrl)
})
// admin routes, require auth and admin users only
rapi.Route("/admin", func(radmin chi.Router) {
radmin.Use(middleware.Timeout(30 * time.Second))
radmin.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
radmin.Use(authMiddleware.Auth, authMiddleware.AdminOnly, matchSiteID)
radmin.Use(middleware.NoCache, logInfoWithBody)
radmin.Delete("/comment/{id}", s.adminRest.deleteCommentCtrl)
radmin.Put("/user/{userid}", s.adminRest.setBlockCtrl)
radmin.Delete("/user/{userid}", s.adminRest.deleteUserCtrl)
radmin.Get("/user/{userid}", s.adminRest.getUserInfoCtrl)
radmin.Get("/deleteme", s.adminRest.deleteMeRequestCtrl)
radmin.Put("/verify/{userid}", s.adminRest.setVerifyCtrl)
radmin.Put("/pin/{id}", s.adminRest.setPinCtrl)
radmin.Get("/blocked", s.adminRest.blockedUsersCtrl)
radmin.Put("/readonly", s.adminRest.setReadOnlyCtrl)
radmin.Put("/title/{id}", s.adminRest.setTitleCtrl)
// migrator
radmin.Get("/export", s.adminRest.migrator.exportCtrl)
radmin.Post("/import", s.adminRest.migrator.importCtrl)
radmin.Post("/import/form", s.adminRest.migrator.importFormCtrl)
radmin.Post("/remap", s.adminRest.migrator.remapCtrl)
radmin.Get("/wait", s.adminRest.migrator.waitCtrl)
})
// protected routes, throttled to 10/s by default, controlled by external UpdateLimiter param
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(10 * time.Second))
rauth.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(s.updateLimiter(), nil)))
rauth.Use(authMiddleware.Auth, matchSiteID, subscribersOnly(s.SubscribersOnly))
rauth.Use(middleware.NoCache, logInfoWithBody)
rauth.Put("/comment/{id}", s.privRest.updateCommentCtrl)
rauth.Post("/preview", s.privRest.previewCommentCtrl)
rauth.Post("/comment", s.privRest.createCommentCtrl)
rauth.Put("/vote/{id}", s.privRest.voteCtrl)
rauth.With(rejectAnonUser).Post("/deleteme", s.privRest.deleteMeCtrl)
rauth.With(rejectAnonUser).Get("/email", s.privRest.getEmailCtrl)
rauth.With(rejectAnonUser).Post("/email/subscribe", s.privRest.sendEmailConfirmationCtrl)
rauth.With(rejectAnonUser).Post("/email/confirm", s.privRest.setConfirmedEmailCtrl)
rauth.With(rejectAnonUser).Delete("/email", s.privRest.deleteEmailCtrl)
rauth.With(rejectAnonUser).Get("/telegram/subscribe", s.privRest.telegramSubscribeCtrl)
rauth.With(rejectAnonUser).Delete("/telegram", s.privRest.deleteTelegramCtrl)
})
// protected routes, anonymous rejected
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(10 * time.Second))
rauth.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(s.updateLimiter(), nil)))
rauth.Use(authMiddleware.Auth, rejectAnonUser, matchSiteID)
rauth.Use(logger.New(logger.Log(log.Default()), logger.Prefix("[DEBUG]"), logger.IPfn(ipFn)).Handler)
rauth.Post("/picture", s.privRest.savePictureCtrl)
})
// protected routes, anonymous rejected
rapi.Group().Route(func(rauth *routegroup.Bundle) {
rauth.Use(R.Timeout(10 * time.Second))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, rejectAnonUser, matchSiteID)
rauth.Use(logger.New(logger.Log(log.Default()), logger.Prefix("[DEBUG]"), logger.IPfn(ipFn)).Handler)
rauth.HandleFunc("POST /picture", s.privRest.savePictureCtrl)
})
// open routes on root level
router.Group(func(rroot chi.Router) {
rroot.Use(middleware.Timeout(10 * time.Second))
rroot.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(50, nil)))
rroot.Get("/robots.txt", s.pubRest.robotsCtrl)
rroot.Get("/email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
rroot.Post("/email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
router.Route(func(rroot *routegroup.Bundle) {
rroot.Use(R.Timeout(10 * time.Second))
rroot.Use(rateLimiter(50))
rroot.HandleFunc("GET /robots.txt", s.pubRest.robotsCtrl)
rroot.With(rejectHead("GET, POST")).HandleFunc("GET /email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
rroot.HandleFunc("POST /email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
})
// file server for static content from s.WebRoot on path /web
addFileServer(router, s.WebFS, s.WebRoot, s.Version)
// file server for /web: the frontend build first, then the assets embedded in the binary.
// the build is embedded under web/ by app/cmd, so that prefix is stripped here. fs.Sub only
// fails for an fs.SubFS that refuses, and a nil result would panic on the first request, so
// serve nothing from the frontend rather than serving it at the wrong paths
embeddedFrontend, err := fs.Sub(s.WebFS, "web")
if err != nil {
log.Printf("[WARN] no embedded frontend, serving built-in assets only: %v", err)
embeddedFrontend = emptyFS{}
}
addFileServer(router, embeddedFrontend, s.WebRoot, s.Version, s.RemarkURL)
return router
}
@@ -369,16 +408,17 @@ func (s *Rest) controllerGroups() (public, private, admin, rss) {
}
privGrp := private{
dataService: s.DataService,
cache: s.Cache,
imageService: s.ImageService,
commentFormatter: s.CommentFormatter,
readOnlyAge: s.ReadOnlyAge,
authenticator: s.Authenticator,
notifyService: s.NotifyService,
telegramService: s.TelegramService,
remarkURL: s.RemarkURL,
anonVote: s.AnonVote,
dataService: s.DataService,
cache: s.Cache,
imageService: s.ImageService,
commentFormatter: s.CommentFormatter,
readOnlyAge: s.ReadOnlyAge,
authenticator: s.Authenticator,
notifyService: s.NotifyService,
telegramService: s.TelegramService,
remarkURL: s.RemarkURL,
anonVote: s.AnonVote,
disableFancyTextFormatting: s.DisableFancyTextFormatting,
}
admGrp := admin{
@@ -417,6 +457,7 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
Version string `json:"version"`
EditDuration int `json:"edit_duration"`
AdminEdit bool `json:"admin_edit"`
MinCommentSize int `json:"min_comment_size"`
MaxCommentSize int `json:"max_comment_size"`
Admins []string `json:"admins"`
AdminEmail string `json:"admin_email"`
@@ -437,6 +478,7 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
Version: s.Version,
EditDuration: int(s.DataService.EditDuration.Seconds()),
AdminEdit: s.DataService.AdminEdits,
MinCommentSize: s.DataService.MinCommentSize,
MaxCommentSize: s.DataService.MaxCommentSize,
Admins: admins,
AdminEmail: emails,
@@ -462,30 +504,39 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
if cnf.Admins == nil { // prevent json serialization to nil
cnf.Admins = []string{}
}
render.Status(r, http.StatusOK)
render.JSON(w, r, cnf)
R.RenderJSON(w, cnf)
}
// serves static files from the webRoot directory or files embedded into the compiled binary if that directory is absent
func addFileServer(r chi.Router, embedFS embed.FS, webRoot, version string) {
var webFS http.Handler
// serves /web from the frontend build, falling back to the assets embedded in the binary for
// names the build does not produce. the frontend build is read from webRoot on disk, or from the
// copy embedded at app/cmd/web when that directory is absent.
func addFileServer(r *routegroup.Bundle, embeddedFrontend fs.FS, webRoot, version, remarkURL string) {
frontendFS := embeddedFrontend
if _, err := os.Stat(webRoot); err == nil {
log.Printf("[INFO] run file server from %s from the disk", webRoot)
webFS = http.FileServer(http.Dir(webRoot))
frontendFS = os.DirFS(webRoot)
} else {
log.Printf("[INFO] run file server, embedded")
var contentFS, _ = fs.Sub(embedFS, "web")
webFS = http.FileServer(http.FS(contentFS))
}
webFS = http.StripPrefix("/web", webFS)
r.Get("/web", http.RedirectHandler("/web/", http.StatusMovedPermanently).ServeHTTP)
// wrapped rather than substituted once at startup: the disk root can change under a running
// server, and the docker image has already substituted its copy, where this is a no-op
sources := templatedFS{
fs: webFiles{frontend: frontendFS, embedded: webassets.FS},
remarkURL: remarkURL,
}
webFS := http.StripPrefix("/web", http.FileServer(http.FS(sources)))
r.HandleFunc("GET /web", http.RedirectHandler("/web/", http.StatusMovedPermanently).ServeHTTP)
r.With(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(20, nil)),
middleware.Timeout(10*time.Second),
cacheControl(time.Hour, version),
).Get("/web/*", func(w http.ResponseWriter, r *http.Request) {
r.With(rateLimiter(20),
R.Timeout(10*time.Second),
// the served body now depends on remarkURL, so it has to be part of the validator. Without
// it an operator who corrects a wrong REMARK_URL and restarts the same binary keeps getting
// 304 on revalidation, and the client keeps a bundle addressed to the old host for good,
// since no-cache means it revalidates rather than aging out
cacheControl(time.Hour, version+":"+remarkURL),
).HandleFunc("GET /web/", func(w http.ResponseWriter, r *http.Request) {
// don't show dirs, just serve files
if strings.HasSuffix(r.URL.Path, "/") && len(r.URL.Path) > 1 && r.URL.Path != ("/web/") {
http.NotFound(w, r)
@@ -495,7 +546,7 @@ func addFileServer(r chi.Router, embedFS embed.FS, webRoot, version string) {
})
}
func encodeJSONWithHTML(v interface{}) ([]byte, error) {
func encodeJSONWithHTML(v any) ([]byte, error) {
buf := &bytes.Buffer{}
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
@@ -541,157 +592,6 @@ func URLKeyWithUser(r *http.Request) string {
return key
}
// rejectAnonUser is a middleware rejecting anonymous users
func rejectAnonUser(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if strings.HasPrefix(user.ID, "anonymous_") {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// matchSiteID is a middleware rejecting users with mismatch between site param and and User.SiteID
func matchSiteID(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
// skip for basic auth user
if user.Name == "admin" && user.ID == "admin" {
next.ServeHTTP(w, r)
return
}
siteID := r.URL.Query().Get("site")
if siteID != "" && user.SiteID != siteID {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// cacheControl is a middleware setting cache expiration. Using url+version as etag
func cacheControl(expiration time.Duration, version string) func(http.Handler) http.Handler {
etag := func(r *http.Request, version string) string {
s := version + ":" + r.URL.String()
return store.EncodeID(s)
}
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
e := `"` + etag(r, version) + `"`
w.Header().Set("Etag", e)
w.Header().Set("Cache-Control", fmt.Sprintf("max-age=%d, no-cache", int(expiration.Seconds())))
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, e) {
w.WriteHeader(http.StatusNotModified)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// frameAncestors is a middleware setting Content-Security-Policy "frame-ancestors host1 host2 ..."
// prevents loading of comments widgets from any other origins. In case if the list of allowed empty, ignored.
func frameAncestors(hosts []string) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if len(hosts) == 0 {
h.ServeHTTP(w, r)
return
}
w.Header().Set("Content-Security-Policy", "frame-ancestors "+strings.Join(hosts, " ")+";")
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// subscribersOnly is a middleware rejecting non-paid_sub users
func subscribersOnly(enable bool) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if enable {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if !user.PaidSub {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// validEmailAuth is a middleware for auth endpoints for email method.
// it rejects login request if user, site or email are suspicious
func validEmailAuth() func(http.Handler) http.Handler {
reUser := regexp.MustCompile(`^[\p{L}\d\s_]{4,64}$`) // matches ui side validation, adding min/max limitation
reSite := regexp.MustCompile(`^[a-zA-Z\d\s_.-]{1,64}$`)
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/auth/email/login" {
// not email login, skip the check
h.ServeHTTP(w, r)
return
}
if u := r.URL.Query().Get("user"); u != "" {
if !reUser.MatchString(u) {
log.Printf("[WARN] suspicious user rejected: %s", u)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if a := r.URL.Query().Get("address"); a != "" {
if _, err := mail.ParseAddress(a); err != nil {
log.Printf("[WARN] suspicious address rejected: %s", a)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if s := r.URL.Query().Get("site"); s != "" {
if !reSite.MatchString(s) {
log.Printf("[WARN] suspicious site rejected: %s", s)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
func parseError(err error, defaultCode int) (code int) {
code = defaultCode
+85 -74
View File
@@ -6,22 +6,21 @@ import (
"crypto/rand"
"crypto/sha1" //nolint:gosec //not used for security
"encoding/json"
"errors"
"fmt"
"html/template"
"io"
"net"
"net/http"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/golang-jwt/jwt"
"github.com/hashicorp/go-multierror"
"github.com/golang-jwt/jwt/v5"
"github.com/umputun/remark42/backend/app/notify"
"github.com/umputun/remark42/backend/app/rest"
@@ -33,16 +32,17 @@ import (
)
type private struct {
dataService privStore
cache LoadingCache
readOnlyAge int
commentFormatter *store.CommentFormatter
imageService *image.Service
notifyService *notify.Service
authenticator *auth.Service
telegramService telegramService
remarkURL string
anonVote bool
dataService privStore
cache LoadingCache
readOnlyAge int
commentFormatter *store.CommentFormatter
imageService *image.Service
notifyService *notify.Service
authenticator *auth.Service
telegramService telegramService
remarkURL string
anonVote bool
disableFancyTextFormatting bool // disables SmartyPants in the comment text rendering of the posted comments
}
// telegramService is a subset of Telegram service used for setting up user telegram notifications
@@ -75,7 +75,7 @@ func (s *private) previewCommentCtrl(w http.ResponseWriter, r *http.Request) {
user := rest.MustGetUserInfo(r)
comment := store.Comment{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &comment); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&comment); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't bind comment", rest.ErrDecode)
return
}
@@ -87,7 +87,7 @@ func (s *private) previewCommentCtrl(w http.ResponseWriter, r *http.Request) {
return
}
comment = s.commentFormatter.Format(comment)
comment = s.commentFormatter.Format(comment, s.disableFancyTextFormatting)
comment.Sanitize()
// check if images are valid, omit proxied images as they are lazy-loaded
@@ -98,14 +98,13 @@ func (s *private) previewCommentCtrl(w http.ResponseWriter, r *http.Request) {
return
}
}
render.HTML(w, r, comment.Text)
rest.HTMLResponse(w, http.StatusOK, comment.Text)
}
// POST /comment - adds comment, resets all immutable fields
func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
comment := store.Comment{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &comment); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&comment); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't bind comment", rest.ErrDecode)
return
}
@@ -120,14 +119,14 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
comment.PrepareUntrusted() // clean all fields user not supposed to set
comment.User = user
comment.User.IP = strings.Split(r.RemoteAddr, ":")[0]
comment.User.IP = extractIP(r.RemoteAddr)
comment.Orig = comment.Text // original comment text, prior to md render
if err := s.dataService.ValidateComment(&comment); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid comment", rest.ErrCommentValidation)
return
}
comment = s.commentFormatter.Format(comment)
comment = s.commentFormatter.Format(comment, s.disableFancyTextFormatting)
// check if images are valid, omit proxied images as they are lazy-loaded
for _, id := range s.imageService.ExtractNonProxiedPictures(comment.Text) {
@@ -150,7 +149,7 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
id, err := s.dataService.Create(comment)
if err == service.ErrRestrictedWordsFound {
if errors.Is(err, service.ErrRestrictedWordsFound) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid comment", rest.ErrCommentRestrictWords)
return
}
@@ -174,8 +173,7 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[DEBUG] created comment %+v", finalComment)
render.Status(r, http.StatusCreated)
render.JSON(w, r, &finalComment)
_ = R.EncodeJSON(w, http.StatusCreated, &finalComment)
}
// PUT /comment/{id}?site=siteID&url=post-url - update comment
@@ -186,14 +184,14 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
Delete bool
}{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &edit); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&edit); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't read comment details from body", rest.ErrDecode)
return
}
user := rest.MustGetUserInfo(r)
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
id := chi.URLParam(r, "id")
id := r.PathValue("id")
log.Printf("[DEBUG] update comment %s", id)
@@ -211,7 +209,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
editReq := service.EditRequest{
Text: s.commentFormatter.FormatText(edit.Text),
Text: s.commentFormatter.FormatText(edit.Text, s.disableFancyTextFormatting),
Orig: edit.Text,
Summary: edit.Summary,
Delete: edit.Delete,
@@ -219,7 +217,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
res, err := s.dataService.EditComment(locator, id, editReq)
if err == service.ErrRestrictedWordsFound {
if errors.Is(err, service.ErrRestrictedWordsFound) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid comment", rest.ErrCommentValidation)
return
}
@@ -231,7 +229,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
s.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.SiteID, locator.URL, lastCommentsScope, user.ID))
render.JSON(w, r, res)
R.RenderJSON(w, res)
}
// GET /user?site=siteID - returns user info
@@ -244,12 +242,12 @@ func (s *private) userInfoCtrl(w http.ResponseWriter, r *http.Request) {
if err != nil {
log.Printf("[WARN] can't read email for %s, %v", user.ID, err)
}
if len(email) > 0 {
if email != "" {
user.EmailSubscription = true
}
}
render.JSON(w, r, user)
R.RenderJSON(w, user)
}
// PUT /vote/{id}?site=siteID&url=post-url&vote=1 - vote for/against comment
@@ -260,7 +258,7 @@ func (s *private) voteCtrl(w http.ResponseWriter, r *http.Request) {
return
}
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
id := chi.URLParam(r, "id")
id := r.PathValue("id")
log.Printf("[DEBUG] vote for comment %s", id)
vote := r.URL.Query().Get("vote") == "1"
@@ -280,7 +278,7 @@ func (s *private) voteCtrl(w http.ResponseWriter, r *http.Request) {
Locator: locator,
CommentID: id,
UserID: user.ID,
UserIP: strings.Split(r.RemoteAddr, ":")[0],
UserIP: extractIP(r.RemoteAddr),
Val: vote,
}
comment, err := s.dataService.Vote(req)
@@ -290,7 +288,7 @@ func (s *private) voteCtrl(w http.ResponseWriter, r *http.Request) {
return
}
s.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL, comment.User.ID))
render.JSON(w, r, R.JSON{"id": comment.ID, "score": comment.Score})
R.RenderJSON(w, R.JSON{"id": comment.ID, "score": comment.Score})
}
// getEmailCtrl gets email address for authenticated user.
@@ -303,7 +301,7 @@ func (s *private) getEmailCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[WARN] can't read email for %s, %v", user.ID, err)
}
render.JSON(w, r, R.JSON{"user": user, "address": address})
R.RenderJSON(w, R.JSON{"user": user, "address": address})
}
// sendEmailConfirmationCtrl gets address and siteID from query, makes confirmation token and sends it to user.
@@ -320,7 +318,7 @@ func (s *private) sendEmailConfirmationCtrl(w http.ResponseWriter, r *http.Reque
Address string
autoConfirm bool
}{autoConfirm: true}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &subscribe); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&subscribe); err != nil {
if err != io.EOF {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't parse request body", rest.ErrDecode)
return
@@ -360,10 +358,10 @@ func (s *private) sendEmailConfirmationCtrl(w http.ResponseWriter, r *http.Reque
claims := token.Claims{
Handshake: &token.Handshake{ID: user.ID + "::" + subscribe.Address},
StandardClaims: jwt.StandardClaims{
Audience: r.URL.Query().Get("site"),
ExpiresAt: time.Now().Add(30 * time.Minute).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{r.URL.Query().Get("site")},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
Issuer: "remark42",
},
}
@@ -383,7 +381,7 @@ func (s *private) sendEmailConfirmationCtrl(w http.ResponseWriter, r *http.Reque
},
)
render.JSON(w, r, R.JSON{"user": user, "address": subscribe.Address, "updated": false})
R.RenderJSON(w, R.JSON{"user": user, "address": subscribe.Address, "updated": false})
}
// telegramSubscribeCtrl generates and verifies telegram notification request
@@ -411,7 +409,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
fmt.Errorf("already subscribed"), "telegram subscription is already set for this user, delete if first to re-subscribe", rest.ErrActionRejected)
return
}
// Generate and send token
// generate and send token
tkn, err := randToken()
if err != nil {
rest.SendErrorJSON(w, r, http.StatusForbidden, err, "failed to generate verification token", rest.ErrInternal)
@@ -421,7 +419,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
s.telegramService.AddToken(tkn, user.ID, siteID, expires)
render.JSON(w, r, R.JSON{"token": tkn, "bot": s.telegramService.GetBotUsername()})
R.RenderJSON(w, R.JSON{"token": tkn, "bot": s.telegramService.GetBotUsername()})
return
}
@@ -443,7 +441,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
return
}
render.JSON(w, r, R.JSON{"updated": true, "address": val})
R.RenderJSON(w, R.JSON{"updated": true, "address": val})
}
// setConfirmedEmailCtrl uses provided token parameter (generated by sendEmailConfirmationCtrl) to set email and add it to user token
@@ -455,7 +453,7 @@ func (s *private) setConfirmedEmailCtrl(w http.ResponseWriter, r *http.Request)
Site string
Token string
}{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &confirm); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&confirm); err != nil {
if err != io.EOF {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't parse request body", rest.ErrDecode)
return
@@ -480,7 +478,7 @@ func (s *private) setConfirmedEmailCtrl(w http.ResponseWriter, r *http.Request)
return
}
// Handshake.ID is user.ID + "::" + address
// handshake.ID is user.ID + "::" + address
elems := strings.Split(confClaims.Handshake.ID, "::")
if len(elems) != 2 || elems[0] != user.ID {
rest.SendErrorJSON(w, r, http.StatusBadRequest, fmt.Errorf("%s", confClaims.Handshake.ID), "invalid handshake token", rest.ErrInternal)
@@ -511,7 +509,7 @@ func (s *private) setEmail(w http.ResponseWriter, r *http.Request, userID, siteI
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "failed to set token", rest.ErrInternal)
return
}
render.JSON(w, r, R.JSON{"updated": true, "address": val})
R.RenderJSON(w, R.JSON{"updated": true, "address": val})
}
// POST/GET /email/unsubscribe.html?site=siteID&tkn=jwt - unsubscribe the user in token from email notifications
@@ -534,7 +532,7 @@ func (s *private) emailUnsubscribeCtrl(w http.ResponseWriter, r *http.Request) {
return
}
// Handshake.ID is user.ID + "::" + address
// handshake.ID is user.ID + "::" + address
elems := strings.Split(confClaims.Handshake.ID, "::")
if len(elems) != 2 {
rest.SendErrorHTML(w, r, http.StatusBadRequest, fmt.Errorf("%s", confClaims.Handshake.ID), "invalid handshake token", rest.ErrInternal)
@@ -579,7 +577,7 @@ func (s *private) emailUnsubscribeCtrl(w http.ResponseWriter, r *http.Request) {
}
// MustExecute behaves like template.Execute, but panics if an error occurs.
MustExecute := func(tmpl *template.Template, wr io.Writer, data interface{}) {
MustExecute := func(tmpl *template.Template, wr io.Writer, data any) {
if err := tmpl.Execute(wr, data); err != nil {
panic(err)
}
@@ -595,7 +593,7 @@ func (s *private) emailUnsubscribeCtrl(w http.ResponseWriter, r *http.Request) {
tmpl := template.Must(template.New("unsubscribe").Parse(tmplstr))
msg := bytes.Buffer{}
MustExecute(tmpl, &msg, nil)
render.HTML(w, r, msg.String())
rest.HTMLResponse(w, http.StatusOK, msg.String())
}
// DELETE /email?site=siteID - removes user's email
@@ -622,7 +620,7 @@ func (s *private) deleteEmailCtrl(w http.ResponseWriter, r *http.Request) {
return
}
}
render.JSON(w, r, R.JSON{"deleted": true})
R.RenderJSON(w, R.JSON{"deleted": true})
}
// DELETE /telegram?site=siteID - removes user's telegram
@@ -636,7 +634,7 @@ func (s *private) deleteTelegramCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete telegram for user", code)
return
}
render.JSON(w, r, R.JSON{"deleted": true})
R.RenderJSON(w, R.JSON{"deleted": true})
}
// GET /userdata?site=siteID - exports all data about the user as a json with user info and list of all comments
@@ -664,13 +662,11 @@ func (s *private) userAllDataCtrl(w http.ResponseWriter, r *http.Request) {
return e
}
var merr error
merr = multierror.Append(merr, write([]byte(`{"info": `))) // send user prefix
merr = multierror.Append(merr, write(userB)) // send user info
merr = multierror.Append(merr, write([]byte(`, "comments":`))) // send comments prefix
// send user prefix, user info and comments prefix
errs := []error{write([]byte(`{"info": `)), write(userB), write([]byte(`, "comments":`))}
// get comments in 100 in each paginated request
for i := 0; i < 100; i++ {
for i := range 100 {
comments, errUser := s.dataService.User(siteID, user.ID, 100, i*100, rest.GetUserOrEmpty(r))
if errUser != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, errUser, "can't get user comments", rest.ErrInternal)
@@ -682,15 +678,15 @@ func (s *private) userAllDataCtrl(w http.ResponseWriter, r *http.Request) {
return
}
merr = multierror.Append(merr, write(b))
errs = append(errs, write(b))
if len(comments) != 100 {
break
}
}
merr = multierror.Append(merr, write([]byte(`}`)))
if merr.(*multierror.Error).ErrorOrNil() != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, merr, "can't write user info", rest.ErrInternal)
errs = append(errs, write([]byte(`}`)))
if err := errors.Join(errs...); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't write user info", rest.ErrInternal)
return
}
}
@@ -702,16 +698,17 @@ func (s *private) deleteMeCtrl(w http.ResponseWriter, r *http.Request) {
siteID := r.URL.Query().Get("site")
claims := token.Claims{
StandardClaims: jwt.StandardClaims{
Audience: siteID,
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{siteID},
Issuer: "remark42",
ExpiresAt: time.Now().AddDate(0, 3, 0).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: jwt.NewNumericDate(time.Now().AddDate(0, 3, 0)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
},
User: &token.User{
ID: user.ID,
Name: user.Name,
Attributes: map[string]interface{}{
ID: user.ID,
Name: user.Name,
Picture: user.Picture, // carried so the avatar can be removed when the request is processed
Attributes: map[string]any{
"delete_me": true, // prevents this token from being used for login
},
},
@@ -724,14 +721,18 @@ func (s *private) deleteMeCtrl(w http.ResponseWriter, r *http.Request) {
}
link := fmt.Sprintf("%s/web/deleteme.html?token=%s", s.remarkURL, tokenStr)
render.JSON(w, r, R.JSON{"site": siteID, "user_id": user.ID, "token": tokenStr, "link": link})
R.RenderJSON(w, R.JSON{"site": siteID, "user_id": user.ID, "token": tokenStr, "link": link})
}
// POST /image - save image with form request
func (s *private) savePictureCtrl(w http.ResponseWriter, r *http.Request) {
user := rest.MustGetUserInfo(r)
if err := r.ParseMultipartForm(5 * 1024 * 1024); err != nil { // 5M max memory, if bigger will make a file
r.Body = http.MaxBytesReader(w, r.Body, 32*1024*1024) // hard cap on upload to prevent memory exhaustion
// gosec G120: r.Body is already bounded by MaxBytesReader on the line above (32 MB),
// so ParseMultipartForm cannot read more than that regardless of the in-memory threshold.
// The 5 MB argument is the soft threshold above which the form is spilled to disk.
if err := r.ParseMultipartForm(5 * 1024 * 1024); err != nil { //nolint:gosec // bounded by MaxBytesReader above
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't parse multipart form", rest.ErrDecode)
return
}
@@ -749,7 +750,7 @@ func (s *private) savePictureCtrl(w http.ResponseWriter, r *http.Request) {
return
}
render.JSON(w, r, R.JSON{"id": id})
R.RenderJSON(w, R.JSON{"id": id})
}
func (s *private) isReadOnly(locator store.Locator) bool {
@@ -773,3 +774,13 @@ func randToken() (string, error) {
}
return fmt.Sprintf("%x", s.Sum(nil)), nil
}
// extractIP returns the IP portion of the remote address, handling both IPv4 and IPv6 formats.
// supports "ip:port", "[ip]:port", and bare "ip" formats.
func extractIP(remoteAddr string) string {
ip, _, err := net.SplitHostPort(remoteAddr)
if err != nil {
return remoteAddr // already a bare IP (no port)
}
return ip
}
+154 -95
View File
@@ -16,11 +16,10 @@ import (
"testing"
"time"
"github.com/go-chi/render"
"github.com/go-pkgz/auth/token"
"github.com/go-pkgz/auth/v2/token"
"github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -39,7 +38,7 @@ func TestRest_Create(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
@@ -50,7 +49,7 @@ func TestRest_Create(t *testing.T) {
c := R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
loc := c["locator"].(map[string]interface{})
loc := c["locator"].(map[string]any)
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.True(t, len(c["id"].(string)) > 8)
@@ -61,7 +60,7 @@ func TestRest_CreateFilteredCode(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "`+"`foo<bar>`"+`", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
@@ -72,7 +71,7 @@ func TestRest_CreateFilteredCode(t *testing.T) {
c := R.JSON{}
err = json.Unmarshal(b, &c)
require.NoError(t, err, string(b))
loc := c["locator"].(map[string]interface{})
loc := c["locator"].(map[string]any)
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.Equal(t, "`foo<bar>`", c["orig"])
@@ -94,6 +93,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
RoutePath: "/api/v1/img",
RemarkURL: srv.RemarkURL,
ImageService: srv.ImageService,
Transport: http.DefaultTransport,
}
srv.CommentFormatter = store.NewCommentFormatter(srv.ImageProxy)
// need to recreate the server with new ImageProxy, otherwise old one will be used
@@ -102,7 +102,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
var pngRead bool
// server with the test PNG image
pngServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
pngServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, e := io.Copy(w, gopherPNG())
assert.NoError(t, e)
pngRead = true
@@ -110,7 +110,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
defer pngServer.Close()
t.Run("create", func(t *testing.T) {
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "![](`+pngServer.URL+`/gopher.png)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
@@ -123,7 +123,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
require.NoError(t, err, string(b))
assert.NotContains(t, c["text"], pngServer.URL)
assert.Contains(t, c["text"], srv.RemarkURL)
loc := c["locator"].(map[string]interface{})
loc := c["locator"].(map[string]any)
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.True(t, len(c["id"].(string)) > 8)
@@ -144,7 +144,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
assert.Equal(t, false, pngRead, "original image is not yet accessed by server")
// retrieve the image from the cache
imgURL := strings.Split(strings.Split(string(b), "src=\"")[1], "\"")[0]
imgURL, _, _ := strings.Cut(strings.Split(string(b), "src=\"")[1], "\"")
// replace srv.RemarkURL with ts.URL
imgURL = strings.ReplaceAll(imgURL, srv.RemarkURL, ts.URL)
resp, err = http.Get(imgURL)
@@ -176,7 +176,7 @@ func TestRest_CreateOldPost(t *testing.T) {
assert.Equal(t, 1, len(comments))
// try to add new comment to the same old post
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "test 123", "locator":{"site": "remark42","url": "https://radio-t.com/blah1"}}`)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
@@ -189,7 +189,7 @@ func TestRest_CreateOldPost(t *testing.T) {
_, err = srv.DataService.Create(old)
assert.NoError(t, err)
resp, err = post(t, ts.URL+"/api/v1/comment",
resp, err = post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "test 123", "locator":{"site": "remark42","url": "https://radio-t.com/blah1"}}`)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
@@ -202,7 +202,7 @@ func TestRest_CreateTooBig(t *testing.T) {
longComment := fmt.Sprintf(`{"text": "%4001s", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`, "Щ")
resp, err := post(t, ts.URL+"/api/v1/comment", longComment)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", longComment)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -215,7 +215,7 @@ func TestRest_CreateTooBig(t *testing.T) {
assert.Equal(t, "invalid comment", c["details"])
veryLongComment := fmt.Sprintf(`{"text": "%70000s", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`, "Щ")
resp, err = post(t, ts.URL+"/api/v1/comment", veryLongComment)
resp, err = post(t, ts.URL+"/api/v1/comment?site=remark42", veryLongComment)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
@@ -235,7 +235,7 @@ func TestRest_CreateWithRestrictedWord(t *testing.T) {
badComment := `{"text": "What the duck is that?", "locator":{"url": "https://radio-t.com/blah1",
"site": "remark42"}}`
resp, err := post(t, ts.URL+"/api/v1/comment", badComment)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", badComment)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -254,7 +254,7 @@ func TestRest_CreateRelativeURL(t *testing.T) {
// check that it's not possible to click insert URL button and not alter the URL in it (which is `url` by default)
relativeURLText := `{"text": "here is a link with relative URL: [google.com](url)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
resp, err := post(t, ts.URL+"/api/v1/comment", relativeURLText)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", relativeURLText)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -273,7 +273,7 @@ func TestRest_CreateRejected(t *testing.T) {
body := `{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
// try to create without auth
resp, err := http.Post(ts.URL+"/api/v1/comment", "", strings.NewReader(body))
resp, err := http.Post(ts.URL+"/api/v1/comment?site=remark42", "", strings.NewReader(body))
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
@@ -281,7 +281,7 @@ func TestRest_CreateRejected(t *testing.T) {
// try with wrong aud
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(body))
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(body))
require.NoError(t, err)
req.Header.Add("X-JWT", devTokenBadAud)
resp, err = client.Do(req)
@@ -295,7 +295,7 @@ func TestRest_CreateWithWrongImage(t *testing.T) {
defer teardown()
// create comment
resp, err := post(t, ts.URL+"/api/v1/comment", fmt.Sprintf(`{"text": "![non-existent.jpg](%s/api/v1/picture/dev_user/bad_picture)", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, srv.RemarkURL))
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", fmt.Sprintf(`{"text": "![non-existent.jpg](%s/api/v1/picture/dev_user/bad_picture)", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, srv.RemarkURL))
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -317,7 +317,7 @@ func TestRest_CreateWithLazyImage(t *testing.T) {
defer teardown()
body := `{"text": "test 123 ![](http://example.com/image.png)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
// create comment
resp, err := post(t, ts.URL+"/api/v1/comment", body)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", body)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -334,7 +334,7 @@ func TestRest_CreateAndGet(t *testing.T) {
defer teardown()
// create comment
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "**test** *123*\n\n http://radio-t.com", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
@@ -368,6 +368,56 @@ func TestRest_CreateAndGet(t *testing.T) {
assert.Equal(t, store.User{Name: "admin", ID: "admin", Admin: true, Blocked: false, IP: ""}, comment.User, "no ip")
}
func TestRest_CreateWithQuotes(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
// create comment with quotes with smartypants
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "smartpants \"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
c := R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
id := c["id"].(string)
// get created comment by id as non-admin
res, code := getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah1", ts.URL, id))
assert.Equal(t, http.StatusOK, code)
comment := store.Comment{}
err = json.Unmarshal([]byte(res), &comment)
assert.NoError(t, err)
assert.Equal(t, "<p>smartpants «quoted» text</p>\n", comment.Text)
assert.Equal(t, "smartpants \"quoted\" text", comment.Orig)
// create comment with quotes without smartypants
srv.privRest.disableFancyTextFormatting = true
resp, err = post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "no_smartpants \"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
c = R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
id = c["id"].(string)
// get created comment by id as non-admin
res, code = getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah1", ts.URL, id))
assert.Equal(t, http.StatusOK, code)
comment = store.Comment{}
err = json.Unmarshal([]byte(res), &comment)
assert.NoError(t, err)
assert.Equal(t, "<p>no_smartpants &#34;quoted&#34; text</p>\n", comment.Text)
assert.Equal(t, "no_smartpants \"quoted\" text", comment.Orig)
}
func TestRest_Update(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
@@ -382,6 +432,7 @@ func TestRest_Update(t *testing.T) {
strings.NewReader(`{"text":"updated text", "summary":"my edit"}`))
assert.NoError(t, err)
req.Header.Add("X-JWT", devToken)
beforeUpdate := time.Now()
b, err := client.Do(req)
assert.NoError(t, err)
body, err := io.ReadAll(b.Body)
@@ -397,7 +448,7 @@ func TestRest_Update(t *testing.T) {
assert.Equal(t, "<p>updated text</p>\n", c2.Text)
assert.Equal(t, "updated text", c2.Orig)
assert.Equal(t, "my edit", c2.Edit.Summary)
assert.True(t, time.Since(c2.Edit.Timestamp) < 1*time.Second)
assert.WithinRange(t, c2.Edit.Timestamp, beforeUpdate, time.Now(), "edit stamped during the update")
// read updated comment
res, code := getWithAdminAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah1", ts.URL, id))
@@ -546,7 +597,7 @@ func TestRest_DeleteChildThenParent(t *testing.T) {
fmt.Sprintf("%s/api/v1/admin/comment/%s?site=remark42&url=https://radio-t.com/blah1", ts.URL, idC2), http.NoBody)
require.NoError(t, err)
requireAdminOnly(t, req)
resp, err = sendReq(t, req, adminUmputunToken)
resp, err = sendReq(req, adminUmputunToken)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -734,7 +785,7 @@ func TestRest_Vote(t *testing.T) {
req, err := http.NewRequest("GET",
fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1), http.NoBody)
assert.NoError(t, err)
resp, err := sendReq(t, req, adminUmputunToken)
resp, err := sendReq(req, adminUmputunToken)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
cr = store.Comment{}
@@ -811,10 +862,10 @@ func TestRest_EmailAndTelegram(t *testing.T) {
// issue good token
claims := token.Claims{
Handshake: &token.Handshake{ID: "provider1_dev::good@example.com"},
StandardClaims: jwt.StandardClaims{
Audience: "remark42",
ExpiresAt: time.Now().Add(10 * time.Minute).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(10 * time.Minute)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
Issuer: "remark42",
},
}
@@ -832,41 +883,43 @@ func TestRest_EmailAndTelegram(t *testing.T) {
body string
}{
{description: "issue delete request without auth", url: "/api/v1/email", method: http.MethodDelete, responseCode: http.StatusUnauthorized, noAuth: true},
{description: "issue delete request without site_id", url: "/api/v1/email", method: http.MethodDelete, responseCode: http.StatusBadRequest},
{description: "issue delete request without site_id", url: "/api/v1/email", method: http.MethodDelete, responseCode: http.StatusForbidden},
{description: "delete non-existent user email", url: "/api/v1/email?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "set user email, token not set", url: "/api/v1/email/confirm", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "set user email, token not set", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "set user email, token not set, old query param", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest},
{description: "send email confirmation without address", url: "/api/v1/email/subscribe", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "send email confirmation without address", url: "/api/v1/email/subscribe?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "send email confirmation without address, old query param", url: "/api/v1/email/subscribe?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest},
{description: "send email confirmation", url: "/api/v1/email/subscribe", method: http.MethodPost, responseCode: http.StatusOK, body: `{"site":"remark42","address":"good@example.com"}`},
{description: "send email confirmation", url: "/api/v1/email/subscribe?site=remark42", method: http.MethodPost, responseCode: http.StatusOK, body: `{"site":"remark42","address":"good@example.com"}`},
{description: "send email confirmation, old query param", url: "/api/v1/email/subscribe?site=remark42&address=good@example.com", method: http.MethodPost, responseCode: http.StatusOK},
{description: "set user email, token is good", url: "/api/v1/email/confirm", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good, old query param", url: fmt.Sprintf("/api/v1/email/confirm?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com"},
{description: "send confirmation with same address", url: "/api/v1/email/subscribe?site=remark42&address=good@example.com", method: http.MethodPost, responseCode: http.StatusConflict},
{description: "get user email", url: "/api/v1/email?site=remark42", method: http.MethodGet, responseCode: http.StatusOK},
{description: "delete user email", url: "/api/v1/email?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "send another confirmation", url: "/api/v1/email/subscribe?site=remark42&address=good@example.com", method: http.MethodPost, responseCode: http.StatusOK},
{description: "set user email, token is good", url: "/api/v1/email/confirm", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good, old query param", url: fmt.Sprintf("/api/v1/email/confirm?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com"},
{description: "unsubscribe user, no token", url: "/email/unsubscribe.html?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest},
{description: "unsubscribe user, wrong token", url: "/email/unsubscribe.html?site=remark42&tkn=jwt", method: http.MethodGet, responseCode: http.StatusForbidden},
{description: "unsubscribe user, good token", url: fmt.Sprintf("/email/unsubscribe.html?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusOK},
{description: "unsubscribe user second time, good token", url: fmt.Sprintf("/email/unsubscribe.html?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusConflict},
{description: "issue delete request without auth", url: "/api/v1/telegram", method: http.MethodDelete, responseCode: http.StatusUnauthorized, noAuth: true},
{description: "issue delete request without site_id", url: "/api/v1/telegram", method: http.MethodDelete, responseCode: http.StatusBadRequest},
{description: "issue delete request without site_id", url: "/api/v1/telegram", method: http.MethodDelete, responseCode: http.StatusForbidden},
{description: "delete non-existent user telegram", url: "/api/v1/telegram?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "send telegram confirmation, no siteID", url: "/api/v1/telegram/subscribe", method: http.MethodGet, responseCode: http.StatusBadRequest},
{description: "send telegram confirmation, no siteID", url: "/api/v1/telegram/subscribe", method: http.MethodGet, responseCode: http.StatusForbidden},
{description: "send telegram confirmation", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodGet, responseCode: http.StatusOK},
{description: "set user telegram, token is good", url: "/api/v1/telegram/subscribe?site=remark42&tkn=good_token", method: http.MethodGet, responseCode: http.StatusOK},
{description: "send confirmation with same address", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodGet, responseCode: http.StatusConflict},
{description: "delete user telegram", url: "/api/v1/telegram?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "send another confirmation", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodGet, responseCode: http.StatusOK},
{description: "set user telegram, token is good", url: "/api/v1/telegram/subscribe?site=remark42&tkn=good_token", method: http.MethodGet, responseCode: http.StatusOK},
// telegramSubscribeCtrl mutates state, so HEAD (which stdlib ServeMux would route to the
// GET handler) must be rejected by rejectHead before it runs
{description: "HEAD is rejected on telegram subscribe", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodHead, responseCode: http.StatusMethodNotAllowed},
}
client := http.Client{}
defer client.CloseIdleConnections()
for _, x := range testData {
x := x
t.Run(x.description, func(t *testing.T) {
reqBody := io.NopCloser(strings.NewReader(x.body))
if x.body == "" {
@@ -907,7 +960,7 @@ func TestRest_EmailNotification(t *testing.T) {
defer client.CloseIdleConnections()
// create new comment from dev user
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 123",
"user": {"name": "provider1_dev::good@example.com"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -921,14 +974,12 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
parentComment := store.Comment{}
require.NoError(t, render.DecodeJSON(strings.NewReader(string(body)), &parentComment))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 1, len(mockDestination.Get()))
require.NoError(t, json.Unmarshal(body, &parentComment))
waitForCount(t, 1, func() int { return len(mockDestination.Get()) })
assert.Empty(t, mockDestination.Get()[0].Emails)
// create child comment from another user, email notification only to admin expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 456",
"pid": %q,
"user": {"name": "other_user"},
@@ -942,15 +993,13 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 2, len(mockDestination.Get()))
waitForCount(t, 2, func() int { return len(mockDestination.Get()) })
assert.Empty(t, mockDestination.Get()[1].Emails)
// send confirmation token for email
req, err = http.NewRequest(
http.MethodPost,
ts.URL+"/api/v1/email/subscribe",
ts.URL+"/api/v1/email/subscribe?site=remark42",
io.NopCloser(strings.NewReader(`{"site": "remark42", "address": "good@example.com"}`)),
)
require.NoError(t, err)
@@ -961,9 +1010,7 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 1, len(mockDestination.GetVerify()))
waitForCount(t, 1, func() int { return len(mockDestination.GetVerify()) })
assert.Equal(t, "good@example.com", mockDestination.GetVerify()[0].Email)
verificationToken := mockDestination.GetVerify()[0].Token
@@ -989,7 +1036,7 @@ func TestRest_EmailNotification(t *testing.T) {
// verify email
req, err = http.NewRequest(
http.MethodPost,
ts.URL+"/api/v1/email/confirm",
ts.URL+"/api/v1/email/confirm?site=remark42",
io.NopCloser(strings.NewReader(fmt.Sprintf(`{"site": "remark42", "token": %q}`, verificationToken))),
)
require.NoError(t, err)
@@ -1021,7 +1068,7 @@ func TestRest_EmailNotification(t *testing.T) {
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, subscribedUser)
// create child comment from another user, email notification expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 789",
"pid": %q,
"user": {"name": "other_user"},
@@ -1035,9 +1082,7 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 3, len(mockDestination.Get()))
waitForCount(t, 3, func() int { return len(mockDestination.Get()) })
assert.Equal(t, []string{"good@example.com"}, mockDestination.Get()[2].Emails)
// delete user's email
@@ -1052,7 +1097,7 @@ func TestRest_EmailNotification(t *testing.T) {
assert.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// create child comment from another user, no email notification
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 321",
"user": {"name": "other_user"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -1065,9 +1110,7 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 4, len(mockDestination.Get()))
waitForCountSettled(t, 4, func() int { return len(mockDestination.Get()) })
assert.Empty(t, mockDestination.Get()[3].Emails)
// confirm email via subscribe call with query params, old behavior, email notification is expected
@@ -1084,9 +1127,7 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 2, len(mockDestination.GetVerify()), "verification email was sent")
waitForCount(t, 2, func() int { return len(mockDestination.GetVerify()) }, "verification email was sent")
// get email user information to verify there is no subscription yet
req, err = http.NewRequest(
@@ -1110,7 +1151,7 @@ func TestRest_EmailNotification(t *testing.T) {
// confirm email via subscribe call, no email notification is expected
req, err = http.NewRequest(
http.MethodPost,
ts.URL+"/api/v1/email/subscribe",
ts.URL+"/api/v1/email/subscribe?site=remark42",
io.NopCloser(strings.NewReader(`{"site": "remark42", "address": "good@example.com"}`)),
)
require.NoError(t, err)
@@ -1121,9 +1162,7 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 2, len(mockDestination.GetVerify()), "no new verification email was sent")
waitForCountSettled(t, 2, func() int { return len(mockDestination.GetVerify()) }, "no new verification email was sent")
// get email user information to verify the subscription happened without the confirmation call
req, err = http.NewRequest(
@@ -1157,7 +1196,7 @@ func TestRest_TelegramNotification(t *testing.T) {
defer client.CloseIdleConnections()
// create new comment from dev user
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 123",
"user": {"name": "provider1_dev::good@example.com"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -1171,14 +1210,12 @@ func TestRest_TelegramNotification(t *testing.T) {
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
parentComment := store.Comment{}
require.NoError(t, render.DecodeJSON(strings.NewReader(string(body)), &parentComment))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 1, len(mockDestination.Get()))
require.NoError(t, json.Unmarshal(body, &parentComment))
waitForCount(t, 1, func() int { return len(mockDestination.Get()) })
assert.Empty(t, mockDestination.Get()[0].Telegrams)
// create child comment from another user, telegram notification only to admin expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 456",
"pid": %q,
"user": {"name": "other_user"},
@@ -1192,9 +1229,7 @@ func TestRest_TelegramNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 2, len(mockDestination.Get()))
waitForCount(t, 2, func() int { return len(mockDestination.Get()) })
assert.Empty(t, mockDestination.Get()[1].Telegrams)
// subscribe to telegram while the telegram destination is absent
@@ -1291,7 +1326,7 @@ func TestRest_TelegramNotification(t *testing.T) {
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, user)
// create child comment from another user, telegram notification expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 789",
"pid": %q,
"user": {"name": "other_user"},
@@ -1305,9 +1340,7 @@ func TestRest_TelegramNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 3, len(mockDestination.Get()))
waitForCount(t, 3, func() int { return len(mockDestination.Get()) })
assert.Equal(t, []string{"good_telegram"}, mockDestination.Get()[2].Telegrams)
// delete user's telegram
@@ -1322,7 +1355,7 @@ func TestRest_TelegramNotification(t *testing.T) {
assert.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// create child comment from another user, no telegram notification
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 321",
"user": {"name": "other_user"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -1335,9 +1368,7 @@ func TestRest_TelegramNotification(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 4, len(mockDestination.Get()))
waitForCountSettled(t, 4, func() int { return len(mockDestination.Get()) })
assert.Empty(t, mockDestination.Get()[3].Telegrams)
}
@@ -1360,7 +1391,7 @@ func TestRest_UserAllData(t *testing.T) {
_, err = srv.DataService.Create(c3)
require.NoError(t, err)
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("GET", ts.URL+"/api/v1/userdata?site=remark42", http.NoBody)
require.NoError(t, err)
@@ -1407,13 +1438,13 @@ func TestRest_UserAllDataManyComments(t *testing.T) {
c := store.Comment{User: user, Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 5, 27, 1, 14, 10, 0, time.Local)}
for i := 0; i < 51; i++ {
for i := range 51 {
c.ID = fmt.Sprintf("id-%03d", i)
c.Timestamp = c.Timestamp.Add(time.Second)
_, err := srv.DataService.Create(c)
require.NoError(t, err)
}
client := &http.Client{Timeout: 1 * time.Second}
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("GET", ts.URL+"/api/v1/userdata?site=remark42", http.NoBody)
require.NoError(t, err)
@@ -1460,6 +1491,8 @@ func TestRest_DeleteMe(t *testing.T) {
claims, err := srv.Authenticator.TokenService().Parse(tkn)
assert.NoError(t, err)
assert.Equal(t, "provider1_dev", claims.User.ID)
assert.Equal(t, "http://example.com/pic.png", claims.User.Picture,
"delete_me token must carry the user's picture so the avatar can be removed when the request is processed")
assert.Equal(t, "https://demo.remark42.com/web/deleteme.html?token="+tkn, m["link"])
req, err = http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/deleteme?site=remark42", ts.URL), http.NoBody)
@@ -1487,7 +1520,7 @@ func TestRest_SavePictureCtrl(t *testing.T) {
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture", ts.URL), bodyBuf)
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture?site=remark42", ts.URL), bodyBuf)
require.NoError(t, err)
req.Header.Add("Content-Type", contentType)
req.Header.Add("X-JWT", devToken)
@@ -1554,7 +1587,9 @@ func TestRest_CreateWithPictures(t *testing.T) {
Staging: "/tmp/remark42/images.staging",
Location: "/tmp/remark42/images",
}, image.ServiceParams{
EditDuration: 100 * time.Millisecond,
// the "not moved yet" checks below run right after the comment POST returns, so the
// commit window has to be wide enough that a stalled runner cannot close it first
EditDuration: 3 * time.Second,
MaxSize: 2000,
ImageAPI: svc.RemarkURL + "/api/v1/picture/",
ProxyAPI: svc.RemarkURL + "/api/v1/img",
@@ -1579,7 +1614,7 @@ func TestRest_CreateWithPictures(t *testing.T) {
require.NoError(t, bodyWriter.Close())
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture", ts.URL), bodyBuf)
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture?site=remark42", ts.URL), bodyBuf)
require.NoError(t, err)
req.Header.Add("Content-Type", contentType)
req.Header.Add("X-JWT", devToken)
@@ -1605,7 +1640,7 @@ func TestRest_CreateWithPictures(t *testing.T) {
text := fmt.Sprintf(`text 123 ![](%s/api/v1/picture/%s) *xxx* ![](%s/api/v1/picture/%s) ![](%s/api/v1/picture/%s)`, svc.RemarkURL, ids[0], svc.RemarkURL, ids[1], svc.RemarkURL, ids[2])
body := fmt.Sprintf(`{"text": %q, "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`, text)
resp, err := post(t, ts.URL+"/api/v1/comment", body)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", body)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
@@ -1617,11 +1652,12 @@ func TestRest_CreateWithPictures(t *testing.T) {
assert.Error(t, err, "picture %d not moved from staging yet", i)
}
time.Sleep(1500 * time.Millisecond)
// the commit runs once EditDuration expires
for i := range ids {
_, err = os.Stat("/tmp/remark42/images/" + ids[i])
assert.NoError(t, err, "picture %d moved from staging and available in permanent location", i)
require.Eventually(t, func() bool {
_, e := os.Stat("/tmp/remark42/images/" + ids[i])
return e == nil
}, waitTimeout, pollInterval, "picture %d moved from staging and available in permanent location", i)
}
}
@@ -1642,3 +1678,26 @@ func (m *mockTelegram) CheckToken(string, string) (telegram, site string, err er
}
return "good_telegram", m.site, nil
}
func TestExtractIP(t *testing.T) {
tbl := []struct {
addr string
exp string
}{
{"127.0.0.1:8080", "127.0.0.1"},
{"127.0.0.1", "127.0.0.1"},
{"192.168.1.1:443", "192.168.1.1"},
{"[::1]:8080", "::1"},
{"::1", "::1"},
{"[2001:db8::1]:8080", "2001:db8::1"},
{"2001:db8::1", "2001:db8::1"},
{"[fe80::1%25eth0]:80", "fe80::1%25eth0"},
{"", ""},
}
for _, tt := range tbl {
t.Run(tt.addr, func(t *testing.T) {
assert.Equal(t, tt.exp, extractIP(tt.addr))
})
}
}
+162 -35
View File
@@ -4,18 +4,19 @@ import (
"bytes"
"crypto/sha1" // nolint
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
"strconv"
"strings"
"time"
"unicode"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/google/uuid"
"github.com/skip2/go-qrcode"
"github.com/umputun/remark42/backend/app/rest"
@@ -48,8 +49,18 @@ type pubStore interface {
Counts(siteID string, postIDs []string) ([]store.PostInfo, error)
}
// GET /find?site=siteID&url=post-url&format=[tree|plain]&sort=[+/-time|+/-score|+/-controversy]&view=[user|all]&since=unix_ts_msec
// find comments for given post. Returns in tree or plain formats, sorted
// GET /find?site=siteID&url=post-url&format=[tree|plain]&sort=[+/-time|+/-score|+/-controversy]&view=[user|all]&since=unix_ts_msec&limit=100&offset_id={id}
// find comments for given post. Returns in tree or plain formats, sorted.
//
// When `url` parameter is not set (e.g. request is for site-wide comments), does not return deleted comments.
//
// When `limit` is set, first {limit} comments are returned. When `offset_id` is set, comments are returned starting
// after the comment with the given id.
// format="tree" limits comments by top-level comments and all their replies,
// and never returns parent comment with only part of replies.
//
// `count` in the response refers to total number of non-deleted comments,
// `count_left` to amount of comments left to be returned _including deleted_.
func (s *public) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
sort := r.URL.Query().Get("sort")
@@ -68,7 +79,24 @@ func (s *public) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
since = time.Time{} // since doesn't make sense for tree
}
log.Printf("[DEBUG] get comments for %+v, sort %s, format %s, since %v", locator, sort, format, since)
limitParam := r.URL.Query().Get("limit")
var limit int
if limitParam != "" {
if limit, err = strconv.Atoi(limitParam); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "bad limit value", rest.ErrCommentNotFound)
return
}
}
offsetID := r.URL.Query().Get("offset_id")
if offsetID != "" {
if _, err = uuid.Parse(offsetID); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "bad offset_id value", rest.ErrCommentNotFound)
return
}
}
log.Printf("[DEBUG] get comments for %+v, sort %s, format %s, since %v, limit %d, offset %s", locator, sort, format, since, limit, offsetID)
key := cache.NewKey(locator.SiteID).ID(URLKeyWithUser(r)).Scopes(locator.SiteID, locator.URL)
data, err := s.cache.Get(key, func() ([]byte, error) {
@@ -77,22 +105,44 @@ func (s *public) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
comments = []store.Comment{} // error should clear comments and continue for post info
}
comments = s.applyView(comments, view)
var commentsInfo store.PostInfo
if info, ee := s.dataService.Info(locator, s.readOnlyAge); ee == nil {
commentsInfo = info
}
if !since.IsZero() { // if since is set, number of comments can be different from total in the DB
commentsInfo.Count = 0
for _, c := range comments {
if !c.Deleted {
commentsInfo.Count++
}
}
}
// post might be readonly without any comments, Info call will fail then and ReadOnly flag should be checked separately
if !commentsInfo.ReadOnly && locator.URL != "" && s.dataService.IsReadOnly(locator) {
commentsInfo.ReadOnly = true
}
var b []byte
switch format {
case "tree":
tree := service.MakeTree(comments, sort, s.readOnlyAge)
if tree.Nodes == nil { // eliminate json nil serialization
tree.Nodes = []*service.Node{}
withInfo := treeWithInfo{Tree: service.MakeTree(comments, sort, limit, offsetID), Info: commentsInfo}
withInfo.Info.CountLeft = withInfo.CountLeft()
withInfo.Info.LastComment = withInfo.LastComment()
if withInfo.Nodes == nil { // eliminate json nil serialization
withInfo.Nodes = []*service.Node{}
}
if s.dataService.IsReadOnly(locator) {
tree.Info.ReadOnly = true
}
b, e = encodeJSONWithHTML(tree)
b, e = encodeJSONWithHTML(withInfo)
default:
withInfo := commentsWithInfo{Comments: comments}
if info, ee := s.dataService.Info(locator, s.readOnlyAge); ee == nil {
withInfo.Info = info
if limit > 0 || offsetID != "" {
comments, commentsInfo.CountLeft = limitComments(comments, limit, offsetID)
}
if limit > 0 && len(comments) > 0 {
commentsInfo.LastComment = comments[len(comments)-1].ID
}
withInfo := commentsWithInfo{Comments: comments, Info: commentsInfo}
b, e = encodeJSONWithHTML(withInfo)
}
return b, e
@@ -137,7 +187,7 @@ func (s *public) lastCommentsCtrl(w http.ResponseWriter, r *http.Request) {
siteID := r.URL.Query().Get("site")
log.Printf("[DEBUG] get last comments for %s", siteID)
limit, err := strconv.Atoi(chi.URLParam(r, "limit"))
limit, err := strconv.Atoi(r.PathValue("limit"))
if err != nil {
limit = 0
}
@@ -171,7 +221,7 @@ func (s *public) lastCommentsCtrl(w http.ResponseWriter, r *http.Request) {
// GET /id/{id}?site=siteID&url=post-url - gets a comment by id
func (s *public) commentByIDCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
id := r.PathValue("id")
siteID := r.URL.Query().Get("site")
url := r.URL.Query().Get("url")
@@ -182,7 +232,6 @@ func (s *public) commentByIDCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get comment by id", rest.ErrCommentNotFound)
return
}
render.Status(r, http.StatusOK)
if err = R.RenderJSONWithHTML(w, r, comment); err != nil {
log.Printf("[WARN] can't render last comments for url=%s, id=%s", url, id)
@@ -247,7 +296,7 @@ func (s *public) countCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get count", rest.ErrPostNotFound)
return
}
render.JSON(w, r, R.JSON{"count": count, "locator": locator})
R.RenderJSON(w, R.JSON{"count": count, "locator": locator})
}
// POST /counts?site=siteID - get number of comments for posts from post body
@@ -255,7 +304,7 @@ func (s *public) countMultiCtrl(w http.ResponseWriter, r *http.Request) {
const countBodyLimit int64 = 1024 * 128 // count request can be big for some site because it lists all urls
siteID := r.URL.Query().Get("site")
posts := []string{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, countBodyLimit), &posts); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, countBodyLimit)).Decode(&posts); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get list of posts from request", rest.ErrSiteNotFound)
return
}
@@ -315,26 +364,81 @@ func (s *public) listCtrl(w http.ResponseWriter, r *http.Request) {
}
}
// safePictureSegment reports whether seg is acceptable as a path segment in
// the picture URL (no traversal markers, no path separators, no control
// characters). Picture IDs are server-generated hashes plus a known
// extension, so any value carrying these characters is hostile and must be
// rejected before reaching the store. Rejecting controls (CR, LF, TAB, NUL,
// etc.) also closes a log-injection vector since the rejected segment is
// echoed into the access log.
func safePictureSegment(seg string) bool {
if seg == "" || seg == "." {
return false
}
if strings.ContainsAny(seg, "/\\") {
return false
}
if strings.Contains(seg, "..") { // also covers seg == ".."
return false
}
for _, r := range seg {
if unicode.IsControl(r) {
return false
}
}
return true
}
// sendPictureError writes a no-store Cache-Control header and delegates to rest.SendErrorJSON.
// Used by every rejection branch in loadPictureCtrl so error responses never inherit the
// 7-day client cache of the success path.
func sendPictureError(w http.ResponseWriter, r *http.Request, status int, err error, details string, code int) {
w.Header().Set("Cache-Control", "no-store")
rest.SendErrorJSON(w, r, status, err, details, code)
}
// GET /picture/{user}/{id} - get picture
func (s *public) loadPictureCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "user") + "/" + chi.URLParam(r, "id")
img, err := s.imageService.Load(id)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get image "+id, rest.ErrAssetNotFound)
rest.SetImageDefenseHeaders(w)
user, imgID := r.PathValue("user"), r.PathValue("id")
if user == "" || imgID == "" || !safePictureSegment(user) || !safePictureSegment(imgID) {
log.Printf("[WARN] rejected picture request with unsafe id segments user=%q id=%q", user, imgID)
sendPictureError(w, r, http.StatusBadRequest, fmt.Errorf("invalid picture id"), "invalid picture id", rest.ErrAssetNotFound)
return
}
// enforce client-side caching
id := user + "/" + imgID
img, err := s.imageService.Load(id)
if err != nil {
log.Printf("[WARN] can't load image %s: %v", id, err)
sendPictureError(w, r, http.StatusBadRequest, fmt.Errorf("image not found"), "can't get image", rest.ErrAssetNotFound)
return
}
contentType, err := rest.SafeImgContentType(img)
if err != nil {
log.Printf("[WARN] rejecting non-image picture %s: %v", id, err)
sendPictureError(w, r, http.StatusUnsupportedMediaType, err, "invalid image content", rest.ErrAssetNotFound)
return
}
// /picture/ does not need a security-version etag prefix — the upload flow
// validates input format (readAndValidateImage) and the serve path re-validates
// the stored bytes via rest.SafeImgContentType. Bytes within the resize dimension
// limits ARE preserved verbatim by resize, so the browser defense relies on the
// response headers (validated Content-Type + nosniff + strict CSP +
// Content-Disposition: inline), not on byte normalization. Picture IDs are limited
// to safePictureSegment (alphanumeric xid-generated guids), so the comma split
// inside rest.EtagMatches cannot collide; if the ID format ever changes, revisit.
etag := `"` + id + `"`
w.Header().Set("Etag", etag)
w.Header().Set("Cache-Control", "max-age=604800") // 7 days
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, etag) {
w.WriteHeader(http.StatusNotModified)
return
}
if match := r.Header.Get("If-None-Match"); match != "" && rest.EtagMatches(match, etag) {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("Content-Type", s.imageService.ImgContentType(img))
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Length", strconv.Itoa(len(img)))
w.WriteHeader(http.StatusOK)
if _, err = io.Copy(w, bytes.NewReader(img)); err != nil {
@@ -343,13 +447,14 @@ func (s *public) loadPictureCtrl(w http.ResponseWriter, r *http.Request) {
}
// GET /robots.txt
func (s *public) robotsCtrl(w http.ResponseWriter, r *http.Request) {
func (s *public) robotsCtrl(w http.ResponseWriter, _ *http.Request) {
allowed := []string{"/find", "/last", "/id", "/count", "/counts", "/list", "/config", "/user",
"/img", "/avatar", "/picture"}
for i := range allowed {
allowed[i] = "Allow: /api/v1" + allowed[i]
}
render.PlainText(w, r, "User-agent: *\nDisallow: /auth/\nDisallow: /api/\n"+strings.Join(allowed, "\n")+"\n")
responseText := fmt.Sprintf("User-agent: *\nDisallow: /auth/\nDisallow: /api/\n%s\n", strings.Join(allowed, "\n"))
rest.PlainTextResponse(w, http.StatusOK, responseText)
}
// GET /qr/telegram - generates QR for provided URL, used for Telegram auth and notifications subscription. The first
@@ -381,7 +486,7 @@ func (s *public) telegramQrCtrl(w http.ResponseWriter, r *http.Request) {
}
w.Header().Set("Content-Type", "image/png")
if _, err = w.Write(png); err != nil {
if _, err = w.Write(png); err != nil { //nolint:gosec // png bytes from go-qrcode, not HTML
log.Printf("[WARN] can't render qr, %v", err)
}
}
@@ -416,3 +521,25 @@ func (s *public) parseSince(r *http.Request) (time.Time, error) {
}
return sinceTS, nil
}
// limitComments returns limited list of comments and count of comments left after limit.
// If offsetID is provided, the list will be sliced starting from the comment with this ID.
// If offsetID is not found, the full list will be returned.
// It's used for only "
func limitComments(c []store.Comment, limit int, offsetID string) (comments []store.Comment, countLeft int) {
if offsetID != "" {
for i, comment := range c {
if comment.ID == offsetID {
c = c[i+1:]
break
}
}
}
if limit > 0 && len(c) > limit {
countLeft = len(c) - limit
c = c[:limit]
}
return c, countLeft
}
+549 -23
View File
@@ -1,21 +1,29 @@
package api
import (
"bytes"
"encoding/json"
"fmt"
"image/png"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"strconv"
"strings"
"testing"
"time"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/image"
"github.com/umputun/remark42/backend/app/store/service"
)
@@ -83,6 +91,24 @@ func TestRest_Preview(t *testing.T) {
string(b),
"/pics-remark42/staging/dev_user/62/bad_picture: no such file or directory\"}\n",
)
// test quotes with and without smartypants
resp, err = post(t, ts.URL+"/api/v1/preview", `{"text": "\"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, "<p>«quoted» text</p>\n", string(b))
srv.privRest.disableFancyTextFormatting = true
resp, err = post(t, ts.URL+"/api/v1/preview", `{"text": "\"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, "<p>&#34;quoted&#34; text</p>\n", string(b))
}
func TestRest_PreviewWithWrongImage(t *testing.T) {
@@ -120,9 +146,9 @@ srv, ts := prep(t)
}
BKT
`
text = strings.Replace(text, "BKT", "```", -1)
text = strings.ReplaceAll(text, "BKT", "```")
j := fmt.Sprintf(`{"text": %q, "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, text)
j = strings.Replace(j, "\n", "\\n", -1)
j = strings.ReplaceAll(j, "\n", "\\n")
resp, err := post(t, ts.URL+"/api/v1/preview", j)
assert.NoError(t, err)
@@ -131,10 +157,10 @@ BKT
assert.NoError(t, err)
assert.Equal(t,
`<h1>h1</h1>
<pre class="chroma"><code><span class="line"><span class="cl">func TestRest_Preview(t *testing.T) {
</span></span><span class="line"><span class="cl">srv, ts := prep(t)
</span></span><span class="line"><span class="cl"> require.NotNil(t, srv)
</span></span><span class="line"><span class="cl">}
<pre class="chroma"><code><span class="line"><span class="cl"><span class="k">func</span> <span class="n">TestRest_Preview</span><span class="p">(</span><span class="n">t</span> <span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"><span class="n">srv</span><span class="p">,</span> <span class="n">ts</span> <span class="p">:</span><span class="o">=</span> <span class="n">prep</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">require</span><span class="o">.</span><span class="n">NotNil</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">srv</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre>`,
string(b))
assert.NoError(t, resp.Body.Close())
@@ -148,17 +174,17 @@ func TestRest_PreviewCode(t *testing.T) {
func main(aa string) int {return 0}
BKT
`
text = strings.Replace(text, "BKT", "```", -1)
text = strings.ReplaceAll(text, "BKT", "```")
j := fmt.Sprintf(`{"text": %q, "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, text)
j = strings.Replace(j, "\n", "\\n", -1)
j = strings.ReplaceAll(j, "\n", "\\n")
resp, err := post(t, ts.URL+"/api/v1/preview", j)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, `<pre class="chroma"><code><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">(</span><span class="nx">aa</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span><span class="k">return</span> <span class="mi">0</span><span class="p">}</span>
</span></span></code></pre>`, string(b))
assert.Equal(t, `<pre class="chroma"><code><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="nx">aa</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="p">{</span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">}</span><span class="w">
</span></span></span></code></pre>`, string(b))
assert.NoError(t, resp.Body.Close())
}
@@ -209,7 +235,7 @@ func TestRest_Find(t *testing.T) {
assert.Equal(t, id2, comments.Comments[0].ID)
// get in tree mode
tree := service.Tree{}
tree := treeWithInfo{}
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, http.StatusOK, code)
err = json.Unmarshal([]byte(res), &tree)
@@ -235,7 +261,7 @@ func TestRest_FindAge(t *testing.T) {
_, err = srv.DataService.Create(c2)
require.NoError(t, err)
tree := service.Tree{}
tree := treeWithInfo{}
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, http.StatusOK, code)
@@ -278,7 +304,7 @@ func TestRest_FindReadOnly(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
tree := service.Tree{}
tree := treeWithInfo{}
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, http.StatusOK, code)
err = json.Unmarshal([]byte(res), &tree)
@@ -286,7 +312,7 @@ func TestRest_FindReadOnly(t *testing.T) {
assert.Equal(t, "https://radio-t.com/blah1", tree.Info.URL)
assert.True(t, tree.Info.ReadOnly, "post is ro")
tree = service.Tree{}
tree = treeWithInfo{}
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah2&format=tree")
assert.Equal(t, http.StatusOK, code)
err = json.Unmarshal([]byte(res), &tree)
@@ -356,11 +382,12 @@ func TestRest_Last(t *testing.T) {
c2 := store.Comment{Text: "test test #2", ParentID: "p1",
Locator: store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah2"}}
// add 3 comments
// add 3 comments, with the clock pushed past a millisecond boundary in between so the two
// "since" values below are distinct
ts1 := time.Now().UnixNano() / 1000000
addComment(t, c1, ts)
id1 := addComment(t, c1, ts)
time.Sleep(10 * time.Millisecond)
waitPastMillisecond(time.Now())
ts2 := time.Now().UnixNano() / 1000000
id2 := addComment(t, c2, ts)
@@ -477,6 +504,298 @@ func TestRest_FindUserComments(t *testing.T) {
}
}
func TestRest_FindUserComments_CWE_918(t *testing.T) {
ts, srv, teardown := startupT(t)
srv.DataService.TitleExtractor = service.NewTitleExtractor(http.Client{Timeout: time.Second}, []string{"radio-t.com"}) // required for extracting the title, bad URL test
defer srv.DataService.TitleExtractor.Close()
defer teardown()
backendRequestedArbitraryServer := false
arbitraryServer := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
t.Logf("request received: %+v", r)
backendRequestedArbitraryServer = true
}))
defer arbitraryServer.Close()
arbitraryURLComment := store.Comment{Text: "arbitrary URL request test",
Locator: store.Locator{SiteID: "remark42", URL: arbitraryServer.URL}}
assert.False(t, backendRequestedArbitraryServer)
addComment(t, arbitraryURLComment, ts)
assert.False(t, backendRequestedArbitraryServer,
"no request is expected to the test server as it's not in the list of the allowed domains for the title extractor")
res, code := get(t, ts.URL+"/api/v1/comments?site=remark42&user=provider1_dev")
assert.Equal(t, http.StatusOK, code)
resp := struct {
Comments []store.Comment
Count int
}{}
err := json.Unmarshal([]byte(res), &resp)
assert.NoError(t, err)
require.Equal(t, 1, len(resp.Comments), "should have 2 comments")
assert.Equal(t, "", resp.Comments[0].PostTitle, "empty from the first post")
assert.Equal(t, arbitraryServer.URL, resp.Comments[0].Locator.URL, "arbitrary URL provided by the request")
}
// waitPastMillisecond blocks until the wall clock moves past ts's millisecond, so whatever is
// created next gets a distinct value for the millisecond-precision "since" filter
func waitPastMillisecond(ts time.Time) {
next := ts.Truncate(time.Millisecond).Add(time.Millisecond)
time.Sleep(time.Until(next) + time.Microsecond) // a non-positive duration returns at once
}
func TestPublic_FindCommentsCtrl_ConsistentCount(t *testing.T) {
// test that comment counting is consistent between tree and plain formats
// the open-route limit is lifted so the subtests below can run back to back
ts, srv, teardown := startupT(t, func(srv *Rest) { srv.openRouteLimiter = 100000 })
defer teardown()
commentLocator := store.Locator{URL: "test-url", SiteID: "remark42"}
// vote for comment multiple times
setScore := func(locator store.Locator, id string, val int) {
abs := func(x int) int {
if x < 0 {
return -x
}
return x
}
for i := 0; i < abs(val); i++ {
_, err := srv.DataService.Vote(service.VoteReq{
Locator: locator,
CommentID: id,
// unique user ID is needed for correct counting of controversial votes
UserID: "user" + strconv.Itoa(val) + strconv.Itoa(i),
Val: val > 0,
})
require.NoError(t, err)
}
}
// adding initial comments (8 to test-url and 1 to another-url) and voting, and delete two of comments to the first post.
// each comment waits for the clock to pass the previous one's millisecond so the "since"
// filter, which has millisecond precision, can tell them apart
ids := make([]string, 9)
timestamps := make([]time.Time, 9)
c1 := store.Comment{Text: "top-level comment 1", Locator: commentLocator}
ids[0], timestamps[0] = addCommentGetCreatedTime(t, c1, ts)
// #3 by score
setScore(commentLocator, ids[0], 1)
waitPastMillisecond(timestamps[0])
c2 := store.Comment{Text: "top-level comment 2", Locator: commentLocator}
ids[1], timestamps[1] = addCommentGetCreatedTime(t, c2, ts)
// #2 by score
setScore(commentLocator, ids[1], 2)
waitPastMillisecond(timestamps[1])
c3 := store.Comment{Text: "second-level comment 1", ParentID: ids[0], Locator: commentLocator}
ids[2], timestamps[2] = addCommentGetCreatedTime(t, c3, ts)
// #1 by score
setScore(commentLocator, ids[2], 10)
waitPastMillisecond(timestamps[2])
c4 := store.Comment{Text: "third-level comment 1", ParentID: ids[2], Locator: commentLocator}
ids[3], timestamps[3] = addCommentGetCreatedTime(t, c4, ts)
// #5 by score, #1 by controversy
setScore(commentLocator, ids[3], 4)
setScore(commentLocator, ids[3], -4)
waitPastMillisecond(timestamps[3])
c5 := store.Comment{Text: "second-level comment 2", ParentID: ids[1], Locator: commentLocator}
ids[4], timestamps[4] = addCommentGetCreatedTime(t, c5, ts)
// #5 by score, #2 by controversy
setScore(commentLocator, ids[4], 2)
setScore(commentLocator, ids[4], -3)
waitPastMillisecond(timestamps[4])
c6 := store.Comment{Text: "deleted third-level comment 2", ParentID: ids[4], Locator: commentLocator}
ids[5], timestamps[5] = addCommentGetCreatedTime(t, c6, ts)
// deleted later so not visible in site-wide requests
setScore(commentLocator, ids[5], 10)
setScore(commentLocator, ids[5], -10)
waitPastMillisecond(timestamps[5])
c7 := store.Comment{Text: "top-level comment 3", Locator: commentLocator}
ids[6], timestamps[6] = addCommentGetCreatedTime(t, c7, ts)
// #6 by score, #4 by controversy
setScore(commentLocator, ids[6], -3)
setScore(commentLocator, ids[6], 1)
waitPastMillisecond(timestamps[6])
c8 := store.Comment{Text: "deleted second-level comment 3", ParentID: ids[6], Locator: commentLocator}
ids[7], timestamps[7] = addCommentGetCreatedTime(t, c8, ts)
// deleted later so not visible in site-wide requests
setScore(commentLocator, ids[7], -20)
c9 := store.Comment{Text: "comment to post 2", Locator: store.Locator{URL: "another-url", SiteID: "remark42"}}
ids[8], timestamps[8] = addCommentGetCreatedTime(t, c9, ts)
// #7 by score
setScore(store.Locator{URL: "another-url", SiteID: "remark42"}, ids[8], -25)
// delete two comments bringing the total from 9 to 6
err := srv.DataService.Delete(commentLocator, ids[7], store.SoftDelete)
assert.NoError(t, err)
err = srv.DataService.Delete(commentLocator, ids[5], store.HardDelete)
assert.NoError(t, err)
srv.Cache.Flush(cache.FlusherRequest{})
commentLocator.URL = "readonly-test"
// set post without comments to read-only
assert.NoError(t, srv.DataService.SetReadOnly(commentLocator, true))
sinceTenSecondsAgo := strconv.FormatInt(time.Now().Add(-time.Second*10).UnixNano()/1000000, 10)
sinceTS := make([]string, 9)
formattedTS := make([]string, 9)
for i, created := range timestamps {
sinceTS[i] = strconv.FormatInt(created.UnixNano()/1000000, 10)
formattedTS[i] = created.Format(time.RFC3339Nano)
}
t.Logf("last timestamp: %v", timestamps[7])
testCases := []struct {
params string
expectedBody string
}{
// test parameters url, format, since, sort
{"", fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url", fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"format=plain", fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"format=plain&url=test-url", fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTenSecondsAgo, fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTenSecondsAgo, fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTS[0], fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTS[0], fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTS[1], fmt.Sprintf(`"info":{"count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTS[1], fmt.Sprintf(`"info":{"url":"test-url","count":5,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTS[4], fmt.Sprintf(`"info":{"count":3,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTS[4], fmt.Sprintf(`"info":{"url":"test-url","count":2,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"format=tree", `"info":{"count":7`},
{"format=tree&url=test-url", `"info":{"url":"test-url","count":6`},
{"format=tree&sort=+time", `"info":{"count":7`},
{"format=tree&url=test-url&sort=+time", `"info":{"url":"test-url","count":6`},
{"format=tree&sort=-score", `"info":{"count":7`},
{"format=tree&url=test-url&sort=-score", `"info":{"url":"test-url","count":6`},
{"sort=+time", fmt.Sprintf(`"score":-25,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[8])},
{"sort=-time", fmt.Sprintf(`"score":1,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[0])},
{"sort=+score", fmt.Sprintf(`"score":10,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[2])},
{"sort=+score&url=test-url", fmt.Sprintf(`"score":10,"vote":0,"time":%q}],"info":{"url":"test-url","count":6`, formattedTS[2])},
{"sort=-score", fmt.Sprintf(`"score":-25,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[8])},
{"sort=-score&url=test-url", fmt.Sprintf(`"score":-2,"vote":0,"controversy":1.5874010519681994,"time":%q}],"info":{"url":"test-url","count":6`, formattedTS[6])},
{"sort=-time&since=" + sinceTS[4], fmt.Sprintf(`"score":-1,"vote":0,"controversy":2.924017738212866,"time":%q}],"info":{"count":3`, formattedTS[4])},
{"sort=-score&since=" + sinceTS[3], fmt.Sprintf(`"score":-25,"vote":0,"time":%q}],"info":{"count":4`, formattedTS[8])},
{"sort=-score&url=test-url&since=" + sinceTS[3], fmt.Sprintf(`"score":-2,"vote":0,"controversy":1.5874010519681994,"time":%q}],"info":{"url":"test-url","count":3`, formattedTS[6])},
{"sort=+controversy&url=test-url&since=" + sinceTS[5], fmt.Sprintf(`"score":-2,"vote":0,"controversy":1.5874010519681994,"time":%q}],"info":{"url":"test-url","count":1`, formattedTS[6])},
// three comments of which last one deleted and doesn't have controversy so returned last
{"sort=-controversy&url=test-url&since=" + sinceTS[5], fmt.Sprintf(`"score":0,"vote":0,"time":%q,"delete":true}],"info":{"url":"test-url","count":1`, formattedTS[7])},
// test readonly status for the post without comments
{"url=readonly-test", `"info":{"count":0,"count_left":0,"read_only":true`},
{"format=tree&url=readonly-test", `"info":{"count":0,"count_left":0,"read_only":true`},
// test parameters limit, offset_id for format=plain
{"limit=bad", `{"code":1,"details":"bad limit value","error":"strconv.Atoi: parsing \"bad\": invalid syntax"}`},
{"offset_id=bad", `{"code":1,"details":"bad offset_id value","error":"invalid UUID length: 3"}`},
{"limit=2", `"info":{"count":7,"count_left":5,"last_comment":"` + ids[1]},
{"limit=6", `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{"limit=7", `"info":{"count":7,"count_left":0,"last_comment":"` + ids[8]},
{"limit=2&url=test-url", `"info":{"url":"test-url","count":6,"count_left":6,"last_comment":"` + ids[1]},
{"limit=6&url=test-url", `"info":{"url":"test-url","count":6,"count_left":2,"last_comment":"` + ids[5]},
{"limit=7&url=test-url", `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("limit=2&offset_id=%s", ids[2]), `"info":{"count":7,"count_left":2,"last_comment":"` + ids[4]},
{fmt.Sprintf("limit=2&offset_id=%s", ids[3]), `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("limit=2&offset_id=%s", ids[4]), `"info":{"count":7,"count_left":0`},
{fmt.Sprintf("limit=1&offset_id=%s", ids[6]), `"info":{"count":7,"count_left":0`},
{fmt.Sprintf("limit=2&offset_id=%s", ids[8]), `"info":{"count":7,"count_left":0`},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[2]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[4]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[3]), `"info":{"url":"test-url","count":6,"count_left":2,"last_comment":"` + ids[5]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[4]), `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("limit=1&url=test-url&offset_id=%s", ids[6]), `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[7]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[8]), `"info":{"url":"test-url","count":6,"count_left":6,`},
// deleted comment, offset is ignored in site-wide request but not for particular URL
{fmt.Sprintf("limit=2&offset_id=%s", ids[5]), `"info":{"count":7,"count_left":5,"last_comment":"` + ids[1]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[5]), `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[7]},
// non-existing comment, offset is ignored, deleted comments included into request with "url"
{fmt.Sprintf("limit=1&offset_id=%s", uuid.New().String()), `"info":{"count":7,"count_left":6,"last_comment":"` + ids[0]},
{fmt.Sprintf("limit=1&url=test-url&offset_id=%s", uuid.New().String()), `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[0]},
// since is ignored for tree format, so we test it only for plain
{"limit=6&since=" + sinceTenSecondsAgo, `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{"limit=1&since=" + sinceTS[4], `"info":{"count":3,"count_left":2,"last_comment":"` + ids[4]},
{"limit=6&url=test-url&since=" + sinceTenSecondsAgo, `"info":{"url":"test-url","count":6,"count_left":2,"last_comment":"` + ids[5]},
{"limit=1&url=test-url&since=" + sinceTS[4], `"info":{"url":"test-url","count":2,"count_left":3,"last_comment":"` + ids[4]},
// start with deleted comment timestamp
{"limit=1&since=" + sinceTS[5], `"info":{"count":2,"count_left":1,"last_comment":"` + ids[6]},
{"limit=1&since=" + sinceTS[6], `"info":{"count":2,"count_left":1,"last_comment":"` + ids[6]},
{"limit=1&url=test-url&since=" + sinceTS[5], `"info":{"url":"test-url","count":1,"count_left":2,"last_comment":"` + ids[5]},
{"limit=1&url=test-url&since=" + sinceTS[6], `"info":{"url":"test-url","count":1,"count_left":1,"last_comment":"` + ids[6]},
// test sort
{"limit=1&sort=+time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[0]},
{"limit=1&sort=-time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[7]},
{"limit=1&sort=+score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[6]},
{"limit=1&sort=-score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[2]},
{"limit=1&sort=+controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[0]},
{"limit=1&sort=-controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[3]},
// test parameters limit, offset_id for format=tree
{"format=tree&limit=bad", `{"code":1,"details":"bad limit value","error":"strconv.Atoi: parsing \"bad\": invalid syntax"}`},
{"format=tree&offset_id=bad", `{"code":1,"details":"bad offset_id value","error":"invalid UUID length: 3"}`},
{"format=tree&limit=2", `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{"format=tree&limit=6", `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{"format=tree&limit=7", `"info":{"count":7,"count_left":0,"last_comment":"` + ids[8]},
{"format=tree&url=test-url&limit=2", `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{"format=tree&url=test-url&limit=6", `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[6]},
{"format=tree&url=test-url&limit=7", `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[6]},
// start after first top-level comment
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[0]), `"info":{"count":7,"count_left":2,"last_comment":"` + ids[1]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[0]), `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[1]},
// start after second top-level comment
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[1]), `"info":{"count":7,"count_left":0,"last_comment":"` + ids[8]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[1]), `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[6]},
// start after third top-level comment, so expect comment to post 2, or no comments on post 1 if "url" is set
{fmt.Sprintf("format=tree&limit=1&offset_id=%s", ids[6]), `"info":{"count":7,"count_left":0,"last_comment":"` + ids[8]},
{fmt.Sprintf("format=tree&url=test-url&limit=1&offset_id=%s", ids[6]), `"info":{"url":"test-url","count":6,"count_left":0`},
// non-root comment IDs or non-existing IDs are ignored
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[2]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[3]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[4]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[7]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=1&offset_id=%s", uuid.New().String()), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[2]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[3]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[4]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[7]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=1&offset_id=%s", uuid.New().String()), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
// test sort
{"format=tree&limit=1&sort=+time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{"format=tree&limit=1&sort=-time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":5,"last_comment":"` + ids[6]},
{"format=tree&limit=1&sort=+score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":5,"last_comment":"` + ids[6]},
{"format=tree&limit=1&sort=-score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":4,"last_comment":"` + ids[1]},
{"format=tree&limit=1&sort=+controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{"format=tree&limit=1&sort=-controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":5,"last_comment":"` + ids[6]},
}
for _, tc := range testCases {
t.Run(tc.params, func(t *testing.T) {
url := fmt.Sprintf(ts.URL+"/api/v1/find?site=remark42&%s", tc.params)
body, code := get(t, url)
// bad-request cases are identified by their error response body rather than
// a "=bad" substring of the params: comment IDs are random UUIDs and one
// starting with "bad" (e.g. offset_id=bad49e60-...) would otherwise be
// misread as a bad request, making this test flaky.
expectedStatus := http.StatusOK
if strings.Contains(tc.expectedBody, `"error":`) {
expectedStatus = http.StatusBadRequest
}
assert.Equal(t, expectedStatus, code)
assert.Contains(t, body, tc.expectedBody)
t.Log(body)
})
}
}
func TestRest_UserInfo(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
@@ -627,7 +946,7 @@ func TestRest_Config(t *testing.T) {
err := json.Unmarshal([]byte(body), &j)
assert.NoError(t, err)
assert.Equal(t, 300.0, j["edit_duration"])
assert.EqualValues(t, []interface{}{"a1", "a2"}, j["admins"])
assert.EqualValues(t, []any{"a1", "a2"}, j["admins"])
assert.Equal(t, "admin@remark-42.com", j["admin_email"])
assert.Equal(t, 4000.0, j["max_comment_size"])
assert.Equal(t, -5.0, j["low_score"])
@@ -668,13 +987,26 @@ func TestRest_QR(t *testing.T) {
assert.Equal(t, "image/png", r.Header.Get("Content-Type"))
assert.Equal(t, http.StatusOK, r.StatusCode)
// compare the image
// compare the decoded image rather than the encoded bytes: the pixels are what the endpoint
// promises, while the byte stream is whatever the toolchain's png encoder produces, and
// pinning that fails on a go release that changes it
fh, err := os.Open("testdata/qr_test.png")
defer func() { assert.NoError(t, fh.Close()) }()
assert.NoError(t, err)
img, err := io.ReadAll(fh)
assert.NoError(t, err)
assert.Equal(t, img, bdy)
require.NoError(t, err)
want, err := png.Decode(fh)
require.NoError(t, err)
got, err := png.Decode(bytes.NewReader(bdy))
require.NoError(t, err, "the endpoint did not return a decodable png")
require.Equal(t, want.Bounds(), got.Bounds(), "the qr code is not the size it used to be")
for y := want.Bounds().Min.Y; y < want.Bounds().Max.Y; y++ {
for x := want.Bounds().Min.X; x < want.Bounds().Max.X; x++ {
if want.At(x, y) != got.At(x, y) {
t.Fatalf("the qr code differs at %d,%d: want %v, got %v", x, y, want.At(x, y), got.At(x, y))
}
}
}
}
func TestRest_Info(t *testing.T) {
@@ -725,3 +1057,197 @@ func TestRest_Robots(t *testing.T) {
"Allow: /api/v1/list\nAllow: /api/v1/config\nAllow: /api/v1/user\nAllow: /api/v1/img\n"+
"Allow: /api/v1/avatar\nAllow: /api/v1/picture\n", body)
}
// TestRest_LoadPictureRejectsPathTraversal reproduces the unauthenticated path-traversal
// vulnerability in GET /api/v1/picture/{user}/{id}. Before the fix, the handler concatenated
// the URL params verbatim into a filesystem path via path.Join, so a request like
// `/api/v1/picture/../remark.db` would resolve to `<base>/../remark.db`, escaping the image
// directory. Even when the file did not exist (default Partitions=100 mitigates direct hits),
// the FS error message leaked the constructed internal path back to the unauthenticated caller.
func TestRest_LoadPictureRejectsPathTraversal(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
cases := []struct {
name string
path string
wantStatus int
}{
// A literal ".." is normalized away by net/http.ServeMux before routing: the request
// is redirected to the cleaned path, which matches no picture route, so it never reaches
// loadPictureCtrl and resolves to 404. The traversal is neutralized at the router level
// (the cleaned path can only ever reach defined routes or the webRoot-bounded file server),
// so nothing is served either way.
{name: "dotdot in user segment", path: "/api/v1/picture/../remark.db", wantStatus: http.StatusNotFound},
// Encoded traversal is not cleaned by the router, so the handler's safePictureSegment
// validation is what rejects it, with 400.
{name: "dotdot in id segment", path: "/api/v1/picture/dev_user/..%2Fremark.db", wantStatus: http.StatusBadRequest},
{name: "encoded dotdot in user segment", path: "/api/v1/picture/%2E%2E/remark.db", wantStatus: http.StatusBadRequest},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
req, err := http.NewRequest(http.MethodGet, ts.URL+c.path, http.NoBody)
require.NoError(t, err)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, c.wantStatus, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
s := string(body)
assert.NotContains(t, s, "..", "error body must not echo traversal marker")
assert.NotContains(t, s, "remark.db", "error body must not echo attacker-supplied filename")
assert.NotContains(t, s, "no such file", "error body must not leak filesystem state")
assert.NotContains(t, s, "/var/", "error body must not leak internal filesystem path")
})
}
}
// TestRest_LoadPictureRejectsControlCharsInSegment makes sure a CRLF / tab / NUL
// in the URL segment is rejected by safePictureSegment. Without the rejection
// the [WARN] log line constructed from %q-formatted segments would still be
// safe (Go's %q escapes control chars), but a future log change to %s would
// turn this into log forgery — and no legitimate picture id ever needs control
// characters, so the right place to slam the door is in the validator.
func TestRest_LoadPictureRejectsControlCharsInSegment(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
cases := []struct {
name string
path string
}{
{name: "lf in user segment", path: "/api/v1/picture/dev%0Auser/abc.png"},
{name: "cr in user segment", path: "/api/v1/picture/dev%0Duser/abc.png"},
{name: "tab in user segment", path: "/api/v1/picture/dev%09user/abc.png"},
{name: "lf in id segment", path: "/api/v1/picture/dev_user/abc%0A.png"},
{name: "nul in id segment", path: "/api/v1/picture/dev_user/abc%00.png"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
req, err := http.NewRequest(http.MethodGet, ts.URL+c.path, http.NoBody)
require.NoError(t, err)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
s := string(body)
assert.Contains(t, s, "invalid picture id", "must reject as invalid input, not fall through to storage")
assert.NotContains(t, s, "no such file", "must not reach the filesystem")
})
}
}
// TestRest_LoadPictureDefenseHeaders saves a real PNG via the standard upload handler
// and asserts that GET /api/v1/picture/{user}/{id} carries the layered defense headers
// (strict CSP, nosniff, Content-Disposition with filename) and that the strict ETag
// matcher does not 304 on a substring-of-the-real-etag (the pre-fix matcher would).
func TestRest_LoadPictureDefenseHeaders(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
// upload a real PNG via /api/v1/picture
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)
fileWriter, err := bodyWriter.CreateFormFile("file", "picture.png")
require.NoError(t, err)
_, err = io.Copy(fileWriter, gopherPNG())
require.NoError(t, err)
contentType := bodyWriter.FormDataContentType()
require.NoError(t, bodyWriter.Close())
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture?site=remark42", ts.URL), bodyBuf)
require.NoError(t, err)
req.Header.Add("Content-Type", contentType)
req.Header.Add("X-JWT", devToken)
resp, err := client.Do(req)
require.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
m := map[string]string{}
require.NoError(t, json.Unmarshal(body, &m))
require.NotEmpty(t, m["id"])
// fetch the picture and assert defense headers
resp, err = http.Get(fmt.Sprintf("%s/api/v1/picture/%s", ts.URL, m["id"]))
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "default-src 'none'; sandbox; frame-ancestors 'none'",
resp.Header.Get("Content-Security-Policy"))
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, `inline; filename="image"`, resp.Header.Get("Content-Disposition"))
assert.Equal(t, "image/png", resp.Header.Get("Content-Type"))
realEtag := resp.Header.Get("Etag")
require.NotEmpty(t, realEtag)
// strict matcher: an If-None-Match value that CONTAINS the real etag as a substring
// but is not equal to it must NOT trigger 304. The pre-fix matcher used
// strings.Contains(header, etag) and would have returned true here.
require.True(t, len(realEtag) > 4)
substringMatch := "prefix-" + realEtag + "-suffix"
req2, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/picture/%s", ts.URL, m["id"]), http.NoBody)
require.NoError(t, err)
req2.Header.Set("If-None-Match", substringMatch)
resp2, err := client.Do(req2)
require.NoError(t, err)
defer resp2.Body.Close()
assert.Equal(t, http.StatusOK, resp2.StatusCode,
"strict etag matcher must NOT 304 when real etag appears only as a substring of If-None-Match; got %q vs real %q", substringMatch, realEtag)
// sanity: the exact real etag DOES validate
req3, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/picture/%s", ts.URL, m["id"]), http.NoBody)
require.NoError(t, err)
req3.Header.Set("If-None-Match", realEtag)
resp3, err := client.Do(req3)
require.NoError(t, err)
defer resp3.Body.Close()
assert.Equal(t, http.StatusNotModified, resp3.StatusCode, "exact etag must round-trip as 304")
}
// TestRest_LoadPictureRejectsNonImage proves the /picture/ handler rejects bytes that
// don't sniff as a real image — even when retrieved successfully from the image store.
// Uses a StoreMock so we can return arbitrary attacker bytes for a valid-looking id.
func TestRest_LoadPictureRejectsNonImage(t *testing.T) {
htmlBody := []byte("<html><body><script>alert(document.domain)</script></body></html>")
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return htmlBody, nil
}}
// minimal public struct on purpose: the reject path only exercises imageService.Load
// (other fields like dataService, cache, commentFormatter are not touched here).
p := &public{imageService: image.NewService(&imageStore, image.ServiceParams{})}
router := routegroup.New(http.NewServeMux())
router.HandleFunc("GET /api/v1/picture/{user}/{id}", p.loadPictureCtrl)
ts := httptest.NewServer(router)
defer ts.Close()
resp, err := http.Get(ts.URL + "/api/v1/picture/dev_user/abc.png")
require.NoError(t, err)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnsupportedMediaType, resp.StatusCode,
"non-image bytes must be rejected as 415")
assert.False(t, strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html"),
"reject response must not be text/html; got %q", resp.Header.Get("Content-Type"))
assert.NotContains(t, string(body), "<script>",
"attacker payload must not be echoed back")
assert.Equal(t, "no-store", resp.Header.Get("Cache-Control"),
"rejection path must not be cacheable")
// defense headers still present on the reject path
assert.Equal(t, "default-src 'none'; sandbox; frame-ancestors 'none'",
resp.Header.Get("Content-Security-Policy"))
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, `inline; filename="image"`, resp.Header.Get("Content-Disposition"))
}
+439 -211
View File
@@ -4,24 +4,28 @@ import (
"bytes"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"math/rand"
"io/fs"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"testing/fstest"
"time"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/provider"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
"github.com/go-pkgz/auth/v2/provider"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
bolt "go.etcd.io/bbolt"
@@ -36,6 +40,7 @@ import (
"github.com/umputun/remark42/backend/app/store/engine"
"github.com/umputun/remark42/backend/app/store/image"
"github.com/umputun/remark42/backend/app/store/service"
"github.com/umputun/remark42/backend/app/webassets"
)
// To generate a token, enter one of the tokens here into https://jwt.io, change the secret to one you're using in your test
@@ -68,22 +73,319 @@ func TestRest_FileServer(t *testing.T) {
_ = os.Remove(testHTMLFile)
}
// TestRest_FileServerStaticAssets covers the static file server behaviors that are
// sensitive to the router: the bare /web -> /web/ redirect, cache headers applied to
// served assets, 404 for missing files, and the directory-listing block.
func TestRest_FileServerStaticAssets(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
require.NoError(t, os.WriteFile(srv.WebRoot+"/asset-test.html", []byte("static body"), 0o600))
require.NoError(t, os.MkdirAll(srv.WebRoot+"/subdir-test", 0o700))
defer func() {
_ = os.Remove(srv.WebRoot + "/asset-test.html")
_ = os.RemoveAll(srv.WebRoot + "/subdir-test")
}()
noRedirect := http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
defer noRedirect.CloseIdleConnections()
t.Run("bare /web redirects to /web/", func(t *testing.T) {
resp, err := noRedirect.Get(ts.URL + "/web")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMovedPermanently, resp.StatusCode)
assert.Equal(t, "/web/", resp.Header.Get("Location"))
})
t.Run("serves an existing asset with cache headers", func(t *testing.T) {
resp, err := noRedirect.Get(ts.URL + "/web/asset-test.html")
require.NoError(t, err)
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "static body", string(body))
assert.NotEmpty(t, resp.Header.Get("Etag"), "cacheControl must set an Etag on served assets")
assert.Contains(t, resp.Header.Get("Cache-Control"), "max-age", "cacheControl must set max-age on served assets")
})
t.Run("missing asset returns 404", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/does-not-exist.html")
assert.Equal(t, http.StatusNotFound, code)
})
t.Run("directory listing is blocked", func(t *testing.T) {
resp, err := noRedirect.Get(ts.URL + "/web/subdir-test/")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusNotFound, resp.StatusCode, "directory listings must be blocked")
})
}
// TestRest_FileServerBackendAssets covers the assets embedded in the binary and the rule that a
// name the frontend build provides is served from there instead. WebRoot is a fresh empty
// directory so the frontend side is known, rather than the shared temp dir startupT defaults to.
func TestRest_FileServerBackendAssets(t *testing.T) {
ts, srv, teardown := startupT(t, func(srv *Rest) { srv.WebRoot = t.TempDir() })
defer teardown()
t.Run("serves every embedded asset byte for byte", func(t *testing.T) {
for _, name := range []string{"privacy.html", "markdown-help.html", "400x400.jpeg"} {
t.Run(name, func(t *testing.T) {
want, err := fs.ReadFile(webassets.FS, name)
require.NoError(t, err)
body, code := get(t, ts.URL+"/web/"+name)
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, string(want), body, "the bytes must come from the embedded assets")
})
}
})
t.Run("serves the image with its own content type", func(t *testing.T) {
resp, err := http.Get(ts.URL + "/web/400x400.jpeg")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "image/jpeg", resp.Header.Get("Content-Type"))
})
t.Run("head is served", func(t *testing.T) {
resp, err := http.Head(ts.URL + "/web/privacy.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
})
t.Run("frontend output wins over the embedded copy", func(t *testing.T) {
require.NoError(t, os.WriteFile(srv.WebRoot+"/privacy.html", []byte("operator's own policy"), 0o600))
t.Cleanup(func() { _ = os.Remove(srv.WebRoot + "/privacy.html") })
body, code := get(t, ts.URL+"/web/privacy.html")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, "operator's own policy", body)
})
t.Run("traversal out of the asset root is refused", func(t *testing.T) {
for _, p := range []string{"/web/../../etc/passwd", "/web/..%2f..%2fetc%2fpasswd", "/web/%2e%2e/%2e%2e/etc/passwd"} {
t.Run(p, func(t *testing.T) {
body, code := get(t, ts.URL+p)
assert.NotContains(t, body, "root:", "must never serve a file outside the served roots")
assert.NotEqual(t, http.StatusInternalServerError, code, "a rejected name must not surface as 500")
})
}
})
t.Run("missing in both still returns 404", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/neither-source-has-this.html")
assert.Equal(t, http.StatusNotFound, code)
})
}
// TestRest_FileServerEmbeddedFrontend covers the branch taken when no web root exists on disk,
// which is how the released binary runs. The frontend stands in for the copy embedded at
// app/cmd/web, so a name it provides and a name only the assets provide are both exercised.
func TestRest_FileServerEmbeddedFrontend(t *testing.T) {
frontend := fstest.MapFS{"index.html": {Data: []byte("embedded frontend index")}}
router := routegroup.New(http.NewServeMux())
addFileServer(router, frontend, filepath.Join(t.TempDir(), "absent"), "test-version", "https://remark.example.com")
ts := httptest.NewServer(router)
defer ts.Close()
t.Run("serves the embedded frontend", func(t *testing.T) {
body, code := get(t, ts.URL+"/web/index.html")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, "embedded frontend index", body)
})
for _, name := range []string{"privacy.html", "markdown-help.html", "400x400.jpeg"} {
t.Run("falls back to "+name, func(t *testing.T) {
want, err := fs.ReadFile(webassets.FS, name)
require.NoError(t, err)
body, code := get(t, ts.URL+"/web/"+name)
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, string(want), body)
})
}
t.Run("a name neither source has is missing", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/nothing-here.html")
assert.Equal(t, http.StatusNotFound, code)
})
t.Run("a name the operating system rejects is missing, not an error", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/a%00b.html")
assert.Equal(t, http.StatusNotFound, code)
})
}
// TestRest_FileServerRoutesEmbedded drives the whole router the released binary runs: no web root
// on disk, and the frontend read from WebFS. It is what pins the web/ prefix routes() strips, which
// a test calling addFileServer directly cannot see.
func TestRest_FileServerRoutesEmbedded(t *testing.T) {
frontend := fstest.MapFS{
"web/index.html": {Data: []byte("embedded index")},
"web/iframe.html": {Data: []byte("embedded iframe")},
"web/remark.mjs": {Data: []byte("embedded bundle")},
}
ts, _, teardown := startupT(t, func(srv *Rest) {
srv.WebRoot = filepath.Join(t.TempDir(), "absent")
srv.WebFS = frontend
})
defer teardown()
t.Run("serves the frontend from under the web prefix", func(t *testing.T) {
for name, want := range map[string]string{
"index.html": "embedded index",
"iframe.html": "embedded iframe",
"remark.mjs": "embedded bundle",
} {
t.Run(name, func(t *testing.T) {
body, code := get(t, ts.URL+"/web/"+name)
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, want, body)
})
}
})
t.Run("the prefix is stripped rather than exposed", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/web/index.html")
assert.Equal(t, http.StatusNotFound, code, "the web/ prefix must not be reachable as a path")
})
t.Run("the embedded assets still answer alongside it", func(t *testing.T) {
want, err := fs.ReadFile(webassets.FS, "privacy.html")
require.NoError(t, err)
body, code := get(t, ts.URL+"/web/privacy.html")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, string(want), body)
})
}
// refusingSubFS is an fs.FS whose Sub refuses, which is the only way fs.Sub returns a nil
// filesystem. routes() has to survive it, since a nil frontend would panic on the first request.
type refusingSubFS struct{}
func (refusingSubFS) Open(name string) (fs.File, error) {
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}
func (refusingSubFS) Sub(string) (fs.FS, error) { return nil, errors.New("refused") }
// TestRest_FileServerFrontendSourceRefused covers the branch where the frontend source cannot be
// sub-rooted: /web must keep serving the embedded assets rather than panicking.
func TestRest_FileServerFrontendSourceRefused(t *testing.T) {
ts, _, teardown := startupT(t, func(srv *Rest) {
srv.WebRoot = filepath.Join(t.TempDir(), "absent")
srv.WebFS = refusingSubFS{}
})
defer teardown()
t.Run("the embedded assets still serve", func(t *testing.T) {
want, err := fs.ReadFile(webassets.FS, "privacy.html")
require.NoError(t, err)
body, code := get(t, ts.URL+"/web/privacy.html")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, string(want), body)
})
t.Run("a frontend name is missing rather than fatal", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/iframe.html")
assert.Equal(t, http.StatusNotFound, code)
})
}
// TestRest_RejectHeadOnDestructiveGET verifies that HEAD is blocked on the state-mutating
// GET routes (which stdlib http.ServeMux would otherwise route to the GET handler) while
// still being served for safe, read-only routes.
func TestRest_RejectHeadOnDestructiveGET(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
client := http.Client{}
defer client.CloseIdleConnections()
t.Run("HEAD is rejected on a destructive GET route", func(t *testing.T) {
req, err := http.NewRequest(http.MethodHead, ts.URL+"/api/v1/admin/deleteme?site=remark42", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode, "HEAD must not reach a state-mutating GET handler")
assert.Equal(t, "GET", resp.Header.Get("Allow"), "405 must carry an Allow header")
})
t.Run("HEAD is rejected on the email unsubscribe route", func(t *testing.T) {
// emailUnsubscribeCtrl deletes the user's email subscription on GET, so HEAD (which
// ServeMux would route to the GET handler) must be rejected before it runs
resp, err := client.Head(ts.URL + "/email/unsubscribe.html?site=remark42")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode, "HEAD must not reach the email-unsubscribe handler")
assert.Equal(t, "GET, POST", resp.Header.Get("Allow"), "Allow must list every method the resource supports")
})
t.Run("HEAD still works on a safe read-only route", func(t *testing.T) {
resp, err := client.Head(ts.URL + "/api/v1/config?site=remark42")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode, "HEAD must still be served for safe read-only routes")
})
t.Run("wrong method on a known route returns 405 with Allow", func(t *testing.T) {
// method-in-pattern is new under ServeMux; a wrong method on a known route must
// still yield 405 with the allowed methods advertised
resp, err := client.Post(ts.URL+"/api/v1/config?site=remark42", "application/json", http.NoBody)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Allow"), "GET", "405 must advertise the allowed methods")
})
}
// TestRest_AvatarMounts verifies both avatar mounts (root /avatar/ and /api/v1/avatar/)
// still route to the avatar handler after the chi Mount -> ServeMux Handle rewiring,
// rather than falling through to a router 404.
func TestRest_AvatarMounts(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
for _, path := range []string{"/api/v1/avatar/nonexistent.image", "/avatar/nonexistent.image"} {
t.Run(path, func(t *testing.T) {
body, code := get(t, ts.URL+path)
// the avatar handler responds (403 "can't load avatar"), not a router 404
assert.Equal(t, http.StatusForbidden, code, "avatar mount must reach the avatar handler")
assert.Contains(t, body, "can't load avatar", "request must reach the avatar handler, not a routing 404")
})
}
}
func TestRest_Shutdown(t *testing.T) {
srv := Rest{Authenticator: &auth.Service{}, ImageProxy: &proxy.Image{}}
port := chooseUnusedPort(t)
done := make(chan bool)
// without waiting for channel close at the end goroutine will stay alive after test finish
// which would create data race with next test
go func() {
time.Sleep(200 * time.Millisecond)
srv.Shutdown()
srv.Run("127.0.0.1", port)
close(done)
}()
st := time.Now()
srv.Run("127.0.0.1", 0)
assert.True(t, time.Since(st).Seconds() < 1, "should take about 100ms")
<-done
defer srv.Shutdown() // a failed readiness wait must not leave srv.Run behind for goleak
waitForServerStart(t, port)
srv.Shutdown()
select {
case <-done:
case <-time.After(serverStopTimeout):
t.Fatal("rest server did not stop after Shutdown")
}
}
func TestRest_filterComments(t *testing.T) {
@@ -102,7 +404,7 @@ func TestRest_filterComments(t *testing.T) {
}
func TestRest_RunStaticSSLMode(t *testing.T) {
sslPort := chooseRandomUnusedPort()
sslPort := chooseUnusedPort(t)
srv := Rest{
Authenticator: auth.NewService(auth.Opts{
AvatarStore: avatar.NewLocalFS("/tmp"),
@@ -119,16 +421,16 @@ func TestRest_RunStaticSSLMode(t *testing.T) {
RemarkURL: fmt.Sprintf("https://localhost:%d", sslPort),
}
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
go func() {
srv.Run("", port)
}()
waitForHTTPSServerStart(sslPort)
waitForServerStart(t, sslPort, port)
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
@@ -157,7 +459,7 @@ func TestRest_RunStaticSSLMode(t *testing.T) {
}
func TestRest_RunAutocertModeHTTPOnly(t *testing.T) {
sslPort := chooseRandomUnusedPort()
sslPort := chooseUnusedPort(t)
srv := Rest{
Authenticator: &auth.Service{},
ImageProxy: &proxy.Image{},
@@ -168,17 +470,17 @@ func TestRest_RunAutocertModeHTTPOnly(t *testing.T) {
RemarkURL: fmt.Sprintf("https://localhost:%d", sslPort),
}
port := chooseRandomUnusedPort()
port := chooseUnusedPort(t)
go func() {
// can't check https server locally, just only http server
srv.Run("", port)
}()
waitForHTTPSServerStart(sslPort)
waitForServerStart(t, sslPort, port)
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
}
@@ -193,28 +495,6 @@ func TestRest_RunAutocertModeHTTPOnly(t *testing.T) {
srv.Shutdown()
}
func TestRest_rejectAnonUser(t *testing.T) {
ts := httptest.NewServer(fakeAuth(rejectAnonUser(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello")
}))))
defer ts.Close()
resp, err := http.Get(ts.URL)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "use not logged in")
resp, err = http.Get(ts.URL + "?fake_id=anonymous_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode, "anon rejected")
resp, err = http.Get(ts.URL + "?fake_id=real_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "real user")
}
func Test_URLKey(t *testing.T) {
tbl := []struct {
url string
@@ -227,7 +507,6 @@ func Test_URLKey(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
r, err := http.NewRequest("GET", tt.url, http.NoBody)
require.NoError(t, err)
@@ -252,7 +531,6 @@ func Test_URLKeyWithUser(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
r, err := http.NewRequest("GET", tt.url, http.NoBody)
require.NoError(t, err)
@@ -279,7 +557,6 @@ func TestRest_parseError(t *testing.T) {
}
for n, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(n), func(t *testing.T) {
res := parseError(tt.err, rest.ErrInternal)
assert.Equal(t, tt.res, res)
@@ -287,150 +564,40 @@ func TestRest_parseError(t *testing.T) {
}
}
func TestRest_cacheControl(t *testing.T) {
tbl := []struct {
url string
version string
exp time.Duration
etag string
maxAge int
}{
{"http://example.com/foo", "v1", time.Hour, "b433be1ea19edaee9dc92ca4b895b6bdf3c058cb", 3600},
{"http://example.com/foo2", "v1", 10 * time.Hour, "6d8466aef3246c1057452561acddf7ad9d0d99e0", 36000},
{"http://example.com/foo", "v2", time.Hour, "481700c52aab0dfbca99f3ffc2a4fbb27884c114", 3600},
{"https://example.com/foo", "v2", time.Hour, "bebd4f1b87f474792c4e75e5affe31fbf67f5778", 3600},
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", tt.url, http.NoBody)
w := httptest.NewRecorder()
h := cacheControl(tt.exp, tt.version)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
t.Logf("%+v", resp.Header)
assert.Equal(t, `"`+tt.etag+`"`, resp.Header.Get("Etag"))
assert.Equal(t, `max-age=`+strconv.Itoa(int(tt.exp.Seconds()))+", no-cache", resp.Header.Get("Cache-Control"))
})
}
}
func TestRest_frameAncestors(t *testing.T) {
tbl := []struct {
hosts []string
header string
}{
{[]string{"http://example.com"}, "frame-ancestors http://example.com;"},
{[]string{}, ""},
{[]string{"http://example.com", "http://example2.com"}, "frame-ancestors http://example.com http://example2.com;"},
}
ts, _, teardown := startupT(t, func(o *Rest) {
o.AllowedAncestors = []string{"'self'", "https://example.com"}
})
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", http.NoBody)
w := httptest.NewRecorder()
// test case with frame-ancestors
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "frame-ancestors 'self' https://example.com;")
// httptest.Server.Close waits on connections still in use, and a deferred close does not run
// until the test ends, so the body has to be released before the server is torn down here
require.NoError(t, resp.Body.Close())
client.CloseIdleConnections()
teardown()
h := frameAncestors(tt.hosts)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
t.Logf("%+v", resp.Header)
assert.Equal(t, tt.header, resp.Header.Get("Content-Security-Policy"))
})
}
}
func TestRest_subscribersOnly(t *testing.T) {
paidSubUser := &token.User{}
paidSubUser.SetPaidSub(true)
tbl := []struct {
subsOnly bool
user token.User
setUser bool
status int
}{
{true, token.User{}, false, http.StatusUnauthorized},
{true, token.User{}, true, http.StatusForbidden},
{false, token.User{}, false, http.StatusOK},
{false, token.User{}, true, http.StatusOK},
{true, *paidSubUser, true, http.StatusOK},
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", http.NoBody)
if tt.setUser {
req = token.SetUserInfo(req, tt.user)
}
w := httptest.NewRecorder()
h := subscribersOnly(tt.subsOnly)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
func Test_validEmailAuth(t *testing.T) {
tbl := []struct {
req string
status int
}{
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=site-with-dash_and_underscore-and.dot&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=Евгений+Умпутун", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=12", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someonelooong+loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong", http.StatusForbidden},
{"/auth/twitter/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun+example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=bad!site&address=umputun%example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooongsite&address=umputun%example.com&user=someone", http.StatusForbidden},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com"+tt.req, http.NoBody)
w := httptest.NewRecorder()
h := validEmailAuth()(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
// randomPath pick a file or folder name which is not in use for sure
func randomPath(tempDir, basename, suffix string) (string, error) {
for i := 0; i < 10; i++ {
fname := fmt.Sprintf("/%s/%s-%d%s", tempDir, basename, rand.Int31(), suffix)
fmt.Printf("fname %q", fname)
_, err := os.Stat(fname)
if err != nil {
return fname, nil
}
}
return "", fmt.Errorf("cannot create temp file in %s", tempDir)
// test case without frame-ancestors
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.AllowedAncestors = []string{}
})
defer teardown()
resp, err = client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "frame-ancestors *;")
}
// startupT runs fully configured testing server
// srvHook is an optional func to set some Rest param after the creation but prior to Run
func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, srv *Rest, teardown func()) {
tmp := os.TempDir()
testDB, err := randomPath(tmp, "test-remark", ".db")
require.NoError(t, err)
testDB := filepath.Join(t.TempDir(), "test-remark.db") // per-test dir, removed when the test ends
_ = os.RemoveAll(tmp + "/ava-remark42")
_ = os.RemoveAll(tmp + "/pics-remark42")
@@ -438,7 +605,7 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
b, err := engine.NewBoltDB(bolt.Options{}, engine.BoltSite{FileName: testDB, SiteID: "remark42"})
require.NoError(t, err)
memCache := cache.NewScache(cache.NewNopCache())
memCache := cache.NewScache[[]byte](cache.NewNopCache[[]byte]())
astore := adminstore.NewStaticStore("123456", []string{"remark42"}, []string{"a1", "a2"}, "admin@remark-42.com")
restrictedWordsMatcher := service.NewRestrictedWordsMatcher(service.StaticRestrictedWordsLister{Words: []string{"duck"}})
@@ -458,7 +625,7 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
DataService: dataStore,
Authenticator: auth.NewService(auth.Opts{
AdminPasswd: "password",
SecretReader: token.SecretFunc(func(aud string) (string, error) { return "secret", nil }),
SecretReader: token.SecretFunc(func(string) (string, error) { return "secret", nil }),
AvatarStore: avatar.NewLocalFS(tmp + "/ava-remark42"),
}),
Cache: memCache,
@@ -487,15 +654,16 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
Cache: memCache,
KeyStore: astore,
},
NotifyService: notify.NopService,
EmojiEnabled: true,
NotifyService: notify.NopService,
EmojiEnabled: true,
openRouteLimiter: 100,
}
srv.ScoreThresholds.Low, srv.ScoreThresholds.Critical = -5, -10
// add some providers. Needed because we don't allow users with unlisted providers to authenticate
providers := []string{"provider1", "anonymous", "github", "email"}
for _, p := range providers {
srv.Authenticator.AddDirectProvider(p, provider.CredCheckerFunc(func(user, password string) (ok bool, err error) {
srv.Authenticator.AddDirectProvider(p, provider.CredCheckerFunc(func(_, _ string) (ok bool, err error) {
return true, nil
}))
}
@@ -504,12 +672,12 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
h(srv)
}
ts = httptest.NewServer(srv.routes())
routes := srv.routes()
ts = httptest.NewServer(routes)
teardown = func() {
ts.Close()
require.NoError(t, srv.DataService.Close())
_ = os.Remove(testDB)
_ = os.RemoveAll(tmp + "/ava-remark42")
_ = os.RemoveAll(tmp + "/pics-remark42")
}
@@ -517,6 +685,44 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
return ts, srv, teardown
}
const (
// outer bound before a wait is called a hang, generous enough for a loaded CI runner
waitTimeout = 30 * time.Second
pollInterval = 10 * time.Millisecond
// budget for a server to stop once asked, tight enough to catch a shutdown that hangs
serverStopTimeout = 10 * time.Second
// connect budget for a single probe, kept off the poll interval so a slow loopback connect
// on a loaded runner does not look like a server that is not listening
probeDialTimeout = time.Second
// window to prove something did not happen
notifySettle = 300 * time.Millisecond
// poll interval for waits that issue an HTTP request. the admin routes allow 10 req/s and
// the open ones 100 in tests, so this stays below the tighter of the two and the poll
// cannot manufacture the 429s it would then have to interpret
httpPoll = 150 * time.Millisecond
)
// waitForCount blocks until got reaches want, failing the test with the last value it saw.
// for work that is delivered asynchronously, such as notifications reaching a mock destination
func waitForCount(t *testing.T, want int, got func() int, msgAndArgs ...any) {
t.Helper()
require.EventuallyWithT(t, func(c *assert.CollectT) {
assert.Equal(c, want, got(), msgAndArgs...)
}, waitTimeout, pollInterval)
}
// waitForCountSettled waits for got to reach want and then holds it there, so a delivery
// arriving late is caught rather than passing because the count was read the instant it matched
func waitForCountSettled(t *testing.T, want int, got func() int, msgAndArgs ...any) {
t.Helper()
waitForCount(t, want, got, msgAndArgs...)
require.Never(t, func() bool { return got() != want }, notifySettle, pollInterval, msgAndArgs...)
}
// fake auth middleware make user authenticated and uses query's fake_id for ID and fake_name for Name
func fakeAuth(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
@@ -540,7 +746,7 @@ func get(t *testing.T, url string) (response string, statusCode int) {
return string(body), r.StatusCode
}
func sendReq(_ *testing.T, r *http.Request, tkn string) (*http.Response, error) {
func sendReq(r *http.Request, tkn string) (*http.Response, error) {
client := http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
if tkn != "" {
@@ -599,13 +805,17 @@ func post(t *testing.T, url, body string) (*http.Response, error) {
return client.Do(req)
}
func addComment(t *testing.T, c store.Comment, ts *httptest.Server) string {
func addCommentGetCreatedTime(t *testing.T, c store.Comment, ts *httptest.Server) (id string, created time.Time) {
b, err := json.Marshal(c)
require.NoError(t, err, "can't marshal comment %+v", c)
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", bytes.NewBuffer(b))
postURL := ts.URL + "/api/v1/comment"
if c.Locator.SiteID != "" {
postURL += "?site=" + c.Locator.SiteID
}
req, err := http.NewRequest("POST", postURL, bytes.NewBuffer(b))
require.NoError(t, err)
req.Header.Add("X-JWT", devToken)
resp, err := client.Do(req)
@@ -618,45 +828,63 @@ func addComment(t *testing.T, c store.Comment, ts *httptest.Server) string {
crResp := R.JSON{}
err = json.Unmarshal(b, &crResp)
require.NoError(t, err)
time.Sleep(time.Nanosecond * 10)
return crResp["id"].(string)
created, err = time.Parse(time.RFC3339, crResp["time"].(string))
require.NoError(t, err)
return crResp["id"].(string), created
}
func addComment(t *testing.T, c store.Comment, ts *httptest.Server) string {
id, _ := addCommentGetCreatedTime(t, c, ts)
return id
}
func requireAdminOnly(t *testing.T, req *http.Request) {
resp, err := sendReq(t, req, "") // no-auth user
resp, err := sendReq(req, "") // no-auth user
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
resp, err = sendReq(t, req, devToken) // non-admin user
resp, err = sendReq(req, devToken) // non-admin user
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
break
}
}
// chooseUnusedPort asks the kernel for a free port from the ephemeral range, which makes a
// collision between concurrently running package test binaries very unlikely
func chooseUnusedPort(t *testing.T) int {
t.Helper()
ln, err := net.Listen("tcp", ":0")
require.NoError(t, err, "no free port available")
port := ln.Addr().(*net.TCPAddr).Port
require.NoError(t, ln.Close())
return port
}
func waitForHTTPSServerStart(port int) {
// wait for up to 3 seconds for HTTPS server to start
for i := 0; i < 300; i++ {
time.Sleep(time.Millisecond * 10)
conn, _ := net.DialTimeout("tcp", fmt.Sprintf("localhost:%d", port), time.Millisecond*10)
if conn != nil {
// waitForServerStart blocks until something accepts on every listed port, failing the test
// naming the port that never came up
func waitForServerStart(t *testing.T, ports ...int) {
t.Helper()
for _, port := range ports {
require.Eventually(t, func() bool {
conn, err := net.DialTimeout("tcp", fmt.Sprintf("localhost:%d", port), probeDialTimeout)
if err != nil {
return false
}
_ = conn.Close()
break
}
return true
}, waitTimeout, pollInterval, "server on port %d didn't start", port)
}
}
func TestMain(m *testing.M) {
goleak.VerifyTestMain(m)
goleak.VerifyTestMain(
m,
// this will be fixed in https://github.com/hashicorp/golang-lru/issues/159
goleak.IgnoreTopFunction("github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1"),
// regexp2, pulled in by chroma for syntax highlighting, keeps one shared clock goroutine
// alive for up to a second after the last match with a timeout, sleeping in 100ms ticks.
// it ends on its own, but a binary that finishes inside that window is reported as leaking
goleak.IgnoreAnyFunction("github.com/dlclark/regexp2/v2.runClock"),
)
}
+4 -5
View File
@@ -5,7 +5,7 @@ import (
"net/http"
"time"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
"github.com/gorilla/feeds"
@@ -56,8 +56,7 @@ func (s *rss) postCommentsCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
if _, err = w.Write(data); err != nil { //nolint:gosec // xml feed bytes from gorilla/feeds, not HTML
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
}
@@ -88,7 +87,7 @@ func (s *rss) siteCommentsCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
if _, err = w.Write(data); err != nil { //nolint:gosec // xml feed bytes from gorilla/feeds, not HTML
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
}
@@ -120,7 +119,7 @@ func (s *rss) repliesCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
if _, err = w.Write(data); err != nil { //nolint:gosec // xml feed bytes from gorilla/feeds, not HTML
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
}
+68 -67
View File
@@ -14,22 +14,29 @@ import (
"github.com/umputun/remark42/backend/app/store"
)
// rssPubTime returns a second-aligned base timestamp and formats it the way the feed does, so
// comment pubDates are pinned rather than dependent on when in the second the test runs.
func rssPubTime() (base time.Time, pubDate string) {
base = time.Now().Truncate(time.Second)
return base, base.Format(time.RFC1123Z)
}
func TestServer_RssPost(t *testing.T) {
ts, rst, teardown := startupT(t)
defer teardown()
waitOnSecChange()
base, pubDate := rssPubTime()
c1 := store.Comment{
ID: "1234567890",
Text: "test 123",
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
ID: "1234567890",
Text: "test 123",
Timestamp: base,
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
}
id1, err := rst.DataService.Create(c1)
require.NoError(t, err)
assert.Equal(t, "1234567890", id1)
pubDate := time.Now().Format(time.RFC1123Z)
res, code := get(t, ts.URL+"/api/v1/rss/post?site=remark42&url=https://radio-t.com/blah1")
assert.Equal(t, http.StatusOK, code)
@@ -63,21 +70,21 @@ func TestServer_RssSite(t *testing.T) {
ts, rst, teardown := startupT(t)
defer teardown()
waitOnSecChange()
pubDate := time.Now().Format(time.RFC1123Z)
base, pubDate := rssPubTime()
c1 := store.Comment{
ID: "comment-id-1",
Text: "test 123",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
ID: "comment-id-1",
Text: "test 123",
Timestamp: base,
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
}
c2 := store.Comment{
ID: "comment-id-2",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah11", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
ID: "comment-id-2",
Text: "xyz test",
Timestamp: base.Add(time.Millisecond),
Locator: store.Locator{URL: "https://radio-t.com/blah11", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
}
_, err := rst.DataService.Create(c1)
@@ -126,22 +133,22 @@ func TestServer_RssWithReply(t *testing.T) {
ts, rst, teardown := startupT(t)
defer teardown()
waitOnSecChange()
pubDate := time.Now().Format(time.RFC1123Z)
base, pubDate := rssPubTime()
c1 := store.Comment{
ID: "comment-id-1",
Text: "test 123",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
ID: "comment-id-1",
Text: "test 123",
Timestamp: base,
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
}
c2 := store.Comment{
ID: "comment-id-2",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
ID: "comment-id-2",
ParentID: "comment-id-1",
Text: "xyz test",
Timestamp: base.Add(time.Millisecond),
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "remark42"},
User: store.User{ID: "u1", Name: "developer one"},
}
_, err := rst.DataService.Create(c1)
@@ -186,42 +193,45 @@ func TestServer_RssReplies(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
waitOnSecChange()
pubDate := time.Now().Format(time.RFC1123Z)
base, pubDate := rssPubTime()
c1 := store.Comment{
ID: "comment-1",
Text: "c1",
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "user1", Name: "user1"},
ID: "comment-1",
Text: "c1",
Timestamp: base,
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "user1", Name: "user1"},
}
c2 := store.Comment{
ID: "comment-2",
Text: "reply to c1 from user2",
ParentID: "comment-1",
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "user2", Name: "user2"},
ID: "comment-2",
Text: "reply to c1 from user2",
ParentID: "comment-1",
Timestamp: base.Add(time.Millisecond),
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "user2", Name: "user2"},
}
c3 := store.Comment{
ID: "comment-3",
Text: "reply to c1 from user3",
ParentID: "comment-1",
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "user3", Name: "user3"},
ID: "comment-3",
Text: "reply to c1 from user3",
ParentID: "comment-1",
Timestamp: base.Add(2 * time.Millisecond),
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "user3", Name: "user3"},
}
c4 := store.Comment{
ID: "comment-4",
Text: "reply to c2 from developer one",
ParentID: "comment-2",
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "dev", Name: "developer one"},
ID: "comment-4",
Text: "reply to c2 from developer one",
ParentID: "comment-2",
Timestamp: base.Add(3 * time.Millisecond),
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "dev", Name: "developer one"},
}
c5 := store.Comment{
ID: "comment-5",
Text: "developer one",
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "dev", Name: "developer one"},
ID: "comment-5",
Text: "developer one",
Timestamp: base.Add(4 * time.Millisecond),
Locator: store.Locator{URL: "https://radio-t.com/blah1", SiteID: "remark42"},
User: store.User{ID: "dev", Name: "developer one"},
}
_, err := srv.DataService.Create(c1)
@@ -270,24 +280,15 @@ func TestServer_RssReplies(t *testing.T) {
assert.Equal(t, http.StatusBadRequest, code)
}
func waitOnSecChange() {
for {
if time.Now().Nanosecond() < 100000000 {
break
}
time.Sleep(10 * time.Nanosecond)
}
}
// clean formatting, i.e. multiple spaces, \t, \n
func cleanRssFormatting(expected, actual string) (cleanExp, cleanAct string) {
reSpaces := regexp.MustCompile(`[\s\p{Zs}]{2,}`)
expected = strings.Replace(expected, "\n", " ", -1)
expected = strings.Replace(expected, "\t", " ", -1)
expected = strings.ReplaceAll(expected, "\n", " ")
expected = strings.ReplaceAll(expected, "\t", " ")
expected = reSpaces.ReplaceAllString(expected, " ")
actual = strings.Replace(actual, "\n", " ", -1)
actual = strings.ReplaceAll(actual, "\n", " ")
actual = reSpaces.ReplaceAllString(actual, " ")
return expected, actual
}
+37 -18
View File
@@ -2,16 +2,16 @@ package api
import (
"crypto/tls"
"fmt"
"net/http"
"net/url"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
log "github.com/go-pkgz/lgr"
"golang.org/x/crypto/acme/autocert"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"golang.org/x/crypto/acme/autocert"
)
// sslMode defines ssl mode for rest server
@@ -40,13 +40,13 @@ type SSLConfig struct {
// httpToHTTPSRouter creates new router which does redirect from http to https server
// with default middlewares. Used in 'static' ssl mode.
func (s *Rest) httpToHTTPSRouter() chi.Router {
log.Printf("[DEBUG] create https-to-http redirect routes")
router := chi.NewRouter()
router.Use(middleware.RealIP, R.Recoverer(log.Default()))
router.Use(middleware.Throttle(1000), middleware.Timeout(60*time.Second))
func (s *Rest) httpToHTTPSRouter() http.Handler {
log.Printf("[DEBUG] create http-to-https redirect routes")
router := routegroup.New(http.NewServeMux())
router.Use(R.Recoverer(log.Default()))
router.Use(R.Throttle(1000), R.Timeout(60*time.Second))
router.Handle("/*", s.redirectHandler())
router.Handle("/", s.redirectHandler())
return router
}
@@ -54,26 +54,45 @@ func (s *Rest) httpToHTTPSRouter() chi.Router {
// with default middlewares. This part is necessary to obtain certificate from LE.
// If it receives not a acme challenge it performs redirect to https server.
// Used in 'auto' ssl mode.
func (s *Rest) httpChallengeRouter(m *autocert.Manager) chi.Router {
func (s *Rest) httpChallengeRouter(m *autocert.Manager) http.Handler {
log.Printf("[DEBUG] create http-challenge routes")
router := chi.NewRouter()
router.Use(middleware.RealIP, R.Recoverer(log.Default()))
router.Use(middleware.Throttle(1000), middleware.Timeout(60*time.Second))
router := routegroup.New(http.NewServeMux())
router.Use(R.Recoverer(log.Default()))
router.Use(R.Throttle(1000), R.Timeout(60*time.Second))
router.Handle("/*", m.HTTPHandler(s.redirectHandler()))
router.Handle("/", m.HTTPHandler(s.redirectHandler()))
return router
}
func (s *Rest) redirectHandler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
newURL := s.RemarkURL + r.URL.Path
if r.URL.RawQuery != "" {
newURL += "?" + r.URL.RawQuery
newURL, err := s.redirectURL(r)
if err != nil {
log.Printf("[WARN] failed to build redirect URL, %s", err)
http.Error(w, "invalid redirect URL", http.StatusInternalServerError)
return
}
http.Redirect(w, r, newURL, http.StatusTemporaryRedirect)
})
}
func (s *Rest) redirectURL(r *http.Request) (string, error) {
baseURL, err := url.Parse(s.RemarkURL)
if err != nil {
return "", fmt.Errorf("parse remark URL: %w", err)
}
if baseURL.Scheme != "http" && baseURL.Scheme != "https" || baseURL.Host == "" {
return "", fmt.Errorf("remark URL must be absolute HTTP(S) URL")
}
basePath := strings.TrimRight(baseURL.Path, "/")
requestPath := "/" + strings.TrimLeft(r.URL.Path, "/")
baseURL.Path = basePath + requestPath
baseURL.RawQuery = r.URL.RawQuery
baseURL.Fragment = ""
return baseURL.String(), nil
}
func (s *Rest) makeAutocertManager() *autocert.Manager {
return &autocert.Manager{
Prompt: autocert.AcceptTOS,
+12 -2
View File
@@ -21,7 +21,7 @@ func TestSSL_Redirect(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
@@ -40,6 +40,16 @@ func TestSSL_Redirect(t *testing.T) {
assert.Equal(t, "https://localhost:443/blah?param=1", resp.Header.Get("Location"))
}
func TestSSL_RedirectURLKeepsConfiguredHost(t *testing.T) {
rest := Rest{RemarkURL: "https://localhost:443/base"}
req, err := http.NewRequest("GET", "http://example.com//evil.test/path?next=//evil.test", http.NoBody)
require.NoError(t, err)
redirectURL, err := rest.redirectURL(req)
require.NoError(t, err)
assert.Equal(t, "https://localhost:443/base/evil.test/path?next=//evil.test", redirectURL)
}
func TestSSL_ACME_HTTPChallengeRouter(t *testing.T) {
rest := Rest{
RemarkURL: "https://localhost:443",
@@ -56,7 +66,7 @@ func TestSSL_ACME_HTTPChallengeRouter(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
}
+116
View File
@@ -0,0 +1,116 @@
{
"version": 1,
"comments": [
{
"commentHex": "e7a2ef4b4aa1414a7ee65a989889aaecd9d5e7e3bca598ea7a967b4dbcaa8e11",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2022-10-25T07:25:46.807555Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "a29e741145daceb4ca5b3e5e279e05b56f73c04703d93b944718ef757e15317f",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-07-26T12:24:55.058552Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "46baf36433830a4e8bda1de56290cf5fd74c08bfa844fee4ec1744985dc77010",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-10-31T11:03:25.403282Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "6d3bb64ff73b5f9d6a959212ffde472a51abf8bdefaa5ed843659796bceef9de",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "46baf36433830a4e8bda1de56290cf5fd74c08bfa844fee4ec1744985dc77010",
"score": 0,
"state": "approved",
"creationDate": "2023-11-01T22:23:47.112062Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "23fcfcd03745ed71a9d23a9b59387a313df57e5c0faad8ba5dc96112766312c5",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-10-23T12:33:03.370182Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "d0ad6f11cf0c5f8e17457a378a6bb789f412c6b7ef7ada4ae06ec8451f7a18aa",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-10-18T01:18:38.193625Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "098960fd01c1fc7c0d3ea428f52fab97ea5c18aa52f3565bba679224daddc687",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "23fcfcd03745ed71a9d23a9b59387a313df57e5c0faad8ba5dc96112766312c5",
"score": 0,
"state": "approved",
"creationDate": "2023-11-01T22:24:04.639965Z",
"direction": 0,
"deleted": false
}
],
"commenters": [
{
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"email": "undefined",
"name": "blank",
"link": "undefined",
"photo": "undefined",
"provider": "anon",
"joinDate": "2022-06-09T15:54:29.865919Z",
"isModerator": false,
"deleted": false
}
]
}
+140
View File
@@ -0,0 +1,140 @@
package api
import (
"bytes"
"errors"
"io"
"io/fs"
"path/filepath"
"strings"
)
// webFiles serves /web from two sources: a name present in the frontend build is served from there,
// and any other name from the assets embedded in the binary.
type webFiles struct {
frontend fs.FS
embedded fs.FS
}
// Open resolves the name against both sources, and answers a missing .js with the .mjs sibling.
// The build stopped emitting .js while integrations still request it; the bundles carry no module
// syntax, so the same bytes serve both names.
func (w webFiles) Open(name string) (fs.File, error) {
// fs.ValidPath alone is not enough: it accepts names the operating system rejects, NUL among
// them, and os.DirFS turns those into fs.ErrInvalid, which renders as 500 rather than 404
if _, err := filepath.Localize(name); err != nil || !fs.ValidPath(name) {
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}
f, err := w.open(name)
if err == nil {
return f, nil
}
if !errors.Is(err, fs.ErrNotExist) || !strings.HasSuffix(name, ".js") {
return nil, err
}
alias, aliasErr := w.open(strings.TrimSuffix(name, ".js") + ".mjs")
if aliasErr == nil {
return alias, nil
}
if !errors.Is(aliasErr, fs.ErrNotExist) {
return nil, aliasErr
}
return nil, err
}
// open looks the name up in the frontend build first. Only a missing file falls through to the
// embedded assets; every other error is returned so an unreadable file keeps reporting as one
// rather than being replaced by the embedded copy or reported as missing.
func (w webFiles) open(name string) (fs.File, error) {
f, err := w.frontend.Open(name)
if err == nil {
return f, nil
}
if !errors.Is(err, fs.ErrNotExist) {
return nil, err
}
if name == "." {
// the embedded set is a flat list of files; only the frontend build answers for the
// directory itself, so a missing web root reports as missing rather than listing them
return nil, err
}
return w.embedded.Open(name)
}
// emptyFS stands in for a frontend source that could not be opened, so a misconfigured one serves
// nothing instead of panicking or serving the build at paths it does not belong at
type emptyFS struct{}
func (emptyFS) Open(name string) (fs.File, error) {
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}
// remarkURLPlaceholder is what the frontend build carries wherever the instance URL belongs. The
// bundler cannot know that URL, so it emits this marker and every distribution fills it in: the
// docker image rewrites the files under the web root at container start, and the binary, which
// serves the build embedded in itself and has nothing to rewrite, does it here.
const remarkURLPlaceholder = "{% REMARK_URL %}"
// templatedFS fills the instance URL into the files carrying the placeholder. Without it the
// binary serves whatever the build baked in, which is a host no visitor can reach, and the widget
// falls back to it whenever a page omits remark_config.host.
type templatedFS struct {
fs fs.FS
remarkURL string
}
// Open substitutes in the file types the frontend templates, and hands everything else through
// untouched so images and stylesheets keep streaming from their original source
func (t templatedFS) Open(name string) (fs.File, error) {
f, err := t.fs.Open(name)
if err != nil || !templatedName(name) {
return f, err
}
info, err := f.Stat()
if err != nil || info.IsDir() {
return f, err
}
body, err := io.ReadAll(f)
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
return nil, err
}
body = bytes.ReplaceAll(body, []byte(remarkURLPlaceholder), []byte(t.remarkURL))
return &memFile{Reader: bytes.NewReader(body), info: sizedInfo{FileInfo: info, size: int64(len(body))}}, nil
}
// templatedName reports whether the frontend templates this file type. It mirrors the set the
// docker image rewrites, so both distributions substitute in the same files
func templatedName(name string) bool {
switch filepath.Ext(name) {
case ".html", ".js", ".mjs":
return true
}
return false
}
// memFile is a substituted file held in memory. The file server needs a seeker to answer range
// requests and to sniff a content type, which a substituted body no longer has on disk
type memFile struct {
*bytes.Reader
info fs.FileInfo
}
func (f *memFile) Stat() (fs.FileInfo, error) { return f.info, nil }
func (f *memFile) Close() error { return nil }
// sizedInfo reports the length after substitution. The file server writes Content-Length from it,
// so reporting the length on disk would truncate the response or leave the client waiting
type sizedInfo struct {
fs.FileInfo
size int64
}
func (i sizedInfo) Size() int64 { return i.size }
+343
View File
@@ -0,0 +1,343 @@
package api
import (
"io"
"io/fs"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"testing/fstest"
"github.com/go-pkgz/routegroup"
"github.com/umputun/remark42/backend/app/webassets"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestWebFiles_Open(t *testing.T) {
frontend := fstest.MapFS{
"both.html": {Data: []byte("from the frontend build")},
"only-frontend.html": {Data: []byte("frontend only")},
}
embedded := fstest.MapFS{
"both.html": {Data: []byte("from the embedded assets")},
"only-embedded.html": {Data: []byte("embedded only")},
}
w := webFiles{frontend: frontend, embedded: embedded}
tbl := []struct {
name string
lookup string
want string
wantErr error
}{
{name: "present in both is served from the frontend build", lookup: "both.html", want: "from the frontend build"},
{name: "frontend only", lookup: "only-frontend.html", want: "frontend only"},
{name: "embedded only", lookup: "only-embedded.html", want: "embedded only"},
{name: "missing in both", lookup: "neither.html", wantErr: fs.ErrNotExist},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
f, err := w.Open(tt.lookup)
if tt.wantErr != nil {
require.Error(t, err)
assert.ErrorIs(t, err, tt.wantErr)
return
}
require.NoError(t, err)
defer f.Close()
b, err := io.ReadAll(f)
require.NoError(t, err)
assert.Equal(t, tt.want, string(b))
})
}
}
func TestWebFiles_OpenJSAlias(t *testing.T) {
frontend := fstest.MapFS{
"embed.mjs": {Data: []byte("module embed")},
"counter.js": {Data: []byte("operator's own counter")},
"counter.mjs": {Data: []byte("module counter")},
"widget.mjs": {Data: []byte("module widget")},
}
embedded := fstest.MapFS{
"legacy.mjs": {Data: []byte("module legacy")},
"widget.js": {Data: []byte("embedded widget")},
}
w := webFiles{frontend: frontend, embedded: embedded}
tbl := []struct {
name string
lookup string
want string
wantErr error
}{
{name: "missing js served from the mjs sibling", lookup: "embed.js", want: "module embed"},
{name: "alias reaches the embedded assets too", lookup: "legacy.js", want: "module legacy"},
{name: "a real js file wins over its sibling", lookup: "counter.js", want: "operator's own counter"},
{name: "an embedded js wins over a frontend sibling", lookup: "widget.js", want: "embedded widget"},
{name: "mjs is still served directly", lookup: "embed.mjs", want: "module embed"},
{name: "neither name present", lookup: "absent.js", wantErr: fs.ErrNotExist},
{name: "only js aliases, not other extensions", lookup: "embed.html", wantErr: fs.ErrNotExist},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
f, err := w.Open(tt.lookup)
if tt.wantErr != nil {
require.Error(t, err)
assert.ErrorIs(t, err, tt.wantErr)
return
}
require.NoError(t, err)
defer f.Close()
b, err := io.ReadAll(f)
require.NoError(t, err)
assert.Equal(t, tt.want, string(b))
})
}
}
func TestWebFiles_OpenJSAliasNamesTheRequestedFile(t *testing.T) {
w := webFiles{frontend: fstest.MapFS{}, embedded: fstest.MapFS{}}
_, err := w.Open("absent.js")
require.Error(t, err)
assert.ErrorIs(t, err, fs.ErrNotExist)
assert.Contains(t, err.Error(), "absent.js")
assert.NotContains(t, err.Error(), "absent.mjs")
}
func TestWebFiles_OpenJSAliasUnreadableSibling(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores file permissions")
}
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "embed.mjs"), []byte("module embed"), 0o000))
w := webFiles{frontend: os.DirFS(dir), embedded: fstest.MapFS{}}
f, err := w.Open("embed.js")
require.Error(t, err)
assert.ErrorIs(t, err, fs.ErrPermission)
assert.NotErrorIs(t, err, fs.ErrNotExist, "an unreadable sibling must not render as 404")
if err == nil {
_ = f.Close()
}
}
// TestEmptyFS_ServesNothing pins the stand-in used when the frontend source cannot be opened:
// every name must report as missing rather than panicking, since it backs a nil-free fallback.
func TestEmptyFS_ServesNothing(t *testing.T) {
for _, name := range []string{".", "index.html", "web/index.html"} {
t.Run(name, func(t *testing.T) {
f, err := emptyFS{}.Open(name)
require.Error(t, err)
assert.ErrorIs(t, err, fs.ErrNotExist)
assert.Nil(t, f)
})
}
}
// TestWebFiles_EmptyFrontendFallsThrough covers the shape routes() builds when fs.Sub refuses:
// the embedded assets must still answer even though the frontend source serves nothing.
func TestWebFiles_EmptyFrontendFallsThrough(t *testing.T) {
w := webFiles{frontend: emptyFS{}, embedded: webassets.FS}
want, err := fs.ReadFile(webassets.FS, "privacy.html")
require.NoError(t, err)
f, err := w.Open("privacy.html")
require.NoError(t, err)
defer f.Close()
got, err := io.ReadAll(f)
require.NoError(t, err)
assert.Equal(t, string(want), string(got))
}
// TestWebFiles_OpenRootIsNotListed keeps the embedded assets from being browsable: they answer
// for their own names only, so a web root that has gone missing reports as missing.
func TestWebFiles_OpenRootIsNotListed(t *testing.T) {
w := webFiles{frontend: os.DirFS(filepath.Join(t.TempDir(), "absent")), embedded: webassets.FS}
f, err := w.Open(".")
require.Error(t, err, "the embedded assets must not answer for the directory itself")
assert.ErrorIs(t, err, fs.ErrNotExist)
if err == nil {
_ = f.Close()
}
// the assets themselves still serve
f, err = w.Open("privacy.html")
require.NoError(t, err)
require.NoError(t, f.Close())
}
// TestWebFiles_OpenInvalidName pins that a name fs rejects reports as missing rather than invalid.
// os.DirFS returns fs.ErrInvalid for these, which http.FileServer renders as 500, so the check has
// to happen before the lookup. A memory filesystem cannot show this: it reports missing either way.
func TestWebFiles_OpenInvalidName(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "privacy.html"), []byte("frontend"), 0o600))
w := webFiles{frontend: os.DirFS(dir), embedded: webassets.FS}
for _, name := range []string{"../escape.html", "/etc/passwd", "a\x00b.html", "./privacy.html"} {
t.Run(name, func(t *testing.T) {
f, err := w.Open(name)
require.Error(t, err)
assert.ErrorIs(t, err, fs.ErrNotExist)
assert.NotErrorIs(t, err, fs.ErrInvalid, "an invalid name must not surface as 500")
if err == nil {
_ = f.Close()
}
})
}
}
// TestWebFiles_OpenUnreadableFrontendFile pins the rule that only a missing file falls through:
// a frontend file that cannot be read must report that, not be masked by the embedded copy.
func TestWebFiles_OpenUnreadableFrontendFile(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores file permissions")
}
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "privacy.html"), []byte("operator's own"), 0o000))
w := webFiles{
frontend: os.DirFS(dir),
embedded: fstest.MapFS{"privacy.html": {Data: []byte("built in")}},
}
f, err := w.Open("privacy.html")
require.Error(t, err, "an unreadable frontend file must not be replaced by the embedded copy")
assert.NotErrorIs(t, err, fs.ErrNotExist, "the error must stay a permission error so it does not render as 404")
assert.ErrorIs(t, err, fs.ErrPermission)
if err == nil {
_ = f.Close()
}
}
func TestTemplatedFS_SubstitutesTheInstanceURL(t *testing.T) {
const placeholder = "host: '" + remarkURLPlaceholder + "'"
source := fstest.MapFS{
"iframe.html": {Data: []byte(placeholder)},
"embed.mjs": {Data: []byte(placeholder)},
"embed.js": {Data: []byte(placeholder)},
"remark.css": {Data: []byte(placeholder)},
"nothing.html": {Data: []byte("no marker here")},
}
tfs := templatedFS{fs: source, remarkURL: "https://remark.example.com"}
tbl := []struct {
name string
want string
}{
{"iframe.html", "host: 'https://remark.example.com'"},
{"embed.mjs", "host: 'https://remark.example.com'"},
{"embed.js", "host: 'https://remark.example.com'"},
// the docker image rewrites html, js and mjs and nothing else, and a stylesheet carrying
// the marker would mean the frontend started templating a file type this does not cover
{"remark.css", placeholder},
{"nothing.html", "no marker here"},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
f, err := tfs.Open(tt.name)
require.NoError(t, err)
defer func() { assert.NoError(t, f.Close()) }()
body, err := io.ReadAll(f)
require.NoError(t, err)
assert.Equal(t, tt.want, string(body))
info, err := f.Stat()
require.NoError(t, err)
assert.Equal(t, int64(len(tt.want)), info.Size(),
"the size has to be the substituted one, or the response is truncated or left hanging")
assert.Equal(t, tt.name, info.Name())
})
}
}
func TestTemplatedFS_PassesErrorsThrough(t *testing.T) {
tfs := templatedFS{fs: fstest.MapFS{}, remarkURL: "https://remark.example.com"}
_, err := tfs.Open("absent.html")
assert.ErrorIs(t, err, fs.ErrNotExist)
}
// TestRest_FileServerFillsInTheInstanceURL covers the reason templatedFS exists: the binary serves
// the frontend build embedded in itself, and nothing else fills the placeholder in for it.
func TestRest_FileServerFillsInTheInstanceURL(t *testing.T) {
frontend := fstest.MapFS{
"embed.mjs": {Data: []byte("host=\"" + remarkURLPlaceholder + "\"")},
"logo.svg": {Data: []byte(remarkURLPlaceholder)},
"plain.html": {Data: []byte("nothing to fill in")},
}
router := routegroup.New(http.NewServeMux())
addFileServer(router, frontend, filepath.Join(t.TempDir(), "absent"), "test-version", "https://remark.example.com")
ts := httptest.NewServer(router)
defer ts.Close()
t.Run("the bundle carries the configured url", func(t *testing.T) {
body, code := get(t, ts.URL+"/web/embed.mjs")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, `host="https://remark.example.com"`, body)
})
t.Run("the legacy js name carries it too", func(t *testing.T) {
body, code := get(t, ts.URL+"/web/embed.js")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, `host="https://remark.example.com"`, body)
})
t.Run("other types are served untouched", func(t *testing.T) {
body, code := get(t, ts.URL+"/web/logo.svg")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, remarkURLPlaceholder, body)
})
t.Run("a file without the marker is unchanged", func(t *testing.T) {
body, code := get(t, ts.URL+"/web/plain.html")
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, "nothing to fill in", body)
})
}
// TestRest_FileServerEtagVariesWithTheInstanceURL covers the case this substitution exists for. An
// operator who notices the widget is addressed to the wrong host corrects REMARK_URL and restarts,
// and the binary and so the version is unchanged. If the validator ignores remarkURL the client
// revalidates, gets 304 and keeps the bundle pointing at the old host. Cache-Control is no-cache,
// so it revalidates every time and never ages out of that state.
func TestRest_FileServerEtagVariesWithTheInstanceURL(t *testing.T) {
frontend := fstest.MapFS{"embed.mjs": {Data: []byte("host=\"" + remarkURLPlaceholder + "\"")}}
etagFor := func(remarkURL string) string {
router := routegroup.New(http.NewServeMux())
addFileServer(router, frontend, filepath.Join(t.TempDir(), "absent"), "test-version", remarkURL)
ts := httptest.NewServer(router)
defer ts.Close()
resp, err := http.Get(ts.URL + "/web/embed.mjs")
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
return resp.Header.Get("Etag")
}
first := etagFor("https://old.example.com")
second := etagFor("https://new.example.com")
require.NotEmpty(t, first, "the file server has to send a validator at all")
assert.NotEqual(t, first, second,
"same version and same path, different instance url: the validator has to change or the "+
"client keeps a bundle addressed to the old host")
}
+21 -6
View File
@@ -10,7 +10,6 @@ import (
"runtime"
"strings"
"github.com/go-chi/render"
log "github.com/go-pkgz/lgr"
"github.com/go-pkgz/rest"
@@ -52,7 +51,7 @@ type errTmplData struct {
// error code is not included in render as it is intended for UI developers and not for the users
func SendErrorHTML(w http.ResponseWriter, r *http.Request, httpStatusCode int, err error, details string, errCode int) {
// MustExecute behaves like template.Execute, but panics if an error occurs.
MustExecute := func(tmpl *template.Template, wr io.Writer, data interface{}) {
MustExecute := func(tmpl *template.Template, wr io.Writer, data any) {
if err = tmpl.Execute(wr, data); err != nil {
panic(err)
}
@@ -67,20 +66,36 @@ func SendErrorHTML(w http.ResponseWriter, r *http.Request, httpStatusCode int, e
tmplstr := MustRead("error_response.html.tmpl")
tmpl := template.Must(template.New("error").Parse(tmplstr))
log.Printf("[WARN] %s", errDetailsMsg(r, httpStatusCode, err, details, errCode))
render.Status(r, httpStatusCode)
msg := bytes.Buffer{}
MustExecute(tmpl, &msg, errTmplData{
Error: err.Error(),
Details: details,
})
render.HTML(w, r, msg.String())
HTMLResponse(w, httpStatusCode, msg.String())
}
// SendErrorJSON makes {error: blah, details: blah, code: 42} json body and responds with error code
func SendErrorJSON(w http.ResponseWriter, r *http.Request, httpStatusCode int, err error, details string, errCode int) {
log.Printf("[WARN] %s", errDetailsMsg(r, httpStatusCode, err, details, errCode))
render.Status(r, httpStatusCode)
render.JSON(w, r, rest.JSON{"error": err.Error(), "details": details, "code": errCode})
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(httpStatusCode)
rest.RenderJSON(w, rest.JSON{"error": err.Error(), "details": details, "code": errCode})
}
// HTMLResponse writes HTML content with the given status code
func HTMLResponse(w http.ResponseWriter, status int, html string) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(status)
_, _ = w.Write([]byte(html))
}
// PlainTextResponse writes plain text content with the given status code
func PlainTextResponse(w http.ResponseWriter, status int, text string) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(status)
_, _ = w.Write([]byte(text))
}
func errDetailsMsg(r *http.Request, httpStatusCode int, err error, details string, errCode int) string {
+76
View File
@@ -0,0 +1,76 @@
package rest
import (
"fmt"
"net/http"
"strings"
)
// StrictImageCSP is the strictest default-deny Content-Security-Policy used both by
// image-serving handlers (/api/v1/img, /api/v1/picture/{user}/{id}) and by the api-wide
// apiCSPMiddleware (covering all /api/v1/* responses — JSON, XML/RSS, images). The name
// keeps the "image" prefix for historical reasons; the policy itself is generic and
// suitable for any non-document API response.
//
// Re-setting the same value inside the image handlers (after the middleware already set
// it) is intentional defense-in-depth: if the middleware ever stops applying (route
// refactor, mount point change), the handlers still emit the header.
const StrictImageCSP = "default-src 'none'; sandbox; frame-ancestors 'none'"
// SafeImgContentType returns the sniffed content type for provided bytes if and only
// if it is in the strict allowlist of image formats safe to serve from a same-origin
// proxy endpoint: image/png, image/jpeg, image/gif, image/webp, image/bmp, image/x-icon.
// Anything else — HTML, XML, SVG, plain text, application/octet-stream, or any future
// image format the stdlib sniffer may learn (e.g. AVIF, HEIC, JXL, TIFF) — is rejected.
// SVG would also be rejected as it sniffs as text/xml or text/plain, never image/svg+xml.
// The previous behavior silently mapped application/octet-stream to image/* and is gone.
func SafeImgContentType(img []byte) (string, error) {
contentType := http.DetectContentType(img)
base, _, _ := strings.Cut(contentType, ";")
base = strings.TrimSpace(base)
switch base {
case "image/png", "image/jpeg", "image/gif", "image/webp", "image/bmp", "image/x-icon":
return base, nil
}
return "", fmt.Errorf("non-image content type %q", contentType)
}
// SetImageDefenseHeaders applies the layered defense headers shared by every response
// from image-serving endpoints (success, 304, or error). Each header survives content-type
// validation regressions, browser sniffing, and top-level navigation:
// - Content-Security-Policy: strict, with sandbox — blocks inline scripts and event handlers
// - X-Content-Type-Options: nosniff — prevents browsers from MIME-overriding the declared type
// - Content-Disposition: inline; filename="image" — frames the response as a file, not a document
//
// CSP is duplicated by apiCSPMiddleware for /api/v1/* — re-setting the same value here is
// harmless and provides defense-in-depth if the middleware is bypassed or moved. The other
// two headers (nosniff, Content-Disposition with filename) are image-specific and not set
// by the middleware.
func SetImageDefenseHeaders(w http.ResponseWriter) {
w.Header().Set("Content-Security-Policy", StrictImageCSP)
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Content-Disposition", `inline; filename="image"`)
}
// EtagMatches reports whether If-None-Match header value contains the given etag.
// Handles the * wildcard, comma-separated etag lists with the W/ weak-validator prefix.
// NOTE: This is intentionally a simple splitter — it does not handle opaque-tags that
// contain commas (allowed by RFC 7232 but never emitted by this codebase, whose etag
// format is `"v2:<base64-url>"` or `"<user>/<xid>"`). If the etag format ever changes
// to include comma-bearing values, revisit this parser.
// Replaces a substring search that could match unrelated entries (e.g. an etag that
// happens to be a prefix of another).
func EtagMatches(header, etag string) bool {
header = strings.TrimSpace(header)
if header == "*" {
return true
}
for tag := range strings.SplitSeq(header, ",") {
tag = strings.TrimSpace(tag)
tag = strings.TrimPrefix(tag, "W/")
if tag == etag {
return true
}
}
return false
}
+103
View File
@@ -0,0 +1,103 @@
package rest
import (
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestEtagMatches covers the strict If-None-Match parser that replaced a substring
// search prone to false positives (etag "abc" being matched inside "fooabc").
func TestEtagMatches(t *testing.T) {
tbl := []struct {
name string
header string
etag string
want bool
}{
{"exact match", `"v2:abc"`, `"v2:abc"`, true},
{"comma-separated, second matches", `"x", "v2:abc"`, `"v2:abc"`, true},
{"weak validator prefix", `W/"v2:abc"`, `"v2:abc"`, true},
{"wildcard matches anything", `*`, `"v2:abc"`, true},
{"leading/trailing whitespace", ` "v2:abc" `, `"v2:abc"`, true},
{"substring not enough", `"v2:abcdef"`, `"v2:abc"`, false},
{"prefix-only mismatch", `"v2:ab"`, `"v2:abc"`, false},
{"pre-fix etag no longer matches v2", `"abc"`, `"v2:abc"`, false},
{"empty header", ``, `"v2:abc"`, false},
{"different etag", `"v2:xyz"`, `"v2:abc"`, false},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, EtagMatches(tt.header, tt.etag))
})
}
}
func TestSetImageDefenseHeaders(t *testing.T) {
w := httptest.NewRecorder()
SetImageDefenseHeaders(w)
assert.Equal(t, StrictImageCSP, w.Header().Get("Content-Security-Policy"))
assert.Equal(t, "nosniff", w.Header().Get("X-Content-Type-Options"))
assert.Equal(t, `inline; filename="image"`, w.Header().Get("Content-Disposition"))
}
// TestSafeImgContentType exercises the strict allowlist. The previous behavior
// (HasPrefix "image/" with an explicit image/svg+xml carve-out) is gone — the
// allowlist is the source of truth, and the explicit svg branch was dead code
// because http.DetectContentType never returns image/svg+xml (real SVG bodies
// sniff as text/xml or text/plain depending on whether they carry an XML decl,
// so they are rejected implicitly by not matching the allowlist).
func TestSafeImgContentType(t *testing.T) {
// minimal magic-byte bodies — verified via http.DetectContentType to produce
// the expected image/* result without needing testdata files for every format
pngMagic := []byte("\x89PNG\r\n\x1a\n")
jpegMagic := []byte("\xff\xd8\xff\xe0\x00\x10JFIF\x00")
gifBytes := []byte("GIF89a")
webpBytes := []byte("RIFF\x00\x00\x00\x00WEBPVP8 ")
bmpBytes := []byte("BM\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")
icoBytes := []byte("\x00\x00\x01\x00\x01\x00")
// SVG with XML decl sniffs as text/xml — rejected because it's not in the allowlist
svgWithXMLDecl := []byte(`<?xml version="1.0"?><svg xmlns="http://www.w3.org/2000/svg" onload="alert(1)"></svg>`)
// SVG without XML decl sniffs as text/plain — also rejected
svgPlain := []byte(`<svg xmlns="http://www.w3.org/2000/svg" width="10"></svg>`)
tbl := []struct {
name string
body []byte
wantCT string
wantErr bool
}{
{name: "nil rejected", body: nil, wantErr: true},
{name: "empty rejected", body: []byte{}, wantErr: true},
{name: "png magic accepted", body: pngMagic, wantCT: "image/png"},
{name: "jpeg magic accepted", body: jpegMagic, wantCT: "image/jpeg"},
{name: "gif accepted", body: gifBytes, wantCT: "image/gif"},
{name: "webp accepted", body: webpBytes, wantCT: "image/webp"},
{name: "bmp accepted", body: bmpBytes, wantCT: "image/bmp"},
{name: "ico accepted", body: icoBytes, wantCT: "image/x-icon"},
{name: "html doc rejected", body: []byte(`<!DOCTYPE html><html></html>`), wantErr: true},
{name: "html fragment rejected", body: []byte(`<body><img></body>`), wantErr: true},
{name: "plain text rejected", body: []byte("hello world"), wantErr: true},
{name: "octet-stream rejected", body: []byte{0x00, 0x01, 0x02, 0x03, 0x04}, wantErr: true},
{name: "svg with xml decl rejected (sniffs as text/xml)", body: svgWithXMLDecl, wantErr: true},
{name: "svg without xml decl rejected (sniffs as text/plain)", body: svgPlain, wantErr: true},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
got, err := SafeImgContentType(tt.body)
if tt.wantErr {
require.Error(t, err)
assert.Empty(t, got)
assert.Contains(t, err.Error(), "non-image content type")
return
}
require.NoError(t, err)
assert.Equal(t, tt.wantCT, got)
// returned type must never carry a charset suffix (the strip code path)
assert.NotContains(t, got, ";")
})
}
}

Some files were not shown because too many files have changed in this diff Show More