* Make backend tests wait on conditions instead of durations
The backend workflow has a long tail of runs that fail once and pass on
a rerun. Every one of them comes down to a test assuming an operation
finishes within some duration rather than waiting for the state it
needs. Three were reproducible and each was reproduced against the old
code before being changed: TestServerAuthHooks minted a token that lived
one second and never tested expiry, so a slow runner turned the first
POST into a 401; TestServerApp_AnonMode saw "connection refused" because
waitForHTTPServerStart returned silently after three seconds and left a
later assertion to fail with something unrelated; TestFsStore_Cleanup
slept 200ms against a 300ms ttl that Cleanup widens to 400ms with its
commit grace, so roughly 100ms of stall collected an image meant to
survive.
Fixed sleeps before asserting on asynchronous work are replaced with
polls on the condition itself, using require.Eventually and
require.EventuallyWithT, and require.Never where the assertion is that
something did not happen. Polling closures assert on the CollectT they
are handed rather than on t, since testify runs them on another
goroutine, and polls that issue HTTP requests stay under the rate limit
on the routes they poll through.
Where a test needs time to have passed, the clock input is pinned
instead: staging ages are stamped with os.Chtimes on both sides of the
cleanup boundary right before each call, which also makes the 100ms
commit grace an exact case rather than something no assertion reaches,
and the RSS tests set store.Comment.Timestamp explicitly rather than
racing the wall clock into the first 100ms of a second so pubDate
matches.
chooseUnusedPort takes a port from the kernel's ephemeral range. Picking
at random out of a fixed 10000-port window let two package binaries,
which go test ./... runs concurrently, land on the same number between
the probe closing and the server binding. The start helpers fail naming
the port they waited on, and the SSL tests wait on the redirect port as
well as the TLS one.
Arbitrary budgets that nothing tests are gone: ten HTTP clients with a
one-second timeout against bolt-backed import and export, the "should
take about 100msec" assertions, and a one-second bound on noticing an
already cancelled context. Shutdown stays bounded at ten seconds so a
hang is still caught.
Two assertions get stronger. TestServerAuthHooks accepted 403 or 401
from a blocked user, an alternative that existed only because the short
token could expire mid-test; it is deterministically 403 now.
TestAdmin_BlockedList asserted two users blocked while one carried the
same 150ms ttl the next step waits to lapse, so the halves raced each
other.
goleak stops reporting the regexp2 clock goroutine, which chroma pulls
in for syntax highlighting and which lives for up to a second after the
last match with a timeout; it ends on its own but a binary finishing
inside that window was reported as leaking, and this suite now finishes
sooner. The ignore for net/http.(*Server).Shutdown goes the other way:
it no longer matches anything, with both packages run fifteen times each
under CPU oversubscription to confirm.
Two gaps the change would otherwise have opened are covered directly
rather than left to the side effects that used to cover them. The
one-second token was the only thing exercising the authenticator's
ClaimsUpd hook on refresh, so TestServerApp_ClaimsUpd now calls the hook
itself and checks admin, blocked, email and restricted-name
impersonation, including the two pass-through cases. Lifting the
open-route limit removed the last incidental exercise of the rate
limiter, so TestRateLimiter drives a burst past the allowance and checks
the refusals and that the limit is per client. Both run without a wall
clock, and both were confirmed to fail when the behaviour they cover is
removed.
Production code is untouched. The two sleeps outside test code, the 429
backoff in cmd/cleanup.go and the submit poll in store/image/image.go,
are left alone: no CI failure implicates them.
Test sleeps drop from 67 to 21, all of them either inside a
testing/synctest bubble or a poll interval. The suite runs in about 22
seconds instead of 46, mostly because
TestPublic_FindCommentsCtrl_ConsistentCount no longer paces a hundred
subtests with an 80ms sleep each to stay under the open route limit. The
300s per-package budget now matches across both workflows, the race_test
target and the documented command, and CLAUDE.md records the convention.
with '#' will be ignored, and an empty message aborts the commit. # #
Date: Sat Aug 22 01:12:31 2026 +0100 # # interactive rebase in progress;
onto 7c312da1 # Last command done (1 command done): # reword deb6cbf1 #
Make backend tests wait on conditions instead of durations # Next
command to do (1 remaining command): # reword 262e6dc2 # Apply go fix
under Go 1.27 # You are currently editing a commit while rebasing branch
'fix/backend-test-flakiness' on '7c312da1'. # # Changes to be committed:
.github/workflows/release.yml # modified: CLAUDE.md # modified: Makefile
modified: backend/_example/memory_store/server/rpc_test.go # modified:
backend/app/cmd/import_test.go # modified:
backend/app/cmd/server_test.go # modified: backend/app/main_test.go #
modified: backend/app/rest/api/admin_test.go # modified:
backend/app/rest/api/middleware_test.go # modified:
backend/app/rest/api/migrator_test.go # modified:
backend/app/rest/api/rest_private_test.go # modified:
backend/app/rest/api/rest_public_test.go # modified:
backend/app/rest/api/rest_test.go # modified:
backend/app/rest/api/rss_test.go # modified:
backend/app/rest/proxy/image_test.go # modified:
backend/app/store/image/fs_store_test.go # modified:
backend/app/store/service/service_test.go # modified:
docs/backlog/api-tests-deadlock-on-macos.md #
* Apply go fix under Go 1.27
Go 1.27 extends go fix with the modernizers, so `go fix ./...` now
rewrites patterns the language has since replaced. Running it across all
three modules produces this: legacy sync/atomic calls on plain integers
become the atomic types (notify.Service.closed, image.Service.term and
submitCount, and several test counters), reverse index loops become
slices.Backward, a Split-then-index becomes strings.Cut, counted loops
become range over an int, and interface{} becomes any in the e2e suite.
The example module needed no changes. The e2e module is behind a build
tag, so it only matches with `go fix -tags e2e ./...`.
One knock-on: prealloc can see the bound of a loop once it is written as
range over an int, so the slice it feeds is now preallocated.
with '#' will be ignored, and an empty message aborts the commit. # #
Date: Sat Aug 22 01:32:09 2026 +0100 # # interactive rebase in progress;
onto 7c312da1 # Last commands done (2 commands done): # reword deb6cbf1
262e6dc2 # Apply go fix under Go 1.27 # No commands remaining. # You are
currently editing a commit while rebasing branch
'fix/backend-test-flakiness' on '7c312da1'. # # Changes to be committed:
backend/app/migrator/native.go # modified: backend/app/notify/notify.go
backend/app/rest/api/rest_private_test.go # modified:
backend/app/store/comment.go # modified:
backend/app/store/image/image.go # modified:
backend/app/store/service/service_test.go # modified:
backend/app/store/service/title_test.go # modified: e2e/e2e_test.go #
modified: e2e/widgets_test.go #
notify/email.go accumulated multi-recipient errors with
multierror.Append(fmt.Errorf(...)) instead of
multierror.Append(result, ...), so the accumulator was overwritten each
iteration and only the last failing recipient's error survived; earlier
failures were silently dropped. The telegram notifier did it correctly.
Replace hashicorp/go-multierror with the stdlib errors.Join everywhere
it was used (notify/email.go, notify/telegram.go, rest/api/rest_private.go,
store/service/service.go, store/image/image.go and store/engine/bolt.go),
which fixes the bug and drops the direct dependency. It stays indirect
because go-pkgz/lcw/v2 still imports it. A regression test in
email_test.go now sends two failing recipients and asserts both errors
are reported.
* fix(security): reject non-image content-types in image proxy and /picture/ to prevent stored XSS
The /api/v1/img proxy and /api/v1/picture/{user}/{id} endpoints emitted
http.DetectContentType on the served bytes as the response Content-Type. A
controlled upstream serving Content-Type: image/png with an HTML body passed
the upstream check (only the response header was inspected, not the body),
and the body bytes then sniffed back to text/html — so the proxy served the
attacker's HTML from the remark42 origin. Browsers honoured the declared
text/html and executed the response as a document with access to cookies and
CSRF tokens. Affected from v1.6.0 (April 2020) through v1.15.0; verified live
via published docker images.
Layered defense applied to both handlers:
- rest.SafeImgContentType (in backend/app/rest/) validates sniffed content
against a strict allowlist: image/png, image/jpeg, image/gif, image/webp,
image/bmp, image/x-icon. Anything else (HTML, XML, SVG, plain text,
octet-stream, or any future image type the stdlib sniffer may learn) is
rejected with no body echo. SVG is implicitly excluded — it sniffs as
text/xml or text/plain, never image/svg+xml, and SVG can execute scripts
when navigated to top-level. The previous octet-stream → image/* fallback
is gone.
- Per-endpoint Content-Security-Policy override sets
"default-src 'none'; sandbox; frame-ancestors 'none'" on every response
(success, 304, or error). Sandbox neuters scripts even if Content-Type
ever regresses. The same policy is also applied to all /api/v1/* via
apiCSPMiddleware as defense-in-depth.
- Content-Disposition: inline; filename="image" frames the response as a
file rather than a renderable document.
- /picture/ rejection paths set Cache-Control: no-store so 4xx responses
are never cached.
The defense headers and the strict ETag matcher are extracted as
rest.SetImageDefenseHeaders and rest.EtagMatches in the shared rest package
(consumed by both proxy/image and api/rest_public — no package cycle).
The /api/v1/img path additionally bumps the ETag to a versioned `"v2:..."`
so revalidating clients (top-level navigation, Ctrl+R, intermediaries) get
a fresh 200 instead of a 304 against poisoned pre-fix cached HTML.
DELIBERATE TRADEOFF: Cache-Control on /api/v1/img success responses remains
max-age=2592000 (30 days), unchanged from before. An aggressive "force
revalidate on every reuse" policy was prototyped during review but reverted
because the perf cost (a server round-trip on every image view, even with
304 saving the body bytes) outweighed the corner-case mitigation. The
realistic exposure of cache carryover is narrow: cache carryover only
affects users who navigated top-level to an attacker URL pre-fix and still
have it in their local cache — the normal <img> embed path cached text/html
but never executed it. Local browser caches that hold pre-fix bytes
continue to serve them until their 30-day TTL expires or are evicted under
memory pressure. The ETag bump reaches all clients that DO revalidate
during the cached lifetime (Ctrl+R, intermediaries, post-expiry use); for
the rest, exposure self-limits via cache expiry. Operators running a
CDN/edge cache in front of remark42 should purge /api/v1/img after deploy.
The /api/v1/img handler short-circuits on a matching current-version
If-None-Match before any store Load or upstream fetch, returning a bodyless
304 with the defense headers set. Safe because the 304 carries no body and
the client's cached bytes came from a prior validated 200; an attacker
fabricating an etag value can only short-circuit fetches for URLs they
themselves crafted. This avoids upstream DoS amplification when clients
revalidate on hot comment pages.
The /api/v1/img route was moved from the "open routes" group (which uses
middleware.NoCache, stripping If-None-Match from incoming requests) to the
"open routes, cached" group alongside /picture/ and /qr/telegram so the
304 revalidation path is no longer broken upstream of the handler.
The /picture/{user}/{id} endpoint does not need the v2 etag prefix. Upload
validates input format via readAndValidateImage and the serve path
re-validates the stored bytes via rest.SafeImgContentType. Bytes within
the resize dimension limits are preserved verbatim, so the browser defense
relies on the response headers (validated Content-Type + nosniff + strict
CSP + Content-Disposition: inline), not on byte normalization.
Global CSP: font-src data: → font-src 'none'. Audit confirmed no @font-face,
no base64 fonts, no icon-font library in the bundle. Drops an unnecessary
attack surface; no behavioural change.
Tests: TestImage_ContentTypeHandling table-tests a real PNG and attack
shapes (HTML claimed as image/png, image/jpeg, image/gif, image/svg+xml,
image/webp; svg with onload; html fragment; polyglot PNG+HTML), proving
the defense holds across arbitrary upstream Content-Type variation.
Polyglot case is intentionally served as image/png — the browser cannot
execute the trailing HTML when the response type is image/png with nosniff.
TestImage_ContentTypeHandling_CacheHit exercises the cache-hit branch with
attacker bytes preloaded into the store. TestImage_PerRequestRevalidation
alternates upstream PNG/HTML across four proxy calls to prove no trust
accumulates between requests. TestImage_RoutesUsingCachedImage asserts
cache-poisoning is caught at serve time. TestImage_EtagVersioned asserts
the v2 prefix invalidates pre-fix etags AND that the revalidation 304
triggers no store Load. TestImage_RevalidationSkipsIO proves the
short-circuit works even with no upstream reachable. TestSafeImgContentType
covers the allowlist directly. TestRest_LoadPictureDefenseHeaders and
TestRest_LoadPictureRejectsNonImage exercise the /picture/ endpoint.
TestRest_apiCSP covers the strict CSP middleware on JSON API + RSS routes;
TestRest_securityHeaders confirms /web/ HTML pages keep the global CSP.
Verified end-to-end against the dev docker image: the original demo URL
(arbitrary HTML claimed as image/png) now returns 415 application/json with
CSP/nosniff/Content-Disposition set, no XSS in the browser.
* fix(security): set Cache-Control: no-store on image-proxy error paths, sync stale route comment
Addresses two review comments on #2067:
1. Cache-Control: max-age=2592000 and Etag were set before the
load/download/validation block, so 404/400/415 error responses inherited
the 30-day cache TTL and the versioned etag — a transient failure (or an
intentionally triggered 415) would be pinned in browser/intermediary
caches for that TTL, keeping users locked out even after the underlying
cause was resolved. Now: etag is computed but not set as a header until
after validation succeeds; error paths route through sendImageProxyError
which sets Cache-Control: no-store and never sets Etag. The 304
short-circuit still sets both because that path serves the same validated
content the client already has cached.
2. The comment at rest.go:282 still described the prototyped
no-cache/must-revalidate Cache-Control policy that was reverted before
the PR landed. Updated to match the actual 30-day max-age behavior.
Tests: TestImage_ContentTypeHandling now asserts reject paths carry
Cache-Control: no-store and have no Etag header, and accept paths carry
the max-age=2592000 + v2: etag.
readAndValidateImage caps the byte size of incoming images but the resize()
helper that follows still called image.Decode unconditionally, allocating
pixel memory proportional to the *declared* image dimensions. A ~100 KB
compressed PNG or GIF that declares 65535x65535 px forces image.Decode to
allocate ~17 GB of raster, OOMing the service on a single comment upload
(or on the proxy's CacheExternal path when caching a malicious upstream).
Hardening:
- maxImagePixels = 16 MP constant. Covers any realistic image (~4096x4096)
while bounding peak allocation.
- resize() now runs image.DecodeConfig first (cheap, no pixel allocation)
to read declared width/height before any full decode.
- Multiplication of width × height uses int64 to defeat 32-bit overflow
(GOARCH=386, 32-bit arm): on those targets, int(cfg.Width)*int(cfg.Height)
could wrap below maxImagePixels and bypass the cap. GIF's 16-bit logical
screen and JPEG's 16-bit SOF dimensions both reach this if int-multiplied.
- Bytes exceeding the cap, or non-image input that fails DecodeConfig,
return nil. prepareImage propagates the rejection as a clear error
instead of storing the malformed/oversized data verbatim.
- The no-resize-needed path returns the validated original bytes verbatim
so animated GIFs round-trip without being flattened to a single frame.
The DecodeConfig precheck applies even when MaxWidth/MaxHeight are 0
(resize disabled) — the dimension cap is unconditional defense-in-depth.
Two adjacent fixes surfaced by the new resize contract:
1. readAndValidateImage previously did `data[:512]` without a bounds check,
panicking on any body shorter than 512 bytes. Now bounded with min().
2. image/webp was listed as an allowed format but no WebP decoder was
registered, so DecodeConfig would refuse legitimate WebP uploads. Added
`_ "golang.org/x/image/webp"` (already in go.mod via x/image/draw) so
the registered decoders match the allowlist.
Tests:
- TestService_resizeRejectsDecompressionBomb builds a 14-byte GIF87a header
declaring 65535x65535 and asserts resize() refuses it both at the unit
level and through SaveWithID end-to-end (no store write).
- TestService_SaveWithIDShortPayload regression-tests the short-body panic.
- TestService_SaveWithIDWebP regression-tests WebP round-trip through
prepareImage with the new DecodeConfig requirement.
- TestService_resize subtests updated to assert non-image bytes are now
refused (previously the helper fell back to returning the raw bytes
verbatim, letting malformed content reach the store).
Apply go fix ./... analysers (Go 1.26) across backend and examples:
- interface{} → any (type alias, no behaviour change)
- for i := 0; i < N; i++ → for range N / for i := range N
- slices.Contains / slices.ContainsFunc replacing manual loops
- strings.SplitSeq replacing strings.Split in range (avoids allocation)
- strings.CutPrefix replacing HasPrefix+TrimPrefix
- min() replacing manual if/else
- fmt.Appendf replacing []byte(fmt.Sprintf(...))
- strings.Builder replacing string += concatenation
- wg.Go(func(){}) replacing wg.Add(1)/go/wg.Done() pattern
- removed redundant ii := i loop variable copies (unnecessary since Go 1.22)
omitempty on struct-typed JSON fields: go fix removed omitempty from
struct-typed fields (time.Time, PostInfo, UserDetailEntry) because
encoding/json's omitempty never applied to struct types — it was always
a no-op. Kept as bare tags (no omitzero replacement) to preserve the
existing serialisation behaviour.
Replace strings.Split(RemoteAddr, ":") with net.SplitHostPort for correct
IPv6 address extraction in vote deduplication and comment IP tracking.
Harden image proxy: add SSRF-safe transport blocking private/reserved IPs
at connection time with DNS rebinding protection, sanitize error messages
to prevent information leakage, add response size limit via io.LimitReader.
Fix shadowed error variables in BlockedUsers, SetTitle, and Delete methods.
Exclude gosec taint analysis false positives at linter config level.
When EditDuration is zero or negative, cleanupTTL becomes zero,
causing time.After(0) to fire immediately in a tight loop.
Block on ctx.Done() instead when edit duration is disabled. Fixes#1991
Previously, images were deleted only from comments deleted
before EditDuration expiration. After this change, any deletion
of the comment deletes images if they are not used elsewhere
in comments under the same page.
Previously, proxied and local images were checked for presence in the
storage before previewing or posting the comment. That logic resulted in
an inability to post with an image when a proxy for images is enabled,
as proxied images are not downloaded to disk before the first time
someone loads them, which could only happen after the user either
previews or posts the message.
After this change, preview and post only checks the local images'
presence and ignore the proxied ones.
Also:
- make commitTTL equal to EditDuration,
so that image is committed to permanent
storage after comment can no longer be edited
- move cleanupTTL to Cleanup function,
as it's not used elsewhere in the code
- add variables to some tests sleeps, so that
instead of being magic numbers they would
rely on timers of structures they suppose
to wait for
That function returns an error in a never
expected condition, and that error would be
logged message on the caller side:
none of the callers handles it.
That change hides that error from the caller
so that function would have a signature that
better fit what it does and how it behaves.
So that image is committed to permanent
storage after comment can no longer be edited.
Also, move cleanupTTL to Cleanup function,
as it's not used elsewhere in the code.
This allows having separate values of TTL for Commit and Cleanup
and moving them apart in time, also clarifying their connection
to EditTime which was previously outside of the package level.
@paskal
* sort imports, add missing copyright
* regenerate engine mock
* make all image.Store interface functions public
* go mod tidy
* make image.Store.Load return []byte instead of io.ReadCloser
* separate memory_store example RPC server to multiple files by handlers groups
* fix potential race on close
* move validate inside
* demote commit and cleanup in image.Store to non-exposed functions.
* replace immediate image commit with delayed via Submit
* minor: remove error logging, rename tests
* minor: err wrapping, comments wording
* clarify FileSystem.Save code
* attempt to fix#584 by making submitted image commits on a half of TTL
Co-authored-by: Dmitry Verkhoturov <paskal.07@gmail.com>
* Add bolt image store support
* Fix lint errors
* Use separate bolt buckets for staging and commited
* Comment bolt store public entities
* Fix spelling
Dmitry VerkhoturovandGitHub
vdimirandUmputun
Artur MullakhmetovandUmputun
Anton PanferovandUmputun