Compare commits

...
110 Commits
Author SHA1 Message Date
Dmitry VerkhoturovandUmputun e62b3c830d fix(trusted-proxy): warn on catch-all, cover more cases, trim wording
Follow-up to the review notes on #2116:
- warn at startup when --trusted-proxy contains a catch-all (0.0.0.0/0 or ::/0),
  which trusts every peer and re-opens the bypass - mirrors the unset-case warning
- realIPMiddleware tests: cover the unparseable-peer and trusted-peer-without-header
  branches, and make the observed values per-call so subtests don't share closure locals
- trim the flag description and shorten the startup warning to the terse [WARN] style
2026-07-09 15:05:05 -05:00
Dmitry VerkhoturovandUmputun b1502801fa fix: add --trusted-proxy to gate client-IP forwarding headers
Rate limiting and (with --votes-ip) vote de-duplication key on the client IP,
recovered from forwarding headers (X-Real-IP / X-Forwarded-For / CF-Connecting-IP)
when behind a reverse proxy. Those headers were accepted from any client, so a
caller could set them to change its apparent IP.

Add --trusted-proxy / TRUSTED_PROXY (comma-separated CIDR/IP): forwarding headers
are honored only when the direct peer is a trusted proxy; other peers keep their
real socket address. Unset preserves the previous trust-all behavior (with a
startup warning) so existing deployments keep working on upgrade.

Docs: a 'Trusted proxies and client IP' section with per-topology guidance, plus a
note in the nginx manual.
2026-07-05 17:47:19 -05:00
Dmitry VerkhoturovandUmputun 2e3a680ca4 fix(deleteme): surface real avatar-store errors, tolerate only not-found
Bumps go-pkgz/auth to v2.1.5, which adds avatar.ErrNotFound. deleteMeRequestCtrl's
avatar removal was best-effort (log and continue on any error) because before the
sentinel there was no portable way to tell an already-removed avatar from a genuine
failure. It now tolerates only errors.Is(err, avatar.ErrNotFound) - keeping the
repeated-request idempotency - and surfaces any other store failure as 500.
2026-07-05 17:28:01 -05:00
Dmitry VerkhoturovandUmputun d8b7f7530c fix: remove user avatar on deleteme request
The delete_me token built by deleteMeCtrl omitted the user's Picture, so the
avatar-removal branch in deleteMeRequestCtrl never ran for real requests and
avatars survived account deletion. Carry Picture in the token so the stored
avatar is removed when the request is processed.

Make the removal best-effort: the avatar stores report an already-missing
avatar as an error with no distinguishable sentinel, and the user's data is
already deleted at that point, so a missing avatar (e.g. a repeated request)
no longer fails the whole deletion with a 400.

Only remove a well-formed avatar id ("<hash>.image") so a malformed picture
can't make a filesystem-backed store target an unexpected path.
2026-07-03 15:40:31 -05:00
Dmitry VerkhoturovandUmputun b33025a76f feat(api): adopt enforcing rest.Timeout, drop local cooperative timeout
go-pkgz/rest v1.22.0 ships an enforcing Timeout middleware (net/http.TimeoutHandler
style): it runs the handler with a deadline and returns 504 at the deadline even if
the handler ignores the context - unlike the local cooperative timeout, which only
cancelled the context and never actually stopped a stuck handler.

Replace the local timeout with rest.Timeout on every route with a bounded response.
The streaming and long-polling routes are deliberately left without it, since the
enforcing timeout buffers the whole response in memory and aborts at the deadline:
- GET /api/v1/userdata and GET /api/v1/admin/export stream gzipped exports
- GET /api/v1/admin/wait long-polls for up to 15m
- POST /api/v1/admin/import[/form] and /remap ingest large uploads

Delete the local timeout middleware and its test; the enforcing behaviour is covered
by go-pkgz/rest. TestRouteTimeout locks the enforcing-vs-exempt contract in this build.
2026-07-03 15:40:10 -05:00
Dmitry VerkhoturovandUmputun c48254a994 chore(deps): bump go-pkgz/rest to v1.22.0, drop local CORS Vary workaround
v1.22.0 includes the preflight Vary fix (https://github.com/go-pkgz/rest/pull/44):
rest.CORS now adds Vary: Access-Control-Request-Method and
Access-Control-Request-Headers on preflight itself, making the local wrapper
that added them redundant. corsMiddleware now returns rest.CORS directly;
TestCorsMiddleware still asserts those preflight Vary headers, now supplied
upstream.

Also tidies the _example/memory_store module for the new version.
2026-07-03 15:40:10 -05:00
Dmitry VerkhoturovandUmputun 3fc5d6b970 fix: make user deletion idempotent for users without comments
deleteUser now succeeds for a user who has no comments (e.g. one who only logged
in) instead of failing on the missing per-user bucket. In hard mode the per-user
bucket is deleted, tolerating bbolt's ErrBucketNotFound so a bucket left behind by
an earlier partial removal is still removed; the comment-deletion failure path now
wraps the actual error.

Because the engine cannot distinguish a valid login-only user from a never-existed
one, deletion is idempotent: /admin/deleteme returns 200 for an unknown (but validly
signed) token rather than 400. The deleteme test is updated to this contract, engine
tests cover hard and soft deletion of login-only and unknown users, and the API docs
note the idempotent behaviour.
2026-07-01 15:05:27 -05:00
Fredrik AppelrosandUmputun 380aa3c828 Allow deleteUser to be called on users with no comments 2026-07-01 15:05:27 -05:00
Fredrik AppelrosandUmputun b6bc8ba675 Fix error handling in deleteUser function to return the correct error when deleting a user bucket. 2026-07-01 15:05:27 -05:00
Dmitry VerkhoturovandGitHub c5121fd402 refactor(api): replace go-chi/chi router with go-pkgz/routegroup (#2103)
Migrate the REST router off go-chi/chi onto go-pkgz/routegroup (backed by the
stdlib http.ServeMux), removing the last use of go-chi from the backend:

- rest.go routes() builds the tree with routegroup (Mount/Group/Route/With) and
  net/http method+path patterns instead of chi's Get/Post/Route/Mount helpers
- chi.URLParam(...) -> r.PathValue(...) in the admin, public and private handlers
- rest_public_test.go loadPictureCtrl test uses routegroup + http.ServeMux
- rest_test.go: add TestRest_FileServerStaticAssets (bare /web -> /web/ redirect,
  cache headers, 404, directory-listing block) and update the path-traversal test
  for ServeMux normalising a literal ".." (encoded traversal is still rejected
  by the handler)
- drop go-chi/chi from go.mod, go.sum and vendor; update the CLAUDE.md reference
2026-07-01 15:04:34 -05:00
Dmitry VerkhoturovandUmputun fff9127976 fix: correct no-providers message grammar, translate it, and cover both branches
Reword "May be" to "Maybe" in the auth.no-providers message and run
translation:generate to register the key in every locale dictionary, then
replace the English placeholders with proper translations for each locale.
Add a test asserting the error is hidden when providers are configured.
2026-06-30 18:23:21 -05:00
Eugene OrlovandUmputun 406df022ba fix: ui error when no auth providers configured 2026-06-30 18:23:21 -05:00
Dmitry VerkhoturovandUmputun 6840a46ac9 Replace go-chi/cors with go-pkgz/rest CORS
Swap the go-chi/cors middleware for rest.CORS (already a dependency),
removing the go-chi/cors module entirely. Behaviour-preserving:
- with AllowedOrigins "*" and credentials enabled, both reflect the request
  Origin into Access-Control-Allow-Origin (a literal "*" is invalid with
  credentials)
- preflight responses also vary on Access-Control-Request-Method/-Headers, not
  just Origin, matching go-chi/cors so caches don't reuse a preflight response
  across different requests

Extract the config into corsMiddleware() in middleware.go and add
TestCorsMiddleware covering origin reflection, credentials, preflight
methods/headers/max-age/Vary, and the no-Origin case. go-chi/cors dropped
from go.mod; the chi router stays until the router migration. go test -race,
vet, golangci-lint, govulncheck clean.
2026-06-30 17:40:03 -05:00
Dmitry VerkhoturovandUmputun 6a50ffd88a Use stdlib http.ServeMux instead of chi in cleanup_test
The cleanup command test builds a self-contained mock HTTP server with
only static routes (and {id} patterns read via r.URL.Path, not URLParam),
so chi.NewRouter is unnecessary — http.NewServeMux (Go 1.22 routing) covers
it. Independent of the main router; drops the chi import from app/cmd.

go test -race and golangci-lint clean.
2026-06-30 17:39:37 -05:00
Dmitry VerkhoturovandUmputun f7dbdae26c Consolidate request middlewares into middleware.go
Pure relocation, no behaviour change: gather all request-scoped middlewares
and their tests into dedicated files instead of scattering them across
rest.go and ssl.go.

  funcs -> app/rest/api/middleware.go:
    timeout (from ssl.go); rejectAnonUser, matchSiteID, cacheControl,
    apiCSPMiddleware, securityHeadersMiddleware, subscribersOnly,
    validEmailAuth, rateLimiter (from rest.go)
  tests -> app/rest/api/middleware_test.go:
    TestTimeout (from ssl_test.go); TestRest_rejectAnonUser,
    TestRest_cacheControl, TestRest_apiCSP, TestRest_securityHeaders,
    TestRest_subscribersOnly, Test_validEmailAuth, TestRest_matchSiteID
    (from rest_test.go)

go test -race, vet, golangci-lint and govulncheck clean; example builds.
2026-06-30 17:15:39 -05:00
Dmitry VerkhoturovandUmputun f4b236c66a Replace chi middleware.Timeout with the timeout helper, drop chi/middleware
middleware.Timeout was the last use of go-chi/chi/v5/middleware (RealIP, the
other user, landed in #2099). Swap it for the local timeout helper (context
deadline + 504 on deadline, matching chi exactly; covered by TestTimeout),
which removes the go-chi/chi/v5/middleware package from the vendor tree.

The go-chi/chi module stays in go.mod — the router (chi.NewRouter etc.) still
uses it, so go.mod only shrinks after the router migration. Build, vet, race
tests and golangci-lint clean.
2026-06-30 17:15:04 -05:00
Dmitry VerkhoturovandUmputun 17365f4304 Replace chi middleware.RealIP with rest.RealIP on the main router
Behaviour-preserving swap: rest.RealIP sets r.RemoteAddr from
X-Real-IP / X-Forwarded-For like chi's middleware.RealIP, removing chi from
the RealIP path without changing the trust model (GHSA-56x6-q882-mf27 stays
present, to be fixed separately). chi/middleware stays imported for Timeout;
whichever of this PR and the Timeout PR (#2097) merges last drops the import.
Drop-in to a tested rest middleware; covered by existing api router tests.
2026-06-30 16:38:12 -05:00
Dmitry VerkhoturovandUmputun 0b6eea68a1 Replace chi middleware.NoCache with rest.NoCache on the main router
go-pkgz/rest.NoCache is borrowed from chi's middleware.NoCache and behaves
identically: same no-cache response headers and the same stripping of
conditional request headers (If-None-Match etc.), which the image-proxy
etag logic relies on. Drop-in swap, chi router left in place.

chi/middleware stays imported for RealIP and Timeout. Build, vet, race
tests and golangci-lint clean.
2026-06-30 16:27:01 -05:00
Dmitry VerkhoturovandUmputun b19e6269c1 Replace chi middleware.Throttle with rest.Throttle on the main router
Drop-in swap of the global concurrency limiter (go-pkgz/rest.Throttle has
the same signature and semantics as chi's middleware.Throttle), with the
chi router left in place. First of the per-middleware swaps that chip away
at go-chi/chi/v5/middleware before the router itself is migrated.

chi/middleware is still imported for RealIP/Timeout/NoCache; build, vet,
race tests and golangci-lint clean.
2026-06-30 16:26:28 -05:00
Dmitry VerkhoturovandUmputun bb6d1450f1 Migrate ssl.go TLS routers from chi to routegroup
First step of the go-chi -> go-pkgz/routegroup migration. The HTTP->HTTPS
redirect and ACME http-01 challenge routers are small, self-contained
http.Handlers separate from the main API router, so they move cleanly:

- chi.NewRouter() -> routegroup.New(http.NewServeMux())
- middleware.Throttle -> rest.Throttle (same concurrency-limit semantics)
- middleware.Timeout -> local timeout helper (context deadline, mirrors chi)
- drop middleware.RealIP: these routers do redirect/challenge only, with no
  per-IP logic, so the spoofable header trust is simply removed here
- return http.Handler instead of chi.Router (callers already take http.Handler)

chi stays a dependency (still used by the main API router); this only removes
its use from ssl.go. go test -race, vet, golangci-lint and govulncheck clean.
2026-06-30 16:26:03 -05:00
dependabot[bot]andUmputun 8318f89dde chore(deps): bump the github-actions-updates group across 1 directory with 4 updates
Bumps the github-actions-updates group with 4 updates in the / directory: [actions/checkout](https://github.com/actions/checkout), [actions/cache](https://github.com/actions/cache), [pnpm/action-setup](https://github.com/pnpm/action-setup) and [codecov/codecov-action](https://github.com/codecov/codecov-action).


Updates `actions/checkout` from 6 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

Updates `actions/cache` from 5 to 6
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v5...v6)

Updates `pnpm/action-setup` from 6.0.4 to 6.0.9
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v6.0.4...v6.0.9)

Updates `codecov/codecov-action` from 6 to 7
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 6.0.9
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-updates
- dependency-name: codecov/codecov-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-30 15:57:05 -05:00
Dmitry VerkhoturovandUmputun 3e18681ca7 Sanitize comment text in email notifications (GHSA-74pc-3r2m-ppx3)
Email notification templates rendered the comment HTML via text/template,
so the store-level UGC sanitizer's permitted <a> and <img> tags reached
the email body verbatim. An authenticated user could plant phishing links
and remote tracking pixels in notification emails sent from the legitimate
remark42 address.

Switch notify to html/template (auto-escaping every non-HTML field) and
add a stricter email-only bluemonday policy that drops <a> and <img> while
keeping basic text formatting; the sanitized comment HTML is passed as
template.HTML. Add regression tests asserting links and images are stripped
while anchor text and formatting survive.
2026-06-30 15:56:36 -05:00
UmputunandGitHub 11d8a978a2 Merge pull request #2094 from umputun/fix/eleventy-outputpath-guard
Guard against falsy outputPath in eleventy htmlmin transform
2026-06-30 15:56:08 -05:00
Dmitry Verkhoturov d7fe27cb97 Guard against falsy outputPath in eleventy htmlmin transform
Eleventy passes a falsy outputPath to transforms for templates rendered
without a written file (e.g. permalink: false); calling .endsWith on it
would throw. Skip minification in that case instead. Pre-existing latent
issue surfaced by Copilot review on #2091.
2026-06-30 20:59:57 +01:00
UmputunandGitHub 7fee12a978 Merge pull request #2091 from umputun/deps/update-frontend
Update frontend and site dependencies to latest, bump pnpm to 10, clear audit alerts
2026-06-30 14:23:48 -05:00
Dmitry Verkhoturov fc3d93c398 Add frontend/CLAUDE.md documenting dependency-update gotchas
Captures what isn't obvious from the diff alone: the ten places a
node/pnpm version is pinned and must move together (including .nvmrc,
which CI never reads and is how the node-16 drift in this PR's first
push went unnoticed), the pnpm-10 layout pins, the msw 1->2 migration,
the deliberately held-back majors, and the abandoned html-minifier
replacement. Written so the next dependency bump doesn't repeat the
same gaps.
2026-06-30 20:11:53 +01:00
Dmitry Verkhoturov 4baf0f4260 Close remaining node/pnpm version drift after the pnpm 10 bump
- frontend/.nvmrc was still pinned to 16, left behind by the node 16->20
  bump everywhere else (Dockerfile, CI matrices). A contributor running
  'nvm use' in frontend/ would land on node 16, which cannot even run
  pnpm 10 (requires node >=18) -- CI never reads .nvmrc, so this was
  invisible to every check.
- pnpm/action-setup 'version: 10' floated the patch release in CI,
  inconsistent with the exact 10.10.0 pin now used in Dockerfile,
  Dockerfile.e2e and packageManager. Pinned all ten occurrences across
  ci-frontend.yml, ci-frontend-api.yml and release.yml to 10.10.0.
2026-06-30 20:10:43 +01:00
Dmitry Verkhoturov b72030114c Address Copilot review feedback on #2091
- Pin pnpm to the exact version (10.10.0) when installing it in the
  production Dockerfile, matching packageManager and Dockerfile.e2e,
  instead of a floating major that can drift the lockfile behaviour.
- Fix mockEndpoint's array header handling in the api test utility:
  append each value instead of joining with a comma, which is how
  multi-value headers (e.g. set-cookie) are actually represented.
- Update apps/remark42's engines to node >=18 / pnpm >=10, matching
  the pnpm 10 requirement instead of the stale node 16 / pnpm 8 range.
2026-06-30 20:05:42 +01:00
Dmitry Verkhoturov 8626e4181f Fix CI for node 20 / pnpm 10: e2e Playwright image and jest arg forwarding
- frontend/Dockerfile.e2e: bump base image to mcr.microsoft.com/playwright:
  v1.61.1-noble to match the Playwright 1.61.1 npm bump (browser revision
  mismatch was failing all e2e specs), and corepack pnpm@8 -> pnpm@10.10.0 to
  match the pnpm bump and the v9 lockfile.
- release.yml validate: pnpm 10 forwards 'test -- --runInBand' literally as
  'jest -- --runInBand' (treated as a path pattern, 0 tests). Drop the extra
  separator: 'pnpm test --runInBand'.
2026-06-30 19:53:10 +01:00
Dmitry Verkhoturov d274724c08 Update site dependencies and clear all yarn audit alerts
yarn audit: 0 vulnerabilities (was 52 findings). Site builds via eleventy +
tailwind on node 20.

Direct bumps: markdown-it 14.2, cross-env 10, date-fns 4.4, prettier 3.9,
@tailwindcss/typography 0.5.20, @11ty/eleventy-plugin-syntaxhighlight 5.0.2.
Replaced abandoned html-minifier (unpatched ReDoS, no fix released) with the
maintained html-minifier-terser fork; the .eleventy.js htmlmin transform is now
async. Transitive vulns patched via yarn resolutions. js-yaml resolves to 3.15.0
(3.x backport) which keeps gray-matter working.

Held: tailwindcss 3.4 (tailwind 4 is a config rewrite) and @11ty/eleventy 2
(eleventy 3 is an ESM migration) - both invasive.

Build output verified against a clean master build: every HTML page differs only
by the build-time ?v= cache-bust query; style.css differs only by an equivalent
refactor of @tailwindcss/typography's prose kbd-shadow variables (same rendered
result). Functionally identical.
2026-06-30 19:47:25 +01:00
Dmitry Verkhoturov f5ccfaa0e1 Update frontend dependencies to latest, bump pnpm to 10, clear all npm audit alerts
pnpm 8.15.9 -> 10.10.0 (packageManager + lockfile regenerated to v9). Frontend
CI (ci-frontend.yml, ci-frontend-api.yml, release.yml) and the production
Dockerfile bumped from node 16 + pnpm 8 to node 20 + pnpm 10 (pnpm 10 requires
node 18+). pnpm audit: no known vulnerabilities (was 63 alerts).

packages/api: bumped to latest including the major test stack - vitest 4, jsdom
29, @vitest/coverage-v8 4, @typescript-eslint 8.62, typescript 5.9, prettier
3.9, @types/node 26, and msw 1 -> 2. Migrated tests/test-utils.ts to the msw 2
http/HttpResponse API (capturing a compatible request shape) and made test base
URLs absolute so node 20's native fetch is intercepted; added the jsdom base
URL. type-check:api, lint:api and coverage:api (45 tests) all pass.

apps/remark42: safe in-major bumps (webpack 5.108, postcss, mini-css-extract,
html-webpack-plugin, ts-loader, webpack-dev-server 5.2.5, core-js, clsx 2,
lodash-es 4.18, dotenv 17, @types/*). Transitive vulns patched via
pnpm.overrides. type-check, lint, build, jest coverage (299 tests) and
translations all pass.

pnpm 10's stricter layout required a few pins to keep the app's preact-compat
setup compiling: preact 10.6.2 (override), react-intl 6.0.5 and
@testing-library/preact 3.2.2 (newer types break the build), tsconfig paths for
preact, @types/minimatch 5.1.2 (6.x is an empty stub) and cheerio 1.0.0-rc.12
(1.2 is ESM and breaks jest 28). Held: react/react-dom (preact compat alias),
babel 7, eslint 8, stylelint 14, jest 28, typescript 4.7 (app),
redux/react-redux - majors that change the bundle or need a config migration.

Build output verified against a clean master build: apps/remark42 output is
functionally identical (the only diffs are webpack module-id numbering and
css-module class tokens from the webpack/css-loader bump; all HTML, CSS values
and translations byte-identical).
2026-06-30 19:47:25 +01:00
UmputunandGitHub c8832e708c Merge pull request #2088 from umputun/deps/update-backend
Update backend dependencies to latest
2026-06-30 12:41:13 -05:00
Dmitry Verkhoturov 07c7926453 Update backend dependencies to latest
Update all Go modules in backend/ and backend/_example/memory_store/ to
their latest versions (chroma 2.27, go-redis 9.21, bbolt 1.5, slack 0.27,
golang.org/x/* and others); re-tidy and re-vendor, keep the example module
in sync.

Hold github.com/go-chi/chi/v5 at v5.2.5: v5.3.0 deprecates
middleware.RealIP (IP-spoofing advisories). Switching off RealIP changes
how the client IP is derived for rate limiting and votes, which is a
security decision better made on its own rather than inside a dependency
bump.

go test -race, go vet, golangci-lint and govulncheck all clean on both
modules.
2026-06-30 18:35:31 +01:00
UmputunandGitHub 34ed97b7a6 Merge pull request #2056 from umputun/dependabot/npm_and_yarn/frontend/postcss-8.5.10
chore(deps-dev): bump postcss from 8.4.14 to 8.5.10 in /frontend
2026-06-01 22:10:26 -05:00
UmputunandGitHub 0868b70fa9 Merge pull request #2063 from umputun/dependabot/npm_and_yarn/frontend/webpack-dev-server-5.2.4
chore(deps-dev): bump webpack-dev-server from 4.9.3 to 5.2.4 in /frontend
2026-06-01 22:10:21 -05:00
Umputun 589e956ade fix: handle REST shutdown before server start 2026-06-01 19:55:14 -05:00
Paul MineevandUmputun a21044738d fix typo in file name 2026-05-28 17:53:37 -05:00
Dmitry VerkhoturovandGitHub 929c06d957 site: fetch latest version client-side instead of embedding at build time (#2072)
* site: fetch latest version client-side instead of embedding at build time

The header version badge was filled in by site/src/data/github.js calling
the GitHub releases API at Eleventy build time and baking data[0].tag_name
into every page. This had three failure modes:

1. Layered cache: Buildx caches the yarn build layer; on a release-triggered
   workflow nothing under ./site changes, so the cached HTML (with the
   previous tag baked in) gets shipped. v1.16.0 went out and remark42.com
   kept showing v1.15.0 until a separate site/ commit landed and naturally
   invalidated the COPY layer.

2. Tag mismatch: the deploy pulls ghcr.io/umputun/remark42-site:master,
   but release events build :v1.16.0 and :latest only. A cache-skip
   workflow tweak wouldn't even reach the served image.

3. API propagation race: the workflow fires ~2s after release publish,
   so even with cache disabled the API might still return the previous
   tag from a stale read replica.

All three vanish if the version is fetched in the browser. GitHub serves
the /releases/latest response with Cache-Control: public, max-age=60 so
per-visitor cost is bounded; failures fall through silently and the
badge stays empty rather than wrong.

Changes:
- header.njk: replace {{ github.latestVersion }} with a
  <span data-remark42-version></span> placeholder.
- inline.js: add a fetch of /releases/latest that fills any
  [data-remark42-version] element on the page. fallback is no-op on any
  network/parse failure.
- delete site/src/data/github.js (Eleventy data file is no longer used).
- drop node-fetch from devDependencies (was used only by github.js).

* site: address PR review on version badge fetch

- gate DOM update on DOMContentLoaded — inline.js is loaded sync in <head>,
  so a cache-hit fetch can resolve before the placeholder span is parsed.
- hide placeholder span by default (`hidden`) so a failed/blocked fetch
  doesn't leave a 0.5rem stray gap before the github icon.
- log fetch failures (rate limit, offline, blocked) instead of silently
  swallowing — matches the prior behaviour of build-time github.js.

* site: cache latest version in sessionStorage with 1h TTL

avoids hitting the GitHub API on every page load — repeated navigations
within a tab read from sessionStorage instead. TTL caps stale display at
1h for very long-lived tabs. cleared on tab close, so each new session
fetches once and reuses the result throughout.

* site: address PR review on header & version fetch

Copilot review on the cache commit raised three points:

1. inline.js had a hard-coded `https://api.github.com/repos/umputun/remark42`
   while the templates use `site.githubUrl`. Rename inline.js → inline.njk
   so nunjucks evaluates it, add `githubApiUrl` to site.json, and template
   the fetch URL from it. One place to update if the repo ever moves.

2. header.njk aria-label said "Remark42's GitHub Repository" but the link
   target is `/releases`. Change to "{{ site.name }} releases on GitHub"
   so screen readers describe the actual destination.

3. console.warn on fetch failure (kept after umputun's prior review noted
   the trade-off): addressed in the PR description, no code change.

* site: actually template fetch URL via site.githubApiUrl

Copilot's second pass caught that 7697dcf3 added site.githubApiUrl,
renamed inline.js → inline.njk, and pointed head.njk at the .njk file
— but the fetch() call itself was never changed to use the template
variable. Build output looked correct because the literal hard-coded
URL happened to match what {{ site.githubApiUrl }} would expand to.

* site: normalise Nunjucks spacing in header.njk

{{ site.githubUrl}}/releases → {{ site.githubUrl }}/releases. Cosmetic
only; matches the spacing used everywhere else in the templates.
2026-05-28 13:10:35 -05:00
Dmitry VerkhoturovandGitHub 198efddb54 fix(frontend): no_footer scrollbar regression introduced in v1.16.0 (#2076)
* fix(frontend): no_footer scrollbar regression introduced in v1.16.0

Two unrelated changes in v1.16.0 combined to surface a scrollbar in
no_footer=true mode:

1. c26f45e5 removed the deprecated `scrolling="no"` iframe attribute
   on the grounds that "overflow is already hidden via CSS". That CSS
   (`overflow: hidden` in createIframe styles) is on the iframe ELEMENT
   in the parent page; it has no effect on the iframe DOCUMENT's own
   scrollbars. The spec-correct replacement is `overflow: hidden` on
   the iframe document's body — added here to global.css.

2. The negative `margin-bottom: -24px` on `.thread:last-child` was a
   trick to tighten the gap to the footer (combined with the footer's
   `margin-top: 48px` it collapsed to a 24px net gap). With no_footer
   the negative margin had no positive-margin sibling to collapse
   against and instead propagated up through .root, leaving body
   ~24px shorter than the visual content. The iframe height calc
   (`body.offsetHeight + 12`) then sized the iframe below the visible
   bottom of the last thread → scrollbar.

   Replace the negative-margin trick with a straight `margin-top: 24px`
   on `.copyright`. Same 24px visual gap when the footer is shown, no
   propagation when it isn't. The mix={styles.thread} on Thread becomes
   a dead reference and is dropped.

Closes #2073

* fix(frontend): drop dead Thread.mix prop after root.tsx removed its only caller

Both Copilot and umputun flagged this in PR review: after the parent
commit on this branch dropped `mix={styles.thread}` from root.tsx, the
`mix?: string` prop and the corresponding entry in the clsx() call in
thread.tsx are dead code — no caller passes it (the recursive Thread
render in thread.tsx:82 never did either). Remove the prop, the
destructure, and the clsx entry.
2026-05-28 13:02:25 -05:00
Dmitry VerkhoturovandGitHub 39408dffe8 fix: parameter docs + --help text inconsistencies (audit) (#2077)
* fix: address parameter docs and --help text inconsistencies

Audit findings from comparing site/src/docs/configuration/parameters/
against the backend flag tags.

Docs (parameters/index.md):
- image.bolt.file default was `/var/pictures.db` (absolute, looks like
  a system path); actual default is `./var/pictures.db` (relative,
  under the working dir).
- notify.webhook.template default was shown as
  `{"text": {{.Text | escapeJSONString}}}` — both the function name
  doesn't exist and the unescaped pipe inside the table cell broke the
  Description column count for that row. Real default is the literal
  `{"text": "{{.Text}}"}`.
- "Custom OAuth2 integration currently supports only one custom
  provider at a time" was a free-standing paragraph wedged between two
  table rows. kramdown terminated the table on that paragraph and
  restarted a new headerless table for the rest of the rows. Moved it
  to its own subsection after the table so the table stays contiguous.

Backend --help text (server.go):
- allowed-hosts description ended with a stray double apostrophe in
  `CSP 'frame-ancestors''` (typo).
- Deprecated auth.email.{port,passwd,user,tls} flag descriptions were
  shuffled — port said "SMTP password", passwd said "SMTP port", user
  said "enable TLS", tls said "SMTP TCP connection timeout". Fixed
  each to match the flag it's actually describing. Docs already had
  the correct descriptions for these deprecated flags.

* fix: webhook template flag default override masking safe fallback

Copilot flagged the audit's "real default" claim and was right. server.go:286
had default:"{\"text\": \"{{.Text}}\"}" — the literal, JSON-unsafe template
that produces invalid JSON if a comment contains a quote or newline. The
notify package (webhook.go:50) has a safer fallback:

    if params.Template == "" {
        params.Template = webhookDefaultTemplate
    }

where webhookDefaultTemplate is {"text": {{.Text | escapeJSONString}}}. But
go-flags applies its default tag at parse time, so the field is never empty
when the user omits --notify.webhook.template, and the safer fallback never
runs.

Drop the unsafe default tag so the webhook package's escapeJSONString-based
default takes effect. Also:
- fix the --help description (was "webhook authentication template", but
  it's a payload template, not an auth one; same for headers).
- update parameters/index.md to document the actual safe default
  ({{.Text | escapeJSONString}}); escape the cell's | as \| so kramdown
  doesn't treat it as a column separator.
- typo: "bellow" -> "below" in the headers env-delim comment.
2026-05-28 12:56:33 -05:00
Dmitry VerkhoturovandUmputun 6961dc24e5 docs: close backtick in smtp.login_auth default cell
opening backtick had no closer in the Default column, so kramdown saw a
broken cell and stopped rendering the parameters table — every row from
smtp.login_auth onward (~40 rows) rendered as raw pipe-delimited text
instead of HTML table cells. closes #2074
2026-05-26 14:12:07 -05:00
Umputun e8b9d70061 docs(site): bump remark42 image tag to v1.16.0 in kubernetes manual
the kubernetes deployment example pinned ghcr.io/umputun/remark42:v1.14.0,
two releases behind.
2026-05-22 16:15:37 -05:00
UmputunandGitHub 556e0a70d5 chore(release): build binary artifacts with GoReleaser (#2070)
replace the Docker artifact build with GoReleaser config and a tag release workflow. Keep local artifact builds snapshot-only and clean generated frontend embed files after release runs.
2026-05-22 13:24:32 -05:00
Dmitry VerkhoturovandGitHub 0e20861419 fix(security): reject non-image content-types in image proxy and /picture/ to prevent stored XSS (#2067)
* fix(security): reject non-image content-types in image proxy and /picture/ to prevent stored XSS

The /api/v1/img proxy and /api/v1/picture/{user}/{id} endpoints emitted
http.DetectContentType on the served bytes as the response Content-Type. A
controlled upstream serving Content-Type: image/png with an HTML body passed
the upstream check (only the response header was inspected, not the body),
and the body bytes then sniffed back to text/html — so the proxy served the
attacker's HTML from the remark42 origin. Browsers honoured the declared
text/html and executed the response as a document with access to cookies and
CSRF tokens. Affected from v1.6.0 (April 2020) through v1.15.0; verified live
via published docker images.

Layered defense applied to both handlers:

- rest.SafeImgContentType (in backend/app/rest/) validates sniffed content
  against a strict allowlist: image/png, image/jpeg, image/gif, image/webp,
  image/bmp, image/x-icon. Anything else (HTML, XML, SVG, plain text,
  octet-stream, or any future image type the stdlib sniffer may learn) is
  rejected with no body echo. SVG is implicitly excluded — it sniffs as
  text/xml or text/plain, never image/svg+xml, and SVG can execute scripts
  when navigated to top-level. The previous octet-stream → image/* fallback
  is gone.
- Per-endpoint Content-Security-Policy override sets
  "default-src 'none'; sandbox; frame-ancestors 'none'" on every response
  (success, 304, or error). Sandbox neuters scripts even if Content-Type
  ever regresses. The same policy is also applied to all /api/v1/* via
  apiCSPMiddleware as defense-in-depth.
- Content-Disposition: inline; filename="image" frames the response as a
  file rather than a renderable document.
- /picture/ rejection paths set Cache-Control: no-store so 4xx responses
  are never cached.

The defense headers and the strict ETag matcher are extracted as
rest.SetImageDefenseHeaders and rest.EtagMatches in the shared rest package
(consumed by both proxy/image and api/rest_public — no package cycle).

The /api/v1/img path additionally bumps the ETag to a versioned `"v2:..."`
so revalidating clients (top-level navigation, Ctrl+R, intermediaries) get
a fresh 200 instead of a 304 against poisoned pre-fix cached HTML.

DELIBERATE TRADEOFF: Cache-Control on /api/v1/img success responses remains
max-age=2592000 (30 days), unchanged from before. An aggressive "force
revalidate on every reuse" policy was prototyped during review but reverted
because the perf cost (a server round-trip on every image view, even with
304 saving the body bytes) outweighed the corner-case mitigation. The
realistic exposure of cache carryover is narrow: cache carryover only
affects users who navigated top-level to an attacker URL pre-fix and still
have it in their local cache — the normal <img> embed path cached text/html
but never executed it. Local browser caches that hold pre-fix bytes
continue to serve them until their 30-day TTL expires or are evicted under
memory pressure. The ETag bump reaches all clients that DO revalidate
during the cached lifetime (Ctrl+R, intermediaries, post-expiry use); for
the rest, exposure self-limits via cache expiry. Operators running a
CDN/edge cache in front of remark42 should purge /api/v1/img after deploy.

The /api/v1/img handler short-circuits on a matching current-version
If-None-Match before any store Load or upstream fetch, returning a bodyless
304 with the defense headers set. Safe because the 304 carries no body and
the client's cached bytes came from a prior validated 200; an attacker
fabricating an etag value can only short-circuit fetches for URLs they
themselves crafted. This avoids upstream DoS amplification when clients
revalidate on hot comment pages.

The /api/v1/img route was moved from the "open routes" group (which uses
middleware.NoCache, stripping If-None-Match from incoming requests) to the
"open routes, cached" group alongside /picture/ and /qr/telegram so the
304 revalidation path is no longer broken upstream of the handler.

The /picture/{user}/{id} endpoint does not need the v2 etag prefix. Upload
validates input format via readAndValidateImage and the serve path
re-validates the stored bytes via rest.SafeImgContentType. Bytes within
the resize dimension limits are preserved verbatim, so the browser defense
relies on the response headers (validated Content-Type + nosniff + strict
CSP + Content-Disposition: inline), not on byte normalization.

Global CSP: font-src data: → font-src 'none'. Audit confirmed no @font-face,
no base64 fonts, no icon-font library in the bundle. Drops an unnecessary
attack surface; no behavioural change.

Tests: TestImage_ContentTypeHandling table-tests a real PNG and attack
shapes (HTML claimed as image/png, image/jpeg, image/gif, image/svg+xml,
image/webp; svg with onload; html fragment; polyglot PNG+HTML), proving
the defense holds across arbitrary upstream Content-Type variation.
Polyglot case is intentionally served as image/png — the browser cannot
execute the trailing HTML when the response type is image/png with nosniff.
TestImage_ContentTypeHandling_CacheHit exercises the cache-hit branch with
attacker bytes preloaded into the store. TestImage_PerRequestRevalidation
alternates upstream PNG/HTML across four proxy calls to prove no trust
accumulates between requests. TestImage_RoutesUsingCachedImage asserts
cache-poisoning is caught at serve time. TestImage_EtagVersioned asserts
the v2 prefix invalidates pre-fix etags AND that the revalidation 304
triggers no store Load. TestImage_RevalidationSkipsIO proves the
short-circuit works even with no upstream reachable. TestSafeImgContentType
covers the allowlist directly. TestRest_LoadPictureDefenseHeaders and
TestRest_LoadPictureRejectsNonImage exercise the /picture/ endpoint.
TestRest_apiCSP covers the strict CSP middleware on JSON API + RSS routes;
TestRest_securityHeaders confirms /web/ HTML pages keep the global CSP.

Verified end-to-end against the dev docker image: the original demo URL
(arbitrary HTML claimed as image/png) now returns 415 application/json with
CSP/nosniff/Content-Disposition set, no XSS in the browser.

* fix(security): set Cache-Control: no-store on image-proxy error paths, sync stale route comment

Addresses two review comments on #2067:

1. Cache-Control: max-age=2592000 and Etag were set before the
   load/download/validation block, so 404/400/415 error responses inherited
   the 30-day cache TTL and the versioned etag — a transient failure (or an
   intentionally triggered 415) would be pinned in browser/intermediary
   caches for that TTL, keeping users locked out even after the underlying
   cause was resolved. Now: etag is computed but not set as a header until
   after validation succeeds; error paths route through sendImageProxyError
   which sets Cache-Control: no-store and never sets Etag. The 304
   short-circuit still sets both because that path serves the same validated
   content the client already has cached.

2. The comment at rest.go:282 still described the prototyped
   no-cache/must-revalidate Cache-Control policy that was reverted before
   the PR landed. Updated to match the actual 30-day max-age behavior.

Tests: TestImage_ContentTypeHandling now asserts reject paths carry
Cache-Control: no-store and have no Etag header, and accept paths carry
the max-age=2592000 + v2: etag.
2026-05-20 22:37:25 -05:00
Dmitry VerkhoturovandUmputun 8224626ed4 fix(image): reject decompression-bomb dimensions before raster decode
readAndValidateImage caps the byte size of incoming images but the resize()
helper that follows still called image.Decode unconditionally, allocating
pixel memory proportional to the *declared* image dimensions. A ~100 KB
compressed PNG or GIF that declares 65535x65535 px forces image.Decode to
allocate ~17 GB of raster, OOMing the service on a single comment upload
(or on the proxy's CacheExternal path when caching a malicious upstream).

Hardening:

- maxImagePixels = 16 MP constant. Covers any realistic image (~4096x4096)
  while bounding peak allocation.
- resize() now runs image.DecodeConfig first (cheap, no pixel allocation)
  to read declared width/height before any full decode.
- Multiplication of width × height uses int64 to defeat 32-bit overflow
  (GOARCH=386, 32-bit arm): on those targets, int(cfg.Width)*int(cfg.Height)
  could wrap below maxImagePixels and bypass the cap. GIF's 16-bit logical
  screen and JPEG's 16-bit SOF dimensions both reach this if int-multiplied.
- Bytes exceeding the cap, or non-image input that fails DecodeConfig,
  return nil. prepareImage propagates the rejection as a clear error
  instead of storing the malformed/oversized data verbatim.
- The no-resize-needed path returns the validated original bytes verbatim
  so animated GIFs round-trip without being flattened to a single frame.

The DecodeConfig precheck applies even when MaxWidth/MaxHeight are 0
(resize disabled) — the dimension cap is unconditional defense-in-depth.

Two adjacent fixes surfaced by the new resize contract:

1. readAndValidateImage previously did `data[:512]` without a bounds check,
   panicking on any body shorter than 512 bytes. Now bounded with min().
2. image/webp was listed as an allowed format but no WebP decoder was
   registered, so DecodeConfig would refuse legitimate WebP uploads. Added
   `_ "golang.org/x/image/webp"` (already in go.mod via x/image/draw) so
   the registered decoders match the allowlist.

Tests:

- TestService_resizeRejectsDecompressionBomb builds a 14-byte GIF87a header
  declaring 65535x65535 and asserts resize() refuses it both at the unit
  level and through SaveWithID end-to-end (no store write).
- TestService_SaveWithIDShortPayload regression-tests the short-body panic.
- TestService_SaveWithIDWebP regression-tests WebP round-trip through
  prepareImage with the new DecodeConfig requirement.
- TestService_resize subtests updated to assert non-image bytes are now
  refused (previously the helper fell back to returning the raw bytes
  verbatim, letting malformed content reach the store).
2026-05-20 21:48:23 -05:00
Dmitry VerkhoturovandUmputun 45c17a913f chore(deps): bump go modules in backend and example
Backend (backend/go.mod):
- github.com/go-pkgz/auth/v2 v2.1.2 → v2.1.4
- github.com/klauspost/compress v1.18.5 → v1.18.6
- github.com/redis/go-redis/v9 v9.18.0 → v9.19.0
- github.com/slack-go/slack v0.21.1 → v0.23.1
- golang.org/x/crypto v0.50.0 → v0.51.0
- golang.org/x/image v0.39.0 → v0.40.0
- golang.org/x/net v0.53.0 → v0.54.0
- golang.org/x/sys v0.43.0 → v0.44.0
- golang.org/x/text v0.36.0 → v0.37.0

Example (backend/_example/memory_store/go.mod):
- golang.org/x/crypto v0.50.0 → v0.51.0
- golang.org/x/image v0.39.0 → v0.40.0
- golang.org/x/net v0.53.0 → v0.54.0
- golang.org/x/sys v0.43.0 → v0.44.0

Transitive cleanup: github.com/dgryski/go-rendezvous is no longer required
after redis/go-redis bump and gets pruned by `go mod tidy`.

`go mod tidy` + `go mod vendor` run on both modules. Both build with -race
and full test suites pass.
2026-05-20 20:09:47 -05:00
Umputun f3a7dea1f1 docs: offer github private vulnerability reporting in security policy
Mention the "Report a vulnerability" button (GitHub private vulnerability
reporting) alongside the existing email contact, now that private reporting
is enabled on the repository.
2026-05-20 13:41:06 -05:00
dependabot[bot]andGitHub e8c106f06b chore(deps-dev): bump webpack-dev-server in /frontend
Bumps [webpack-dev-server](https://github.com/webpack/webpack-dev-server) from 4.9.3 to 5.2.4.
- [Release notes](https://github.com/webpack/webpack-dev-server/releases)
- [Changelog](https://github.com/webpack/webpack-dev-server/blob/main/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-server/compare/v4.9.3...v5.2.4)

---
updated-dependencies:
- dependency-name: webpack-dev-server
  dependency-version: 5.2.4
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-19 19:47:06 +00:00
dependabot[bot]andGitHub 54b7b3fdd4 chore(deps-dev): bump postcss from 8.4.14 to 8.5.10 in /frontend
Bumps [postcss](https://github.com/postcss/postcss) from 8.4.14 to 8.5.10.
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.4.14...8.5.10)

---
updated-dependencies:
- dependency-name: postcss
  dependency-version: 8.5.10
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-06 06:54:50 +00:00
UmputunandGitHub c0636f204e Merge pull request #2053 from umputun/dependabot/github_actions/github-actions-updates-512a575e1a
chore(deps): bump pnpm/action-setup from 5.0.0 to 6.0.4 in the github-actions-updates group
2026-05-06 01:53:15 -05:00
UmputunandGitHub c418f8ec00 Merge pull request #2052 from umputun/dependabot/go_modules/backend/go-modules-updates-47fdc5c9f4
chore(deps): bump the go-modules-updates group in /backend with 2 updates
2026-05-06 01:53:10 -05:00
dependabot[bot]andDmitry Verkhoturov e9ad5dcc09 chore(deps): bump the go-modules-updates group
Bumps the go-modules-updates group in /backend with 2 updates: [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) and [github.com/go-pkgz/auth/v2](https://github.com/go-pkgz/auth).

Updates `github.com/alecthomas/chroma/v2` from 2.23.1 to 2.24.1
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.23.1...v2.24.1)

Updates `github.com/go-pkgz/auth/v2` from 2.1.2-0.20260421203319-686683f19cf7 to 2.1.2
- [Release notes](https://github.com/go-pkgz/auth/releases)
- [Commits](https://github.com/go-pkgz/auth/commits/v2.1.2)

---
updated-dependencies:
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-version: 2.24.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/auth/v2
  dependency-version: 2.1.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-01 22:52:12 +01:00
dependabot[bot]andGitHub d072f34a67 chore(deps): bump pnpm/action-setup in the github-actions-updates group
Bumps the github-actions-updates group with 1 update: [pnpm/action-setup](https://github.com/pnpm/action-setup).


Updates `pnpm/action-setup` from 5.0.0 to 6.0.4
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v5.0.0...v6.0.4)

---
updated-dependencies:
- dependency-name: pnpm/action-setup
  dependency-version: 6.0.4
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-01 13:46:19 +00:00
Dmitry VerkhoturovandGitHub a4c5e17bbb Probe /auth/status from frontend to avoid 401 on /user (closes #1188) (#1763)
* Probe /auth/status from frontend to avoid 401 console noise on /user

GET /api/v1/user requires auth and returns 401 for anonymous visitors,
which the browser logs to console even when JS catches it. Probe
/auth/status first (always 200), then fetch /user only when logged in.
Stale auth cookies are cleared when status reports "not logged in" to
preserve the cleanup-on-probe behaviour previously triggered by /user 401.

Closes #1188.

* Don't clear auth cookies when /auth/status probe itself fails

A transient network/5xx on the /auth/status probe used to fall through
into the cookie-clear branch and silently log the user out on the next
page load. Distinguish "probe failed" (null) from explicit "not logged
in"; only the latter clears JWT/XSRF cookies. Lock the distinction with
a negative assertion in the probe-failure test.

Also align packages/api prettier config with apps/remark42 (trailingComma: 'es5')
so future edits don't sweep unrelated trailing commas into the diff.
2026-04-30 19:32:49 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8d9290ea1f Bump picomatch from 2.3.1 to 2.3.2 in /site (#2028)
Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/picomatch/compare/2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 2.3.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:44 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
07d89202b4 chore(deps): bump liquidjs from 10.19.0 to 10.25.6 in /site (#2050)
Bumps [liquidjs](https://github.com/harttle/liquidjs) from 10.19.0 to 10.25.6.
- [Release notes](https://github.com/harttle/liquidjs/releases)
- [Changelog](https://github.com/harttle/liquidjs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/harttle/liquidjs/compare/v10.19.0...v10.25.6)

---
updated-dependencies:
- dependency-name: liquidjs
  dependency-version: 10.25.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:41 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5d6599237d Bump the github-actions-updates group across 1 directory with 7 updates (#2034)
Bumps the github-actions-updates group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) | `3` | `4` |
| [pnpm/action-setup](https://github.com/pnpm/action-setup) | `4.2.0` | `5.0.0` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5` | `6` |
| [docker/login-action](https://github.com/docker/login-action) | `3` | `4` |
| [docker/build-push-action](https://github.com/docker/build-push-action) | `6` | `7` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `6` | `7` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7` | `8` |



Updates `docker/setup-buildx-action` from 3 to 4
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

Updates `pnpm/action-setup` from 4.2.0 to 5.0.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v4.2.0...v5.0.0)

Updates `codecov/codecov-action` from 5 to 6
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v5...v6)

Updates `docker/login-action` from 3 to 4
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

Updates `docker/build-push-action` from 6 to 7
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

Updates `actions/upload-artifact` from 6 to 7
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v6...v7)

Updates `actions/download-artifact` from 7 to 8
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7...v8)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: codecov/codecov-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:38 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b13b737461 Bump handlebars from 4.7.8 to 4.7.9 in /site (#2030)
Bumps [handlebars](https://github.com/handlebars-lang/handlebars.js) from 4.7.8 to 4.7.9.
- [Release notes](https://github.com/handlebars-lang/handlebars.js/releases)
- [Changelog](https://github.com/handlebars-lang/handlebars.js/blob/v4.7.9/release-notes.md)
- [Commits](https://github.com/handlebars-lang/handlebars.js/compare/v4.7.8...v4.7.9)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-version: 4.7.9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-21 19:29:27 -05:00
Dmitry VerkhoturovandGitHub c9ba8520c7 fix(auth): close OAuth open-redirect by wiring AllowedRedirectHosts (#2049)
* fix(auth): close OAuth open-redirect by wiring AllowedRedirectHosts

Bump go-pkgz/auth/v2 to master (v2.1.2-0.20260421203319-686683f19cf7)
which carries the `from` redirect validator from go-pkgz/auth#275.

The library default with a nil AllowedRedirectHosts is permissive
(preserves legacy behavior for existing consumers on a dep bump), so
just bumping the dep leaves remark42 vulnerable — a crafted
/auth/<provider>/login?from=https://evil.example.com/... still issues
the 307 to the attacker host after the user completes legitimate
OAuth. Verified end-to-end against a local dev-auth instance before
and after this commit.

Wire Opts.AllowedRedirectHosts in getAuthenticator to the operator's
existing --allowed-hosts config, stripping the CSP "self" sentinel
which is not a real hostname. RemarkURL's own host is always implicit
per the library contract, so a default single-site deployment gains
the protection with no config change. Multi-host embeds work as soon
as their embedding hosts are added to AllowedHosts (they already need
to be there for CSP frame-ancestors).

Refreshed vendor tree to match the new module version.

* chore(lint): suppress G703 false positives on image Save

CI's newer gosec flags os.MkdirAll/os.WriteFile in FileSystem.Save with
G703 because id flows in from the caller. id is validated at the HTTP
layer (safePictureSegment in rest_public.go) and dst is derived via
f.location — not a real traversal. Targeted //nolint with reason.

* fix(auth): normalise AllowedRedirectHosts entries + add unit test

Address Copilot review on PR #2049. The previous closure passed raw
s.AllowedHosts entries straight to the auth library, but --allowed-hosts
holds CSP frame-ancestors source expressions: scheme-prefixed values
(https://blog.example.com), entries with ports, and wildcards
(*.cdn.example.com) are all valid there but the auth library compares
against u.Hostname() and would silently drop them — breaking legitimate
redirects on multi-host deployments.

Extract getAllowedRedirectHosts that:
* trims whitespace, drops empty / 'self' / "self" / wildcard entries
* prepends https:// if scheme missing then url.Parse to extract Hostname
* logs a warning on parse failure rather than poisoning the allowlist

Wire the closure in getAuthenticator to call the helper.

Test_getAllowedRedirectHosts covers all the edge cases Copilot flagged
(scheme stripping, port handling, self spellings, wildcards, empty,
mixed real-world).

* fix(auth): preserve explicit port in AllowedRedirectHosts + clarify fs_store nolint

Address Copilot follow-up on PR #2049:

* getAllowedRedirectHosts stripped explicit ports via u.Hostname(), which
  broadened the allowlist. The auth validator checks both Hostname() and
  Host, so an entry like admin.example.com:8443 can and should be kept
  host:port — allowing only that port, not any. Emit u.Host when
  u.Port() != "", u.Hostname() otherwise. Updated tests.

* fs_store Save nolint rationale said "id validated at HTTP layer", but
  Save is reached via image.Service.Save and SaveWithID (cache), neither
  of which is HTTP validation. id is actually a server-generated hash in
  both paths. Updated the comment.
2026-04-21 19:09:26 -05:00
Dmitry VerkhoturovGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>umputun
ee782785f0 test: use testing/synctest to eliminate wall-clock sleeps (#2048)
Go 1.25's testing/synctest package (GA) provides a fake clock bubble
for deterministic goroutine and timer testing. Convert tests that
waited on real-time durations to use synctest, removing most wall-clock
time.Sleep workarounds.

Converted (11 tests, 9 files):
- notify/notify_test.go — all tests, replaced 17 time.Sleep(110ms) with synctest.Wait()
- store/service/service_test.go — VoteSameIPWithDuration, UserReplies, submitImages,
  ResubmitStagingImages, deleteImagesOnCommentDelete
- store/image/{image,bolt_store}_test.go — Cleanup, Submit, SubmitDelay
- store/engine/bolt_test.go — FlagListBlocked
- providers/telegram_test.go — DispatchTelegramUpdates
- migrator/backup_test.go — TestBackup_Do
- _example/memory_store/accessor/data_test.go — FlagListBlocked

Simplifications along the way:
- notify/notify_mock.go: dropped the 10ms time.After delay and
  ctx.Done select in MockDest — the artificial I/O simulation is
  pointless and blocked synctest.Wait from draining the queue
- Removed three dead-code time.Sleep(1s) calls in EditCommentDurationFailed,
  EditCommentAdmin, and Info tests: prepopulated comments from 2017
  already exceed any EditDuration/ReadOnlyAge under real clock, making
  the sleeps meaningless
- UserReplies: replaced the Eventually+Sleep+mutex polling with a
  direct time.Sleep under fake clock

Skipped (incompatible with synctest):
- fs_store_test.go: relies on OS file mtime (real wall clock)
- rss_test.go: needs real wall-clock second boundary for pubDate
- admin/rest_private/rest_public tests: httptest network I/O
- cmd/server_test.go: real HTTP server startup polling

Notes on quirks encountered:
- synctest.Wait() does NOT advance fake time, contrary to what one
  might expect. It only returns once all other bubble goroutines are
  durably blocked. To advance the fake clock, the test goroutine must
  itself call time.Sleep
- BoltDB keys the "last" bucket by comment.Timestamp nanosecond string.
  Rapid b.Create calls under frozen fake time produce identical keys
  and overwrite each other. TestService_UserReplies adds
  time.Sleep(time.Nanosecond) between Creates to advance the clock
- Bolt image Cleanup uses strict age > ttl. Under fake time the
  age-ttl delta is exactly zero at the boundary, so subtract 1ms from
  the passed ttl to stay strictly under

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: umputun <535880+umputun@users.noreply.github.com>
2026-04-18 02:44:21 -05:00
Dmitry VerkhoturovandUmputun 3b1d7be6fc fix(safehttp): clone http.DefaultTransport, sharpen Image.Transport contract
Address review feedback on PR #2044.

safehttp.Transport():
* Clone http.DefaultTransport instead of building a bare &http.Transport{} so
  Proxy, ForceAttemptHTTP2, MaxIdleConns, IdleConnTimeout, TLSHandshakeTimeout
  and ExpectContinueTimeout are inherited (the bare struct loses them all).
  Verified by new TestTransport_PreservesDefaultTransportSettings.
* TestTransport_AllowsPublic: bound the dial of TEST-NET-3 with a 100ms
  context so the test does not depend on real-world routing of 203.0.113.0/24,
  and drop the dead dialer var.

proxy/image.go:
* Document Image.Transport contract: nil installs safehttp.Transport (SSRF-safe);
  caller-supplied transport is the caller's responsibility.
* Replace the misleading "SSRF mitigated by safehttp.Transport" nolint comments
  with one that points at the documented contract above.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun e98657a88a chore(lint): cap multipart upload size and suppress remaining gosec G70x
Address all golangci-lint v2.10.1 (CI's version) findings:

* Add http.MaxBytesReader hard cap to ParseMultipartForm sites in
  rest_private.savePictureCtrl (32MB) and api/migrator (256MB) — fixes
  G120 by bounding total request body before form parsing.

* Suppress G70x in CLI subcommands cmd/{backup,cleanup,import,remap}.go:
  all four issue HTTP requests against operator-supplied RemarkURL/CLI
  flags, never user input. Each suppression carries a one-line reason.

* Suppress G122 in image fs_store cleanup walk: staging directory tree
  is server-only, no untrusted symlinks land there.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun a96bddcb8d chore(lint): suppress gosec G70x false positives in admin/CLI paths
CI's golangci-lint v2.10.1 (newer rule set than my local 2.11.4) flags
four G70x cases the previous run missed. All are false positives:
backup.go and cleanup.go drive HTTP requests against the operator's own
RemarkURL from CLI flags (not user input); migrator.go removes a temp
file whose name was returned by os.CreateTemp (server-controlled). Add
targeted //nolint:gosec comments naming the reason at each site.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun 5ff5059db3 chore(lint): re-enable gosec G703/G704/G705 with targeted suppressions
Commit aca0cff3 silenced the path-traversal, SSRF and XSS taint rules
project-wide as "false positives" while fixing image-proxy SSRF. With
the path-traversal and TitleExtractor SSRF gaps now closed, restore the
rules so future regressions get flagged. The four genuine false positives
that remain (image proxy http.NewRequest, QR png Write, two RSS XML
Writes) get individual //nolint:gosec comments naming the reason.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun ff85bbc5ea fix(ssrf): apply ssrf-safe transport to TitleExtractor
The image proxy got an ssrfSafeTransport in commit aca0cff3 that resolves
DNS first, blocks any IP in private/reserved CIDRs, then dials by IP to
defeat DNS rebinding. The TitleExtractor used to construct comments'
PostTitle from Locator.URL — a user-supplied field — was missed by that
fix and kept using http.DefaultTransport. The hostname allowlist there
checks the parsed URL host but never the IP it resolves to, so a domain
suffix-matching an allowed host (or 127.0.0.1 itself when AllowedHosts
is empty) reaches the metadata service or any other internal endpoint.

The same gosec rule (G704) was excluded globally in .golangci.yml as part
of aca0cff3, so this gap was not caught by the linter either.

Extract the transport into a new safehttp package so it lives in one
place and can be reused, then pass safehttp.Transport() into the
TitleExtractor's http.Client at construction (cmd/server.go). The image
proxy switches to safehttp.Transport() too — same behaviour, no longer
duplicated.

Reproduction in title_test.go uses the production-style client to hit
an httptest.Server (always 127.0.0.1) and asserts the dialer refuses
even though "127.0.0.1" is in the allowed-domains list. A control case
shows the same setup without safehttp.Transport returns the page —
making the original vulnerability explicit.
2026-04-18 02:32:31 -05:00
Dmitry VerkhoturovandUmputun 5d88c1b2fa fix(api): drop QR-write nolint dup + trim dead .. check
Address PR #2045 review (umputun):

* The //nolint:gosec on telegramQrCtrl's w.Write(png) was byte-identical
  to the same line in #2044 (gosec-rule restoration). Drop it here so
  the two PRs do not conflict; #2044 owns it.
* `seg == ".."` in safePictureSegment was already covered by the
  strings.Contains(seg, "..") check two lines down — trim and add an
  inline comment so the cover-by-superset is explicit.
2026-04-18 02:15:53 -05:00
Dmitry VerkhoturovandUmputun 114a1be2e9 fix(api): reject control characters in /picture URL segments
Address PR #2045 review feedback (Copilot #2045-1). The previous
safePictureSegment allowed CR/LF/TAB through, so a request such as
GET /api/v1/picture/dev%0Auser/abc.png would inject literal newlines
into the access log line ("GET - /api/v1/picture/dev\nuser/abc.png ...")
— a log-forgery primitive against any operator parsing those logs.

Reject any unicode.IsControl rune in either segment (NUL was already
caught via strings.ContainsAny). New TestRest_LoadPictureRejectsControlCharsInSegment
covers LF, CR, TAB, NUL across both segments.
2026-04-18 02:15:53 -05:00
Dmitry VerkhoturovandUmputun 59c92f8c4d fix(api): reject path traversal and sanitise error in /picture/{user}/{id}
The unauthenticated GET /api/v1/picture/{user}/{id} handler concatenated the
two URL params verbatim into a filesystem path via path.Join, so a request
like /api/v1/picture/../remark.db resolved to <base>/../remark.db, escaping
the image directory. With Partitions=0 (a documented option) this is a
direct arbitrary-file read; with the default Partitions=100 the constructed
path lands in a CRC-derived subdirectory but the server still leaks the
internal filesystem path back to the unauthenticated caller via the JSON
error body — confirmed against demo.remark42.com (master-80c12a3) which
returned `stat /var/folders/.../staging/.../remark.db` for `..` requests.

Validate both URL segments via safePictureSegment (no traversal markers,
no path separators, no NULs) at the handler entry, and replace the raw
storage error with a generic "image not found" response. The original
error is logged for operators.

Reproduction test asserts that ../remark.db, foo/..%2Fremark.db and
%2E%2E/remark.db all return 400 with no internal path leaked.
2026-04-18 02:15:53 -05:00
Dmitry VerkhoturovandUmputun ddcb2c7b5f test(store): use time.UTC in test fixtures to be timezone-agnostic
The store tests stored timestamps with time.Local in their fixtures and
asserted equality against returned values that the engine round-trips
through UTC. assert.Equal compares zone identity, so on UTC machines
(CI, most cloud envs) Local==UTC and the tests passed; on a developer
machine in any other timezone (here BST, UTC+1) TestService_Put,
TestService_List, TestBoltDB_InfoPost, TestBoltDB_InfoList and several
others would fail with same wall-clock numbers but mismatched zones.

Replace time.Local with time.UTC across store/comment_test.go,
store/formatter_test.go, store/service/service_test.go,
store/engine/bolt_test.go, store/engine/engine_test.go. Production code
is untouched.
2026-04-17 19:38:11 -05:00
Dmitry VerkhoturovandUmputun f8ba38779b fix(api): require explicit ?site= in matchSiteID middleware
matchSiteID guarded most authenticated and admin routes with
`if siteID != "" && user.SiteID != siteID`. Dropping the ?site= query
parameter made the check no-op and any authenticated user passed the
middleware. Downstream handlers fell back to reading site from the JSON
body or just used the empty string, so on email/telegram subscribe
endpoints (which read site from body) a user authenticated to siteA
could perform actions targeting siteB without the cross-site guard
ever firing.

Require ?site= to be present and to match user.SiteID. Body-only site
flows are still supported provided the URL also carries the matching
?site= — both must agree, which removes the bypass and keeps the
declared site visible to the middleware.

Reproduction TestRest_matchSiteID enumerates four cases (matching,
mismatched, missing, empty). Existing test calls that relied on the
implicit pass had to add ?site=remark42 to the URL: the addComment
helper now derives the param from c.Locator.SiteID, picture upload
URL gets the param explicitly, and the email/telegram subscribe table
adds it to every endpoint. The negative cases that previously asserted
StatusBadRequest from the handler now correctly assert StatusForbidden
from the middleware.
2026-04-17 19:35:50 -05:00
AlexMa233andGitHub 94d1f6e224 feat: custom oauth2 provider (#2006)
* feat: add configurable custom OAuth2 provider and icons

* fix: reserve built-in custom provider names

* fix: add nolint directive for sha1 import

* fix: harden custom oauth provider validation
2026-04-16 23:10:05 -05:00
Adán Román RuizandUmputun ba3df171d1 #2025 Fix typo in Spanish localization for sort-by 2026-04-14 16:11:30 -05:00
Amir MohamadandGitHub 7ec5af8068 Fix Firefox dark mode white background on comment iframe (#2023)
* fix(embed): set color-scheme on iframe to fix Firefox dark mode

Firefox renders a white background in dark mode when color-scheme is 'none' on the iframe. Set color-scheme to match the active theme on both the outer iframe element and the inner document root, so Firefox uses the correct rendering mode from the start and on theme changes.

* fix(embed): default iframe color-scheme to light when no theme set

Changes the fallback from 'light dark' to 'light' to match the inner document's default behavior, which always defaults to light when no theme is specified.
2026-04-14 15:35:41 -05:00
Dmitry VerkhoturovandUmputun fc6f15534e fix(frontend): preserve orig verbatim in edit textarea (#2040)
The edit textarea was running `data.orig` through the browser's HTML
parser via a detached `<span>.innerHTML` to "decode entities", which
turned user-typed `&lt;`/`&gt;` into real `<`/`>`. On save, blackfriday
then saw a real `<script>` tag, bluemonday stripped it, and the comment
body collapsed to an empty string.

The decode block predates commit 243c835 (2022) which stopped the
backend from sanitising `orig` with bluemonday. Before 243c835, orig
came back HTML-escaped from the API and the frontend compensated.
After 243c835 the backend stores and returns orig byte-for-byte, but
the frontend decode was never removed — so it has been silently
corrupting user input containing entities for ~3.5 years.

The backend contract is clear: `orig` is the raw user input, never
rendered as HTML. The frontend should echo it back into the textarea
unchanged. This change removes the decode and adds 45 table-driven
regression tests covering entity round-trips, unicode edge cases,
and markdown constructs.
2026-04-12 11:55:24 -05:00
Dmitry VerkhoturovandUmputun 80c12a3f10 chore(deps): update Go modules
Bump Go dependencies in both backend/ and backend/_example/memory_store.

Notable updates:
- github.com/go-pkgz/lgr v0.12.1 -> v0.12.3
- github.com/klauspost/compress v1.18.2 -> v1.18.5
- github.com/PuerkitoBio/goquery v1.11.0 -> v1.12.0
- github.com/montanaflynn/stats v0.7.1 -> v0.9.0
- github.com/redis/go-redis/v9 v9.17.2 -> v9.18.0
- github.com/slack-go/slack v0.17.3 -> v0.21.1
- go.mongodb.org/mongo-driver v1.17.6 -> v1.17.9
- golang.org/x/crypto v0.48.0 -> v0.50.0
- golang.org/x/net v0.49.0 -> v0.53.0
- golang.org/x/image v0.36.0 -> v0.39.0
- golang.org/x/sys v0.41.0 -> v0.43.0
- golang.org/x/{oauth2,sync,text} minor bumps

Key markdown/sanitisation libs (bluemonday v1.0.27,
alecthomas/chroma/v2 v2.23.1, russross/blackfriday/v2 v2.1.0,
Depado/bfchroma/v2 v2.0.0) are already at the latest available
versions and were not bumped.

Verified the Chroma span-class allowlist regex in
backend/app/store/comment.go:128-131 is still fully in sync with
chroma/v2 types.go StandardTypes map (86 classes, byte-equal after
sorting). The inline comment references commit c263f6f which is
stale (Chroma is at v2 now), but the class list content is current.

Ran `go mod tidy` + `go mod vendor` + full race test suite on both
modules. All green. Added a reminder in CLAUDE.md that updating
backend/ Go modules also requires `go mod tidy` in
backend/_example/memory_store since the example module uses a
local replace directive and inherits indirect deps from the main
module.
2026-04-12 11:52:57 -05:00
UmputunandGitHub bea67f0136 Merge pull request #2032 from umputun/dependabot/go_modules/backend/_example/memory_store/golang.org/x/image-0.38.0
Bump golang.org/x/image from 0.36.0 to 0.38.0 in /backend/_example/memory_store
2026-04-04 23:19:12 -05:00
dependabot[bot]andGitHub 8b9e5c6c8c Bump golang.org/x/image in /backend/_example/memory_store
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.36.0 to 0.38.0.
- [Commits](https://github.com/golang/image/compare/v0.36.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/image
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-30 16:29:36 +00:00
Dmitry VerkhoturovandGitHub 06436ff9b0 Migrate batch 1 components from BEM to CSS Modules (#2014)
* feat: migrate batch 1 components from BEM to CSS Modules

Migrate 8 components from BEM to CSS Modules:
- button (7 BEM files -> 1 module)
- dropdown (7 BEM files -> 1 module)
- thread (3 BEM files -> 1 module)
- auth-panel (2 BEM files -> 1 module)
- dropdown-item, list-comments, subscribe-by-rss, settings (from batch 0 PR #2013)

Consolidates 19 BEM CSS files into 8 CSS Module files. Uses clsx for
conditional class composition, replacing bem-react-helper's b() calls.
Class naming follows the established convention: BEM block = .root,
elements = camelCase, modifiers = camelCase.

Visual regression verification on built artefacts:
- remark.css: 43,779 -> 43,299 bytes (480 bytes smaller)
- last-comments.css: 18,792 -> 18,776 bytes (16 bytes smaller)
- remark.js: 256,709 -> 304,837 bytes (48KB larger, expected: CSS Module
  classname mappings now live in JS instead of plain strings)
- Dark theme: pixel-identical (zero difference)
- Light theme: pixel-identical (0.21% diff is the native demo page
  "Toggle theme" button, not any remark42 widget element)

Also updates CLAUDE.md CSS guideline to reflect the migration status.

* Migrate remaining BEM components to CSS Modules (final batch)

Migrate the last 4 BEM components to CSS Modules, completing the
migration and removing bem-react-helper from the project entirely.

Components migrated:
- subscribe-by-email (1 BEM CSS file -> 1 module)
- comment-form + markdown-toolbar (20 BEM CSS files -> 2 modules)
- comment (19 BEM CSS files -> expanded existing module)
- root (10 BEM CSS files -> expanded existing module)

Consolidates ~50 BEM CSS files into 4 new + 2 expanded CSS Module files.
Removes bem-react-helper dependency — all components now use clsx for
conditional class composition.

Dead CSS cleanup during migration:
- Orphaned comment-actions selectors in comment theme CSS (already migrated)
- Dead BEM modifiers: comment_disabled, comment_pinned, comment_guest
- Dead element: comment__user-id (CSS existed but never used in TSX)
- Dead button type classes: comment-form__button_type_preview/_send
- Dead mix values: auth-email-login-form__back-button, comment-form__email-dropdown

Key implementation details:
- comment_highlighting stays global via :global() (imperatively added by classList)
- Bare .dark/.light theme class preserved on root wrapper (8+ modules depend on it)
- raw-content.css kept as global utility CSS (syntax highlighting)

Visual regression verification on built artefacts:
- remark.css: 43,779 -> 36,106 bytes (-17.5%)
- last-comments.css: 18,792 -> 13,955 bytes (-25.7%)
- remark.js: 256,709 -> 253,637 bytes (-1.2%)
- last-comments.js: 121,726 -> 120,795 bytes (-0.8%)
- Total: 441,006 -> 424,493 bytes (-3.7%)
- Screenshot comparison: pixel-identical across light/dark themes
2026-03-25 16:53:32 -05:00
Dmitry VerkhoturovandGitHub b888a53759 Migrate dropdown-item, list-comments, subscribe-by-rss, and settings from BEM to CSS Modules (#2013)
Consolidate legacy BEM CSS files into CSS Modules for 4 components:
- dropdown/__item: 1 CSS file → dropdown-item.module.css
- list-comments: 1 CSS file → list-comments.module.css (removed unused
  comments-list class that had no CSS rules)
- comment-form/__subscribe-by-rss: 1 CSS file → subscribe-by-rss.module.css,
  removed dead titleClass prop and dead __rss-link directory
- settings: 10 CSS files → settings.module.css, removed dead
  .settings__blocked-users-username CSS rule

Built artefact comparison (master vs branch):
- 83 of 89 files in /srv/web/ are byte-identical (all locale bundles,
  SVGs, HTML pages unchanged)
- 6 files differ: remark.css/js/mjs and last-comments.css/js/mjs
- CSS changes are class name hash shifts (e.g. G_A → H_A) caused by
  webpack's module ordering, plus 3 new var() fallback values added
  by the CSS modules build; all property:value pairs are preserved
- JS changes are minified variable name shifts (O ↔ A, I ↔ L) from
  changed import order; no logic changes
- Visual comparison (pixel-by-pixel screenshots of both light and dark
  themes on the demo page) shows 0 different pixels
- Bundle sizes: remark.css -626 bytes, remark.js -512 bytes,
  last-comments.css -16 bytes (dead CSS removed)
2026-03-25 16:42:44 -05:00
Dmitry VerkhoturovandGitHub c26f45e55e Clean up deprecated CSS and fix silent CSS bugs in frontend (#2012)
* frontend: remove deprecated iframe attrs and non-standard CSS

Three separate cleanups:

1. remove deprecated HTML attributes from iframe creation (create-iframe.ts)
   - frameborder="0": deprecated since HTML5; border is already set to none via CSS
   - allowtransparency="true": non-standard Microsoft attribute never in any spec;
     transparency is handled by body { background: transparent } in CSS instead
   - scrolling="no": deprecated since HTML5; overflow is already hidden via CSS
   - horizontalscrolling/verticalscrolling: non-standard IE-era attributes with
     no effect in modern browsers; remove without replacement

2. replace allowtransparency with explicit CSS (global.css)
   - add background: transparent to body; this is the spec-correct way to make
     an iframe document transparent, as documented by MDN

3. drop -moz-touch-enabled media query prefix (5 comment CSS files)
   - -moz-touch-enabled was a Firefox-only non-standard media feature removed
     in Firefox 58 (2018); pointer: coarse is the standard equivalent and was
     already present as the second condition in every query, so removing the
     dead -moz prefix reduces the media query to just (pointer: coarse)

note: colorScheme: 'none' in create-iframe.ts is intentionally left unchanged;
it is tracked by #1430 and requires a broader color-scheme implementation

* frontend: fix CSS bugs and replace deprecated properties

Bugs fixed:

- comment-votes.module.css: add missing comma between transition values;
  without it the shorthand was invalid and colour transitions on vote
  buttons were silently ignored

- icon-button.module.css: fix "transfrom" typo (should be "transform");
  the misspelling made the transition declaration a no-op, so the hover
  scale animation jumped instantly instead of easing

- auth.module.css: remove doubly-nested rgb(rgb(var(…))) call; the outer
  rgb() rejected the inner rgb() result, so the .title element's colour
  fell back to inherited instead of the intended --secondary-text-color

Deprecated properties replaced:

- comment-form__markdown-toolbar.css: replace deprecated clip: rect()
  with clip-path: inset(50%); clip was deprecated in CSS Masking Level 1

- raw-content.css: replace word-wrap with overflow-wrap; word-wrap was
  renamed in CSS Text Level 3, all current browsers support overflow-wrap

- global.css: remove redundant literal-colour fallback lines before
  var() declarations in .preloader and .preloader_view_iframe; the var()
  calls already have inline fallback values (e.g. var(--color6, #fff)),
  making the preceding duplicate property and its stylelint-disable
  comment unnecessary since IE11 EOL

* move border:none from inline style to widget__comments-frame class
2026-03-25 16:42:40 -05:00
Dmitry VerkhoturovandGitHub ba7c3aed94 refactor: modernise Go code with go fix and manual improvements (#2027)
Apply go fix ./... analysers (Go 1.26) across backend and examples:
- interface{} → any (type alias, no behaviour change)
- for i := 0; i < N; i++ → for range N / for i := range N
- slices.Contains / slices.ContainsFunc replacing manual loops
- strings.SplitSeq replacing strings.Split in range (avoids allocation)
- strings.CutPrefix replacing HasPrefix+TrimPrefix
- min() replacing manual if/else
- fmt.Appendf replacing []byte(fmt.Sprintf(...))
- strings.Builder replacing string += concatenation
- wg.Go(func(){}) replacing wg.Add(1)/go/wg.Done() pattern
- removed redundant ii := i loop variable copies (unnecessary since Go 1.22)

omitempty on struct-typed JSON fields: go fix removed omitempty from
struct-typed fields (time.Time, PostInfo, UserDetailEntry) because
encoding/json's omitempty never applied to struct types — it was always
a no-op. Kept as bare tags (no omitzero replacement) to preserve the
existing serialisation behaviour.
2026-03-25 16:42:37 -05:00
UmputunandGitHub 8aafc8fcd7 Merge pull request #2020 from paskal/ci/add-pnpm-cache
ci: add node dependency caching
2026-03-16 01:38:00 -05:00
Dmitry Verkhoturov ab9e6675cf fix type check failure in @remark42/api package
Add skipLibCheck to skip type checking of .d.ts files in node_modules,
matching the setting already used by the main remark42 app. Fixes
@types/eslint-scope vs @types/eslint type incompatibility.
2026-03-07 21:37:46 +00:00
Dmitry Verkhoturov ed67390dea ci: add pnpm dependency caching via setup-node
Replace manual actions/cache steps with built-in setup-node cache support.
Add cache: pnpm and cache-dependency-path to all setup-node steps in both
ci-frontend.yml and ci-frontend-api.yml. Move pnpm install before setup-node
as required for pnpm caching to work.
2026-03-07 21:22:25 +00:00
Umputun aca0cff399 fix: IPv6 address truncation and image proxy SSRF vulnerabilities
Replace strings.Split(RemoteAddr, ":") with net.SplitHostPort for correct
IPv6 address extraction in vote deduplication and comment IP tracking.

Harden image proxy: add SSRF-safe transport blocking private/reserved IPs
at connection time with DNS rebinding protection, sanitize error messages
to prevent information leakage, add response size limit via io.LimitReader.

Fix shadowed error variables in BlockedUsers, SetTitle, and Delete methods.
Exclude gosec taint analysis false positives at linter config level.
2026-02-28 04:13:07 -06:00
Dmitry VerkhoturovandUmputun f359256489 docs: document placeholder support in the remark42 div (#1990)
Clarify that any content placed inside the `<div id="remark42">` is
automatically removed once the iframe signals it has initialised.
Update all code examples across getting-started, frontend config, and
Astro/Gatsby integration guides to use "Comments loading..." as the
placeholder so the feature is visible by default.
2026-02-22 17:54:43 -06:00
Dmitry VerkhoturovandUmputun 336f17e7b7 Document EDIT_TIME=0 behavior in parameters
Setting edit-time to 0 disables both comment editing and staged image cleanup.
2026-02-22 17:54:19 -06:00
Dmitry VerkhoturovandUmputun 0105bc2314 Drop GitHub token permissions on deploy jobs
Deploy jobs only curl an external updater URL and need no GitHub API
access. Without an explicit permissions block they inherit the workflow
default, which may include contents:write, packages:write, etc.
Setting permissions to {} limits the blast radius if a job is
compromised.
2026-02-21 20:16:00 -06:00
Dmitry VerkhoturovandUmputun 78d6de6bce Add X-Content-Type-Options and Referrer-Policy security headers
Add two missing security headers to the existing securityHeadersMiddleware:

- X-Content-Type-Options: nosniff — prevents browsers from MIME-sniffing
  responses away from the declared Content-Type, stopping e.g. a
  user-uploaded image from being reinterpreted as executable HTML/JS

- Referrer-Policy: strict-origin-when-cross-origin — limits URL information
  leaked in the Referer header on cross-origin requests to just the origin
  (no path), and sends nothing at all on HTTPS-to-HTTP downgrades
2026-02-21 20:14:44 -06:00
dependabot[bot]GitHubpaskaldependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
638fa63e81 Bump the go-modules-updates group in /backend with 7 updates (#1995)
* Bump the go-modules-updates group in /backend with 7 updates

Bumps the go-modules-updates group in /backend with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.21.1` | `2.23.1` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.2.3` | `5.2.4` |
| [github.com/go-pkgz/rest](https://github.com/go-pkgz/rest) | `1.20.6` | `1.21.0` |
| [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) | `5.3.0` | `5.3.1` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.46.0` | `0.47.0` |
| [golang.org/x/image](https://github.com/golang/image) | `0.34.0` | `0.35.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.48.0` | `0.49.0` |


Updates `github.com/alecthomas/chroma/v2` from 2.21.1 to 2.23.1
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.21.1...v2.23.1)

Updates `github.com/go-chi/chi/v5` from 5.2.3 to 5.2.4
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.3...v5.2.4)

Updates `github.com/go-pkgz/rest` from 1.20.6 to 1.21.0
- [Release notes](https://github.com/go-pkgz/rest/releases)
- [Commits](https://github.com/go-pkgz/rest/compare/v1.20.6...v1.21.0)

Updates `github.com/golang-jwt/jwt/v5` from 5.3.0 to 5.3.1
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.3.0...v5.3.1)

Updates `golang.org/x/crypto` from 0.46.0 to 0.47.0
- [Commits](https://github.com/golang/crypto/compare/v0.46.0...v0.47.0)

Updates `golang.org/x/image` from 0.34.0 to 0.35.0
- [Commits](https://github.com/golang/image/compare/v0.34.0...v0.35.0)

Updates `golang.org/x/net` from 0.48.0 to 0.49.0
- [Commits](https://github.com/golang/net/compare/v0.48.0...v0.49.0)

---
updated-dependencies:
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-version: 2.23.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.2.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/rest
  dependency-version: 1.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-version: 5.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-version: 0.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-version: 0.35.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-version: 0.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>

* Run go mod tidy in examples directory

Co-authored-by: paskal <712534+paskal@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: paskal <712534+paskal@users.noreply.github.com>
2026-02-14 19:48:24 -06:00
UmputunandGitHub e3b0d63648 Merge pull request #1999 from umputun/configurable-microsoft-tenant
feat: make Microsoft Entra ID tenant configurable
2026-02-10 22:53:31 -06:00
UmputunandGitHub b38d91cb0f Merge pull request #2000 from umputun/fix/quick-fixes-1946-1991-1996
Fix email encoding, image cleanup CPU spin, and demo template paths
2026-02-10 22:52:40 -06:00
UmputunandGitHub d6d53ff2e0 Merge pull request #2001 from umputun/fix/admin-edit-frontend-1986
Fix frontend not respecting ADMIN_EDIT config
2026-02-10 22:51:32 -06:00
UmputunandGitHub 195becc6ee Merge pull request #2002 from umputun/fix/placeholder-clearing-1990
Clear user placeholder content when comments iframe loads
2026-02-10 22:50:51 -06:00
UmputunandGitHub 5bc5167a31 Merge pull request #2003 from umputun/docs/email-template-variables
Document email template variables and plain-text email setup
2026-02-10 22:50:20 -06:00
UmputunandGitHub 1320b1f055 Merge pull request #1984 from umputun/dependabot/github_actions/github-actions-updates-35b2a8182b
Bump the github-actions-updates group with 3 updates
2026-02-10 22:40:25 -06:00
UmputunandGitHub 283e2c19c7 Merge pull request #1994 from umputun/dependabot/npm_and_yarn/frontend/lodash-es-4.17.23
Bump lodash-es from 4.17.21 to 4.17.23 in /frontend
2026-02-10 22:40:18 -06:00
UmputunandGitHub 55d9e22373 Merge pull request #1997 from umputun/dependabot/npm_and_yarn/frontend/webpack-5.104.1
Bump webpack from 5.73.0 to 5.104.1 in /frontend
2026-02-10 22:40:09 -06:00
Dmitry Verkhoturov 31e20fc26d feat: make Microsoft Entra ID tenant configurable
Add AUTH_MICROSOFT_TENANT env var to allow configuring the Azure AD
tenant for single-tenant Entra ID applications, which cannot use the
default /common endpoint.

Depends on go-pkgz/auth#266

Closes #1998
2026-02-11 00:45:38 +00:00
Dmitry Verkhoturov c2cc2305c1 Document email template variables and plain-text email setup 2026-02-11 00:08:05 +00:00
Dmitry Verkhoturov 4d0bd29b45 Clear placeholder content when comments iframe loads
Remove non-iframe child nodes from the root element once the
iframe signals it has initialised, allowing users to add
loading placeholders that get cleaned up automatically. Fixes #1990
2026-02-10 23:52:22 +00:00
Dmitry Verkhoturov a1215d87d9 Fix frontend not respecting ADMIN_EDIT for comment editing
Add admin_edit field to frontend Config types and use it in
comment component to give admins unlimited edit time and allow
editing comments with replies. Hide countdown timer when
editDeadline is Infinity. Fixes #1986
2026-02-10 23:52:17 +00:00
Dmitry Verkhoturov 3c2679a1d5 Fix hardcoded /web paths in demo template
Use REMARK_URL template variable for widget links so demo page
works with non-root path prefixes. Fixes #1996
2026-02-10 23:52:12 +00:00
Dmitry Verkhoturov 79177e52f9 Fix 100% CPU when EDIT_TIME=0 in image cleanup
When EditDuration is zero or negative, cleanupTTL becomes zero,
causing time.After(0) to fire immediately in a tight loop.
Block on ctx.Done() instead when edit duration is disabled. Fixes #1991
2026-02-10 23:52:12 +00:00
Dmitry Verkhoturov baa615a0d1 Fix NOTIFY_EMAIL_FROM plus sign encoding in mailto URLs
URL-encode e.From in mailto query parameters so that + characters
are preserved instead of being decoded as spaces. Fixes #1946
2026-02-10 23:52:12 +00:00
dependabot[bot]andGitHub e665dcf9e2 Bump webpack from 5.73.0 to 5.104.1 in /frontend
Bumps [webpack](https://github.com/webpack/webpack) from 5.73.0 to 5.104.1.
- [Release notes](https://github.com/webpack/webpack/releases)
- [Changelog](https://github.com/webpack/webpack/blob/main/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack/compare/v5.73.0...v5.104.1)

---
updated-dependencies:
- dependency-name: webpack
  dependency-version: 5.104.1
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-06 23:31:33 +00:00
Dmitry VerkhoturovandUmputun b7a13a6636 Fix site rebuild on release
The paths filter was applied to tag events, preventing site rebuilds
when releases don't include site changes. Switch to release event
trigger which always fires on new releases, ensuring the site fetches
the latest version from GitHub API.

Closes #1992
2026-02-05 11:06:58 -06:00
dependabot[bot]andGitHub 218570cfad Bump lodash-es from 4.17.21 to 4.17.23 in /frontend
Bumps [lodash-es](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.21...4.17.23)

---
updated-dependencies:
- dependency-name: lodash-es
  dependency-version: 4.17.23
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-21 23:06:53 +00:00
dependabot[bot]andGitHub 4c9a791d6d Bump the github-actions-updates group with 3 updates
Bumps the github-actions-updates group with 3 updates: [actions/cache](https://github.com/actions/cache), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/cache` from 4 to 5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

Updates `actions/upload-artifact` from 5 to 6
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v5...v6)

Updates `actions/download-artifact` from 6 to 7
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-01-01 11:09:55 +00:00
Umputun cdad560df3 Update backend base image to buildgo-v1.17.0 in Dockerfile for artifacts build 2025-12-24 02:55:38 -06:00
939 changed files with 87641 additions and 105259 deletions
+3 -3
View File
@@ -25,7 +25,7 @@ jobs:
contents: read
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
with:
persist-credentials: false
@@ -60,13 +60,13 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: "v2.6.0"
version: "v2.10.1"
working-directory: backend/app
- name: golangci-lint on example directory
uses: golangci/golangci-lint-action@v9
with:
version: "v2.6.0"
version: "v2.10.1"
args: --config ../../.golangci.yml
working-directory: backend/_example/memory_store
+3 -3
View File
@@ -23,15 +23,15 @@ jobs:
contents: read
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: expose GitHub Actions cache
uses: actions/cache@v4
uses: actions/cache@v6
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
+31 -64
View File
@@ -22,37 +22,26 @@ jobs:
contents: read
strategy:
matrix:
node: [ 16 ]
node: [ 20 ]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -69,37 +58,26 @@ jobs:
contents: read
strategy:
matrix:
node: [ 16 ]
node: [ 20 ]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -116,37 +94,26 @@ jobs:
contents: read
strategy:
matrix:
node: [ 16 ]
node: [ 20 ]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -157,7 +124,7 @@ jobs:
working-directory: ./frontend
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v7
with:
token: ${{ secrets.CODECOV_TOKEN }}
working-directory: ./frontend
+44 -89
View File
@@ -22,37 +22,26 @@ jobs:
contents: read
strategy:
matrix:
node: [16]
node: [20]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -69,37 +58,26 @@ jobs:
contents: read
strategy:
matrix:
node: [16]
node: [20]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -116,37 +94,26 @@ jobs:
contents: read
strategy:
matrix:
node: [16]
node: [20]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -167,15 +134,14 @@ jobs:
CI_JOB_NUMBER: 1
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
uses: pnpm/action-setup@v6.0.9
with:
version: 8
version: 10.10.0
run_install: false
- name: Check bundle size
@@ -192,37 +158,26 @@ jobs:
contents: read
strategy:
matrix:
node: [16]
node: [20]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: Install node
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v4
with:
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: Install dependencies
run: pnpm i
@@ -233,7 +188,7 @@ jobs:
working-directory: ./frontend/apps/remark42
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v7
with:
token: ${{ secrets.CODECOV_TOKEN }}
working-directory: ./frontend/apps/remark42
+12 -10
View File
@@ -1,10 +1,11 @@
name: site
on:
release:
types: [published]
push:
branches:
- master
tags:
paths:
- ".github/workflows/ci-site.yml"
- "site/**"
@@ -38,15 +39,15 @@ jobs:
steps:
- name: checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -54,7 +55,7 @@ jobs:
- name: build and push by digest
id: build
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: ./site
platforms: ${{ matrix.platform }}
@@ -69,7 +70,7 @@ jobs:
touch "/tmp/digests/${digest#sha256:}"
- name: upload digest
uses: actions/upload-artifact@v5
uses: actions/upload-artifact@v7
with:
name: site-digests-${{ matrix.artifact }}
path: /tmp/digests/*
@@ -85,7 +86,7 @@ jobs:
steps:
- name: download digests
uses: actions/download-artifact@v6
uses: actions/download-artifact@v8
with:
path: /tmp/digests
pattern: site-digests-*
@@ -103,10 +104,10 @@ jobs:
echo "All $expected digests present"
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
@@ -133,7 +134,8 @@ jobs:
name: Deploy site
runs-on: ubuntu-latest
needs: merge
if: github.ref == 'refs/heads/master'
if: github.ref == 'refs/heads/master' || github.event_name == 'release'
permissions: {} # only calls an external URL via curl, no GitHub API access needed
steps:
- name: trigger deployment
+14 -13
View File
@@ -34,23 +34,23 @@ jobs:
steps:
- name: checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
ref: ${{ github.event.workflow_run.head_sha }}
persist-credentials: false
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: login to DockerHub
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: umputun
password: ${{ secrets.DOCKER_HUB_TOKEN }}
@@ -64,7 +64,7 @@ jobs:
- name: build and push to ghcr.io by digest
id: build-ghcr
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: .
platforms: ${{ matrix.platform }}
@@ -81,7 +81,7 @@ jobs:
- name: build and push to DockerHub by digest
id: build-dockerhub
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: .
platforms: ${{ matrix.platform }}
@@ -104,14 +104,14 @@ jobs:
touch "/tmp/digests/dockerhub/${digest_dockerhub#sha256:}"
- name: upload ghcr digest
uses: actions/upload-artifact@v5
uses: actions/upload-artifact@v7
with:
name: digests-ghcr-${{ matrix.artifact }}
path: /tmp/digests/ghcr/*
retention-days: 1
- name: upload dockerhub digest
uses: actions/upload-artifact@v5
uses: actions/upload-artifact@v7
with:
name: digests-dockerhub-${{ matrix.artifact }}
path: /tmp/digests/dockerhub/*
@@ -127,14 +127,14 @@ jobs:
steps:
- name: download ghcr digests
uses: actions/download-artifact@v6
uses: actions/download-artifact@v8
with:
path: /tmp/digests/ghcr
pattern: digests-ghcr-*
merge-multiple: true
- name: download dockerhub digests
uses: actions/download-artifact@v6
uses: actions/download-artifact@v8
with:
path: /tmp/digests/dockerhub
pattern: digests-dockerhub-*
@@ -154,17 +154,17 @@ jobs:
echo "All digests present for both registries"
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: login to ghcr.io
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: login to DockerHub
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: umputun
password: ${{ secrets.DOCKER_HUB_TOKEN }}
@@ -206,6 +206,7 @@ jobs:
runs-on: ubuntu-latest
needs: merge
if: github.event.workflow_run.head_branch == 'master'
permissions: {} # only calls an external URL via curl, no GitHub API access needed
steps:
- name: trigger deployment
+2 -2
View File
@@ -26,7 +26,7 @@ jobs:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
persist-credentials: false
@@ -34,7 +34,7 @@ jobs:
id: tests
run: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -f compose-e2e-test.yml up --build --quiet-pull --exit-code-from tests
- uses: actions/upload-artifact@v5
- uses: actions/upload-artifact@v7
if: always()
with:
name: playwright-report
+143
View File
@@ -0,0 +1,143 @@
name: release
on:
push:
tags:
- "v*"
pull_request:
paths:
- ".github/workflows/release.yml"
- ".goreleaser.yml"
- "Makefile"
- "scripts/**"
- "backend/**"
- "frontend/**"
- "README.md"
- "LICENSE"
- "CLAUDE.md"
- "site/src/docs/getting-started/installation/index.md"
permissions:
contents: read
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- name: install go
uses: actions/setup-go@v6
with:
go-version: "1.25"
cache-dependency-path: backend/go.sum
- name: install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: install node
uses: actions/setup-node@v6
with:
node-version: 20
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: test and build backend
run: |
go test -race -timeout=120s ./...
go build -race ./...
working-directory: backend/app
env:
TZ: "America/Chicago"
- name: test examples
run: |
go test -race ./...
go build -race ./...
working-directory: backend/_example/memory_store
env:
TZ: "America/Chicago"
- name: install frontend dependencies
run: pnpm install --frozen-lockfile
working-directory: frontend
env:
CI: "true"
- name: check frontend
run: |
pnpm lint
pnpm type-check
pnpm test --runInBand
working-directory: frontend/apps/remark42
env:
CI: "true"
- name: check goreleaser snapshot
if: github.event_name == 'pull_request'
uses: goreleaser/goreleaser-action@v7
with:
version: latest
args: release --snapshot --clean --skip=publish
env:
SKIP_PNPM_INSTALL: "true"
- name: clean generated release assets
if: always()
run: ./scripts/cleanup-release-assets.sh
release:
if: github.event_name == 'push'
needs: validate
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
persist-credentials: false
- name: install go
uses: actions/setup-go@v6
with:
go-version: "1.25"
cache-dependency-path: backend/go.sum
- name: install pnpm
uses: pnpm/action-setup@v6.0.9
with:
version: 10.10.0
run_install: false
- name: install node
uses: actions/setup-node@v6
with:
node-version: 20
cache: "pnpm"
cache-dependency-path: frontend/pnpm-lock.yaml
- name: install frontend dependencies
run: pnpm install --frozen-lockfile
working-directory: frontend
env:
CI: "true"
- name: run goreleaser
uses: goreleaser/goreleaser-action@v7
with:
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SKIP_PNPM_INSTALL: "true"
- name: clean generated release assets
if: always()
run: ./scripts/cleanup-release-assets.sh
+4
View File
@@ -15,6 +15,7 @@ debug.test
.mongo
remark42
/bin/
/dist/
/backend/var/
/backend/app/var/
/backend/app/cmd/web/
@@ -26,3 +27,6 @@ compose-private.yml
http-client.env.json
/playwright-report/
/backend/app/cmd/var
# ralphex progress logs
.ralphex/progress/
+69
View File
@@ -0,0 +1,69 @@
version: 2
project_name: remark42
git:
ignore_tags:
- backend/*
before:
hooks:
- ./scripts/prepare-release-assets.sh
builds:
- id: remark42
dir: backend
main: ./app
binary: "remark42.{{ .Os }}-{{ .Arch }}"
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
- freebsd
- windows
goarch:
- amd64
- arm64
- "386"
- arm
goarm:
- "7"
ignore:
- goos: darwin
goarch: "386"
- goos: darwin
goarch: arm
- goos: freebsd
goarch: arm64
- goos: freebsd
goarch: "386"
- goos: freebsd
goarch: arm
- goos: windows
goarch: arm64
- goos: windows
goarch: "386"
- goos: windows
goarch: arm
ldflags:
- -s -w -X main.revision={{ .Tag }}-{{ .ShortCommit }}-{{ trimsuffix (replace (replace .CommitDate "-" "") ":" "") "Z" }}
archives:
- id: remark42
ids:
- remark42
name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}"
formats:
- tar.gz
format_overrides:
- goos: windows
formats:
- zip
files:
- LICENSE
- README.md
release:
name_template: "Version {{ .Version }}"
mode: keep-existing
+25 -2
View File
@@ -17,15 +17,38 @@
- **IMPORTANT**: Example lint: `cd backend/_example/memory_store && golangci-lint run --config ../../.golangci.yml`
- Frontend: `cd frontend && pnpm lint`
- **Before committing**: Always run tests and linter on both main backend AND examples
- **Dependency Updates**:
- When updating Go modules in `backend/`, also run `go mod tidy` (and `go mod vendor`) in `backend/_example/memory_store` to keep indirect deps in sync. The example module replaces `github.com/umputun/remark42/backend` with `../../` so stale indirect deps there will break the example build.
## Release Procedure
Remark42 uses two tags for each release:
- `vX.Y.Z` - product release tag used by GitHub releases, GoReleaser binary artifacts, and Docker image publishing.
- `backend/vX.Y.Z` - nested Go module tag for `github.com/umputun/remark42/backend`.
Release flow:
1. Create the GitHub release for `vX.Y.Z` with title `Version X.Y.Z`. The GitHub release must exist before the `vX.Y.Z` tag reaches the remote; `gh release create vX.Y.Z` satisfies this because it creates and pushes the tag.
2. The `vX.Y.Z` tag triggers GoReleaser, which builds and uploads binary artifacts to the existing release.
3. Create and push the matching backend module tag pointing at the same commit:
```bash
git fetch origin --tags
git tag backend/vX.Y.Z vX.Y.Z
git push origin backend/vX.Y.Z
```
GoReleaser must ignore `backend/*` tags in `.goreleaser.yml` so release notes and current-tag detection use only product tags. Docker image publishing stays separate and is handled by the existing Docker workflow.
For local artifact runs, install GoReleaser, Go 1.25, Node 16+, PNPM 8, and Perl, then use `make release`. The target runs a snapshot/no-publish GoReleaser build, leaves local artifacts and metadata in `dist/`, and cleans generated frontend embed files after GoReleaser exits. Do not run raw `goreleaser release` for local artifacts unless you also run `./scripts/cleanup-release-assets.sh` afterward.
## Code Style
- **Backend**: Formatting with golangci-lint, strict error handling
- **Frontend**: TypeScript with ESLint, Stylelint and Prettier
- **Imports**: Group stdlib, external packages, then internal packages
- **CSS**: CSS Modules for new components (`component.module.css`)
- **CSS**: All components use CSS Modules (`component.module.css`). Class naming: BEM block = `.root`, elements = camelCase, modifiers = camelCase. Use `clsx` for conditional class composition. `raw-content.css` is the only global CSS file (syntax highlighting utility). Root wrapper keeps bare `.dark`/`.light` theme class — 8+ module CSS files depend on `:global(.dark)` ancestor. `comment_highlighting` uses `:global()` for imperative `classList` usage in root.tsx
## Key Backend Packages
- **Web/API**: `github.com/go-chi/chi/v5`, `github.com/go-pkgz/rest`
- **Web/API**: `github.com/go-pkgz/routegroup`, `github.com/go-pkgz/rest`
- **Auth**: `github.com/go-pkgz/auth/v2`
- **Logging**: `github.com/go-pkgz/lgr`
- **Testing**: `github.com/stretchr/testify`
+2 -2
View File
@@ -1,4 +1,4 @@
FROM --platform=$BUILDPLATFORM node:16.20-alpine AS frontend-deps
FROM --platform=$BUILDPLATFORM node:20-alpine AS frontend-deps
ARG SKIP_FRONTEND_TEST
ARG SKIP_FRONTEND_BUILD
@@ -11,7 +11,7 @@ COPY ./frontend/apps/remark42/package.json /srv/frontend/apps/remark42/
RUN \
if [[ -z "$SKIP_FRONTEND_BUILD" || -z "$SKIP_FRONTEND_TEST" ]]; then \
apk add --no-cache --update git && \
npm i -g pnpm@8; \
npm i -g pnpm@10.10.0; \
fi
RUN --mount=type=cache,id=pnpm,target=/root/.pnpm-store/v3 \
-64
View File
@@ -1,64 +0,0 @@
FROM node:16-alpine AS frontend-deps
ENV CI=true
WORKDIR /srv/frontend
COPY ./frontend/package.json ./frontend/pnpm-lock.yaml ./frontend/pnpm-workspace.yaml /srv/frontend/
COPY ./frontend/apps/remark42/package.json /srv/frontend/apps/remark42/package.json
RUN apk add --no-cache --update git && npm i -g pnpm@8
RUN --mount=type=cache,id=pnpm,target=/root/.pnpm-store/v3 pnpm i
FROM frontend-deps AS build-frontend
ENV NODE_ENV=production
ENV CI=true
WORKDIR /srv/frontend/apps/remark42/
COPY ./frontend/apps/remark42/ /srv/frontend/apps/remark42/
RUN pnpm build
FROM umputun/baseimage:buildgo-v1.14.0 AS build-backend
ARG GITHUB_TOKEN
ARG GITHUB_REF
ARG GITHUB_SHA
WORKDIR /build/backend
ADD backend /build/backend
ADD README.md /build/
ADD LICENSE /build/
COPY --from=build-frontend /srv/frontend/apps/remark42/public/ /build/backend/app/cmd/web/
RUN find /build/backend/app/cmd/web/ -regex '.*\.\(html\|js\|mjs\)$' -print -exec sed -i "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
RUN \
version=$("/script/version.sh") && echo "version=${version}" && \
GOOS=linux GOARCH=amd64 go build -o remark42.linux-amd64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=linux GOARCH=386 go build -o remark42.linux-386 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=linux GOARCH=arm go build -o remark42.linux-arm -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=linux GOARCH=arm64 go build -o remark42.linux-arm64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=windows GOARCH=amd64 go build -o remark42.windows-amd64.exe -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=darwin GOARCH=amd64 go build -o remark42.darwin-amd64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=darwin GOARCH=arm64 go build -o remark42.darwin-arm64 -ldflags "-X main.revision=${version} -s -w" ./app && \
GOOS=freebsd GOARCH=amd64 go build -o remark42.freebsd-amd64 -ldflags "-X main.revision=${version} -s -w" ./app
RUN \
apk add --no-cache --update zip && \
cp ../LICENSE ./LICENSE && cp ../README.md ./README.md && \
tar cvzf remark42.linux-amd64.tar.gz remark42.linux-amd64 LICENSE README.md && \
tar cvzf remark42.linux-386.tar.gz remark42.linux-386 LICENSE README.md && \
tar cvzf remark42.linux-arm.tar.gz remark42.linux-arm LICENSE README.md && \
tar cvzf remark42.linux-arm64.tar.gz remark42.linux-arm64 LICENSE README.md && \
tar cvzf remark42.darwin-amd64.tar.gz remark42.darwin-amd64 LICENSE README.md && \
tar cvzf remark42.darwin-arm64.tar.gz remark42.darwin-arm64 LICENSE README.md && \
tar cvzf remark42.freebsd-amd64.tar.gz remark42.freebsd-amd64 LICENSE README.md && \
zip remark42.windows-amd64.zip remark42.windows-amd64.exe LICENSE README.md
FROM alpine
COPY --from=build-backend /build/backend/remark42.* /artifacts/
RUN ls -la /artifacts/*
CMD ["sleep", "100"]
+9 -19
View File
@@ -2,13 +2,13 @@ OS=linux
ARCH=amd64
GITHUB_REF=$(shell git rev-parse --symbolic-full-name HEAD)
GITHUB_SHA=$(shell git rev-parse --short HEAD)
CLEANUP_RELEASE_ASSETS=$(CURDIR)/scripts/cleanup-release-assets.sh
bin:
docker build -f Dockerfile.artifacts -t remark42.bin .
- @docker rm -f remark42.bin 2>/dev/null || exit 0
docker run -d --name=remark42.bin remark42.bin
docker cp remark42.bin:/artifacts/remark42.$(OS)-$(ARCH) remark42
docker rm -f remark42.bin
@set -e; \
./scripts/prepare-release-assets.sh; \
trap '$(CLEANUP_RELEASE_ASSETS)' EXIT; \
cd backend && CGO_ENABLED=0 GOOS=$(OS) GOARCH=$(ARCH) go build -o ../remark42 -ldflags "-X main.revision=$(GITHUB_REF)-$(GITHUB_SHA) -s -w" ./app
docker:
DOCKER_BUILDKIT=1 docker build -t umputun/remark42 -t ghcr.io/umputun/remark42 --build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) \
@@ -21,19 +21,9 @@ dockerx:
-t ghcr.io/umputun/remark42:master -t umputun/remark42:master .
release:
docker build -f Dockerfile.artifacts --no-cache --pull --build-arg CI=true \
--build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) -t remark42.bin .
- @docker rm -f remark42.bin 2>/dev/null || exit 0
- @mkdir -p bin
docker run -d --name=remark42.bin remark42.bin
docker cp remark42.bin:/artifacts/remark42.linux-amd64.tar.gz bin/remark42.linux-amd64.tar.gz
docker cp remark42.bin:/artifacts/remark42.linux-386.tar.gz bin/remark42.linux-386.tar.gz
docker cp remark42.bin:/artifacts/remark42.linux-arm64.tar.gz bin/remark42.linux-arm64.tar.gz
docker cp remark42.bin:/artifacts/remark42.darwin-amd64.tar.gz bin/remark42.darwin-amd64.tar.gz
docker cp remark42.bin:/artifacts/remark42.darwin-arm64.tar.gz bin/remark42.darwin-arm64.tar.gz
docker cp remark42.bin:/artifacts/remark42.freebsd-amd64.tar.gz bin/remark42.freebsd-amd64.tar.gz
docker cp remark42.bin:/artifacts/remark42.windows-amd64.zip bin/remark42.windows-amd64.zip
docker rm -f remark42.bin
@set -e; \
trap '$(CLEANUP_RELEASE_ASSETS)' EXIT; \
goreleaser release --snapshot --clean --skip=publish
race_test:
cd backend/app && go test -race -timeout=60s -count 1 ./...
@@ -52,4 +42,4 @@ rundev:
e2e:
docker compose -f compose-e2e-test.yml up --build --quiet-pull --exit-code-from tests
.PHONY: bin backend
.PHONY: bin docker dockerx release race_test backend frontend rundev e2e
+1 -1
View File
@@ -2,7 +2,7 @@
Remark42 is a self-hosted, lightweight and simple (yet functional) comment engine, which doesn't spy on users. It can be embedded into blogs, articles, or any other place where readers add comments.
* Social login via Google, Facebook, Microsoft, GitHub, Apple, Yandex, Patreon, Discord and Telegram
* Social login via Google, Facebook, Microsoft, GitHub, Apple, Yandex, Patreon, Discord, Telegram and custom OAuth2 providers
* Login via email
* Optional anonymous access
* Multi-level nested comments with both tree and plain presentations
+1 -1
View File
@@ -12,4 +12,4 @@ We release patches for security vulnerabilities.
## Reporting a Vulnerability
Please report (suspected) security vulnerabilities to umputun@gmail.com. You will receive a response from us within 48 hours. If the issue is confirmed, we will release a patch as soon as possible depending on complexity but historically within a few days.
Please report (suspected) security vulnerabilities either by using GitHub's [private vulnerability reporting](https://github.com/umputun/remark42/security/advisories/new) (click the "Report a vulnerability" button on the [Security tab](https://github.com/umputun/remark42/security)) or by emailing umputun@gmail.com. You will receive a response within 48 hours. If the issue is confirmed, we will release a patch as soon as possible depending on complexity but historically within a few days.
+6
View File
@@ -23,6 +23,9 @@ linters:
goconst:
min-len: 2
min-occurrences: 2
gosec:
excludes:
- G117 # false positive: struct field name matches "secret" pattern
gocritic:
disabled-checks:
- wrapperFunc
@@ -51,6 +54,9 @@ linters:
- linters:
- revive
text: 'var-naming: avoid meaningless package names'
- linters:
- revive
text: 'var-naming: avoid package names that conflict with Go standard library package names'
- linters:
- dupl
- gosec
@@ -251,11 +251,11 @@ func (m *MemData) Flag(req engine.FlagRequest) (val bool, err error) {
// ListFlags get list of flagged keys, like blocked & verified user
// works for full locator (post flags) or with userID
func (m *MemData) ListFlags(req engine.FlagRequest) (res []interface{}, err error) {
func (m *MemData) ListFlags(req engine.FlagRequest) (res []any, err error) {
m.mu.RLock()
defer m.mu.RUnlock()
res = []interface{}{}
res = []any{}
switch req.Flag {
case engine.Verified:
@@ -10,6 +10,7 @@ import (
"fmt"
"sort"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -198,7 +199,7 @@ func TestMemData_FindForUserPagination(t *testing.T) {
}
// write 200 comments
for i := 0; i < 200; i++ {
for i := range 200 {
c.ID = fmt.Sprintf("idd-%d", i)
c.Text = fmt.Sprintf("text #%d", i)
c.Timestamp = time.Date(2017, 12, 20, 15, 18, i, 0, time.Local)
@@ -484,7 +485,7 @@ func TestMemData_FlagVerified(t *testing.T) {
func TestMemData_FlagListVerified(t *testing.T) {
b := prepMem(t)
toIDs := func(inp []interface{}) (res []string) {
toIDs := func(inp []any) (res []string) {
res = make([]string, len(inp))
for i, v := range inp {
vv, ok := v.(string)
@@ -521,51 +522,52 @@ func TestMemData_FlagListVerified(t *testing.T) {
}
func TestMemData_FlagListBlocked(t *testing.T) {
b := prepMem(t)
setBlocked := func(site, user string, status engine.FlagStatus, ttl time.Duration) error {
req := engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: site}, UserID: user, Update: status,
TTL: ttl}
_, err := b.Flag(req)
return err
}
toBlocked := func(inp []interface{}) (res []store.BlockedUser) {
res = make([]store.BlockedUser, len(inp))
for i, v := range inp {
vv, ok := v.(store.BlockedUser)
require.True(t, ok)
res[i] = vv
synctest.Test(t, func(t *testing.T) {
b := prepMem(t)
setBlocked := func(site, user string, status engine.FlagStatus, ttl time.Duration) error {
req := engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: site}, UserID: user, Update: status,
TTL: ttl}
_, err := b.Flag(req)
return err
}
return res
}
assert.NoError(t, setBlocked("radio-t", "user1", engine.FlagTrue, 0))
assert.NoError(t, setBlocked("radio-t", "user2", engine.FlagTrue, 50*time.Millisecond))
assert.NoError(t, setBlocked("radio-t", "user3", engine.FlagFalse, 0))
vv, err := b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
toBlocked := func(inp []any) (res []store.BlockedUser) {
res = make([]store.BlockedUser, len(inp))
for i, v := range inp {
vv, ok := v.(store.BlockedUser)
require.True(t, ok)
res[i] = vv
}
return res
}
assert.NoError(t, setBlocked("radio-t", "user1", engine.FlagTrue, 0))
assert.NoError(t, setBlocked("radio-t", "user2", engine.FlagTrue, 50*time.Millisecond))
assert.NoError(t, setBlocked("radio-t", "user3", engine.FlagFalse, 0))
blockedList := toBlocked(vv)
var blockedIDs = make([]string, len(blockedList))
for i, x := range blockedList {
blockedIDs[i] = x.ID
}
require.Equal(t, 2, len(blockedList), b.metaUsers)
assert.ElementsMatch(t, []string{"user1", "user2"}, blockedIDs)
t.Logf("%+v", blockedList)
vv, err := b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
// check block expiration
time.Sleep(50 * time.Millisecond)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
blockedList = toBlocked(vv)
require.Equal(t, 1, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
blockedList := toBlocked(vv)
var blockedIDs = make([]string, len(blockedList))
for i, x := range blockedList {
blockedIDs[i] = x.ID
}
require.Equal(t, 2, len(blockedList), b.metaUsers)
assert.ElementsMatch(t, []string{"user1", "user2"}, blockedIDs)
t.Logf("%+v", blockedList)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "bad"}})
assert.NoError(t, err)
assert.Equal(t, 0, len(vv))
// check block expiration
time.Sleep(50 * time.Millisecond)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
blockedList = toBlocked(vv)
require.Equal(t, 1, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
vv, err = b.ListFlags(engine.FlagRequest{Flag: engine.Blocked, Locator: store.Locator{SiteID: "bad"}})
assert.NoError(t, err)
assert.Equal(t, 0, len(vv))
})
}
func TestMemData_DeleteComment(t *testing.T) {
+12 -12
View File
@@ -1,10 +1,10 @@
module github.com/umputun/remark42/memory_store
go 1.25
go 1.25.0
require (
github.com/go-pkgz/jrpc v0.4.0
github.com/go-pkgz/lgr v0.12.1
github.com/go-pkgz/lgr v0.12.3
github.com/jessevdk/go-flags v1.6.1
github.com/stretchr/testify v1.11.1
github.com/umputun/remark42/backend v1.1000.0
@@ -12,13 +12,13 @@ require (
require (
github.com/Depado/bfchroma/v2 v2.0.0 // indirect
github.com/PuerkitoBio/goquery v1.11.0 // indirect
github.com/alecthomas/chroma/v2 v2.21.1 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/PuerkitoBio/goquery v1.12.0 // indirect
github.com/alecthomas/chroma/v2 v2.27.0 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/go-pkgz/rest v1.20.6 // indirect
github.com/dlclark/regexp2/v2 v2.2.2 // indirect
github.com/go-pkgz/rest v1.22.0 // indirect
github.com/go-pkgz/routegroup v1.6.0 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
@@ -29,11 +29,11 @@ require (
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
go.etcd.io/bbolt v1.4.3 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/image v0.34.0 // indirect
golang.org/x/net v0.48.0 // indirect
golang.org/x/sys v0.39.0 // indirect
go.etcd.io/bbolt v1.5.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/image v0.43.0 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/sys v0.46.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
+24 -89
View File
@@ -1,31 +1,30 @@
github.com/Depado/bfchroma/v2 v2.0.0 h1:IRpN9BPkNwEpR6w1ectIcNWOuhDSLx+8f1pn83fzxx8=
github.com/Depado/bfchroma/v2 v2.0.0/go.mod h1:wFwW/Pw8Tnd0irzgO9Zxtxgzp3aPS8qBWlyadxujxmw=
github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw=
github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ=
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/chroma/v2 v2.21.1 h1:FaSDrp6N+3pphkNKU6HPCiYLgm8dbe5UXIXcoBhZSWA=
github.com/alecthomas/chroma/v2 v2.21.1/go.mod h1:NqVhfBR0lte5Ouh3DcthuUCTUpDC9cxBOfyMbMQPs3o=
github.com/alecthomas/chroma/v2 v2.27.0 h1:FodwmyOBgJULFYmDqibcp9pvfDLWdtPRh9v/r5BXYZs=
github.com/alecthomas/chroma/v2 v2.27.0/go.mod h1:NjJ3ciIgrqBNeIkWZ4e46nseoLDslxU1LmfCoL+wcY8=
github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dlclark/regexp2/v2 v2.2.2 h1:MYWvNYw8okuqNhwTYO587EZMiDruVa2vhV6fsGpfya0=
github.com/dlclark/regexp2/v2 v2.2.2/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU=
github.com/go-pkgz/jrpc v0.4.0 h1:oD7xiGrzDkndkuCjeHGugQXxbggLSV7O1QmHhoc5pYY=
github.com/go-pkgz/jrpc v0.4.0/go.mod h1:JFoY3bRjRyx4M3CbEVDFQStMB1m2gmQ7OjqFK7q3kOo=
github.com/go-pkgz/lgr v0.12.1 h1:8GVfG2rSARq3Eaj5PP158rtBR2LHVGkwioIkQBGbvKg=
github.com/go-pkgz/lgr v0.12.1/go.mod h1:A4AxjOthFVFK6jRnVYMeusno5SeDAxcLVHd0kI/lN/Y=
github.com/go-pkgz/rest v1.20.6 h1:O/IhQ3I2cS4bJYvL1TLcy63w2OcXZTTBG3R+wTIqPS4=
github.com/go-pkgz/rest v1.20.6/go.mod h1:NY+MX1is2kJckJt+nHDNovS/5j9jmF4yQuSno4qg7XU=
github.com/go-pkgz/lgr v0.12.3 h1:QDug7kRkEsuQtruT9fNF5PVT2kZUqCDPc4GmsgS3fP8=
github.com/go-pkgz/lgr v0.12.3/go.mod h1:lpCDgVvCIxBHZp8+sGCj9MPctIzKZyZ3QdE19ddqd54=
github.com/go-pkgz/rest v1.22.0 h1:d3XFKlmAGBiU9MQER9/n46iXpyUr8IQUtfjU8JlqkkY=
github.com/go-pkgz/rest v1.22.0/go.mod h1:+AHzjHazq7Z3Tk/kRWOhbbAz/YZlUV40feC1Hf4NtbE=
github.com/go-pkgz/routegroup v1.6.0 h1:44XHZgF6JIIldRlv+zjg6SygULASmjifnfIQjwCT0e4=
github.com/go-pkgz/routegroup v1.6.0/go.mod h1:Pmu04fhgWhRtBMIJ8HXppnnzOPjnL/IEPBIdO2zmeqg=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=
github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
@@ -53,82 +52,18 @@ github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/image v0.34.0 h1:33gCkyw9hmwbZJeZkct8XyR11yH889EQt/QH4VmXMn8=
golang.org/x/image v0.34.0/go.mod h1:2RNFBZRB+vnwwFil8GkMdRvrJOFd1AzdZI6vOY+eJVU=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/image v0.43.0 h1:FLxcP4ec2350nTfOC8ysKtqYSIFbk/QGjw1ZHNP4tsY=
golang.org/x/image v0.43.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
@@ -73,7 +73,7 @@ func (s *RPC) admEnabledHndl(id uint64, params json.RawMessage) (rr jrpc.Respons
// onEvent returns nothing, callback to OnEvent
func (s *RPC) admEventHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
var siteID string
var ps []interface{}
var ps []any
if err := json.Unmarshal(params, &ps); err != nil {
return jrpc.Response{Error: err.Error()}
}
@@ -217,7 +217,7 @@ func TestRPC_listFlagsHndl(t *testing.T) {
flags, err = re.ListFlags(verifyFlagReq)
require.NoError(t, err)
assert.Equal(t, []interface{}{"u1"}, flags)
assert.Equal(t, []any{"u1"}, flags)
verifiedUsers := make([]string, 0, len(flags))
for _, v := range flags {
verifiedUsers = append(verifiedUsers, v.(string))
@@ -21,7 +21,7 @@ import (
)
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
for range 10 {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
@@ -34,7 +34,7 @@ func chooseRandomUnusedPort() (port int) {
func waitForHTTPServerStart(port int) {
// wait for up to 3 seconds for server to start before returning it
client := http.Client{Timeout: time.Second}
for i := 0; i < 300; i++ {
for range 300 {
time.Sleep(time.Millisecond * 10)
if resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port)); err == nil {
_ = resp.Body.Close()
+1 -1
View File
@@ -47,7 +47,7 @@ func (ec *BackupCommand) Execute(_ []string) error {
req.SetBasicAuth("admin", ec.AdminPasswd)
// get with timeout
resp, err := client.Do(req.WithContext(ctx))
resp, err := client.Do(req.WithContext(ctx)) //nolint:gosec // exportURL is built from operator-supplied CLI flags, not user input
if err != nil {
return fmt.Errorf("request failed for %s: %w", exportURL, err)
}
+3 -3
View File
@@ -179,7 +179,7 @@ func (cc *CleanupCommand) listComments(postURL string) ([]store.Comment, error)
commentsWithInfo := struct {
Comments []store.Comment `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}{}
if err = json.NewDecoder(r.Body).Decode(&commentsWithInfo); err != nil {
@@ -199,7 +199,7 @@ func (cc *CleanupCommand) deleteComment(c store.Comment) error { //nolint:dupl /
client := http.Client{}
defer client.CloseIdleConnections()
r, err := client.Do(req)
r, err := client.Do(req) //nolint:gosec // RemarkURL comes from operator CLI flag, not user input
if err != nil {
return fmt.Errorf("delete request failed for comment %s, %s: %w", c.ID, c.Locator.URL, err)
}
@@ -221,7 +221,7 @@ func (cc *CleanupCommand) setTitle(c store.Comment) error { //nolint:dupl // not
client := http.Client{}
defer client.CloseIdleConnections()
r, err := client.Do(req)
r, err := client.Do(req) //nolint:gosec // RemarkURL comes from operator CLI flag, not user input
if err != nil {
return fmt.Errorf("title request failed for comment %s, %s: %w", c.ID, c.Locator.URL, err)
}
+6 -7
View File
@@ -9,7 +9,6 @@ import (
"testing"
"time"
"github.com/go-chi/chi/v5"
"github.com/jessevdk/go-flags"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -58,7 +57,7 @@ func TestCleanup_IsSpam(t *testing.T) {
}
func TestCleanup_postsInRange(t *testing.T) {
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, nil)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -81,7 +80,7 @@ func TestCleanup_postsInRange(t *testing.T) {
}
func TestCleanup_listComments(t *testing.T) {
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, nil)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -107,7 +106,7 @@ func TestCleanup_listComments(t *testing.T) {
func TestCleanup_ExecuteSpam(t *testing.T) {
cleaned := cleanedComments{}
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, &cleaned)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -126,7 +125,7 @@ func TestCleanup_ExecuteSpam(t *testing.T) {
func TestCleanup_ExecuteTitle(t *testing.T) {
titledComments := cleanedComments{}
r := chi.NewRouter()
r := http.NewServeMux()
cleanupRoutes(t, r, &titledComments)
ts := httptest.NewServer(r)
defer ts.Close()
@@ -142,7 +141,7 @@ func TestCleanup_ExecuteTitle(t *testing.T) {
assert.Equal(t, []string{"/api/v1/admin/title/1", "/api/v1/admin/title/2", "/api/v1/admin/title/3", "/api/v1/admin/title/11"}, titledComments.ids)
}
func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
func cleanupRoutes(t *testing.T, r *http.ServeMux, c *cleanedComments) {
r.HandleFunc("/api/v1/list", func(w http.ResponseWriter, r *http.Request) {
require.Equal(t, "GET", r.Method)
require.Equal(t, "site=remark&limit=10000", r.URL.RawQuery)
@@ -173,7 +172,7 @@ func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
commentsWithInfo := struct {
Comments []store.Comment `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}{}
switch r.URL.Query().Get("url") {
+1 -1
View File
@@ -124,7 +124,7 @@ func responseError(resp *http.Response) error {
// mkdir -p for all dirs
func makeDirs(dirs ...string) error {
for _, dir := range dirs {
if err := os.MkdirAll(dir, 0o700); err != nil { // If path is already a directory, MkdirAll does nothing
if err := os.MkdirAll(dir, 0o700); err != nil { // if path is already a directory, MkdirAll does nothing
return fmt.Errorf("can't make directory %s: %w", dir, err)
}
}
+1 -1
View File
@@ -42,7 +42,7 @@ func (ic *ImportCommand) Execute(_ []string) error {
}
req.SetBasicAuth("admin", ic.AdminPasswd)
resp, err := client.Do(req.WithContext(ctx)) // closes request's reader
resp, err := client.Do(req.WithContext(ctx)) //nolint:gosec // importURL built from operator CLI flags, not user input; closes request's reader
if err != nil {
return fmt.Errorf("request failed for %s: %w", importURL, err)
}
+2 -2
View File
@@ -34,13 +34,13 @@ func (rc *RemapCommand) Execute(_ []string) error {
ctx, cancel := context.WithTimeout(context.Background(), rc.Timeout)
defer cancel()
remapURL := fmt.Sprintf("%s/api/v1/admin/remap?site=%s", rc.RemarkURL, rc.Site)
req, err := http.NewRequest(http.MethodPost, remapURL, rulesReader)
req, err := http.NewRequest(http.MethodPost, remapURL, rulesReader) //nolint:gosec // RemarkURL is operator CLI flag, not user input
if err != nil {
return fmt.Errorf("can't make remap request for %s: %w", remapURL, err)
}
req.SetBasicAuth("admin", rc.AdminPasswd)
resp, err := client.Do(req.WithContext(ctx))
resp, err := client.Do(req.WithContext(ctx)) //nolint:gosec // see above
if err != nil {
return fmt.Errorf("request failed for %s: %w", remapURL, err)
}
+231 -32
View File
@@ -2,7 +2,9 @@ package cmd
import (
"context"
"crypto/sha1" //nolint:gosec // used only for stable ID hashing, not for security
"embed"
"encoding/json"
"fmt"
"net"
"net/http"
@@ -11,6 +13,7 @@ import (
"os/signal"
"path"
"regexp"
"slices"
"strings"
"syscall"
"time"
@@ -22,6 +25,7 @@ import (
"github.com/golang-jwt/jwt/v5"
"github.com/kyokomi/emoji/v2"
bolt "go.etcd.io/bbolt"
"golang.org/x/oauth2"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
@@ -35,6 +39,7 @@ import (
"github.com/umputun/remark42/backend/app/providers"
"github.com/umputun/remark42/backend/app/rest/api"
"github.com/umputun/remark42/backend/app/rest/proxy"
"github.com/umputun/remark42/backend/app/safehttp"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/admin"
"github.com/umputun/remark42/backend/app/store/engine"
@@ -74,18 +79,19 @@ type ServerCommand struct {
CriticalScore int `long:"critical-score" env:"CRITICAL_SCORE" default:"-10" description:"critical score threshold"`
PositiveScore bool `long:"positive-score" env:"POSITIVE_SCORE" description:"enable positive score only"`
ReadOnlyAge int `long:"read-age" env:"READONLY_AGE" default:"0" description:"read-only age of comments, days"`
EditDuration time.Duration `long:"edit-time" env:"EDIT_TIME" default:"5m" description:"edit window"`
EditDuration time.Duration `long:"edit-time" env:"EDIT_TIME" default:"5m" description:"edit window; set to 0 to disable comment editing and staged image cleanup"`
AdminEdit bool `long:"admin-edit" env:"ADMIN_EDIT" description:"unlimited edit for admins"`
Port int `long:"port" env:"REMARK_PORT" default:"8080" description:"port"`
Address string `long:"address" env:"REMARK_ADDRESS" default:"" description:"listening address"`
WebRoot string `long:"web-root" env:"REMARK_WEB_ROOT" default:"./web" description:"web root directory"`
UpdateLimit float64 `long:"update-limit" env:"UPDATE_LIMIT" default:"0.5" description:"updates/sec limit"`
TrustedProxies []string `long:"trusted-proxy" env:"TRUSTED_PROXY" description:"reverse-proxy networks (CIDR or IP) trusted to set the client IP; if unset, trusted from any client (see docs)" env-delim:","`
RestrictedWords []string `long:"restricted-words" env:"RESTRICTED_WORDS" description:"words prohibited to use in comments" env-delim:","`
RestrictedNames []string `long:"restricted-names" env:"RESTRICTED_NAMES" description:"names prohibited to use by user" env-delim:","`
EnableEmoji bool `long:"emoji" env:"EMOJI" description:"enable emoji"`
SimpleView bool `long:"simple-view" env:"SIMPLE_VIEW" description:"minimal comment editor mode"`
ProxyCORS bool `long:"proxy-cors" env:"PROXY_CORS" description:"disable internal CORS and delegate it to proxy"`
AllowedHosts []string `long:"allowed-hosts" env:"ALLOWED_HOSTS" description:"limit hosts/sources allowed to embed comments via CSP 'frame-ancestors''" env-delim:","`
AllowedHosts []string `long:"allowed-hosts" env:"ALLOWED_HOSTS" description:"limit hosts/sources allowed to embed comments via CSP 'frame-ancestors'" env-delim:","`
SubscribersOnly bool `long:"subscribers-only" env:"SUBSCRIBERS_ONLY" description:"enable commenting only for Patreon subscribers"`
DisableSignature bool `long:"disable-signature" env:"DISABLE_SIGNATURE" description:"disable server signature in headers"`
DisableFancyTextFormatting bool `long:"disable-fancy-text-formatting" env:"DISABLE_FANCY_TEXT_FORMATTING" description:"disable fancy comments text formatting (replacement of quotes, dashes, fractions, etc)"`
@@ -99,28 +105,29 @@ type ServerCommand struct {
SendJWTHeader bool `long:"send-jwt-header" env:"SEND_JWT_HEADER" description:"send JWT as a header instead of server-set cookie; with this enabled, frontend stores the JWT in a client-side cookie (note: increases vulnerability to XSS attacks)"`
SameSite string `long:"same-site" env:"SAME_SITE" description:"set same site policy for cookies" choice:"default" choice:"none" choice:"lax" choice:"strict" default:"default"` // nolint
Apple AppleGroup `group:"apple" namespace:"apple" env-namespace:"APPLE" description:"Apple OAuth"`
Google AuthGroup `group:"google" namespace:"google" env-namespace:"GOOGLE" description:"Google OAuth"`
Github AuthGroup `group:"github" namespace:"github" env-namespace:"GITHUB" description:"Github OAuth"`
Facebook AuthGroup `group:"facebook" namespace:"facebook" env-namespace:"FACEBOOK" description:"Facebook OAuth"`
Microsoft AuthGroup `group:"microsoft" namespace:"microsoft" env-namespace:"MICROSOFT" description:"Microsoft OAuth"`
Yandex AuthGroup `group:"yandex" namespace:"yandex" env-namespace:"YANDEX" description:"Yandex OAuth"`
Twitter AuthGroup `group:"twitter" namespace:"twitter" env-namespace:"TWITTER" description:"[deprecated, doesn't work] Twitter OAuth"`
Patreon AuthGroup `group:"patreon" namespace:"patreon" env-namespace:"PATREON" description:"Patreon OAuth"`
Discord AuthGroup `group:"discord" namespace:"discord" env-namespace:"DISCORD" description:"Discord OAuth"`
Telegram bool `long:"telegram" env:"TELEGRAM" description:"Enable Telegram auth (using token from telegram.token)"`
Dev bool `long:"dev" env:"DEV" description:"enable dev (local) oauth2"`
Anonymous bool `long:"anon" env:"ANON" description:"enable anonymous login"`
Apple AppleGroup `group:"apple" namespace:"apple" env-namespace:"APPLE" description:"Apple OAuth"`
Google AuthGroup `group:"google" namespace:"google" env-namespace:"GOOGLE" description:"Google OAuth"`
Github AuthGroup `group:"github" namespace:"github" env-namespace:"GITHUB" description:"Github OAuth"`
Facebook AuthGroup `group:"facebook" namespace:"facebook" env-namespace:"FACEBOOK" description:"Facebook OAuth"`
Microsoft MicrosoftAuthGroup `group:"microsoft" namespace:"microsoft" env-namespace:"MICROSOFT" description:"Microsoft OAuth"`
Yandex AuthGroup `group:"yandex" namespace:"yandex" env-namespace:"YANDEX" description:"Yandex OAuth"`
Twitter AuthGroup `group:"twitter" namespace:"twitter" env-namespace:"TWITTER" description:"[deprecated, doesn't work] Twitter OAuth"`
Patreon AuthGroup `group:"patreon" namespace:"patreon" env-namespace:"PATREON" description:"Patreon OAuth"`
Discord AuthGroup `group:"discord" namespace:"discord" env-namespace:"DISCORD" description:"Discord OAuth"`
Custom CustomAuthGroup `group:"custom" namespace:"custom" env-namespace:"CUSTOM" description:"Custom OAuth2 provider"`
Telegram bool `long:"telegram" env:"TELEGRAM" description:"Enable Telegram auth (using token from telegram.token)"`
Dev bool `long:"dev" env:"DEV" description:"enable dev (local) oauth2"`
Anonymous bool `long:"anon" env:"ANON" description:"enable anonymous login"`
Email struct {
Enable bool `long:"enable" env:"ENABLE" description:"enable auth via email"`
From string `long:"from" env:"FROM" description:"from email address"`
Subject string `long:"subj" env:"SUBJ" default:"remark42 confirmation" description:"email's subject"`
ContentType string `long:"content-type" env:"CONTENT_TYPE" default:"text/html" description:"content type"`
Host string `long:"host" env:"HOST" description:"[deprecated, use --smtp.host] SMTP host"`
Port int `long:"port" env:"PORT" description:"[deprecated, use --smtp.port] SMTP password"`
SMTPPassword string `long:"passwd" env:"PASSWD" description:"[deprecated, use --smtp.password] SMTP port"`
SMTPUserName string `long:"user" env:"USER" description:"[deprecated, use --smtp.username] enable TLS"`
TLS bool `long:"tls" env:"TLS" description:"[deprecated, use --smtp.tls] SMTP TCP connection timeout"`
Port int `long:"port" env:"PORT" description:"[deprecated, use --smtp.port] SMTP port"`
SMTPPassword string `long:"passwd" env:"PASSWD" description:"[deprecated, use --smtp.password] SMTP password"`
SMTPUserName string `long:"user" env:"USER" description:"[deprecated, use --smtp.username] SMTP user name"`
TLS bool `long:"tls" env:"TLS" description:"[deprecated, use --smtp.tls] enable TLS"`
TimeOut time.Duration `long:"timeout" env:"TIMEOUT" default:"10s" description:"[deprecated, use --smtp.timeout] SMTP TCP connection timeout"`
MsgTemplate string `long:"template" env:"TEMPLATE" description:"[deprecated] message template file" default:"email_confirmation_login.html.tmpl"`
} `group:"email" namespace:"email" env-namespace:"EMAIL"`
@@ -152,6 +159,28 @@ type AuthGroup struct {
CSEC string `long:"csec" env:"CSEC" description:"OAuth client secret"`
}
// MicrosoftAuthGroup defines options group for Microsoft auth params
type MicrosoftAuthGroup struct {
CID string `long:"cid" env:"CID" description:"OAuth client ID"`
CSEC string `long:"csec" env:"CSEC" description:"OAuth client secret"`
Tenant string `long:"tenant" env:"TENANT" description:"Azure AD tenant ID, domain, or 'common' (default)" default:"common"`
}
// CustomAuthGroup defines options group for custom OAuth2 provider params
type CustomAuthGroup struct {
Name string `long:"name" env:"NAME" description:"custom provider name used in auth route"`
CID string `long:"cid" env:"CID" description:"OAuth client ID"`
CSEC string `long:"csec" env:"CSEC" description:"OAuth client secret"`
AuthURL string `long:"auth-url" env:"AUTH_URL" description:"OAuth authorization endpoint"`
TokenURL string `long:"token-url" env:"TOKEN_URL" description:"OAuth token endpoint"`
InfoURL string `long:"info-url" env:"INFO_URL" description:"OAuth user info endpoint"`
Scopes []string `long:"scopes" env:"SCOPES" env-delim:"," description:"OAuth scopes"`
IDField string `long:"id-field" env:"ID_FIELD" default:"sub" description:"user info field used as unique id"`
NameField string `long:"name-field" env:"NAME_FIELD" default:"name" description:"user info field used as display name"`
PictureField string `long:"picture-field" env:"PICTURE_FIELD" default:"picture" description:"user info field used as avatar url"`
EmailField string `long:"email-field" env:"EMAIL_FIELD" default:"email" description:"user info field used as email"`
}
// StoreGroup defines options group for store params
type StoreGroup struct {
Type string `long:"type" env:"TYPE" description:"type of storage" choice:"bolt" choice:"rpc" default:"bolt"` // nolint
@@ -255,8 +284,8 @@ type NotifyGroup struct {
} `group:"slack" namespace:"slack" env-namespace:"SLACK"`
Webhook struct {
URL string `long:"url" env:"URL" description:"webhook URL for admin notifications"`
Template string `long:"template" env:"TEMPLATE" description:"webhook authentication template" default:"{\"text\": \"{{.Text}}\"}"`
Headers []string `long:"headers" description:"webhook authentication headers in format --notify.webhook.headers=Header1:Value1,Value2,... [$NOTIFY_WEBHOOK_HEADERS]"` // env NOTIFY_WEBHOOK_HEADERS split in code bellow to allow , inside ""
Template string `long:"template" env:"TEMPLATE" description:"webhook payload template (Go text/template); falls back to {\"text\": {{.Text | escapeJSONString}}} when empty"`
Headers []string `long:"headers" description:"webhook headers in format --notify.webhook.headers=Header1:Value1,Value2,... [$NOTIFY_WEBHOOK_HEADERS]"` // env NOTIFY_WEBHOOK_HEADERS split in code below to allow , inside ""
Timeout time.Duration `long:"timeout" env:"TIMEOUT" description:"webhook timeout" default:"5s"`
} `group:"webhook" namespace:"webhook" env-namespace:"WEBHOOK"`
}
@@ -323,6 +352,7 @@ func (s *ServerCommand) Execute(_ []string) error {
"AUTH_YANDEX_CSEC",
"AUTH_PATREON_CSEC",
"AUTH_DISCORD_CSEC",
"AUTH_CUSTOM_CSEC",
"TELEGRAM_TOKEN",
"SMTP_PASSWORD",
"ADMIN_PASSWD",
@@ -472,12 +502,87 @@ func stringsSetAndDifferent(s1, s2 string) bool {
}
func contains(s string, a []string) bool {
for _, t := range a {
if t == s {
return true
return slices.Contains(a, s)
}
var reservedCustomProviderNames = map[string]struct{}{
"email": {},
"anonymous": {},
"google": {},
"github": {},
"facebook": {},
"yandex": {},
"twitter": {},
"microsoft": {},
"patreon": {},
"discord": {},
"telegram": {},
"dev": {},
"apple": {},
}
var validCustomProviderName = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]*$`)
func isReservedCustomProviderName(name string) bool {
_, ok := reservedCustomProviderNames[name]
return ok
}
func isValidCustomProviderName(name string) bool {
return validCustomProviderName.MatchString(name)
}
func customProviderSourceID(data provider.UserData, cfg CustomAuthGroup) string {
sourceID := data.Value(cfg.IDField)
if sourceID == "" {
sourceID = data.Value(cfg.EmailField)
}
if sourceID == "" {
sourceID = data.Value(cfg.NameField)
}
if sourceID == "" {
sourceID = data.Value(cfg.PictureField)
}
if sourceID == "" {
payload, err := json.Marshal(data)
if err != nil {
log.Printf("[WARN] failed to serialize custom oauth user data for ID fallback: %v", err)
} else {
sourceID = string(payload)
}
}
return false
if sourceID == "" || sourceID == "{}" {
log.Printf("[WARN] custom oauth provider returned no stable user identifier fields, falling back to hashed payload")
}
return sourceID
}
func (c CustomAuthGroup) isConfigured() bool {
return c.Name != "" || c.CID != "" || c.CSEC != "" || c.AuthURL != "" || c.TokenURL != "" || c.InfoURL != "" ||
len(c.Scopes) > 0 || c.IDField != "sub" || c.NameField != "name" || c.PictureField != "picture" || c.EmailField != "email"
}
func (c CustomAuthGroup) missingRequired() []string {
missing := []string{}
if c.Name == "" {
missing = append(missing, "AUTH_CUSTOM_NAME")
}
if c.CID == "" {
missing = append(missing, "AUTH_CUSTOM_CID")
}
if c.CSEC == "" {
missing = append(missing, "AUTH_CUSTOM_CSEC")
}
if c.AuthURL == "" {
missing = append(missing, "AUTH_CUSTOM_AUTH_URL")
}
if c.TokenURL == "" {
missing = append(missing, "AUTH_CUSTOM_TOKEN_URL")
}
if c.InfoURL == "" {
missing = append(missing, "AUTH_CUSTOM_INFO_URL")
}
return missing
}
// newServerApp prepares application and return it with all active parts
@@ -492,6 +597,18 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
}
log.Printf("[INFO] root url=%s", s.RemarkURL)
// parse trusted proxies up front so a bad CIDR fails before any resource is allocated
trustedProxies, err := api.ParseTrustedProxies(s.TrustedProxies)
if err != nil {
return nil, fmt.Errorf("invalid --trusted-proxy: %w", err)
}
switch {
case len(trustedProxies) == 0:
log.Printf("[WARN] --trusted-proxy not set: forwarding headers are trusted from any client and can be spoofed to bypass rate limiting / vote dedup; set it behind a reverse proxy (see docs)")
case api.TrustsAnyPeer(trustedProxies):
log.Printf("[WARN] --trusted-proxy has a catch-all (0.0.0.0/0 or ::/0): forwarding headers are trusted from any client, re-opening the spoofing bypass; scope it to your proxy network")
}
storeEngine, err := s.makeDataStore()
if err != nil {
return nil, fmt.Errorf("failed to make data store engine: %w", err)
@@ -518,7 +635,7 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
MaxVotes: s.MaxVotes,
PositiveScore: s.PositiveScore,
ImageService: imageService,
TitleExtractor: service.NewTitleExtractor(http.Client{Timeout: time.Second * 5}, s.getAllowedDomains()),
TitleExtractor: service.NewTitleExtractor(http.Client{Timeout: time.Second * 5, Transport: safehttp.Transport()}, s.getAllowedDomains()),
RestrictedWordsMatcher: service.NewRestrictedWordsMatcher(service.StaticRestrictedWordsLister{Words: s.RestrictedWords}),
}
dataService.RestrictSameIPVotes.Enabled = s.RestrictVoteIP
@@ -599,6 +716,7 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
Migrator: migr,
ReadOnlyAge: s.ReadOnlyAge,
SharedSecret: s.SharedSecret,
TrustedProxies: trustedProxies,
Authenticator: authenticator,
Cache: loadingCache,
NotifyService: notifyService,
@@ -678,9 +796,9 @@ func (s *ServerCommand) getAllowedDomains() []string {
continue
}
// Only for RemarkURL if domain is not IP and has more than two levels, extract second level domain.
// For AllowedHosts we don't do this as they are exact list of domains which can host comments, but
// RemarkURL might be on a subdomain and we must allow parent domain to be used for TitleExtract.
// only for RemarkURL if domain is not IP and has more than two levels, extract second level domain.
// for AllowedHosts we don't do this as they are exact list of domains which can host comments, but
// remarkURL might be on a subdomain and we must allow parent domain to be used for TitleExtract.
if rawURL == s.RemarkURL && net.ParseIP(domain) == nil && len(strings.Split(domain, ".")) > 2 {
domain = strings.Join(strings.Split(domain, ".")[len(strings.Split(domain, "."))-2:], ".")
}
@@ -690,6 +808,42 @@ func (s *ServerCommand) getAllowedDomains() []string {
return allowedDomains
}
// getAllowedRedirectHosts normalises s.AllowedHosts into the form that
// go-pkgz/auth's redirect validator expects. Strips http(s) schemes and
// paths; preserves explicit ports (the validator matches both host-only
// and host:port, so an entry without a port accepts any port while an
// entry with a port restricts to that port). Skips CSP sentinels
// ('self' / "self") and wildcard entries (*, *.example.com) that are
// valid CSP source expressions but not valid hostnames.
func (s *ServerCommand) getAllowedRedirectHosts() []string {
out := make([]string, 0, len(s.AllowedHosts))
for _, raw := range s.AllowedHosts {
raw = strings.TrimSpace(raw)
if raw == "" || raw == "self" || raw == "'self'" || raw == `"self"` {
continue
}
if strings.ContainsRune(raw, '*') { // CSP wildcard, not a host
continue
}
// add scheme so url.Parse populates Hostname()/Host consistently for bare hosts
toParse := raw
if !strings.HasPrefix(toParse, "http://") && !strings.HasPrefix(toParse, "https://") {
toParse = "https://" + toParse
}
u, err := url.Parse(toParse)
if err != nil || u.Hostname() == "" {
log.Printf("[WARN] skipping invalid AllowedHosts entry %q for redirect allowlist: %v", raw, err)
continue
}
if u.Port() != "" {
out = append(out, u.Host) // preserve explicit host:port so allowlist is port-specific
continue
}
out = append(out, u.Hostname())
}
return out
}
// Run all application objects
func (a *serverApp) run(ctx context.Context) error {
if a.AdminPasswd != "" {
@@ -939,7 +1093,7 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
providersCount++
}
if s.Auth.Microsoft.CID != "" && s.Auth.Microsoft.CSEC != "" {
authenticator.AddProvider("microsoft", s.Auth.Microsoft.CID, s.Auth.Microsoft.CSEC)
authenticator.AddMicrosoftProvider(s.Auth.Microsoft.CID, s.Auth.Microsoft.CSEC, s.Auth.Microsoft.Tenant)
providersCount++
}
if s.Auth.Yandex.CID != "" && s.Auth.Yandex.CSEC != "" {
@@ -959,6 +1113,45 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
providersCount++
}
if s.Auth.Custom.isConfigured() {
missing := s.Auth.Custom.missingRequired()
if len(missing) > 0 {
return fmt.Errorf("custom oauth provider configuration is incomplete, missing: %s", strings.Join(missing, ", "))
}
customName := strings.ToLower(strings.TrimSpace(s.Auth.Custom.Name))
if !isValidCustomProviderName(customName) {
return fmt.Errorf("custom oauth provider name %q is invalid, expected pattern %q", customName, validCustomProviderName.String())
}
if isReservedCustomProviderName(customName) {
return fmt.Errorf("custom oauth provider name %q is reserved", customName)
}
authenticator.AddCustomProvider(customName, auth.Client{Cid: s.Auth.Custom.CID, Csecret: s.Auth.Custom.CSEC}, provider.CustomHandlerOpt{
Endpoint: oauth2.Endpoint{
AuthURL: s.Auth.Custom.AuthURL,
TokenURL: s.Auth.Custom.TokenURL,
},
InfoURL: s.Auth.Custom.InfoURL,
Scopes: s.Auth.Custom.Scopes,
MapUserFn: func(data provider.UserData, _ []byte) token.User {
sourceID := customProviderSourceID(data, s.Auth.Custom)
hashID := token.HashID(sha1.New(), sourceID) //nolint:gosec // stable provider user id hash
user := token.User{
ID: customName + "_" + hashID,
Name: data.Value(s.Auth.Custom.NameField),
Picture: data.Value(s.Auth.Custom.PictureField),
Email: data.Value(s.Auth.Custom.EmailField),
}
if user.Name == "" {
user.Name = "noname_" + hashID[:4]
}
return user
},
})
providersCount++
}
if s.Auth.Dev {
log.Print("[INFO] dev access enabled")
u, errURL := url.Parse(s.RemarkURL)
@@ -1020,7 +1213,7 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
}
return true, nil
}),
// Custom user ID generator, used to distinguish anonymous users with the same login
// custom user ID generator, used to distinguish anonymous users with the same login
// coming from different IPs
func(user string, r *http.Request) string {
return user + r.RemoteAddr
@@ -1105,7 +1298,7 @@ func (s *ServerCommand) makeNotifyDestinations(authenticator *auth.Service) ([]n
VerificationSubject: s.Notify.Email.VerificationSubject,
UnsubscribeURL: s.RemarkURL + "/email/unsubscribe.html",
// TODO: uncomment after #560 frontend part is ready and URL is known
// SubscribeURL: s.RemarkURL + "/subscribe.html?token=",
// subscribeURL: s.RemarkURL + "/subscribe.html?token=",
TokenGenFn: func(userID, email, site string) (string, error) {
claims := token.Claims{
Handshake: &token.Handshake{ID: userID + "::" + email},
@@ -1208,6 +1401,12 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
SendJWTHeader: s.Auth.SendJWTHeader,
SameSiteCookie: s.parseSameSite(s.Auth.SameSite),
SecureCookies: strings.HasPrefix(s.RemarkURL, "https://"),
// enable the `from` redirect allowlist in go-pkgz/auth v2.1.2+ — limits
// post-auth redirects to RemarkURL's own host plus any configured
// AllowedHosts. Prevents the OAuth open-redirect / phishing vector.
AllowedRedirectHosts: token.AllowedHostsFunc(func() ([]string, error) {
return s.getAllowedRedirectHosts(), nil
}),
SecretReader: token.SecretFunc(func(aud string) (string, error) { // get secret per site
return admns.Key(aud)
}),
@@ -1215,7 +1414,7 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
if c.User == nil {
return c
}
// Audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
// audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
if len(c.Audience) != 1 {
return c
}
+191 -11
View File
@@ -15,6 +15,7 @@ import (
"testing"
"time"
"github.com/go-pkgz/auth/v2/provider"
"github.com/go-pkgz/auth/v2/token"
"github.com/golang-jwt/jwt/v5"
"github.com/jessevdk/go-flags"
@@ -47,7 +48,7 @@ func TestServerApp(t *testing.T) {
// add comment
client := http.Client{Timeout: 10 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
@@ -95,6 +96,30 @@ func TestServerApp_DevMode(t *testing.T) {
app.Wait()
}
func TestServerApp_CustomOAuthProvider(t *testing.T) {
port := chooseRandomUnusedPort()
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
o.Auth.Custom.Name = "oidc"
o.Auth.Custom.CID = "cid"
o.Auth.Custom.CSEC = "csec"
o.Auth.Custom.AuthURL = "https://example.com/oauth2/authorize"
o.Auth.Custom.TokenURL = "https://example.com/oauth2/token"
o.Auth.Custom.InfoURL = "https://example.com/oauth2/userinfo"
return o
})
go func() { _ = app.run(ctx) }()
waitForHTTPServerStart(port)
providers := app.restSrv.Authenticator.Providers()
require.Equal(t, 11+1, len(providers), "extra auth provider")
assert.Equal(t, "oidc", providers[len(providers)-2].Name(), "custom auth provider")
cancel()
app.Wait()
}
func TestServerApp_AnonMode(t *testing.T) {
port := chooseRandomUnusedPort()
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
@@ -129,7 +154,7 @@ func TestServerApp_AnonMode(t *testing.T) {
assert.Equal(t, http.StatusOK, resp.StatusCode)
// try to add a comment as good anonymous
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
@@ -194,7 +219,7 @@ func TestServerApp_AnonMode(t *testing.T) {
// try to add a comment as anonymous with admin name
time.Sleep(time.Second)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
@@ -352,6 +377,16 @@ func TestServerApp_Failed(t *testing.T) {
assert.EqualError(t, err, "invalid remark42 url demo.remark42.com")
t.Log(err)
// invalid trusted proxy CIDR fails fast, before any resource is created
opts = ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p = flags.NewParser(&opts, flags.Default)
_, err = p.ParseArgs([]string{"--backup=/tmp", "--trusted-proxy=nonsense"})
assert.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err, `invalid --trusted-proxy: invalid trusted proxy "nonsense"`)
t.Log(err)
// wrong store type
opts = ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
@@ -389,6 +424,95 @@ func TestServerApp_Failed(t *testing.T) {
"failed to make authenticator: an AppleProvider creating failed: "+
"provided private key is not ECDSA")
t.Log(err)
// incomplete custom oauth config
opts = ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p = flags.NewParser(&opts, flags.Default)
_, err = p.ParseArgs([]string{"--store.bolt.path=/tmp", "--backup=/tmp", "--image.fs.path=/tmp", "--auth.custom.name=oidc", "--auth.custom.cid=123"})
assert.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err,
"failed to make authenticator: custom oauth provider configuration is incomplete, missing: "+
"AUTH_CUSTOM_CSEC, AUTH_CUSTOM_AUTH_URL, AUTH_CUSTOM_TOKEN_URL, AUTH_CUSTOM_INFO_URL")
t.Log(err)
}
func TestIsReservedCustomProviderName(t *testing.T) {
reserved := []string{
"email", "anonymous", "google", "github", "facebook", "yandex", "twitter",
"microsoft", "patreon", "discord", "telegram", "dev", "apple",
}
for _, name := range reserved {
t.Run(name, func(t *testing.T) {
assert.True(t, isReservedCustomProviderName(name))
})
}
assert.False(t, isReservedCustomProviderName("oidc"))
}
func TestIsValidCustomProviderName(t *testing.T) {
valid := []string{"oidc", "codeberg", "provider_1", "provider-1", "a1"}
for _, name := range valid {
t.Run("valid_"+name, func(t *testing.T) {
assert.True(t, isValidCustomProviderName(name))
})
}
invalid := []string{"", " has-space", "has space", "Uppercase", "provider!", "-provider", "_provider"}
for _, name := range invalid {
t.Run("invalid_"+strings.ReplaceAll(name, " ", "_"), func(t *testing.T) {
assert.False(t, isValidCustomProviderName(name))
})
}
}
func TestCustomProviderSourceID(t *testing.T) {
cfg := CustomAuthGroup{IDField: "sub", EmailField: "email", NameField: "name", PictureField: "picture"}
assert.Equal(t, "user-1", customProviderSourceID(provider.UserData{"sub": "user-1", "email": "a@example.com"}, cfg))
assert.Equal(t, "a@example.com", customProviderSourceID(provider.UserData{"email": "a@example.com"}, cfg))
assert.Equal(t, "alice", customProviderSourceID(provider.UserData{"name": "alice"}, cfg))
assert.Equal(t, "https://example.com/avatar.png", customProviderSourceID(provider.UserData{"picture": "https://example.com/avatar.png"}, cfg))
assert.Equal(t, `{"login":"alice"}`, customProviderSourceID(provider.UserData{"login": "alice"}, cfg))
assert.Equal(t, "{}", customProviderSourceID(provider.UserData{}, cfg))
}
func TestServerApp_InvalidCustomOAuthProviderName(t *testing.T) {
baseArgs := []string{
"--store.bolt.path=/tmp",
"--backup=/tmp",
"--image.fs.path=/tmp",
"--auth.custom.cid=123",
"--auth.custom.csec=456",
"--auth.custom.auth-url=https://example.com/oauth2/authorize",
"--auth.custom.token-url=https://example.com/oauth2/token",
"--auth.custom.info-url=https://example.com/oauth2/userinfo",
}
t.Run("reserved", func(t *testing.T) {
opts := ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p := flags.NewParser(&opts, flags.Default)
_, err := p.ParseArgs(append(baseArgs, "--auth.custom.name=twitter"))
require.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err, `failed to make authenticator: custom oauth provider name "twitter" is reserved`)
})
t.Run("not_url_safe", func(t *testing.T) {
opts := ServerCommand{}
opts.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
p := flags.NewParser(&opts, flags.Default)
_, err := p.ParseArgs(append(baseArgs, "--auth.custom.name=bad name"))
require.NoError(t, err)
_, err = opts.newServerApp(context.Background())
assert.EqualError(t, err, `failed to make authenticator: custom oauth provider name "bad name" is invalid, expected pattern "^[a-z0-9][a-z0-9_-]*$"`)
})
}
func TestServerApp_Shutdown(t *testing.T) {
@@ -434,6 +558,34 @@ func TestServerApp_MainSignal(t *testing.T) {
assert.True(t, time.Since(st).Seconds() < 5, "should take under five sec", time.Since(st).Seconds())
}
func TestServerApp_RunCanceledBeforeRESTStart(t *testing.T) {
port := chooseRandomUnusedPort()
app, ctx, cancel := prepServerApp(t, func(o ServerCommand) ServerCommand {
o.Port = port
return o
})
cancel()
errCh := make(chan error, 1)
go func() { errCh <- app.run(ctx) }()
select {
case err := <-errCh:
require.NoError(t, err)
app.Wait()
case <-time.After(time.Second):
waitForHTTPServerStart(port)
app.restSrv.Shutdown()
select {
case <-errCh:
app.Wait()
case <-time.After(time.Second):
t.Fatal("server app did not stop after forced REST shutdown")
}
t.Fatal("server app should exit when context is canceled before REST server starts")
}
}
func TestServerApp_DeprecatedArgs(t *testing.T) {
s := ServerCommand{}
s.SetCommon(CommonOpts{RemarkURL: "https://demo.remark42.com", SharedSecret: "123456"})
@@ -628,7 +780,7 @@ func TestServerAuthHooks(t *testing.T) {
defer client.CloseIdleConnections()
// add comment
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err := http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-630/", "site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tk)
@@ -643,7 +795,7 @@ func TestServerAuthHooks(t *testing.T) {
tkNoAud, err := tkService.Token(badClaimsNoAud)
require.NoError(t, err)
t.Logf("no-aud claims: %s", tkNoAud)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
@@ -661,7 +813,7 @@ func TestServerAuthHooks(t *testing.T) {
tkMultipleAuds, err := tkService.Token(badClaimsMultipleAud)
require.NoError(t, err)
t.Logf("multiple aud claims: %s", tkMultipleAuds)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
@@ -680,7 +832,7 @@ func TestServerAuthHooks(t *testing.T) {
tkNoUser, err := tkService.Token(badClaimsNoUser)
require.NoError(t, err)
t.Logf("no user claims: %s", tkNoUser)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
@@ -706,7 +858,7 @@ func TestServerAuthHooks(t *testing.T) {
t.Log(string(b))
// try add a comment with blocked user
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment?site=remark", port),
strings.NewReader(`{"text": "test 123 blah", "locator":{"url": "https://radio-t.com/blah1", "site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tk)
@@ -788,8 +940,36 @@ func Test_getAllowedDomains(t *testing.T) {
}
}
func Test_getAllowedRedirectHosts(t *testing.T) {
tbl := []struct {
name string
hosts []string
want []string
}{
{name: "empty", hosts: nil, want: []string{}},
{name: "bare hostnames pass through", hosts: []string{"example.com", "admin.example.com"}, want: []string{"example.com", "admin.example.com"}},
{name: "https scheme stripped", hosts: []string{"https://example.com"}, want: []string{"example.com"}},
{name: "http scheme stripped", hosts: []string{"http://example.com"}, want: []string{"example.com"}},
{name: "scheme with path strips path", hosts: []string{"https://example.com/embed"}, want: []string{"example.com"}},
{name: "explicit port preserved as host:port", hosts: []string{"example.com:8080"}, want: []string{"example.com:8080"}},
{name: "scheme with explicit port preserved", hosts: []string{"https://example.com:8443"}, want: []string{"example.com:8443"}},
{name: "scheme without port stays bare host", hosts: []string{"https://example.com"}, want: []string{"example.com"}},
{name: "self sentinel filtered", hosts: []string{"'self'", "self", `"self"`, "example.com"}, want: []string{"example.com"}},
{name: "wildcards filtered", hosts: []string{"*", "*.example.com", "https://*.example.com", "example.com"}, want: []string{"example.com"}},
{name: "empty entries filtered", hosts: []string{"", " ", "example.com"}, want: []string{"example.com"}},
{name: "mixed real-world", hosts: []string{"'self'", "https://blog.example.com", "admin.example.com:8443", "*.cdn.example.com"},
want: []string{"blog.example.com", "admin.example.com:8443"}},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
s := ServerCommand{AllowedHosts: tt.hosts}
assert.Equal(t, tt.want, s.getAllowedRedirectHosts())
})
}
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
for range 10 {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
@@ -803,7 +983,7 @@ func waitForHTTPServerStart(port int) {
// wait for up to 3 seconds for server to start before returning it
client := http.Client{Timeout: time.Second}
defer client.CloseIdleConnections()
for i := 0; i < 300; i++ {
for range 300 {
time.Sleep(time.Millisecond * 10)
if resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port)); err == nil {
_ = resp.Body.Close()
@@ -814,7 +994,7 @@ func waitForHTTPServerStart(port int) {
func waitForHTTPSServerStart(port int) {
// wait for up to 3 seconds for HTTPS server to start
for i := 0; i < 300; i++ {
for range 300 {
time.Sleep(time.Millisecond * 10)
conn, _ := net.DialTimeout("tcp", fmt.Sprintf("localhost:%d", port), time.Millisecond*10)
if conn != nil {
+1 -4
View File
@@ -92,10 +92,7 @@ func logDeprecatedParams(params []cmd.DeprecatedFlag) {
func getDump() string {
maxSize := 5 * 1024 * 1024
stacktrace := make([]byte, maxSize)
length := runtime.Stack(stacktrace, true)
if length > maxSize {
length = maxSize
}
length := min(runtime.Stack(stacktrace, true), maxSize)
return string(stacktrace[:length])
}
+2 -2
View File
@@ -130,7 +130,7 @@ func TestGetDump(t *testing.T) {
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
for range 10 {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
@@ -144,7 +144,7 @@ func waitForHTTPServerStart(port int) {
// wait for up to 10 seconds for server to start before returning it
client := http.Client{Timeout: time.Second}
defer client.CloseIdleConnections()
for i := 0; i < 100; i++ {
for range 100 {
time.Sleep(time.Millisecond * 100)
if resp, err := client.Get(fmt.Sprintf("http://localhost:%d", port)); err == nil {
_ = resp.Body.Close()
+14 -11
View File
@@ -6,6 +6,7 @@ import (
"io"
"os"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -59,19 +60,21 @@ func TestBackup_Do(t *testing.T) {
defer os.RemoveAll(loc)
assert.NoError(t, os.MkdirAll(loc, 0o700))
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(time.Second)
cancel()
}()
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(time.Second)
cancel()
}()
bk := AutoBackup{BackupLocation: loc, SiteID: "site1", KeepMax: 3, Exporter: &mockExporter{}, Duration: 600 * time.Millisecond}
bk.Do(ctx)
bk := AutoBackup{BackupLocation: loc, SiteID: "site1", KeepMax: 3, Exporter: &mockExporter{}, Duration: 600 * time.Millisecond}
bk.Do(ctx)
expFile := fmt.Sprintf("/tmp/remark-backups.test/backup-site1-%s.gz", time.Now().Format("20060102"))
fi, err := os.Lstat(expFile)
assert.NoError(t, err)
assert.Equal(t, int64(52), fi.Size())
expFile := fmt.Sprintf("/tmp/remark-backups.test/backup-site1-%s.gz", time.Now().Format("20060102"))
fi, err := os.Lstat(expFile)
assert.NoError(t, err)
assert.Equal(t, int64(52), fi.Size())
})
}
type mockExporter struct{}
+1 -1
View File
@@ -48,7 +48,7 @@ type commentoCommenter struct {
Link string `json:"link"`
Photo string `json:"photo"`
Provider string `json:"provider,omitempty"`
JoinDate time.Time `json:"joinDate,omitempty"`
JoinDate time.Time `json:"joinDate"`
IsModerator bool `json:"isModerator"`
}
+1 -1
View File
@@ -122,7 +122,7 @@ func TestDisqus_Convert(t *testing.T) {
require.NoError(t, err)
ch := d.convert(fh, "test")
res := []store.Comment{}
res := make([]store.Comment, 0, 4)
for comment := range ch {
res = append(res, comment)
}
+3 -3
View File
@@ -38,7 +38,7 @@ func (u *URLMapper) loadRules(reader io.Reader) error {
u.rules = make(map[string]string)
for _, row := range strings.Split(rulesText, "\n") {
for row := range strings.SplitSeq(rulesText, "\n") {
row = strings.TrimSpace(row)
urls := strings.Split(row, " ")
if len(urls) != 2 {
@@ -64,8 +64,8 @@ func (u *URLMapper) URL(url string) string {
}
oldURL = strings.TrimSuffix(oldURL, "*")
newURL = strings.TrimSuffix(newURL, "*")
if strings.HasPrefix(url, oldURL) {
return newURL + strings.TrimPrefix(url, oldURL)
if after, ok := strings.CutPrefix(url, oldURL); ok {
return newURL + after
}
}
// search failed, return given url
+1 -1
View File
@@ -162,7 +162,7 @@ func TestNative_ImportManyWithError(t *testing.T) {
buf := &bytes.Buffer{}
buf.WriteString(`{"version":1, "users":[], "posts":[]}` + "\n")
for i := 0; i < 100; i++ {
for i := range 100 {
fmt.Fprintf(buf, goodRec, i)
}
buf.WriteString("{}\n")
+2 -2
View File
@@ -72,7 +72,7 @@ func TestWordPress_Convert(t *testing.T) {
wp := WordPress{}
ch := wp.convert(strings.NewReader(xmlTestWP), "testWP")
comments := []store.Comment{}
comments := make([]store.Comment, 0, 3)
for c := range ch {
comments = append(comments, c)
}
@@ -100,7 +100,7 @@ func TestWP_Convert_MD(t *testing.T) {
wp := WordPress{}
ch := wp.convert(strings.NewReader(xmlTestWPmd), "siteID")
comments := []store.Comment{}
comments := make([]store.Comment, 0, 3)
for c := range ch {
comments = append(comments, c)
}
+33 -8
View File
@@ -4,14 +4,15 @@ import (
"bytes"
"context"
"fmt"
"html/template"
"net/url"
"text/template"
"time"
log "github.com/go-pkgz/lgr"
ntf "github.com/go-pkgz/notify"
"github.com/go-pkgz/repeater/v2"
"github.com/hashicorp/go-multierror"
"github.com/microcosm-cc/bluemonday"
"github.com/umputun/remark42/backend/app/templates"
)
@@ -26,7 +27,7 @@ type EmailParams struct {
SubscribeURL string // full subscribe handler URL
UnsubscribeURL string // full unsubscribe handler URL
TokenGenFn func(userID, email, site string) (string, error) // Unsubscribe token generation function
TokenGenFn func(userID, email, site string) (string, error) // unsubscribe token generation function
}
// Email implements notify.Destination for email
@@ -42,12 +43,12 @@ type Email struct {
type msgTmplData struct {
UserName string
UserPicture string
CommentText string
CommentText template.HTML
CommentLink string
CommentDate time.Time
ParentUserName string
ParentUserPicture string
ParentCommentText string
ParentCommentText template.HTML
ParentCommentLink string
ParentCommentDate time.Time
PostTitle string
@@ -56,6 +57,30 @@ type msgTmplData struct {
ForAdmin bool
}
// emailCommentPolicy sanitizes comment HTML for inclusion in notification emails.
// It is intentionally stricter than the store-level UGC policy used for web rendering:
// links (<a>) and images (<img>) are dropped so a comment can't smuggle phishing links
// or remote tracking pixels into an email sent from the legitimate remark42 address,
// while basic inline and block text formatting is preserved.
var emailCommentPolicy = func() *bluemonday.Policy {
p := bluemonday.NewPolicy()
p.AllowElements(
"p", "br", "hr", "div", "span",
"b", "strong", "i", "em", "u", "s", "strike", "del", "ins", "sub", "sup", "mark", "small",
"blockquote", "q", "cite",
"code", "pre", "kbd", "samp", "var",
"ul", "ol", "li", "dl", "dt", "dd",
"h1", "h2", "h3", "h4", "h5", "h6",
)
return p
}()
// emailSafeHTML strips links and images from pre-rendered comment HTML and returns
// it as template.HTML so html/template renders the remaining safe formatting as-is.
func emailSafeHTML(commentHTML string) template.HTML {
return template.HTML(emailCommentPolicy.Sanitize(commentHTML)) //nolint:gosec // sanitized above: <a>/<img> dropped, only formatting tags survive
}
// verifyTmplData store data for verification message template execution
type verifyTmplData struct {
User string
@@ -168,7 +193,7 @@ func (e *Email) buildAndSendMessage(ctx context.Context, req Request, email stri
ctx,
fmt.Sprintf("mailto:%s?from=%s&unsubscribeLink=%s&subject=%s",
email,
e.From,
url.QueryEscape(e.From),
url.QueryEscape(msg.unsubscribeLink),
url.QueryEscape(msg.subject),
),
@@ -203,7 +228,7 @@ func (e *Email) SendVerification(ctx context.Context, req VerificationRequest) e
ctx,
fmt.Sprintf("mailto:%s?from=%s&subject=%s",
req.Email,
e.From,
url.QueryEscape(e.From),
url.QueryEscape(e.VerificationSubject),
),
msg,
@@ -257,7 +282,7 @@ func (e *Email) buildMessageFromRequest(req Request, email string, forAdmin bool
tmplData := msgTmplData{
UserName: req.Comment.User.Name,
UserPicture: req.Comment.User.Picture,
CommentText: req.Comment.Text,
CommentText: emailSafeHTML(req.Comment.Text),
CommentLink: commentURLPrefix + req.Comment.ID,
CommentDate: req.Comment.Timestamp,
PostTitle: req.Comment.PostTitle,
@@ -269,7 +294,7 @@ func (e *Email) buildMessageFromRequest(req Request, email string, forAdmin bool
if req.Comment.ParentID != "" {
tmplData.ParentUserName = req.parent.User.Name
tmplData.ParentUserPicture = req.parent.User.Picture
tmplData.ParentCommentText = req.parent.Text
tmplData.ParentCommentText = emailSafeHTML(req.parent.Text)
tmplData.ParentCommentLink = commentURLPrefix + req.parent.ID
tmplData.ParentCommentDate = req.parent.Timestamp
}
+46 -2
View File
@@ -3,8 +3,8 @@ package notify
import (
"context"
"fmt"
"html/template"
"testing"
"text/template"
ntf "github.com/go-pkgz/notify"
"github.com/stretchr/testify/assert"
@@ -164,7 +164,7 @@ User: test_user
01.01.0001 at 00:00
Comment:
test@example.org for parent_user
Unsubscribe link: https://remark42.com/api/v1/email/unsubscribe?site=&tkn=token
Unsubscribe link: https://remark42.com/api/v1/email/unsubscribe?site=&amp;tkn=token
`, msg.body)
assert.Equal(t, "https://remark42.com/api/v1/email/unsubscribe?site=&tkn=token", msg.unsubscribeLink)
assert.Equal(t, `New reply to your comment for "test_title"`, msg.subject)
@@ -190,6 +190,50 @@ admin@example.org
assert.Empty(t, msg.unsubscribeLink)
}
func TestEmail_CommentTextSanitizedForEmail(t *testing.T) {
// comment HTML reaching the email path is sanitized by the store-level UGC policy,
// which permits <a> and <img>. The email must drop both so a comment can't inject
// phishing links or remote tracking pixels into a notification (GHSA-74pc-3r2m-ppx3).
email, err := NewEmail(EmailParams{
From: "from@example.org",
MsgTemplatePath: "testdata/msg.html.tmpl",
}, ntf.SMTPParams{})
require.NoError(t, err)
email.TokenGenFn = TokenGenFn
malicious := `hello <a href="https://phishing.example/verify">click to verify</a>` +
` <img src="https://attacker.example/track.png" width="1" height="1"> <b>kept</b>`
req := Request{
Comment: store.Comment{ID: "999", User: store.User{ID: "1", Name: "test_user"}, PostTitle: "test_title", Text: malicious},
Emails: []string{"test@example.org"},
}
msg, err := email.buildMessageFromRequest(req, req.Emails[0], false)
require.NoError(t, err)
assert.NotContains(t, msg.body, "phishing.example", "phishing link must be stripped")
assert.NotContains(t, msg.body, "attacker.example", "tracking pixel must be stripped")
assert.NotContains(t, msg.body, "<img", "no image tags in email body")
assert.NotContains(t, msg.body, "<a ", "no anchor tags in email body")
assert.Contains(t, msg.body, "click to verify", "anchor text is preserved, only the link is dropped")
assert.Contains(t, msg.body, "<b>kept</b>", "basic formatting is preserved")
}
// emailSafeHTML drops links/images while keeping inline/block formatting and escaping nothing extra.
func TestEmailSafeHTML(t *testing.T) {
tbl := []struct{ name, in, want string }{
{"strips anchor keeps text", `<a href="http://evil">x</a>`, "x"},
{"strips image entirely", `a<img src="http://evil/t.png">b`, "ab"},
{"keeps bold/italic/code", `<b>b</b><i>i</i><code>c</code>`, `<b>b</b><i>i</i><code>c</code>`},
{"keeps blockquote and lists", `<blockquote>q</blockquote><ul><li>x</li></ul>`, `<blockquote>q</blockquote><ul><li>x</li></ul>`},
{"drops onclick handlers", `<span onclick="alert(1)">s</span>`, `<span>s</span>`},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, string(emailSafeHTML(tt.in)))
})
}
}
func TestEmail_SendVerification(t *testing.T) {
email, err := NewEmail(EmailParams{
From: "from@example.org",
+8 -11
View File
@@ -4,7 +4,6 @@ import (
"context"
"fmt"
"sync"
"time"
log "github.com/go-pkgz/lgr"
)
@@ -22,14 +21,13 @@ type MockDest struct {
func (m *MockDest) Send(ctx context.Context, r Request) error {
m.lock.Lock()
defer m.lock.Unlock()
select {
case <-time.After(10 * time.Millisecond):
m.data = append(m.data, r)
log.Printf("sent %s -> %d", r.Comment.ID, m.id)
case <-ctx.Done():
if err := ctx.Err(); err != nil {
log.Printf("ctx closed %d", m.id)
m.closed = true
return nil
}
m.data = append(m.data, r)
log.Printf("sent %s -> %d", r.Comment.ID, m.id)
return nil
}
@@ -37,14 +35,13 @@ func (m *MockDest) Send(ctx context.Context, r Request) error {
func (m *MockDest) SendVerification(ctx context.Context, v VerificationRequest) error {
m.lock.Lock()
defer m.lock.Unlock()
select {
case <-time.After(10 * time.Millisecond):
m.verificationData = append(m.verificationData, v)
log.Printf("sent verification %s -> %d", v.User, m.id)
case <-ctx.Done():
if err := ctx.Err(); err != nil {
log.Printf("verification ctx closed %d", m.id)
m.closed = true
return nil
}
m.verificationData = append(m.verificationData, v)
log.Printf("sent verification %s -> %d", v.User, m.id)
return nil
}
+195 -190
View File
@@ -2,10 +2,9 @@ package notify
import (
"fmt"
"math/rand"
"sync/atomic"
"testing"
"time"
"testing/synctest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -26,250 +25,256 @@ func TestService_NoDestinations(t *testing.T) {
}
func TestService_WithDestinations(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
s.Submit(Request{Comment: store.Comment{ID: "100"}})
time.Sleep(time.Millisecond * 110)
s.Submit(Request{Comment: store.Comment{ID: "101"}})
time.Sleep(time.Millisecond * 110)
s.Submit(Request{Comment: store.Comment{ID: "102"}})
time.Sleep(time.Millisecond * 110)
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "100"}})
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "101"}})
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "102"}})
synctest.Wait()
s.Close()
require.Equal(t, 3, len(d1.Get()), "got all comments to d1")
require.Equal(t, 3, len(d2.Get()), "got all comments to d2")
require.Equal(t, 3, len(d1.Get()), "got all comments to d1")
require.Equal(t, 3, len(d2.Get()), "got all comments to d2")
assert.Equal(t, "100", d1.Get()[0].Comment.ID)
assert.Equal(t, "101", d1.Get()[1].Comment.ID)
assert.Equal(t, "102", d1.Get()[2].Comment.ID)
assert.Equal(t, "100", d1.Get()[0].Comment.ID)
assert.Equal(t, "101", d1.Get()[1].Comment.ID)
assert.Equal(t, "102", d1.Get()[2].Comment.ID)
})
}
func TestService_WithDrops(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
s.Submit(Request{Comment: store.Comment{ID: "100"}})
s.Submit(Request{Comment: store.Comment{ID: "101"}})
s.Submit(Request{Comment: store.Comment{ID: "102"}})
time.Sleep(time.Millisecond * 21)
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "100"}})
s.Submit(Request{Comment: store.Comment{ID: "101"}})
s.Submit(Request{Comment: store.Comment{ID: "102"}})
synctest.Wait()
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "111"}}) // safe to send after close
s.Submit(Request{Comment: store.Comment{ID: "111"}}) // safe to send after close
assert.LessOrEqual(t, len(d1.Get()), 2, "at least one comment from three dropped from d1, got: %v", d1.Get())
assert.LessOrEqual(t, len(d2.Get()), 2, "at least one comment from three dropped from d2, got: %v", d2.Get())
assert.LessOrEqual(t, len(d1.Get()), 2, "at least one comment from three dropped from d1, got: %v", d1.Get())
assert.LessOrEqual(t, len(d2.Get()), 2, "at least one comment from three dropped from d2, got: %v", d2.Get())
})
}
func TestService_SubmitVerificationWithDrops(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 1, d1, d2)
assert.NotNil(t, s)
s.SubmitVerification(VerificationRequest{
SiteID: "remark",
User: "testUser",
Email: "test@example.org",
Token: "testToken",
s.SubmitVerification(VerificationRequest{
SiteID: "remark",
User: "testUser",
Email: "test@example.org",
Token: "testToken",
})
s.SubmitVerification(VerificationRequest{})
s.SubmitVerification(VerificationRequest{})
synctest.Wait()
s.Close()
s.SubmitVerification(VerificationRequest{}) // safe to send after close
assert.LessOrEqual(t, len(d2.GetVerify()), 2, "one request from three dropped from d2, got: %v", d2.GetVerify())
verifyDest := d1.GetVerify()
require.LessOrEqual(t, len(verifyDest), 2, "one request from three dropped from d1, got: %v", verifyDest)
assert.Equal(t, "remark", verifyDest[0].SiteID)
assert.Equal(t, "testUser", verifyDest[0].User)
assert.Equal(t, "test@example.org", verifyDest[0].Email)
assert.Equal(t, "testToken", verifyDest[0].Token)
})
s.SubmitVerification(VerificationRequest{})
s.SubmitVerification(VerificationRequest{})
time.Sleep(time.Millisecond * 21)
s.Close()
s.SubmitVerification(VerificationRequest{}) // safe to send after close
assert.LessOrEqual(t, len(d2.GetVerify()), 2, "one request from three dropped from d2, got: %v", d2.GetVerify())
verifyDest := d1.GetVerify()
require.LessOrEqual(t, len(verifyDest), 2, "one request from three dropped from d1, got: %v", verifyDest)
assert.Equal(t, "remark", verifyDest[0].SiteID)
assert.Equal(t, "testUser", verifyDest[0].User)
assert.Equal(t, "test@example.org", verifyDest[0].Email)
assert.Equal(t, "testToken", verifyDest[0].Token)
}
func TestService_Many(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 5, d1, d2)
assert.NotNil(t, s)
synctest.Test(t, func(t *testing.T) {
d1, d2 := &MockDest{id: 1}, &MockDest{id: 2}
s := NewService(nil, 5, d1, d2)
assert.NotNil(t, s)
for i := 0; i < 10; i++ {
s.Submit(Request{Comment: store.Comment{ID: fmt.Sprintf("%d", 100+i)}})
s.SubmitVerification(VerificationRequest{User: fmt.Sprintf("%d", 100+i)})
time.Sleep(time.Millisecond * time.Duration(rand.Int31n(20)))
}
s.Close()
for i := range 10 {
s.Submit(Request{Comment: store.Comment{ID: fmt.Sprintf("%d", 100+i)}})
s.SubmitVerification(VerificationRequest{User: fmt.Sprintf("%d", 100+i)})
}
s.Close()
// wait for destinations to close
assert.Eventually(t, func() bool { return d1.IsClosed() && d2.IsClosed() }, 100*time.Millisecond, 10*time.Millisecond)
assert.NotEqual(t, 10, len(d1.Get()), "some comments dropped from d1")
assert.NotEqual(t, 10, len(d1.GetVerify()), "some verifications dropped from d1")
assert.NotEqual(t, 10, len(d2.Get()), "some comments dropped from d2")
assert.NotEqual(t, 10, len(d2.GetVerify()), "some verifications dropped from d2")
assert.True(t, d1.IsClosed())
assert.True(t, d2.IsClosed())
assert.Equal(t, "mock id=1, closed=true", d1.String())
assert.NotEqual(t, 10, len(d1.Get()), "some comments dropped from d1")
assert.NotEqual(t, 10, len(d1.GetVerify()), "some verifications dropped from d1")
assert.NotEqual(t, 10, len(d2.Get()), "some comments dropped from d2")
assert.NotEqual(t, 10, len(d2.GetVerify()), "some verifications dropped from d2")
})
}
func TestService_WithParent(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}}
synctest.Test(t, func(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}}
dataStore.data["p1"] = store.Comment{ID: "p1"}
dataStore.data["p2"] = store.Comment{ID: "p2"}
dataStore.data["p1"] = store.Comment{ID: "p1"}
dataStore.data["p2"] = store.Comment{ID: "p2"}
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s.Submit(Request{Comment: store.Comment{ID: "c1", ParentID: "p1"}})
time.Sleep(time.Millisecond * 110)
s.Submit(Request{Comment: store.Comment{ID: "c11", ParentID: "p11"}})
time.Sleep(time.Millisecond * 110)
s.Close()
s.Submit(Request{Comment: store.Comment{ID: "c1", ParentID: "p1"}})
synctest.Wait()
s.Submit(Request{Comment: store.Comment{ID: "c11", ParentID: "p11"}})
synctest.Wait()
s.Close()
destRes := dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ParentID)
assert.Equal(t, "p1", destRes[0].parent.ID)
assert.Equal(t, "p11", destRes[1].Comment.ParentID)
assert.Equal(t, "", destRes[1].parent.ID)
destRes := dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ParentID)
assert.Equal(t, "p1", destRes[0].parent.ID)
assert.Equal(t, "p11", destRes[1].Comment.ParentID)
assert.Equal(t, "", destRes[1].parent.ID)
})
}
func TestService_EmailRetrieval(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
synctest.Test(t, func(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.userDetails["u1"] = "u1@example.com"
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.userDetails["u1"] = "u1@example.com"
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
// one comment, one notification
s.Submit(Request{Comment: dataStore.data["p1"]})
time.Sleep(time.Millisecond * 110)
// one comment, one notification
s.Submit(Request{Comment: dataStore.data["p1"]})
synctest.Wait()
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
// reply to the first comment, same comment as one in original comment
s.Submit(Request{Comment: dataStore.data["p2"]})
time.Sleep(time.Millisecond * 110)
// reply to the first comment, same comment as one in original comment
s.Submit(Request{Comment: dataStore.data["p2"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.Empty(t, destRes[1].Emails, "u1 is not notified they are the one who left the comment")
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.Empty(t, destRes[1].Emails, "u1 is not notified they are the one who left the comment")
// another reply to the first comment, another user
s.Submit(Request{Comment: dataStore.data["p3"]})
time.Sleep(time.Millisecond * 110)
// another reply to the first comment, another user
s.Submit(Request{Comment: dataStore.data["p3"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p1", destRes[2].parent.ID)
assert.Equal(t, "u1", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p1", destRes[2].parent.ID)
assert.Equal(t, "u1", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
// reply to the last comment by another user, should trigger email retrieval error
s.Submit(Request{Comment: dataStore.data["p4"]})
time.Sleep(time.Millisecond * 110)
// reply to the last comment by another user, should trigger email retrieval error
s.Submit(Request{Comment: dataStore.data["p4"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u2", destRes[3].parent.User.ID)
assert.Empty(t, destRes[3].Emails, "no email can be retrieved for u2")
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u2", destRes[3].parent.User.ID)
assert.Empty(t, destRes[3].Emails, "no email can be retrieved for u2")
s.Close()
s.Close()
})
}
func TestService_Recursive(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
synctest.Test(t, func(t *testing.T) {
dest := &MockDest{id: 1}
dataStore := &mockStore{data: map[string]store.Comment{}, userDetails: map[string]string{}}
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p2", User: store.User{ID: "u3"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.data["p5"] = store.Comment{ID: "p5", ParentID: "p4", User: store.User{ID: "u4"}}
dataStore.userDetails["u1"] = "u1@example.com"
// second comment goes without email address for notification
dataStore.userDetails["u3"] = "u3@example.com"
dataStore.data["p1"] = store.Comment{ID: "p1", User: store.User{ID: "u1"}}
dataStore.data["p2"] = store.Comment{ID: "p2", ParentID: "p1", User: store.User{ID: "u2"}}
dataStore.data["p3"] = store.Comment{ID: "p3", ParentID: "p2", User: store.User{ID: "u3"}}
dataStore.data["p4"] = store.Comment{ID: "p4", ParentID: "p3", User: store.User{ID: "u1"}}
dataStore.data["p5"] = store.Comment{ID: "p5", ParentID: "p4", User: store.User{ID: "u4"}}
dataStore.userDetails["u1"] = "u1@example.com"
// second comment goes without email address for notification
dataStore.userDetails["u3"] = "u3@example.com"
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
s := NewService(dataStore, 1, dest)
assert.NotNil(t, s)
// one comment from u1 with email set
s.Submit(Request{Comment: dataStore.data["p1"]})
time.Sleep(time.Millisecond * 110)
// one comment from u1 with email set
s.Submit(Request{Comment: dataStore.data["p1"]})
synctest.Wait()
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
destRes := dest.Get()
require.Equal(t, 1, len(destRes), "one comment notified")
assert.Equal(t, "p1", destRes[0].Comment.ID)
assert.Empty(t, destRes[0].parent)
assert.Empty(t, destRes[0].Emails)
// reply to the first comment from u2 without email set
s.Submit(Request{Comment: dataStore.data["p2"]})
time.Sleep(time.Millisecond * 110)
// reply to the first comment from u2 without email set
s.Submit(Request{Comment: dataStore.data["p2"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[1].Emails)
destRes = dest.Get()
require.Equal(t, 2, len(destRes), "two comment notified")
assert.Equal(t, "p2", destRes[1].Comment.ID)
assert.Equal(t, "p1", destRes[1].parent.ID)
assert.Equal(t, "u1", destRes[1].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[1].Emails)
// reply to the second comment from u3 with email set
s.Submit(Request{Comment: dataStore.data["p3"]})
time.Sleep(time.Millisecond * 110)
// reply to the second comment from u3 with email set
s.Submit(Request{Comment: dataStore.data["p3"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p2", destRes[2].parent.ID)
assert.Equal(t, "u2", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
destRes = dest.Get()
require.Equal(t, 3, len(destRes), "three comment notified")
assert.Equal(t, "p3", destRes[2].Comment.ID)
assert.Equal(t, "p2", destRes[2].parent.ID)
assert.Equal(t, "u2", destRes[2].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com"}, destRes[2].Emails)
// reply to the third comment from u1 (author of the first comment), only u3 should be notified
s.Submit(Request{Comment: dataStore.data["p4"]})
time.Sleep(time.Millisecond * 110)
// reply to the third comment from u1 (author of the first comment), only u3 should be notified
s.Submit(Request{Comment: dataStore.data["p4"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified once each")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u3", destRes[3].parent.User.ID)
assert.ElementsMatch(t, []string{"u3@example.com"}, destRes[3].Emails, "u1 is not notified they are the one who left the comment")
destRes = dest.Get()
require.Equal(t, 4, len(destRes), "four comment notified once each")
assert.Equal(t, "p4", destRes[3].Comment.ID)
assert.Equal(t, "p3", destRes[3].parent.ID)
assert.Equal(t, "u3", destRes[3].parent.User.ID)
assert.ElementsMatch(t, []string{"u3@example.com"}, destRes[3].Emails, "u1 is not notified they are the one who left the comment")
// reply to the fourth comment from u4, u1 and u3 should be notified once as a result
s.Submit(Request{Comment: dataStore.data["p5"]})
time.Sleep(time.Millisecond * 110)
// reply to the fourth comment from u4, u1 and u3 should be notified once as a result
s.Submit(Request{Comment: dataStore.data["p5"]})
synctest.Wait()
destRes = dest.Get()
require.Equal(t, 5, len(destRes), "four comment notified once each")
assert.Equal(t, "p5", destRes[4].Comment.ID)
assert.Equal(t, "p4", destRes[4].parent.ID)
assert.Equal(t, "u1", destRes[4].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com", "u3@example.com"}, destRes[4].Emails, "u3 and u1 notified once")
destRes = dest.Get()
require.Equal(t, 5, len(destRes), "four comment notified once each")
assert.Equal(t, "p5", destRes[4].Comment.ID)
assert.Equal(t, "p4", destRes[4].parent.ID)
assert.Equal(t, "u1", destRes[4].parent.User.ID)
assert.ElementsMatch(t, []string{"u1@example.com", "u3@example.com"}, destRes[4].Emails, "u3 and u1 notified once")
s.Close()
s.Close()
})
}
func TestService_Nop(t *testing.T) {
+3 -3
View File
@@ -15,7 +15,7 @@ import (
)
type tgRequester interface {
Request(ctx context.Context, method string, b []byte, data interface{}) error
Request(ctx context.Context, method string, b []byte, data any) error
}
// TGUpdatesReceiver used to dispatch telegram updates to multiple receivers
@@ -27,8 +27,8 @@ type TGUpdatesReceiver interface {
// DispatchTelegramUpdates dispatches telegram updates to provided list of receivers
// Blocks caller
func DispatchTelegramUpdates(ctx context.Context, requester tgRequester, receivers []TGUpdatesReceiver, period time.Duration) {
// Identifier of the first update to be requested.
// Should be equal to LastSeenUpdateID + 1
// identifier of the first update to be requested.
// should be equal to LastSeenUpdateID + 1
// See https://core.telegram.org/bots/api#getupdates
var updateOffset int
+10 -7
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"testing"
"testing/synctest"
"time"
ntf "github.com/go-pkgz/notify"
@@ -12,12 +13,14 @@ import (
)
func TestDispatchTelegramUpdates(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
poolPeriod := time.Millisecond * 100
go DispatchTelegramUpdates(ctx, &mockTGRequester{t: t}, []TGUpdatesReceiver{&mockTGUpdatesReceiver{t: t}}, poolPeriod)
time.Sleep(poolPeriod * 3)
cancel()
time.Sleep(poolPeriod)
synctest.Test(t, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
poolPeriod := time.Millisecond * 100
go DispatchTelegramUpdates(ctx, &mockTGRequester{t: t}, []TGUpdatesReceiver{&mockTGUpdatesReceiver{t: t}}, poolPeriod)
time.Sleep(poolPeriod * 3)
cancel()
synctest.Wait()
})
}
const getUpdatesResp = `{
@@ -39,7 +42,7 @@ type mockTGRequester struct {
t *testing.T
}
func (m *mockTGRequester) Request(_ context.Context, _ string, _ []byte, data interface{}) error {
func (m *mockTGRequester) Request(_ context.Context, _ string, _ []byte, data any) error {
if m.hit < 2 {
m.hit++
assert.NoError(m.t, json.Unmarshal([]byte(getUpdatesResp), data))
+32 -14
View File
@@ -1,13 +1,15 @@
package api
import (
"errors"
"fmt"
"net/http"
"path"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
@@ -43,7 +45,7 @@ type adminStore interface {
// DELETE /comment/{id}?site=siteID&url=post-url - removes comment
func (a *admin) deleteCommentCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
id := r.PathValue("id")
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
log.Printf("[INFO] delete comment %s", id)
@@ -58,7 +60,7 @@ func (a *admin) deleteCommentCtrl(w http.ResponseWriter, r *http.Request) {
// DELETE /user/{userid}?site=side-id - delete all user comments for requested userid
func (a *admin) deleteUserCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
log.Printf("[INFO] delete all user comments for %s, site %s", userID, siteID)
@@ -72,7 +74,7 @@ func (a *admin) deleteUserCtrl(w http.ResponseWriter, r *http.Request) {
// GET /user/{userid}?site=side-id - get user info for requested userid
func (a *admin) getUserInfoCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
log.Printf("[INFO] get user info for %s, site %s", userID, siteID)
@@ -103,7 +105,7 @@ func (a *admin) deleteMeRequestCtrl(w http.ResponseWriter, r *http.Request) {
return
}
// Audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
// audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
if len(claims.Audience) != 1 {
rest.SendErrorJSON(w, r, http.StatusBadRequest, fmt.Errorf("bad request"), "can't process token, claims.Audience expected to be a single element but it's not", rest.ErrActionRejected)
return
@@ -123,10 +125,15 @@ func (a *admin) deleteMeRequestCtrl(w http.ResponseWriter, r *http.Request) {
}
if claims.User.Picture != "" && a.authenticator.AvatarProxy() != nil {
avatarStore := a.authenticator.AvatarProxy().Store
if err = avatarStore.Remove(path.Base(claims.User.Picture)); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete user's avatar", rest.ErrInternal)
return
if avatarID := avatarIDFromPicture(claims.User.Picture); avatarID != "" {
// an already-removed avatar is fine (a repeated request stays idempotent), but a genuine
// store failure is surfaced now that avatar.ErrNotFound lets us tell the two apart
if err = a.authenticator.AvatarProxy().Store.Remove(avatarID); err != nil && !errors.Is(err, avatar.ErrNotFound) {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't delete user's avatar", rest.ErrInternal)
return
}
} else {
log.Printf("[WARN] unexpected avatar picture %q for user %s on site %s, skipping removal", claims.User.Picture, claims.User.ID, audience)
}
}
@@ -134,9 +141,20 @@ func (a *admin) deleteMeRequestCtrl(w http.ResponseWriter, r *http.Request) {
R.RenderJSON(w, R.JSON{"user_id": claims.User.ID, "site_id": claims.Audience})
}
// avatarIDFromPicture returns the avatar-store object id for a user picture, or "" if the picture
// does not resolve to a well-formed id (the store names its objects "<hash>.image"). Guarding on the
// id shape keeps a malformed picture, e.g. a path sentinel, from making a filesystem-backed store
// target an unexpected path.
func avatarIDFromPicture(picture string) string {
if id := path.Base(picture); strings.HasSuffix(id, ".image") {
return id
}
return ""
}
// PUT /user/{userid}?site=side-id&block=1&ttl=7d - block or unblock user
func (a *admin) setBlockCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
blockStatus := r.URL.Query().Get("block") == "1"
@@ -202,7 +220,7 @@ func (a *admin) setReadOnlyCtrl(w http.ResponseWriter, r *http.Request) {
// PUT /title/{id}?site=siteID&url=post-url - set comment PostTitle to page's title
func (a *admin) setTitleCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
id := r.PathValue("id")
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
c, err := a.dataService.SetTitle(locator, id)
@@ -216,9 +234,9 @@ func (a *admin) setTitleCtrl(w http.ResponseWriter, r *http.Request) {
R.RenderJSON(w, R.JSON{"id": id, "locator": locator})
}
// PUT /verify?site=siteID&url=post-url&ro=1 - set or reset read-only status for the post
// PUT /verify/{userid}?site=siteID&verified=1 - set or reset verified status for the user
func (a *admin) setVerifyCtrl(w http.ResponseWriter, r *http.Request) {
userID := chi.URLParam(r, "userid")
userID := r.PathValue("userid")
siteID := r.URL.Query().Get("site")
verifyStatus := r.URL.Query().Get("verified") == "1"
@@ -233,7 +251,7 @@ func (a *admin) setVerifyCtrl(w http.ResponseWriter, r *http.Request) {
// PUT /pin/{id}?site=siteID&url=post-url&pin=1
// mark/unmark comment as a special
func (a *admin) setPinCtrl(w http.ResponseWriter, r *http.Request) {
commentID := chi.URLParam(r, "id")
commentID := r.PathValue("id")
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
pinStatus := r.URL.Query().Get("pin") == "1"
+127 -8
View File
@@ -87,7 +87,7 @@ func TestAdmin_Delete(t *testing.T) {
// check count updated
res, code = get(t, ts.URL+"/api/v1/count?site=remark42&url=https://radio-t.com/blah")
assert.Equal(t, http.StatusOK, code)
b := map[string]interface{}{}
b := map[string]any{}
err = json.Unmarshal([]byte(res), &b)
assert.NoError(t, err)
t.Logf("%#v", b)
@@ -465,7 +465,7 @@ func TestAdmin_ReadOnly(t *testing.T) {
Locator: store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah"}}
b, err := json.Marshal(c)
assert.NoError(t, err, "can't marshal comment %+v", c)
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", bytes.NewBuffer(b))
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", bytes.NewBuffer(b))
require.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
require.NoError(t, err)
@@ -489,7 +489,7 @@ func TestAdmin_ReadOnly(t *testing.T) {
Locator: store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah"}}
b, err = json.Marshal(c)
assert.NoError(t, err, "can't marshal comment %+v", c)
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", bytes.NewBuffer(b))
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site="+c.Locator.SiteID, bytes.NewBuffer(b))
require.NoError(t, err)
resp, err = sendReq(t, req, adminUmputunToken)
require.NoError(t, err)
@@ -717,8 +717,8 @@ func TestAdmin_DeleteMeRequest(t *testing.T) {
},
User: &token.User{
ID: "user1",
Picture: "pic.image",
Attributes: map[string]interface{}{
Picture: "https://demo.remark42.com/api/v1/avatar/pic.image", // production-shaped URL: removal must path.Base it to the avatar id
Attributes: map[string]any{
"delete_me": true,
},
},
@@ -747,6 +747,124 @@ func TestAdmin_DeleteMeRequest(t *testing.T) {
email, err = srv.DataService.GetUserEmail("remark42", "user1")
assert.NoError(t, err)
assert.Empty(t, email, "user1 email was deleted")
assert.NoFileExists(t, os.TempDir()+"/ava-remark42/42/pic.image", "user's avatar should be removed on deleteme")
}
// a delete_me request whose token carries a picture must still succeed when the avatar is
// already gone from the store: the user data is deleted and a missing avatar is tolerated
func TestAdmin_DeleteMeRequestMissingAvatar(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
c1 := store.Comment{Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user3 name", ID: "user3"}}
_, err := srv.DataService.Create(c1)
require.NoError(t, err)
claims := token.Claims{
SessionOnly: true,
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "2345678",
Issuer: "remark42",
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user3",
Picture: "missing.image", // no avatar file exists for this picture in the store
Attributes: map[string]any{
"delete_me": true,
},
},
}
tkn, err := srv.Authenticator.TokenService().Token(claims)
require.NoError(t, err)
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "a missing avatar must not fail the deletion")
_, err = srv.DataService.User("remark42", "user3", 0, 0, store.User{})
assert.EqualError(t, err, "no comments for user user3 in store", "user3 comments should be deleted")
}
// a genuine (non not-found) avatar-store failure must now surface, not be silently swallowed:
// avatar.ErrNotFound lets deleteMeRequestCtrl tell an already-gone avatar from a real error
func TestAdmin_DeleteMeRequestAvatarRemoveError(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
c1 := store.Comment{Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user5 name", ID: "user5"}}
_, err := srv.DataService.Create(c1)
require.NoError(t, err)
// put a non-empty directory where the avatar file is expected, so Store.Remove fails with a real
// error (directory not empty), not os.ErrNotExist - "pic" hashes to partition 42
require.NoError(t, os.MkdirAll(os.TempDir()+"/ava-remark42/42/pic.image", 0o700))
require.NoError(t, os.WriteFile(os.TempDir()+"/ava-remark42/42/pic.image/child", []byte("x"), 0o600))
claims := token.Claims{
SessionOnly: true,
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "4567890",
Issuer: "remark42",
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user5",
Picture: "https://demo.remark42.com/api/v1/avatar/pic.image",
Attributes: map[string]any{
"delete_me": true,
},
},
}
tkn, err := srv.Authenticator.TokenService().Token(claims)
require.NoError(t, err)
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusInternalServerError, resp.StatusCode, "a real avatar-store failure must surface, not be swallowed")
}
func TestAvatarIDFromPicture(t *testing.T) {
tbl := []struct {
name string
picture string
want string
}{
{"local avatar url", "https://demo.remark42.com/api/v1/avatar/cb42ff493ade696d88a3a590f136ae9e34de7c1b.image", "cb42ff493ade696d88a3a590f136ae9e34de7c1b.image"},
{"bare avatar id", "pic.image", "pic.image"},
{"parent sentinel", "https://demo.remark42.com/api/v1/avatar/..", ""},
{"trailing slash", "https://demo.remark42.com/api/v1/avatar/", ""},
{"root", "/", ""},
{"dotdot", "..", ""},
{"empty", "", ""},
{"provider url without image suffix", "https://example.com/pic.png", ""},
}
for _, tc := range tbl {
t.Run(tc.name, func(t *testing.T) {
assert.Equal(t, tc.want, avatarIDFromPicture(tc.picture))
})
}
}
func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
@@ -786,7 +904,7 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
},
User: &token.User{
ID: "provider1_user1",
Attributes: map[string]interface{}{
Attributes: map[string]any{
"delete_me": true,
},
},
@@ -802,7 +920,8 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try bad user
// unknown user: deletion is idempotent, so a valid (signed) delete_me token for a user with
// no stored data is a no-op success rather than an error
badClaimsUser := claims
badClaimsUser.User.ID = "no-such-id"
tkn, err = srv.Authenticator.TokenService().Token(badClaimsUser)
@@ -813,7 +932,7 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
resp, err = client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusBadRequest, resp.StatusCode, resp.Status)
assert.Equal(t, http.StatusOK, resp.StatusCode, resp.Status)
badClaimsUser.User.ID = "provider1_user1"
// try without deleteme flag
+353
View File
@@ -0,0 +1,353 @@
// Package api middleware: request-scoped HTTP middlewares used by the REST router.
package api
import (
"fmt"
"net"
"net/http"
"net/mail"
"regexp"
"strings"
"time"
"github.com/didip/tollbooth/v8"
"github.com/didip/tollbooth/v8/limiter"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/store"
)
// ipForwardingHeaders are the request headers R.RealIP derives the client IP from.
var ipForwardingHeaders = []string{"X-Real-IP", "X-Forwarded-For", "CF-Connecting-IP"}
// realIPMiddleware derives the client IP from forwarding headers (X-Real-IP / X-Forwarded-For /
// CF-Connecting-IP) via R.RealIP, but honors those headers only for requests whose direct peer
// is one of the trusted proxies. For any other peer it drops those headers and pins RemoteAddr to
// the real socket IP, so an untrusted client can't spoof the IP that per-IP controls (rate limiting,
// vote dedup, comment IP, anonymous id) and the request log key on.
//
// With no trusted proxies configured it falls back to trusting the headers from any client (the
// historical behavior). That is spoofable by design, so operators running behind a reverse proxy
// should set --trusted-proxy to the proxy's network — see the "trusted proxy" docs.
func realIPMiddleware(trustedProxies []*net.IPNet) func(http.Handler) http.Handler {
if len(trustedProxies) == 0 {
return R.RealIP
}
return func(next http.Handler) http.Handler {
fromTrusted := R.RealIP(next) // rewrites RemoteAddr from the forwarding headers
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
peer := directPeerIP(r.RemoteAddr)
if peer != nil && cidrsContain(trustedProxies, peer) {
fromTrusted.ServeHTTP(w, r) // trusted proxy: honor the forwarding headers
return
}
// untrusted peer: drop the forwarding headers and pin RemoteAddr to the real socket IP,
// so nothing downstream can be fooled by a spoofed header (R.RealIP normalizes
// RemoteAddr to a bare IP for trusted peers; do the same here for consistency)
for _, h := range ipForwardingHeaders {
r.Header.Del(h)
}
if peer != nil {
r.RemoteAddr = peer.String()
}
next.ServeHTTP(w, r)
})
}
}
// directPeerIP extracts the IP from a "host:port" (or bare host) RemoteAddr, or nil if unparseable.
func directPeerIP(remoteAddr string) net.IP {
host, _, err := net.SplitHostPort(remoteAddr)
if err != nil {
host = remoteAddr // may already be a bare IP with no port
}
return net.ParseIP(host)
}
// TrustsAnyPeer reports whether the trusted-proxy list contains a catch-all (0.0.0.0/0 or ::/0),
// which trusts forwarding headers from every client and re-opens the IP-spoofing bypass.
func TrustsAnyPeer(cidrs []*net.IPNet) bool {
for _, c := range cidrs {
if ones, _ := c.Mask.Size(); ones == 0 {
return true
}
}
return false
}
// cidrsContain reports whether ip falls within any of the CIDRs.
func cidrsContain(cidrs []*net.IPNet, ip net.IP) bool {
for _, c := range cidrs {
if c.Contains(ip) {
return true
}
}
return false
}
// ParseTrustedProxies parses a list of trusted-proxy entries into CIDRs. Each entry may be a CIDR
// (e.g. 172.16.0.0/12) or a bare IP (treated as a single host). Blank entries are skipped; a
// malformed entry is a hard error so a typo can't silently disable proxy trust.
func ParseTrustedProxies(entries []string) ([]*net.IPNet, error) {
var out []*net.IPNet
for _, e := range entries {
e = strings.TrimSpace(e)
if e == "" {
continue
}
if !strings.Contains(e, "/") { // bare IP -> single-host CIDR
ip := net.ParseIP(e)
if ip == nil {
return nil, fmt.Errorf("invalid trusted proxy %q", e)
}
// build the network from the normalized IP so a v4-mapped IPv6 (e.g. ::ffff:10.0.0.1)
// yields the intended /32 host, not a huge ::/32 range
bits := 128
if v4 := ip.To4(); v4 != nil {
ip, bits = v4, 32
}
out = append(out, &net.IPNet{IP: ip, Mask: net.CIDRMask(bits, bits)})
continue
}
_, network, err := net.ParseCIDR(e)
if err != nil {
return nil, fmt.Errorf("invalid trusted proxy CIDR %q: %w", e, err)
}
out = append(out, network)
}
return out, nil
}
// corsMiddleware builds the CORS middleware for the public API. With AllowedOrigins
// "*" and credentials enabled, rest.CORS reflects the request Origin into
// Access-Control-Allow-Origin (rather than a literal "*"), which browsers require
// for credentialed cross-origin requests.
func corsMiddleware() func(http.Handler) http.Handler {
return R.CORS(
R.CorsAllowedOrigins("*"),
R.CorsAllowedMethods("GET", "POST", "PUT", "DELETE", "OPTIONS"),
R.CorsAllowedHeaders("Accept", "Authorization", "Content-Type", "X-XSRF-Token", "X-JWT"),
R.CorsExposedHeaders("Authorization"),
R.CorsAllowCredentials(true),
R.CorsMaxAge(300),
)
}
// rejectHead rejects HEAD requests with 405, advertising the given allowed methods in
// the Allow header. net/http.ServeMux routes HEAD to a "GET ..." handler, but per RFC
// 9110 GET/HEAD are safe methods; this guard is applied to the few GET routes whose
// handlers mutate state so they cannot be triggered by a (nominally side-effect-free)
// HEAD, preserving the pre-routegroup behavior. allow lists every method the resource
// supports (e.g. "GET" or "GET, POST") so the 405 Allow header is accurate.
func rejectHead(allow string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodHead {
w.Header().Set("Allow", allow)
http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
return
}
next.ServeHTTP(w, r)
})
}
}
// rejectAnonUser is a middleware rejecting anonymous users
func rejectAnonUser(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if strings.HasPrefix(user.ID, "anonymous_") {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// matchSiteID is a middleware rejecting users with mismatch between site param and and User.SiteID
func matchSiteID(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
// skip for basic auth user
if user.Name == "admin" && user.ID == "admin" {
next.ServeHTTP(w, r)
return
}
siteID := r.URL.Query().Get("site")
// require an explicit site so the user.SiteID check below cannot be bypassed
// by simply omitting the query parameter
if siteID == "" || user.SiteID != siteID {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// cacheControl is a middleware setting cache expiration. Using url+version as etag
func cacheControl(expiration time.Duration, version string) func(http.Handler) http.Handler {
etag := func(r *http.Request, version string) string {
s := version + ":" + r.URL.String()
return store.EncodeID(s)
}
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
e := `"` + etag(r, version) + `"`
w.Header().Set("Etag", e)
w.Header().Set("Cache-Control", fmt.Sprintf("max-age=%d, no-cache", int(expiration.Seconds())))
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, e) {
w.WriteHeader(http.StatusNotModified)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// apiCSPMiddleware overrides the global Content-Security-Policy on /api/v1 routes
// with a strict, default-deny policy. The global CSP (securityHeadersMiddleware) keeps
// 'self' 'unsafe-inline' for script-src/style-src because the widget HTML pages
// (/web/*.html) need inline bootstrap blocks. API responses serve JSON, XML/RSS, or
// images — none of those should ever execute scripts when rendered, so they get the
// strictest policy available as defense-in-depth against future trust-boundary bugs.
//
// Image-serving handlers (/api/v1/img, /api/v1/picture/{user}/{id}) re-apply the same
// rest.StrictImageCSP value at the handler level and additionally set Content-Disposition:
// inline; filename="image" (framing the response as a file rather than a renderable
// document) and X-Content-Type-Options: nosniff. The CSP re-apply is intentional belt-and-
// braces: if a future route refactor bypasses this middleware, the image handlers still
// emit the policy.
func apiCSPMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Security-Policy", rest.StrictImageCSP)
next.ServeHTTP(w, r)
})
}
// securityHeadersMiddleware sets security-related headers:
// - Content-Security-Policy: controls which resources the browser is allowed to load
// - Permissions-Policy: disables browser features (camera, mic, etc.) not needed by a comment widget
// - X-Content-Type-Options: prevents browsers from MIME-sniffing responses away from the declared type,
// stopping e.g. a user-uploaded image from being reinterpreted as executable HTML/JS
// - Referrer-Policy: controls how much URL information leaks in the Referer header on cross-origin
// requests; "strict-origin-when-cross-origin" sends only the origin (no path) to other domains
// and nothing at all on HTTPS→HTTP downgrades
func securityHeadersMiddleware(imageProxyEnabled bool, allowedAncestors []string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
imgSrc := "*"
if imageProxyEnabled {
imgSrc = "'self'"
}
frameAncestors := "*"
if len(allowedAncestors) > 0 {
frameAncestors = strings.Join(allowedAncestors, " ")
}
// font-src is set to 'none' (no @font-face / no base64 fonts in the bundle).
w.Header().Set("Content-Security-Policy", fmt.Sprintf("default-src 'none'; base-uri 'none'; form-action 'none'; connect-src 'self'; frame-src 'self' mailto:; img-src %s; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; font-src 'none'; object-src 'none'; frame-ancestors %s;", imgSrc, frameAncestors))
w.Header().Set("Permissions-Policy", "accelerometer=(), autoplay=(), camera=(), cross-origin-isolated=(), display-capture=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), unload=(), window-management=()")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Referrer-Policy", "strict-origin-when-cross-origin")
next.ServeHTTP(w, r)
})
}
}
// subscribersOnly is a middleware rejecting non-paid_sub users
func subscribersOnly(enable bool) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if enable {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if !user.PaidSub {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// validEmailAuth is a middleware for auth endpoints for email method.
// it rejects login request if user, site or email are suspicious
func validEmailAuth() func(http.Handler) http.Handler {
reUser := regexp.MustCompile(`^[\p{L}\d\s_]{4,64}$`) // matches ui side validation, adding min/max limitation
reSite := regexp.MustCompile(`^[a-zA-Z\d\s_.-]{1,64}$`)
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/auth/email/login" {
// not email login, skip the check
h.ServeHTTP(w, r)
return
}
if u := r.URL.Query().Get("user"); u != "" {
if !reUser.MatchString(u) {
log.Printf("[WARN] suspicious user rejected: %s", u)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if a := r.URL.Query().Get("address"); a != "" {
if _, err := mail.ParseAddress(a); err != nil {
log.Printf("[WARN] suspicious address rejected: %s", a)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if s := r.URL.Query().Get("site"); s != "" {
if !reSite.MatchString(s) {
log.Printf("[WARN] suspicious site rejected: %s", s)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// rateLimiter creates a rate limiting middleware with proper IP lookup configuration.
// tollbooth v8 requires explicit IP lookup method to be set.
// keys on RemoteAddr, which realIPMiddleware sets to the client IP (from the forwarding
// headers for trusted proxies, otherwise the real socket IP).
func rateLimiter(maxReq float64) func(http.Handler) http.Handler {
lmt := tollbooth.NewLimiter(maxReq, nil)
lmt.SetIPLookup(limiter.IPLookup{
Name: "RemoteAddr",
IndexFromRight: 0,
})
return tollbooth.HTTPMiddleware(lmt)
}
+406
View File
@@ -0,0 +1,406 @@
package api
import (
"fmt"
"net"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"
"github.com/go-pkgz/auth/v2/token"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/store"
)
// routes() wraps bounded routes with the enforcing rest.Timeout and deliberately leaves the
// streaming/long-polling routes (GET /export, /userdata, /wait) without it. This checks that
// contract holds against the vendored middleware: a slow handler under R.Timeout is aborted with
// 504 at the deadline, while a route left without it runs to completion.
func TestRouteTimeout(t *testing.T) {
slow := func(d time.Duration) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
select {
case <-r.Context().Done(): // return promptly once the enforcing timeout cancels the context
case <-time.After(d):
}
w.WriteHeader(http.StatusOK)
}
}
router := routegroup.New(http.NewServeMux())
router.With(R.Timeout(20*time.Millisecond)).HandleFunc("GET /bounded", slow(time.Second))
router.HandleFunc("GET /streaming", slow(30*time.Millisecond)) // no timeout, like /export and /wait
ts := httptest.NewServer(router)
defer ts.Close()
resp, err := http.Get(ts.URL + "/bounded")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusGatewayTimeout, resp.StatusCode, "route under R.Timeout is aborted at the deadline")
resp, err = http.Get(ts.URL + "/streaming")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "route without R.Timeout runs to completion")
}
func TestRealIPMiddleware(t *testing.T) {
// call runs mw with the given peer and (optional) X-Real-IP header and returns what the
// downstream handler observes; state is per-call, so subtests don't share closure locals.
call := func(mw func(http.Handler) http.Handler, remoteAddr, xRealIP string) (addr, hdr string) {
next := http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
addr, hdr = r.RemoteAddr, r.Header.Get("X-Real-IP")
})
req := httptest.NewRequest(http.MethodGet, "/", http.NoBody)
req.RemoteAddr = remoteAddr
if xRealIP != "" {
req.Header.Set("X-Real-IP", xRealIP)
}
mw(next).ServeHTTP(httptest.NewRecorder(), req)
return addr, hdr
}
trusted, err := ParseTrustedProxies([]string{"172.16.0.0/12", "2001:db8::/32"})
require.NoError(t, err)
t.Run("no trusted proxies trusts the header from anyone (legacy)", func(t *testing.T) {
addr, _ := call(realIPMiddleware(nil), "203.0.113.9:1234", "8.8.8.8")
assert.Equal(t, "8.8.8.8", addr)
})
t.Run("trusted v4 peer: forwarding header sets the client IP", func(t *testing.T) {
addr, _ := call(realIPMiddleware(trusted), "172.18.0.5:5555", "8.8.8.8")
assert.Equal(t, "8.8.8.8", addr)
})
t.Run("trusted v6 peer: forwarding header honored", func(t *testing.T) {
addr, _ := call(realIPMiddleware(trusted), "[2001:db8::5]:5555", "8.8.8.8")
assert.Equal(t, "8.8.8.8", addr)
})
t.Run("trusted peer without a forwarding header falls back to the socket IP", func(t *testing.T) {
addr, _ := call(realIPMiddleware(trusted), "172.18.0.5:5555", "")
assert.Equal(t, "172.18.0.5", addr, "no header to honor, so the bare socket IP is used")
})
t.Run("untrusted peer: header stripped, RemoteAddr pinned to bare socket IP", func(t *testing.T) {
addr, hdr := call(realIPMiddleware(trusted), "203.0.113.9:1234", "8.8.8.8")
assert.Equal(t, "203.0.113.9", addr, "real socket IP with the port stripped")
assert.Empty(t, hdr, "spoofed forwarding header removed so nothing downstream can read it")
})
t.Run("unparseable RemoteAddr is treated as untrusted, header stripped", func(t *testing.T) {
addr, hdr := call(realIPMiddleware(trusted), "garbage", "8.8.8.8")
assert.Equal(t, "garbage", addr, "unparseable peer left as-is, not overwritten")
assert.Empty(t, hdr, "forwarding header still stripped for a non-trusted peer")
})
}
func TestParseTrustedProxies(t *testing.T) {
t.Run("cidr, bare v4, bare v6, blanks", func(t *testing.T) {
got, err := ParseTrustedProxies([]string{"172.16.0.0/12", " 10.0.0.1 ", "", "2001:db8::/32"})
require.NoError(t, err)
require.Len(t, got, 3)
assert.True(t, got[0].Contains(net.ParseIP("172.18.0.5")))
assert.True(t, got[1].Contains(net.ParseIP("10.0.0.1")))
assert.False(t, got[1].Contains(net.ParseIP("10.0.0.2")), "a bare IP is a single host")
assert.True(t, got[2].Contains(net.ParseIP("2001:db8::1")))
})
t.Run("v4-mapped IPv6 bare entry resolves to the v4 host", func(t *testing.T) {
got, err := ParseTrustedProxies([]string{"::ffff:10.0.0.1"})
require.NoError(t, err)
require.Len(t, got, 1)
assert.True(t, got[0].Contains(net.ParseIP("10.0.0.1")), "the intended /32 host")
assert.False(t, got[0].Contains(net.ParseIP("10.0.0.2")), "not a wider range")
})
t.Run("malformed entry is a hard error", func(t *testing.T) {
_, err := ParseTrustedProxies([]string{"172.16.0.0/12", "nonsense"})
require.Error(t, err)
_, err = ParseTrustedProxies([]string{"10.0.0.0/999"})
require.Error(t, err)
})
t.Run("all blank yields nil", func(t *testing.T) {
got, err := ParseTrustedProxies([]string{"", " "})
require.NoError(t, err)
assert.Empty(t, got)
})
}
func TestTrustsAnyPeer(t *testing.T) {
catchAll := func(entries ...string) bool {
cidrs, err := ParseTrustedProxies(entries)
require.NoError(t, err)
return TrustsAnyPeer(cidrs)
}
assert.True(t, catchAll("10.0.0.0/8", "0.0.0.0/0"), "v4 catch-all")
assert.True(t, catchAll("::/0"), "v6 catch-all")
assert.False(t, catchAll("172.16.0.0/12", "10.0.0.5"), "scoped ranges are not catch-all")
assert.False(t, catchAll(), "empty is not catch-all")
}
func TestRest_rejectAnonUser(t *testing.T) {
ts := httptest.NewServer(fakeAuth(rejectAnonUser(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprintln(w, "Hello")
}))))
defer ts.Close()
resp, err := http.Get(ts.URL)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "use not logged in")
resp, err = http.Get(ts.URL + "?fake_id=anonymous_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode, "anon rejected")
resp, err = http.Get(ts.URL + "?fake_id=real_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "real user")
}
func TestRest_cacheControl(t *testing.T) {
tbl := []struct {
url string
version string
exp time.Duration
etag string
maxAge int
}{
{"http://example.com/foo", "v1", time.Hour, "b433be1ea19edaee9dc92ca4b895b6bdf3c058cb", 3600},
{"http://example.com/foo2", "v1", 10 * time.Hour, "6d8466aef3246c1057452561acddf7ad9d0d99e0", 36000},
{"http://example.com/foo", "v2", time.Hour, "481700c52aab0dfbca99f3ffc2a4fbb27884c114", 3600},
{"https://example.com/foo", "v2", time.Hour, "bebd4f1b87f474792c4e75e5affe31fbf67f5778", 3600},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", tt.url, http.NoBody)
w := httptest.NewRecorder()
h := cacheControl(tt.exp, tt.version)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
t.Logf("%+v", resp.Header)
assert.Equal(t, `"`+tt.etag+`"`, resp.Header.Get("Etag"))
assert.Equal(t, `max-age=`+strconv.Itoa(int(tt.exp.Seconds()))+", no-cache", resp.Header.Get("Cache-Control"))
})
}
}
// TestRest_apiCSP locks in that /api/v1/* responses get a strict default-src 'none'
// override regardless of what the global CSP allows. The widget HTML pages
// (/web/*.html) still get the global CSP (with 'unsafe-inline' for bootstrap),
// so the test asserts the two policies diverge across origins.
func TestRest_apiCSP(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
client := http.Client{}
// JSON API endpoint — must carry the strict policy
resp, err := client.Get(ts.URL + "/api/v1/config")
require.NoError(t, err)
defer resp.Body.Close()
csp := resp.Header.Get("Content-Security-Policy")
assert.Contains(t, csp, "default-src 'none'",
"API responses must override the global CSP with default-src 'none'; got %q", csp)
assert.Contains(t, csp, "sandbox", "API CSP must include sandbox; got %q", csp)
assert.NotContains(t, csp, "'unsafe-inline'",
"API CSP must not allow inline scripts/styles; got %q", csp)
// RSS/XML endpoint — same strict policy, and the XML response itself must still be served
respRSS, err := client.Get(ts.URL + "/api/v1/rss/site?site=remark42")
require.NoError(t, err)
defer respRSS.Body.Close()
assert.Equal(t, http.StatusOK, respRSS.StatusCode, "RSS must still respond OK under strict CSP")
cspRSS := respRSS.Header.Get("Content-Security-Policy")
assert.Contains(t, cspRSS, "default-src 'none'", "RSS responses must carry the strict API CSP")
assert.Contains(t, cspRSS, "sandbox", "RSS CSP must include sandbox")
// widget HTML — must keep the global CSP (unchanged, lax to support inline bootstrap)
resp2, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp2.Body.Close()
csp2 := resp2.Header.Get("Content-Security-Policy")
assert.Contains(t, csp2, "'unsafe-inline'",
"widget HTML CSP must keep unsafe-inline for bootstrap; got %q", csp2)
}
// check CSP, img-src should be 'self' with proxy enabled and * without it
func TestRest_securityHeaders(t *testing.T) {
ts, _, teardown := startupT(t)
// with proxy disabled
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src *;")
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, "strict-origin-when-cross-origin", resp.Header.Get("Referrer-Policy"))
teardown()
// check CSP with proxy enabled
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.ExternalImageProxy = true
})
defer teardown()
resp, err = client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src 'self';")
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, "strict-origin-when-cross-origin", resp.Header.Get("Referrer-Policy"))
}
func TestRest_subscribersOnly(t *testing.T) {
paidSubUser := &token.User{}
paidSubUser.SetPaidSub(true)
tbl := []struct {
subsOnly bool
user token.User
setUser bool
status int
}{
{true, token.User{}, false, http.StatusUnauthorized},
{true, token.User{}, true, http.StatusForbidden},
{false, token.User{}, false, http.StatusOK},
{false, token.User{}, true, http.StatusOK},
{true, *paidSubUser, true, http.StatusOK},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", http.NoBody)
if tt.setUser {
req = token.SetUserInfo(req, tt.user)
}
w := httptest.NewRecorder()
h := subscribersOnly(tt.subsOnly)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
func Test_validEmailAuth(t *testing.T) {
tbl := []struct {
req string
status int
}{
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=site-with-dash_and_underscore-and.dot&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=Евгений+Умпутун", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=12", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someonelooong+loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong", http.StatusForbidden},
{"/auth/twitter/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun+example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=bad!site&address=umputun%example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooongsite&address=umputun%example.com&user=someone", http.StatusForbidden},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com"+tt.req, http.NoBody)
w := httptest.NewRecorder()
h := validEmailAuth()(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
// TestRest_matchSiteID reproduces the multi-tenant isolation gap in the matchSiteID
// middleware. Before the fix, the check `if siteID != "" && user.SiteID != siteID`
// silently allowed any authenticated request that omitted the ?site= query param.
// On admin and user-mutation routes this meant the cross-site check was bypassable
// just by dropping the parameter. The fix requires ?site= to be present and to match
// the user's bound site.
func TestRest_matchSiteID(t *testing.T) {
wrapped := matchSiteID(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
}))
cases := []struct {
name string
userSite string
query string
want int
}{
{name: "matching site allowed", userSite: "site-a", query: "?site=site-a", want: http.StatusOK},
{name: "mismatched site forbidden", userSite: "site-a", query: "?site=site-b", want: http.StatusForbidden},
{name: "missing site param rejected", userSite: "site-a", query: "", want: http.StatusForbidden},
{name: "empty site param rejected", userSite: "site-a", query: "?site=", want: http.StatusForbidden},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r = rest.SetUserInfo(r, store.User{ID: "u", Name: "u", SiteID: c.userSite})
wrapped.ServeHTTP(w, r)
})
ts := httptest.NewServer(h)
defer ts.Close()
resp, err := http.Get(ts.URL + c.query)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, c.want, resp.StatusCode)
})
}
}
func TestCorsMiddleware(t *testing.T) {
h := corsMiddleware()(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
}))
t.Run("credentialed cross-origin reflects the request origin", func(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/", http.NoBody)
req.Header.Set("Origin", "https://example.com")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
// AllowedOrigins "*" with credentials must reflect the origin, never a literal "*"
assert.Equal(t, "https://example.com", rec.Header().Get("Access-Control-Allow-Origin"))
assert.Equal(t, "true", rec.Header().Get("Access-Control-Allow-Credentials"))
assert.Equal(t, "Authorization", rec.Header().Get("Access-Control-Expose-Headers"))
})
t.Run("preflight advertises configured methods, headers and max-age", func(t *testing.T) {
req := httptest.NewRequest(http.MethodOptions, "/", http.NoBody)
req.Header.Set("Origin", "https://example.com")
req.Header.Set("Access-Control-Request-Method", "POST")
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
assert.Equal(t, http.StatusNoContent, rec.Code)
assert.Contains(t, rec.Header().Get("Access-Control-Allow-Methods"), "POST")
assert.Contains(t, rec.Header().Get("Access-Control-Allow-Headers"), "X-JWT")
assert.Equal(t, "300", rec.Header().Get("Access-Control-Max-Age"))
// preflight responses must vary on origin and the request method/headers so caches
// don't reuse one preflight across different requests
vary := rec.Header().Values("Vary")
assert.Contains(t, vary, "Origin")
assert.Contains(t, vary, "Access-Control-Request-Method")
assert.Contains(t, vary, "Access-Control-Request-Headers")
})
t.Run("same-origin request (no Origin) gets no CORS headers", func(t *testing.T) {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", http.NoBody))
assert.Empty(t, rec.Header().Get("Access-Control-Allow-Origin"))
})
}
+31 -11
View File
@@ -73,21 +73,41 @@ func (m *Migrator) importFormCtrl(w http.ResponseWriter, r *http.Request) {
return
}
if err := r.ParseMultipartForm(20 * 1024 * 1024); err != nil { // 20M max memory, if bigger will make a file
r.Body = http.MaxBytesReader(w, r.Body, 256*1024*1024) // hard cap on upload to prevent memory exhaustion
reader, err := r.MultipartReader()
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't parse multipart form", rest.ErrDecode)
return
}
file, _, err := r.FormFile("file")
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't get import file from the request", rest.ErrInternal)
return
}
defer func() { _ = file.Close() }()
tmpfile := ""
for {
part, err := reader.NextPart()
if err == io.EOF {
break
}
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't parse multipart form", rest.ErrDecode)
return
}
if part.FormName() != "file" {
_ = part.Close()
continue
}
tmpfile, err := m.saveTemp(file)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't save request to temp file", rest.ErrInternal)
tmpfile, err = m.saveTemp(part)
if closeErr := part.Close(); err == nil && closeErr != nil {
err = closeErr
}
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't save request to temp file", rest.ErrInternal)
return
}
break
}
if tmpfile == "" {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, fmt.Errorf("file field missing"),
"can't get import file from the request", rest.ErrInternal)
return
}
@@ -180,7 +200,7 @@ func (m *Migrator) remapCtrl(w http.ResponseWriter, r *http.Request) {
return
}
defer func() {
if e = os.Remove(fh.Name()); e != nil {
if e = os.Remove(fh.Name()); e != nil { //nolint:gosec // fh.Name() is from os.CreateTemp, server-controlled
log.Printf("[WARN] failed to remove temp file %+v", e)
}
}()
+3 -3
View File
@@ -276,8 +276,8 @@ func TestMigrator_ImportDouble(t *testing.T) {
"picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}`
recs := []string{}
for i := 0; i < 50; i++ {
recs := make([]string, 0, 50)
for i := range 50 {
recs = append(recs, fmt.Sprintf(tmpl, i))
}
r := strings.NewReader(`{"version":1}` + strings.Join(recs, "\n")) // reader with 10k records
@@ -329,7 +329,7 @@ func TestMigrator_ImportWaitExpired(t *testing.T) {
"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}`
nRecs := 50
recs := make([]string, 0, nRecs)
for i := 0; i < nRecs; i++ {
for i := range nRecs {
recs = append(recs, fmt.Sprintf(tmpl, i))
}
r := strings.NewReader(`{"version":1}` + strings.Join(recs, "\n")) // reader with `nRecs` records
+164 -307
View File
@@ -7,24 +7,19 @@ import (
"encoding/json"
"fmt"
"io/fs"
"net"
"net/http"
"net/mail"
"os"
"regexp"
"strings"
"sync"
"time"
"github.com/didip/tollbooth/v8"
"github.com/didip/tollbooth/v8/limiter"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/go-chi/cors"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/rest/logger"
"github.com/go-pkgz/routegroup"
"github.com/umputun/remark42/backend/app/notify"
"github.com/umputun/remark42/backend/app/rest"
@@ -54,6 +49,7 @@ type Rest struct {
RemarkURL string
ReadOnlyAge int
SharedSecret string
TrustedProxies []*net.IPNet // reverse-proxy networks whose forwarding headers (X-Real-IP, X-Forwarded-For, ...) are trusted
ScoreThresholds struct {
Low int
Critical int
@@ -71,10 +67,11 @@ type Rest struct {
DisableFancyTextFormatting bool // disables SmartyPants in the comment text rendering of the posted comments
ExternalImageProxy bool
SSLConfig SSLConfig
httpsServer *http.Server
httpServer *http.Server
lock sync.Mutex
SSLConfig SSLConfig
httpsServer *http.Server
httpServer *http.Server
shutdownRequested bool
lock sync.Mutex
pubRest public
privRest private
@@ -96,12 +93,12 @@ const lastCommentsScope = "last"
type commentsWithInfo struct {
Comments []store.Comment `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}
type treeWithInfo struct {
*service.Tree
Info store.PostInfo `json:"info,omitempty"`
Info store.PostInfo `json:"info"`
}
// Run the lister and request's router, activate rest server
@@ -117,6 +114,11 @@ func (s *Rest) Run(address string, port int) {
s.lock.Lock()
s.httpServer = s.makeHTTPServer(address, port, s.routes())
s.httpServer.ErrorLog = log.ToStdLogger(log.Default(), "WARN")
if s.shutdownRequested {
s.lock.Unlock()
log.Print("[WARN] rest server start canceled")
return
}
s.lock.Unlock()
err := s.httpServer.ListenAndServe()
@@ -130,6 +132,11 @@ func (s *Rest) Run(address string, port int) {
s.httpServer = s.makeHTTPServer(address, port, s.httpToHTTPSRouter())
s.httpServer.ErrorLog = log.ToStdLogger(log.Default(), "WARN")
if s.shutdownRequested {
s.lock.Unlock()
log.Print("[WARN] rest server start canceled")
return
}
s.lock.Unlock()
go func() {
@@ -150,6 +157,11 @@ func (s *Rest) Run(address string, port int) {
s.httpServer = s.makeHTTPServer(address, port, s.httpChallengeRouter(m))
s.httpServer.ErrorLog = log.ToStdLogger(log.Default(), "WARN")
if s.shutdownRequested {
s.lock.Unlock()
log.Print("[WARN] rest server start canceled")
return
}
s.lock.Unlock()
@@ -171,6 +183,7 @@ func (s *Rest) Shutdown() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
s.lock.Lock()
s.shutdownRequested = true
if s.httpServer != nil {
if err := s.httpServer.Shutdown(ctx); err != nil {
log.Printf("[DEBUG] http shutdown error, %s", err)
@@ -198,13 +211,13 @@ func (s *Rest) makeHTTPServer(address string, port int, router http.Handler) *ht
}
}
func (s *Rest) routes() chi.Router {
func (s *Rest) routes() http.Handler {
if s.openRouteLimiter == 0 {
// set the default open route limiter. Just a safety measure as it should be set by Run method anyway
s.openRouteLimiter = openRouteLimiter
}
router := chi.NewRouter()
router.Use(middleware.Throttle(1000), middleware.RealIP, R.Recoverer(log.Default()))
router := routegroup.New(http.NewServeMux())
router.Use(R.Throttle(1000), realIPMiddleware(s.TrustedProxies), R.Recoverer(log.Default()))
router.Use(securityHeadersMiddleware(s.ExternalImageProxy, s.AllowedAncestors))
if !s.DisableSignature {
router.Use(R.AppInfo("remark42", "umputun", s.Version))
@@ -216,15 +229,7 @@ func (s *Rest) routes() chi.Router {
if s.ProxyCORS {
log.Printf("[WARN] internal CORS disabled")
} else {
corsMiddleware := cors.New(cors.Options{
AllowedOrigins: []string{"*"},
AllowedMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS"},
AllowedHeaders: []string{"Accept", "Authorization", "Content-Type", "X-XSRF-Token", "X-JWT"},
ExposedHeaders: []string{"Authorization"},
AllowCredentials: true,
MaxAge: 300,
})
router.Use(corsMiddleware.Handler)
router.Use(corsMiddleware())
}
ipFn := func(ip string) string { return store.HashValue(ip, s.SharedSecret)[:12] } // logger uses it for anonymization
@@ -232,133 +237,152 @@ func (s *Rest) routes() chi.Router {
authHandler, avatarHandler := s.Authenticator.Handlers()
router.Group(func(r chi.Router) {
r.Use(middleware.Timeout(5 * time.Second))
r.Use(logInfoWithBody, rateLimiter(2), middleware.NoCache)
router.Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(5 * time.Second))
r.Use(logInfoWithBody, rateLimiter(2), R.NoCache)
r.Use(validEmailAuth()) // reject suspicious email logins
r.Mount("/auth", authHandler)
r.Handle("/auth/", authHandler)
})
router.Group(func(r chi.Router) {
r.Use(middleware.Timeout(5 * time.Second))
router.Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(5 * time.Second))
r.Use(rateLimiter(100))
r.Mount("/avatar", avatarHandler)
r.Handle("/avatar/", avatarHandler)
})
authMiddleware := s.Authenticator.Middleware()
// api routes
router.Route("/api/v1", func(rapi chi.Router) {
rapi.Group(func(rava chi.Router) {
rava.Use(middleware.Timeout(5 * time.Second))
rava.Use(rateLimiter(100))
rava.Mount("/avatar", avatarHandler)
})
rapi := router.Mount("/api/v1")
rapi.Use(apiCSPMiddleware)
// open routes
rapi.Group(func(ropen chi.Router) {
ropen.Use(middleware.Timeout(30 * time.Second))
ropen.Use(rateLimiter(s.openRouteLimiter))
ropen.Use(authMiddleware.Trace, middleware.NoCache, logInfoWithBody)
ropen.Get("/config", s.configCtrl)
ropen.Get("/find", s.pubRest.findCommentsCtrl)
ropen.Get("/id/{id}", s.pubRest.commentByIDCtrl)
ropen.Get("/comments", s.pubRest.findUserCommentsCtrl)
ropen.Get("/last/{limit}", s.pubRest.lastCommentsCtrl)
ropen.Get("/count", s.pubRest.countCtrl)
ropen.Post("/counts", s.pubRest.countMultiCtrl)
ropen.Get("/list", s.pubRest.listCtrl)
ropen.Get("/info", s.pubRest.infoCtrl)
ropen.Get("/img", s.ImageProxy.Handler)
rapi.Group().Route(func(rava *routegroup.Bundle) {
rava.Use(R.Timeout(5 * time.Second))
rava.Use(rateLimiter(100))
rava.Handle("/avatar/", avatarHandler)
})
ropen.Route("/rss", func(rrss chi.Router) {
rrss.Get("/post", s.rssRest.postCommentsCtrl)
rrss.Get("/site", s.rssRest.siteCommentsCtrl)
rrss.Get("/reply", s.rssRest.repliesCtrl)
})
})
// open routes
rapi.Group().Route(func(ropen *routegroup.Bundle) {
ropen.Use(R.Timeout(30 * time.Second))
ropen.Use(rateLimiter(s.openRouteLimiter))
ropen.Use(authMiddleware.Trace, R.NoCache, logInfoWithBody)
ropen.HandleFunc("GET /config", s.configCtrl)
ropen.HandleFunc("GET /find", s.pubRest.findCommentsCtrl)
ropen.HandleFunc("GET /id/{id}", s.pubRest.commentByIDCtrl)
ropen.HandleFunc("GET /comments", s.pubRest.findUserCommentsCtrl)
ropen.HandleFunc("GET /last/{limit}", s.pubRest.lastCommentsCtrl)
ropen.HandleFunc("GET /count", s.pubRest.countCtrl)
ropen.HandleFunc("POST /counts", s.pubRest.countMultiCtrl)
ropen.HandleFunc("GET /list", s.pubRest.listCtrl)
ropen.HandleFunc("GET /info", s.pubRest.infoCtrl)
// open routes, cached
rapi.Group(func(ropen chi.Router) {
ropen.Use(middleware.Timeout(30 * time.Second))
ropen.Use(rateLimiter(10))
ropen.Use(authMiddleware.Trace, logInfoWithBody)
ropen.Get("/picture/{user}/{id}", s.pubRest.loadPictureCtrl)
ropen.Get("/qr/telegram", s.pubRest.telegramQrCtrl)
})
// protected routes, require auth
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(30 * time.Second))
rauth.Use(rateLimiter(10))
rauth.Use(authMiddleware.Auth, matchSiteID, middleware.NoCache, logInfoWithBody)
rauth.Get("/user", s.privRest.userInfoCtrl)
rauth.Get("/userdata", s.privRest.userAllDataCtrl)
})
// admin routes, require auth and admin users only
rapi.Route("/admin", func(radmin chi.Router) {
radmin.Use(middleware.Timeout(30 * time.Second))
radmin.Use(rateLimiter(10))
radmin.Use(authMiddleware.Auth, authMiddleware.AdminOnly, matchSiteID)
radmin.Use(middleware.NoCache, logInfoWithBody)
radmin.Delete("/comment/{id}", s.adminRest.deleteCommentCtrl)
radmin.Put("/user/{userid}", s.adminRest.setBlockCtrl)
radmin.Delete("/user/{userid}", s.adminRest.deleteUserCtrl)
radmin.Get("/user/{userid}", s.adminRest.getUserInfoCtrl)
radmin.Get("/deleteme", s.adminRest.deleteMeRequestCtrl)
radmin.Put("/verify/{userid}", s.adminRest.setVerifyCtrl)
radmin.Put("/pin/{id}", s.adminRest.setPinCtrl)
radmin.Get("/blocked", s.adminRest.blockedUsersCtrl)
radmin.Put("/readonly", s.adminRest.setReadOnlyCtrl)
radmin.Put("/title/{id}", s.adminRest.setTitleCtrl)
// migrator
radmin.Get("/export", s.adminRest.migrator.exportCtrl)
radmin.Post("/import", s.adminRest.migrator.importCtrl)
radmin.Post("/import/form", s.adminRest.migrator.importFormCtrl)
radmin.Post("/remap", s.adminRest.migrator.remapCtrl)
radmin.Get("/wait", s.adminRest.migrator.waitCtrl)
})
// protected routes, throttled to 10/s by default, controlled by external UpdateLimiter param
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(10 * time.Second))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, matchSiteID, subscribersOnly(s.SubscribersOnly))
rauth.Use(middleware.NoCache, logInfoWithBody)
rauth.Put("/comment/{id}", s.privRest.updateCommentCtrl)
rauth.Post("/preview", s.privRest.previewCommentCtrl)
rauth.Post("/comment", s.privRest.createCommentCtrl)
rauth.Put("/vote/{id}", s.privRest.voteCtrl)
rauth.With(rejectAnonUser).Post("/deleteme", s.privRest.deleteMeCtrl)
rauth.With(rejectAnonUser).Get("/email", s.privRest.getEmailCtrl)
rauth.With(rejectAnonUser).Post("/email/subscribe", s.privRest.sendEmailConfirmationCtrl)
rauth.With(rejectAnonUser).Post("/email/confirm", s.privRest.setConfirmedEmailCtrl)
rauth.With(rejectAnonUser).Delete("/email", s.privRest.deleteEmailCtrl)
rauth.With(rejectAnonUser).Get("/telegram/subscribe", s.privRest.telegramSubscribeCtrl)
rauth.With(rejectAnonUser).Delete("/telegram", s.privRest.deleteTelegramCtrl)
})
// protected routes, anonymous rejected
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(10 * time.Second))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, rejectAnonUser, matchSiteID)
rauth.Use(logger.New(logger.Log(log.Default()), logger.Prefix("[DEBUG]"), logger.IPfn(ipFn)).Handler)
rauth.Post("/picture", s.privRest.savePictureCtrl)
ropen.Mount("/rss").Route(func(rrss *routegroup.Bundle) {
rrss.HandleFunc("GET /post", s.rssRest.postCommentsCtrl)
rrss.HandleFunc("GET /site", s.rssRest.siteCommentsCtrl)
rrss.HandleFunc("GET /reply", s.rssRest.repliesCtrl)
})
})
// open routes, cached. /img lives here (not in the NoCache group above) because
// R.NoCache strips If-None-Match from incoming requests, which would
// defeat the proxy handler's 304 short-circuit. The handler sets a 30-day
// max-age on validated success responses (with a versioned etag for cache
// invalidation on revalidation); error responses get Cache-Control: no-store
// so transient failures aren't pinned in the cache.
rapi.Group().Route(func(ropen *routegroup.Bundle) {
ropen.Use(R.Timeout(30 * time.Second))
ropen.Use(rateLimiter(10))
ropen.Use(authMiddleware.Trace, logInfoWithBody)
ropen.HandleFunc("GET /img", s.ImageProxy.Handler)
ropen.HandleFunc("GET /picture/{user}/{id}", s.pubRest.loadPictureCtrl)
ropen.HandleFunc("GET /qr/telegram", s.pubRest.telegramQrCtrl)
})
// protected routes, require auth
rapi.Group().Route(func(rauth *routegroup.Bundle) {
rauth.Use(rateLimiter(10))
rauth.Use(authMiddleware.Auth, matchSiteID, R.NoCache, logInfoWithBody)
// GET /userdata streams a gzipped export of the user's data straight to the client, so it
// deliberately runs without R.Timeout: that middleware buffers the whole response in memory
// before sending and aborts at the deadline, which would hold a full export in RAM and truncate it.
rauth.HandleFunc("GET /userdata", s.privRest.userAllDataCtrl)
rauth.Group().Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(30 * time.Second))
r.HandleFunc("GET /user", s.privRest.userInfoCtrl)
})
})
// admin routes, require auth and admin users only
rapi.Mount("/admin").Route(func(radmin *routegroup.Bundle) {
radmin.Use(rateLimiter(10))
radmin.Use(authMiddleware.Auth, authMiddleware.AdminOnly, matchSiteID)
radmin.Use(R.NoCache, logInfoWithBody)
// bounded admin operations return small responses and get the enforcing request timeout
radmin.Group().Route(func(r *routegroup.Bundle) {
r.Use(R.Timeout(30 * time.Second))
r.HandleFunc("DELETE /comment/{id}", s.adminRest.deleteCommentCtrl)
r.HandleFunc("PUT /user/{userid}", s.adminRest.setBlockCtrl)
r.HandleFunc("DELETE /user/{userid}", s.adminRest.deleteUserCtrl)
r.HandleFunc("GET /user/{userid}", s.adminRest.getUserInfoCtrl)
r.With(rejectHead("GET")).HandleFunc("GET /deleteme", s.adminRest.deleteMeRequestCtrl)
r.HandleFunc("PUT /verify/{userid}", s.adminRest.setVerifyCtrl)
r.HandleFunc("PUT /pin/{id}", s.adminRest.setPinCtrl)
r.HandleFunc("GET /blocked", s.adminRest.blockedUsersCtrl)
r.HandleFunc("PUT /readonly", s.adminRest.setReadOnlyCtrl)
r.HandleFunc("PUT /title/{id}", s.adminRest.setTitleCtrl)
})
// migrator routes deliberately run without R.Timeout: GET /export streams a full-site
// backup, GET /wait long-polls for up to 15m, and import/remap ingest large uploads. The
// enforcing timeout buffers the whole response and aborts at the deadline, which would
// truncate backups, break waiting, and reject large imports.
radmin.HandleFunc("GET /export", s.adminRest.migrator.exportCtrl)
radmin.HandleFunc("POST /import", s.adminRest.migrator.importCtrl)
radmin.HandleFunc("POST /import/form", s.adminRest.migrator.importFormCtrl)
radmin.HandleFunc("POST /remap", s.adminRest.migrator.remapCtrl)
radmin.HandleFunc("GET /wait", s.adminRest.migrator.waitCtrl)
})
// protected routes, throttled to 10/s by default, controlled by external UpdateLimiter param
rapi.Group().Route(func(rauth *routegroup.Bundle) {
rauth.Use(R.Timeout(10 * time.Second))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, matchSiteID, subscribersOnly(s.SubscribersOnly))
rauth.Use(R.NoCache, logInfoWithBody)
rauth.HandleFunc("PUT /comment/{id}", s.privRest.updateCommentCtrl)
rauth.HandleFunc("POST /preview", s.privRest.previewCommentCtrl)
rauth.HandleFunc("POST /comment", s.privRest.createCommentCtrl)
rauth.HandleFunc("PUT /vote/{id}", s.privRest.voteCtrl)
rauth.With(rejectAnonUser).HandleFunc("POST /deleteme", s.privRest.deleteMeCtrl)
rauth.With(rejectAnonUser).HandleFunc("GET /email", s.privRest.getEmailCtrl)
rauth.With(rejectAnonUser).HandleFunc("POST /email/subscribe", s.privRest.sendEmailConfirmationCtrl)
rauth.With(rejectAnonUser).HandleFunc("POST /email/confirm", s.privRest.setConfirmedEmailCtrl)
rauth.With(rejectAnonUser).HandleFunc("DELETE /email", s.privRest.deleteEmailCtrl)
rauth.With(rejectAnonUser, rejectHead("GET")).HandleFunc("GET /telegram/subscribe", s.privRest.telegramSubscribeCtrl)
rauth.With(rejectAnonUser).HandleFunc("DELETE /telegram", s.privRest.deleteTelegramCtrl)
})
// protected routes, anonymous rejected
rapi.Group().Route(func(rauth *routegroup.Bundle) {
rauth.Use(R.Timeout(10 * time.Second))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, rejectAnonUser, matchSiteID)
rauth.Use(logger.New(logger.Log(log.Default()), logger.Prefix("[DEBUG]"), logger.IPfn(ipFn)).Handler)
rauth.HandleFunc("POST /picture", s.privRest.savePictureCtrl)
})
// open routes on root level
router.Group(func(rroot chi.Router) {
rroot.Use(middleware.Timeout(10 * time.Second))
router.Route(func(rroot *routegroup.Bundle) {
rroot.Use(R.Timeout(10 * time.Second))
rroot.Use(rateLimiter(50))
rroot.Get("/robots.txt", s.pubRest.robotsCtrl)
rroot.Get("/email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
rroot.Post("/email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
rroot.HandleFunc("GET /robots.txt", s.pubRest.robotsCtrl)
rroot.With(rejectHead("GET, POST")).HandleFunc("GET /email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
rroot.HandleFunc("POST /email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
})
// file server for static content from s.WebRoot on path /web
@@ -476,7 +500,7 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
}
// serves static files from the webRoot directory or files embedded into the compiled binary if that directory is absent
func addFileServer(r chi.Router, embedFS embed.FS, webRoot, version string) {
func addFileServer(r *routegroup.Bundle, embedFS embed.FS, webRoot, version string) {
var webFS http.Handler
if _, err := os.Stat(webRoot); err == nil {
@@ -489,12 +513,12 @@ func addFileServer(r chi.Router, embedFS embed.FS, webRoot, version string) {
}
webFS = http.StripPrefix("/web", webFS)
r.Get("/web", http.RedirectHandler("/web/", http.StatusMovedPermanently).ServeHTTP)
r.HandleFunc("GET /web", http.RedirectHandler("/web/", http.StatusMovedPermanently).ServeHTTP)
r.With(rateLimiter(20),
middleware.Timeout(10*time.Second),
R.Timeout(10*time.Second),
cacheControl(time.Hour, version),
).Get("/web/*", func(w http.ResponseWriter, r *http.Request) {
).HandleFunc("GET /web/", func(w http.ResponseWriter, r *http.Request) {
// don't show dirs, just serve files
if strings.HasSuffix(r.URL.Path, "/") && len(r.URL.Path) > 1 && r.URL.Path != ("/web/") {
http.NotFound(w, r)
@@ -504,7 +528,7 @@ func addFileServer(r chi.Router, embedFS embed.FS, webRoot, version string) {
})
}
func encodeJSONWithHTML(v interface{}) ([]byte, error) {
func encodeJSONWithHTML(v any) ([]byte, error) {
buf := &bytes.Buffer{}
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
@@ -550,160 +574,6 @@ func URLKeyWithUser(r *http.Request) string {
return key
}
// rejectAnonUser is a middleware rejecting anonymous users
func rejectAnonUser(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if strings.HasPrefix(user.ID, "anonymous_") {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// matchSiteID is a middleware rejecting users with mismatch between site param and and User.SiteID
func matchSiteID(next http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
// skip for basic auth user
if user.Name == "admin" && user.ID == "admin" {
next.ServeHTTP(w, r)
return
}
siteID := r.URL.Query().Get("site")
if siteID != "" && user.SiteID != siteID {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
next.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
// cacheControl is a middleware setting cache expiration. Using url+version as etag
func cacheControl(expiration time.Duration, version string) func(http.Handler) http.Handler {
etag := func(r *http.Request, version string) string {
s := version + ":" + r.URL.String()
return store.EncodeID(s)
}
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
e := `"` + etag(r, version) + `"`
w.Header().Set("Etag", e)
w.Header().Set("Cache-Control", fmt.Sprintf("max-age=%d, no-cache", int(expiration.Seconds())))
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, e) {
w.WriteHeader(http.StatusNotModified)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// securityHeadersMiddleware sets security-related headers: Content-Security-Policy and Permissions-Policy
func securityHeadersMiddleware(imageProxyEnabled bool, allowedAncestors []string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
imgSrc := "*"
if imageProxyEnabled {
imgSrc = "'self'"
}
frameAncestors := "*"
if len(allowedAncestors) > 0 {
frameAncestors = strings.Join(allowedAncestors, " ")
}
w.Header().Set("Content-Security-Policy", fmt.Sprintf("default-src 'none'; base-uri 'none'; form-action 'none'; connect-src 'self'; frame-src 'self' mailto:; img-src %s; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; font-src data:; object-src 'none'; frame-ancestors %s;", imgSrc, frameAncestors))
w.Header().Set("Permissions-Policy", "accelerometer=(), autoplay=(), camera=(), cross-origin-isolated=(), display-capture=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), unload=(), window-management=()")
next.ServeHTTP(w, r)
})
}
}
// subscribersOnly is a middleware rejecting non-paid_sub users
func subscribersOnly(enable bool) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if enable {
user, err := rest.GetUserInfo(r)
if err != nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if !user.PaidSub {
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
// validEmailAuth is a middleware for auth endpoints for email method.
// it rejects login request if user, site or email are suspicious
func validEmailAuth() func(http.Handler) http.Handler {
reUser := regexp.MustCompile(`^[\p{L}\d\s_]{4,64}$`) // matches ui side validation, adding min/max limitation
reSite := regexp.MustCompile(`^[a-zA-Z\d\s_.-]{1,64}$`)
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/auth/email/login" {
// not email login, skip the check
h.ServeHTTP(w, r)
return
}
if u := r.URL.Query().Get("user"); u != "" {
if !reUser.MatchString(u) {
log.Printf("[WARN] suspicious user rejected: %s", u)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if a := r.URL.Query().Get("address"); a != "" {
if _, err := mail.ParseAddress(a); err != nil {
log.Printf("[WARN] suspicious address rejected: %s", a)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
if s := r.URL.Query().Get("site"); s != "" {
if !reSite.MatchString(s) {
log.Printf("[WARN] suspicious site rejected: %s", s)
http.Error(w, "Access denied", http.StatusForbidden)
return
}
}
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
}
}
func parseError(err error, defaultCode int) (code int) {
code = defaultCode
@@ -727,16 +597,3 @@ func parseError(err error, defaultCode int) (code int) {
return code
}
// rateLimiter creates a rate limiting middleware with proper IP lookup configuration.
// tollbooth v8 requires explicit IP lookup method to be set.
// uses RemoteAddr which is set by chi's middleware.RealIP to the real client IP
// from X-Forwarded-For, X-Real-IP, or True-Client-IP headers.
func rateLimiter(maxReq float64) func(http.Handler) http.Handler {
lmt := tollbooth.NewLimiter(maxReq, nil)
lmt.SetIPLookup(limiter.IPLookup{
Name: "RemoteAddr",
IndexFromRight: 0,
})
return tollbooth.HTTPMiddleware(lmt)
}
+29 -14
View File
@@ -10,11 +10,11 @@ import (
"fmt"
"html/template"
"io"
"net"
"net/http"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
@@ -120,7 +120,7 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
comment.PrepareUntrusted() // clean all fields user not supposed to set
comment.User = user
comment.User.IP = strings.Split(r.RemoteAddr, ":")[0]
comment.User.IP = extractIP(r.RemoteAddr)
comment.Orig = comment.Text // original comment text, prior to md render
if err := s.dataService.ValidateComment(&comment); err != nil {
@@ -192,7 +192,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
user := rest.MustGetUserInfo(r)
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
id := chi.URLParam(r, "id")
id := r.PathValue("id")
log.Printf("[DEBUG] update comment %s", id)
@@ -259,7 +259,7 @@ func (s *private) voteCtrl(w http.ResponseWriter, r *http.Request) {
return
}
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
id := chi.URLParam(r, "id")
id := r.PathValue("id")
log.Printf("[DEBUG] vote for comment %s", id)
vote := r.URL.Query().Get("vote") == "1"
@@ -279,7 +279,7 @@ func (s *private) voteCtrl(w http.ResponseWriter, r *http.Request) {
Locator: locator,
CommentID: id,
UserID: user.ID,
UserIP: strings.Split(r.RemoteAddr, ":")[0],
UserIP: extractIP(r.RemoteAddr),
Val: vote,
}
comment, err := s.dataService.Vote(req)
@@ -410,7 +410,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
fmt.Errorf("already subscribed"), "telegram subscription is already set for this user, delete if first to re-subscribe", rest.ErrActionRejected)
return
}
// Generate and send token
// generate and send token
tkn, err := randToken()
if err != nil {
rest.SendErrorJSON(w, r, http.StatusForbidden, err, "failed to generate verification token", rest.ErrInternal)
@@ -479,7 +479,7 @@ func (s *private) setConfirmedEmailCtrl(w http.ResponseWriter, r *http.Request)
return
}
// Handshake.ID is user.ID + "::" + address
// handshake.ID is user.ID + "::" + address
elems := strings.Split(confClaims.Handshake.ID, "::")
if len(elems) != 2 || elems[0] != user.ID {
rest.SendErrorJSON(w, r, http.StatusBadRequest, fmt.Errorf("%s", confClaims.Handshake.ID), "invalid handshake token", rest.ErrInternal)
@@ -533,7 +533,7 @@ func (s *private) emailUnsubscribeCtrl(w http.ResponseWriter, r *http.Request) {
return
}
// Handshake.ID is user.ID + "::" + address
// handshake.ID is user.ID + "::" + address
elems := strings.Split(confClaims.Handshake.ID, "::")
if len(elems) != 2 {
rest.SendErrorHTML(w, r, http.StatusBadRequest, fmt.Errorf("%s", confClaims.Handshake.ID), "invalid handshake token", rest.ErrInternal)
@@ -578,7 +578,7 @@ func (s *private) emailUnsubscribeCtrl(w http.ResponseWriter, r *http.Request) {
}
// MustExecute behaves like template.Execute, but panics if an error occurs.
MustExecute := func(tmpl *template.Template, wr io.Writer, data interface{}) {
MustExecute := func(tmpl *template.Template, wr io.Writer, data any) {
if err := tmpl.Execute(wr, data); err != nil {
panic(err)
}
@@ -669,7 +669,7 @@ func (s *private) userAllDataCtrl(w http.ResponseWriter, r *http.Request) {
merr = multierror.Append(merr, write([]byte(`, "comments":`))) // send comments prefix
// get comments in 100 in each paginated request
for i := 0; i < 100; i++ {
for i := range 100 {
comments, errUser := s.dataService.User(siteID, user.ID, 100, i*100, rest.GetUserOrEmpty(r))
if errUser != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, errUser, "can't get user comments", rest.ErrInternal)
@@ -708,9 +708,10 @@ func (s *private) deleteMeCtrl(w http.ResponseWriter, r *http.Request) {
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
},
User: &token.User{
ID: user.ID,
Name: user.Name,
Attributes: map[string]interface{}{
ID: user.ID,
Name: user.Name,
Picture: user.Picture, // carried so the avatar can be removed when the request is processed
Attributes: map[string]any{
"delete_me": true, // prevents this token from being used for login
},
},
@@ -730,7 +731,11 @@ func (s *private) deleteMeCtrl(w http.ResponseWriter, r *http.Request) {
func (s *private) savePictureCtrl(w http.ResponseWriter, r *http.Request) {
user := rest.MustGetUserInfo(r)
if err := r.ParseMultipartForm(5 * 1024 * 1024); err != nil { // 5M max memory, if bigger will make a file
r.Body = http.MaxBytesReader(w, r.Body, 32*1024*1024) // hard cap on upload to prevent memory exhaustion
// gosec G120: r.Body is already bounded by MaxBytesReader on the line above (32 MB),
// so ParseMultipartForm cannot read more than that regardless of the in-memory threshold.
// The 5 MB argument is the soft threshold above which the form is spilled to disk.
if err := r.ParseMultipartForm(5 * 1024 * 1024); err != nil { //nolint:gosec // bounded by MaxBytesReader above
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't parse multipart form", rest.ErrDecode)
return
}
@@ -772,3 +777,13 @@ func randToken() (string, error) {
}
return fmt.Sprintf("%x", s.Sum(nil)), nil
}
// extractIP returns the IP portion of the remote address, handling both IPv4 and IPv6 formats.
// supports "ip:port", "[ip]:port", and bare "ip" formats.
func extractIP(remoteAddr string) string {
ip, _, err := net.SplitHostPort(remoteAddr)
if err != nil {
return remoteAddr // already a bare IP (no port)
}
return ip
}
+71 -42
View File
@@ -38,7 +38,7 @@ func TestRest_Create(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
@@ -49,7 +49,7 @@ func TestRest_Create(t *testing.T) {
c := R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
loc := c["locator"].(map[string]interface{})
loc := c["locator"].(map[string]any)
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.True(t, len(c["id"].(string)) > 8)
@@ -60,7 +60,7 @@ func TestRest_CreateFilteredCode(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "`+"`foo<bar>`"+`", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
@@ -71,7 +71,7 @@ func TestRest_CreateFilteredCode(t *testing.T) {
c := R.JSON{}
err = json.Unmarshal(b, &c)
require.NoError(t, err, string(b))
loc := c["locator"].(map[string]interface{})
loc := c["locator"].(map[string]any)
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.Equal(t, "`foo<bar>`", c["orig"])
@@ -93,6 +93,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
RoutePath: "/api/v1/img",
RemarkURL: srv.RemarkURL,
ImageService: srv.ImageService,
Transport: http.DefaultTransport,
}
srv.CommentFormatter = store.NewCommentFormatter(srv.ImageProxy)
// need to recreate the server with new ImageProxy, otherwise old one will be used
@@ -109,7 +110,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
defer pngServer.Close()
t.Run("create", func(t *testing.T) {
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "![](`+pngServer.URL+`/gopher.png)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
@@ -122,7 +123,7 @@ func TestRest_CreateAndPreviewWithImage(t *testing.T) {
require.NoError(t, err, string(b))
assert.NotContains(t, c["text"], pngServer.URL)
assert.Contains(t, c["text"], srv.RemarkURL)
loc := c["locator"].(map[string]interface{})
loc := c["locator"].(map[string]any)
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.True(t, len(c["id"].(string)) > 8)
@@ -175,7 +176,7 @@ func TestRest_CreateOldPost(t *testing.T) {
assert.Equal(t, 1, len(comments))
// try to add new comment to the same old post
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "test 123", "locator":{"site": "remark42","url": "https://radio-t.com/blah1"}}`)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
@@ -188,7 +189,7 @@ func TestRest_CreateOldPost(t *testing.T) {
_, err = srv.DataService.Create(old)
assert.NoError(t, err)
resp, err = post(t, ts.URL+"/api/v1/comment",
resp, err = post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "test 123", "locator":{"site": "remark42","url": "https://radio-t.com/blah1"}}`)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
@@ -201,7 +202,7 @@ func TestRest_CreateTooBig(t *testing.T) {
longComment := fmt.Sprintf(`{"text": "%4001s", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`, "Щ")
resp, err := post(t, ts.URL+"/api/v1/comment", longComment)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", longComment)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -214,7 +215,7 @@ func TestRest_CreateTooBig(t *testing.T) {
assert.Equal(t, "invalid comment", c["details"])
veryLongComment := fmt.Sprintf(`{"text": "%70000s", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`, "Щ")
resp, err = post(t, ts.URL+"/api/v1/comment", veryLongComment)
resp, err = post(t, ts.URL+"/api/v1/comment?site=remark42", veryLongComment)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
@@ -234,7 +235,7 @@ func TestRest_CreateWithRestrictedWord(t *testing.T) {
badComment := `{"text": "What the duck is that?", "locator":{"url": "https://radio-t.com/blah1",
"site": "remark42"}}`
resp, err := post(t, ts.URL+"/api/v1/comment", badComment)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", badComment)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -253,7 +254,7 @@ func TestRest_CreateRelativeURL(t *testing.T) {
// check that it's not possible to click insert URL button and not alter the URL in it (which is `url` by default)
relativeURLText := `{"text": "here is a link with relative URL: [google.com](url)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
resp, err := post(t, ts.URL+"/api/v1/comment", relativeURLText)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", relativeURLText)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -272,7 +273,7 @@ func TestRest_CreateRejected(t *testing.T) {
body := `{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
// try to create without auth
resp, err := http.Post(ts.URL+"/api/v1/comment", "", strings.NewReader(body))
resp, err := http.Post(ts.URL+"/api/v1/comment?site=remark42", "", strings.NewReader(body))
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
@@ -280,7 +281,7 @@ func TestRest_CreateRejected(t *testing.T) {
// try with wrong aud
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(body))
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(body))
require.NoError(t, err)
req.Header.Add("X-JWT", devTokenBadAud)
resp, err = client.Do(req)
@@ -294,7 +295,7 @@ func TestRest_CreateWithWrongImage(t *testing.T) {
defer teardown()
// create comment
resp, err := post(t, ts.URL+"/api/v1/comment", fmt.Sprintf(`{"text": "![non-existent.jpg](%s/api/v1/picture/dev_user/bad_picture)", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, srv.RemarkURL))
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", fmt.Sprintf(`{"text": "![non-existent.jpg](%s/api/v1/picture/dev_user/bad_picture)", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, srv.RemarkURL))
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -316,7 +317,7 @@ func TestRest_CreateWithLazyImage(t *testing.T) {
defer teardown()
body := `{"text": "test 123 ![](http://example.com/image.png)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
// create comment
resp, err := post(t, ts.URL+"/api/v1/comment", body)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", body)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
@@ -333,7 +334,7 @@ func TestRest_CreateAndGet(t *testing.T) {
defer teardown()
// create comment
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "**test** *123*\n\n http://radio-t.com", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
@@ -372,7 +373,7 @@ func TestRest_CreateWithQuotes(t *testing.T) {
defer teardown()
// create comment with quotes with smartypants
resp, err := post(t, ts.URL+"/api/v1/comment",
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "smartpants \"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
@@ -395,7 +396,7 @@ func TestRest_CreateWithQuotes(t *testing.T) {
// create comment with quotes without smartypants
srv.privRest.disableFancyTextFormatting = true
resp, err = post(t, ts.URL+"/api/v1/comment",
resp, err = post(t, ts.URL+"/api/v1/comment?site=remark42",
`{"text": "no_smartpants \"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
@@ -881,36 +882,39 @@ func TestRest_EmailAndTelegram(t *testing.T) {
body string
}{
{description: "issue delete request without auth", url: "/api/v1/email", method: http.MethodDelete, responseCode: http.StatusUnauthorized, noAuth: true},
{description: "issue delete request without site_id", url: "/api/v1/email", method: http.MethodDelete, responseCode: http.StatusBadRequest},
{description: "issue delete request without site_id", url: "/api/v1/email", method: http.MethodDelete, responseCode: http.StatusForbidden},
{description: "delete non-existent user email", url: "/api/v1/email?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "set user email, token not set", url: "/api/v1/email/confirm", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "set user email, token not set", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "set user email, token not set, old query param", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest},
{description: "send email confirmation without address", url: "/api/v1/email/subscribe", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "send email confirmation without address", url: "/api/v1/email/subscribe?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest, body: `{"site":"remark42"}`},
{description: "send email confirmation without address, old query param", url: "/api/v1/email/subscribe?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest},
{description: "send email confirmation", url: "/api/v1/email/subscribe", method: http.MethodPost, responseCode: http.StatusOK, body: `{"site":"remark42","address":"good@example.com"}`},
{description: "send email confirmation", url: "/api/v1/email/subscribe?site=remark42", method: http.MethodPost, responseCode: http.StatusOK, body: `{"site":"remark42","address":"good@example.com"}`},
{description: "send email confirmation, old query param", url: "/api/v1/email/subscribe?site=remark42&address=good@example.com", method: http.MethodPost, responseCode: http.StatusOK},
{description: "set user email, token is good", url: "/api/v1/email/confirm", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good, old query param", url: fmt.Sprintf("/api/v1/email/confirm?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com"},
{description: "send confirmation with same address", url: "/api/v1/email/subscribe?site=remark42&address=good@example.com", method: http.MethodPost, responseCode: http.StatusConflict},
{description: "get user email", url: "/api/v1/email?site=remark42", method: http.MethodGet, responseCode: http.StatusOK},
{description: "delete user email", url: "/api/v1/email?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "send another confirmation", url: "/api/v1/email/subscribe?site=remark42&address=good@example.com", method: http.MethodPost, responseCode: http.StatusOK},
{description: "set user email, token is good", url: "/api/v1/email/confirm", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good", url: "/api/v1/email/confirm?site=remark42", method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com", body: fmt.Sprintf(`{"site":"remark42","token":%q}`, goodToken)},
{description: "set user email, token is good, old query param", url: fmt.Sprintf("/api/v1/email/confirm?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusOK, cookieEmail: "good@example.com"},
{description: "unsubscribe user, no token", url: "/email/unsubscribe.html?site=remark42", method: http.MethodPost, responseCode: http.StatusBadRequest},
{description: "unsubscribe user, wrong token", url: "/email/unsubscribe.html?site=remark42&tkn=jwt", method: http.MethodGet, responseCode: http.StatusForbidden},
{description: "unsubscribe user, good token", url: fmt.Sprintf("/email/unsubscribe.html?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusOK},
{description: "unsubscribe user second time, good token", url: fmt.Sprintf("/email/unsubscribe.html?site=remark42&tkn=%s", goodToken), method: http.MethodPost, responseCode: http.StatusConflict},
{description: "issue delete request without auth", url: "/api/v1/telegram", method: http.MethodDelete, responseCode: http.StatusUnauthorized, noAuth: true},
{description: "issue delete request without site_id", url: "/api/v1/telegram", method: http.MethodDelete, responseCode: http.StatusBadRequest},
{description: "issue delete request without site_id", url: "/api/v1/telegram", method: http.MethodDelete, responseCode: http.StatusForbidden},
{description: "delete non-existent user telegram", url: "/api/v1/telegram?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "send telegram confirmation, no siteID", url: "/api/v1/telegram/subscribe", method: http.MethodGet, responseCode: http.StatusBadRequest},
{description: "send telegram confirmation, no siteID", url: "/api/v1/telegram/subscribe", method: http.MethodGet, responseCode: http.StatusForbidden},
{description: "send telegram confirmation", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodGet, responseCode: http.StatusOK},
{description: "set user telegram, token is good", url: "/api/v1/telegram/subscribe?site=remark42&tkn=good_token", method: http.MethodGet, responseCode: http.StatusOK},
{description: "send confirmation with same address", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodGet, responseCode: http.StatusConflict},
{description: "delete user telegram", url: "/api/v1/telegram?site=remark42", method: http.MethodDelete, responseCode: http.StatusOK},
{description: "send another confirmation", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodGet, responseCode: http.StatusOK},
{description: "set user telegram, token is good", url: "/api/v1/telegram/subscribe?site=remark42&tkn=good_token", method: http.MethodGet, responseCode: http.StatusOK},
// telegramSubscribeCtrl mutates state, so HEAD (which stdlib ServeMux would route to the
// GET handler) must be rejected by rejectHead before it runs
{description: "HEAD is rejected on telegram subscribe", url: "/api/v1/telegram/subscribe?site=remark42", method: http.MethodHead, responseCode: http.StatusMethodNotAllowed},
}
client := http.Client{}
defer client.CloseIdleConnections()
@@ -955,7 +959,7 @@ func TestRest_EmailNotification(t *testing.T) {
defer client.CloseIdleConnections()
// create new comment from dev user
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 123",
"user": {"name": "provider1_dev::good@example.com"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -976,7 +980,7 @@ func TestRest_EmailNotification(t *testing.T) {
assert.Empty(t, mockDestination.Get()[0].Emails)
// create child comment from another user, email notification only to admin expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 456",
"pid": %q,
"user": {"name": "other_user"},
@@ -998,7 +1002,7 @@ func TestRest_EmailNotification(t *testing.T) {
// send confirmation token for email
req, err = http.NewRequest(
http.MethodPost,
ts.URL+"/api/v1/email/subscribe",
ts.URL+"/api/v1/email/subscribe?site=remark42",
io.NopCloser(strings.NewReader(`{"site": "remark42", "address": "good@example.com"}`)),
)
require.NoError(t, err)
@@ -1037,7 +1041,7 @@ func TestRest_EmailNotification(t *testing.T) {
// verify email
req, err = http.NewRequest(
http.MethodPost,
ts.URL+"/api/v1/email/confirm",
ts.URL+"/api/v1/email/confirm?site=remark42",
io.NopCloser(strings.NewReader(fmt.Sprintf(`{"site": "remark42", "token": %q}`, verificationToken))),
)
require.NoError(t, err)
@@ -1069,7 +1073,7 @@ func TestRest_EmailNotification(t *testing.T) {
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, subscribedUser)
// create child comment from another user, email notification expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 789",
"pid": %q,
"user": {"name": "other_user"},
@@ -1100,7 +1104,7 @@ func TestRest_EmailNotification(t *testing.T) {
assert.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// create child comment from another user, no email notification
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 321",
"user": {"name": "other_user"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -1158,7 +1162,7 @@ func TestRest_EmailNotification(t *testing.T) {
// confirm email via subscribe call, no email notification is expected
req, err = http.NewRequest(
http.MethodPost,
ts.URL+"/api/v1/email/subscribe",
ts.URL+"/api/v1/email/subscribe?site=remark42",
io.NopCloser(strings.NewReader(`{"site": "remark42", "address": "good@example.com"}`)),
)
require.NoError(t, err)
@@ -1205,7 +1209,7 @@ func TestRest_TelegramNotification(t *testing.T) {
defer client.CloseIdleConnections()
// create new comment from dev user
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 123",
"user": {"name": "provider1_dev::good@example.com"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -1226,7 +1230,7 @@ func TestRest_TelegramNotification(t *testing.T) {
assert.Empty(t, mockDestination.Get()[0].Telegrams)
// create child comment from another user, telegram notification only to admin expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 456",
"pid": %q,
"user": {"name": "other_user"},
@@ -1339,7 +1343,7 @@ func TestRest_TelegramNotification(t *testing.T) {
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, user)
// create child comment from another user, telegram notification expected
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(fmt.Sprintf(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(fmt.Sprintf(
`{"text": "test 789",
"pid": %q,
"user": {"name": "other_user"},
@@ -1370,7 +1374,7 @@ func TestRest_TelegramNotification(t *testing.T) {
assert.Equal(t, http.StatusOK, resp.StatusCode, string(body))
// create child comment from another user, no telegram notification
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
req, err = http.NewRequest("POST", ts.URL+"/api/v1/comment?site=remark42", strings.NewReader(
`{"text": "test 321",
"user": {"name": "other_user"},
"locator":{"url": "https://radio-t.com/blah1",
@@ -1455,7 +1459,7 @@ func TestRest_UserAllDataManyComments(t *testing.T) {
c := store.Comment{User: user, Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 5, 27, 1, 14, 10, 0, time.Local)}
for i := 0; i < 51; i++ {
for i := range 51 {
c.ID = fmt.Sprintf("id-%03d", i)
c.Timestamp = c.Timestamp.Add(time.Second)
_, err := srv.DataService.Create(c)
@@ -1508,6 +1512,8 @@ func TestRest_DeleteMe(t *testing.T) {
claims, err := srv.Authenticator.TokenService().Parse(tkn)
assert.NoError(t, err)
assert.Equal(t, "provider1_dev", claims.User.ID)
assert.Equal(t, "http://example.com/pic.png", claims.User.Picture,
"delete_me token must carry the user's picture so the avatar can be removed when the request is processed")
assert.Equal(t, "https://demo.remark42.com/web/deleteme.html?token="+tkn, m["link"])
req, err = http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/deleteme?site=remark42", ts.URL), http.NoBody)
@@ -1535,7 +1541,7 @@ func TestRest_SavePictureCtrl(t *testing.T) {
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture", ts.URL), bodyBuf)
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture?site=remark42", ts.URL), bodyBuf)
require.NoError(t, err)
req.Header.Add("Content-Type", contentType)
req.Header.Add("X-JWT", devToken)
@@ -1627,7 +1633,7 @@ func TestRest_CreateWithPictures(t *testing.T) {
require.NoError(t, bodyWriter.Close())
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture", ts.URL), bodyBuf)
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture?site=remark42", ts.URL), bodyBuf)
require.NoError(t, err)
req.Header.Add("Content-Type", contentType)
req.Header.Add("X-JWT", devToken)
@@ -1653,7 +1659,7 @@ func TestRest_CreateWithPictures(t *testing.T) {
text := fmt.Sprintf(`text 123 ![](%s/api/v1/picture/%s) *xxx* ![](%s/api/v1/picture/%s) ![](%s/api/v1/picture/%s)`, svc.RemarkURL, ids[0], svc.RemarkURL, ids[1], svc.RemarkURL, ids[2])
body := fmt.Sprintf(`{"text": %q, "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`, text)
resp, err := post(t, ts.URL+"/api/v1/comment", body)
resp, err := post(t, ts.URL+"/api/v1/comment?site=remark42", body)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
@@ -1690,3 +1696,26 @@ func (m *mockTelegram) CheckToken(string, string) (telegram, site string, err er
}
return "good_telegram", m.site, nil
}
func TestExtractIP(t *testing.T) {
tbl := []struct {
addr string
exp string
}{
{"127.0.0.1:8080", "127.0.0.1"},
{"127.0.0.1", "127.0.0.1"},
{"192.168.1.1:443", "192.168.1.1"},
{"[::1]:8080", "::1"},
{"::1", "::1"},
{"[2001:db8::1]:8080", "2001:db8::1"},
{"2001:db8::1", "2001:db8::1"},
{"[fe80::1%25eth0]:80", "fe80::1%25eth0"},
{"", ""},
}
for _, tt := range tbl {
t.Run(tt.addr, func(t *testing.T) {
assert.Equal(t, tt.exp, extractIP(tt.addr))
})
}
}
+70 -15
View File
@@ -11,8 +11,8 @@ import (
"strconv"
"strings"
"time"
"unicode"
"github.com/go-chi/chi/v5"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
@@ -187,7 +187,7 @@ func (s *public) lastCommentsCtrl(w http.ResponseWriter, r *http.Request) {
siteID := r.URL.Query().Get("site")
log.Printf("[DEBUG] get last comments for %s", siteID)
limit, err := strconv.Atoi(chi.URLParam(r, "limit"))
limit, err := strconv.Atoi(r.PathValue("limit"))
if err != nil {
limit = 0
}
@@ -221,7 +221,7 @@ func (s *public) lastCommentsCtrl(w http.ResponseWriter, r *http.Request) {
// GET /id/{id}?site=siteID&url=post-url - gets a comment by id
func (s *public) commentByIDCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
id := r.PathValue("id")
siteID := r.URL.Query().Get("site")
url := r.URL.Query().Get("url")
@@ -364,26 +364,81 @@ func (s *public) listCtrl(w http.ResponseWriter, r *http.Request) {
}
}
// safePictureSegment reports whether seg is acceptable as a path segment in
// the picture URL (no traversal markers, no path separators, no control
// characters). Picture IDs are server-generated hashes plus a known
// extension, so any value carrying these characters is hostile and must be
// rejected before reaching the store. Rejecting controls (CR, LF, TAB, NUL,
// etc.) also closes a log-injection vector since the rejected segment is
// echoed into the access log.
func safePictureSegment(seg string) bool {
if seg == "" || seg == "." {
return false
}
if strings.ContainsAny(seg, "/\\") {
return false
}
if strings.Contains(seg, "..") { // also covers seg == ".."
return false
}
for _, r := range seg {
if unicode.IsControl(r) {
return false
}
}
return true
}
// sendPictureError writes a no-store Cache-Control header and delegates to rest.SendErrorJSON.
// Used by every rejection branch in loadPictureCtrl so error responses never inherit the
// 7-day client cache of the success path.
func sendPictureError(w http.ResponseWriter, r *http.Request, status int, err error, details string, code int) {
w.Header().Set("Cache-Control", "no-store")
rest.SendErrorJSON(w, r, status, err, details, code)
}
// GET /picture/{user}/{id} - get picture
func (s *public) loadPictureCtrl(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "user") + "/" + chi.URLParam(r, "id")
img, err := s.imageService.Load(id)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get image "+id, rest.ErrAssetNotFound)
rest.SetImageDefenseHeaders(w)
user, imgID := r.PathValue("user"), r.PathValue("id")
if user == "" || imgID == "" || !safePictureSegment(user) || !safePictureSegment(imgID) {
log.Printf("[WARN] rejected picture request with unsafe id segments user=%q id=%q", user, imgID)
sendPictureError(w, r, http.StatusBadRequest, fmt.Errorf("invalid picture id"), "invalid picture id", rest.ErrAssetNotFound)
return
}
// enforce client-side caching
id := user + "/" + imgID
img, err := s.imageService.Load(id)
if err != nil {
log.Printf("[WARN] can't load image %s: %v", id, err)
sendPictureError(w, r, http.StatusBadRequest, fmt.Errorf("image not found"), "can't get image", rest.ErrAssetNotFound)
return
}
contentType, err := rest.SafeImgContentType(img)
if err != nil {
log.Printf("[WARN] rejecting non-image picture %s: %v", id, err)
sendPictureError(w, r, http.StatusUnsupportedMediaType, err, "invalid image content", rest.ErrAssetNotFound)
return
}
// /picture/ does not need a security-version etag prefix — the upload flow
// validates input format (readAndValidateImage) and the serve path re-validates
// the stored bytes via rest.SafeImgContentType. Bytes within the resize dimension
// limits ARE preserved verbatim by resize, so the browser defense relies on the
// response headers (validated Content-Type + nosniff + strict CSP +
// Content-Disposition: inline), not on byte normalization. Picture IDs are limited
// to safePictureSegment (alphanumeric xid-generated guids), so the comma split
// inside rest.EtagMatches cannot collide; if the ID format ever changes, revisit.
etag := `"` + id + `"`
w.Header().Set("Etag", etag)
w.Header().Set("Cache-Control", "max-age=604800") // 7 days
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, etag) {
w.WriteHeader(http.StatusNotModified)
return
}
if match := r.Header.Get("If-None-Match"); match != "" && rest.EtagMatches(match, etag) {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("Content-Type", s.imageService.ImgContentType(img))
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Length", strconv.Itoa(len(img)))
w.WriteHeader(http.StatusOK)
if _, err = io.Copy(w, bytes.NewReader(img)); err != nil {
@@ -431,7 +486,7 @@ func (s *public) telegramQrCtrl(w http.ResponseWriter, r *http.Request) {
}
w.Header().Set("Content-Type", "image/png")
if _, err = w.Write(png); err != nil {
if _, err = w.Write(png); err != nil { //nolint:gosec // png bytes from go-qrcode, not HTML
log.Printf("[WARN] can't render qr, %v", err)
}
}
+201 -3
View File
@@ -1,9 +1,11 @@
package api
import (
"bytes"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
@@ -14,11 +16,13 @@ import (
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/image"
"github.com/umputun/remark42/backend/app/store/service"
)
@@ -562,8 +566,8 @@ func TestPublic_FindCommentsCtrl_ConsistentCount(t *testing.T) {
}
}
// Adding initial comments (8 to test-url and 1 to another-url) and voting, and delete two of comments to the first post.
// With sleep so that at least few millisecond pass between each comment
// adding initial comments (8 to test-url and 1 to another-url) and voting, and delete two of comments to the first post.
// with sleep so that at least few millisecond pass between each comment
// and later we would be able to use that in "since" filter with millisecond precision
ids := make([]string, 9)
timestamps := make([]time.Time, 9)
@@ -930,7 +934,7 @@ func TestRest_Config(t *testing.T) {
err := json.Unmarshal([]byte(body), &j)
assert.NoError(t, err)
assert.Equal(t, 300.0, j["edit_duration"])
assert.EqualValues(t, []interface{}{"a1", "a2"}, j["admins"])
assert.EqualValues(t, []any{"a1", "a2"}, j["admins"])
assert.Equal(t, "admin@remark-42.com", j["admin_email"])
assert.Equal(t, 4000.0, j["max_comment_size"])
assert.Equal(t, -5.0, j["low_score"])
@@ -1028,3 +1032,197 @@ func TestRest_Robots(t *testing.T) {
"Allow: /api/v1/list\nAllow: /api/v1/config\nAllow: /api/v1/user\nAllow: /api/v1/img\n"+
"Allow: /api/v1/avatar\nAllow: /api/v1/picture\n", body)
}
// TestRest_LoadPictureRejectsPathTraversal reproduces the unauthenticated path-traversal
// vulnerability in GET /api/v1/picture/{user}/{id}. Before the fix, the handler concatenated
// the URL params verbatim into a filesystem path via path.Join, so a request like
// `/api/v1/picture/../remark.db` would resolve to `<base>/../remark.db`, escaping the image
// directory. Even when the file did not exist (default Partitions=100 mitigates direct hits),
// the FS error message leaked the constructed internal path back to the unauthenticated caller.
func TestRest_LoadPictureRejectsPathTraversal(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
cases := []struct {
name string
path string
wantStatus int
}{
// A literal ".." is normalized away by net/http.ServeMux before routing: the request
// is redirected to the cleaned path, which matches no picture route, so it never reaches
// loadPictureCtrl and resolves to 404. The traversal is neutralized at the router level
// (the cleaned path can only ever reach defined routes or the webRoot-bounded file server),
// so nothing is served either way.
{name: "dotdot in user segment", path: "/api/v1/picture/../remark.db", wantStatus: http.StatusNotFound},
// Encoded traversal is not cleaned by the router, so the handler's safePictureSegment
// validation is what rejects it, with 400.
{name: "dotdot in id segment", path: "/api/v1/picture/dev_user/..%2Fremark.db", wantStatus: http.StatusBadRequest},
{name: "encoded dotdot in user segment", path: "/api/v1/picture/%2E%2E/remark.db", wantStatus: http.StatusBadRequest},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
req, err := http.NewRequest(http.MethodGet, ts.URL+c.path, http.NoBody)
require.NoError(t, err)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, c.wantStatus, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
s := string(body)
assert.NotContains(t, s, "..", "error body must not echo traversal marker")
assert.NotContains(t, s, "remark.db", "error body must not echo attacker-supplied filename")
assert.NotContains(t, s, "no such file", "error body must not leak filesystem state")
assert.NotContains(t, s, "/var/", "error body must not leak internal filesystem path")
})
}
}
// TestRest_LoadPictureRejectsControlCharsInSegment makes sure a CRLF / tab / NUL
// in the URL segment is rejected by safePictureSegment. Without the rejection
// the [WARN] log line constructed from %q-formatted segments would still be
// safe (Go's %q escapes control chars), but a future log change to %s would
// turn this into log forgery — and no legitimate picture id ever needs control
// characters, so the right place to slam the door is in the validator.
func TestRest_LoadPictureRejectsControlCharsInSegment(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
cases := []struct {
name string
path string
}{
{name: "lf in user segment", path: "/api/v1/picture/dev%0Auser/abc.png"},
{name: "cr in user segment", path: "/api/v1/picture/dev%0Duser/abc.png"},
{name: "tab in user segment", path: "/api/v1/picture/dev%09user/abc.png"},
{name: "lf in id segment", path: "/api/v1/picture/dev_user/abc%0A.png"},
{name: "nul in id segment", path: "/api/v1/picture/dev_user/abc%00.png"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
req, err := http.NewRequest(http.MethodGet, ts.URL+c.path, http.NoBody)
require.NoError(t, err)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
s := string(body)
assert.Contains(t, s, "invalid picture id", "must reject as invalid input, not fall through to storage")
assert.NotContains(t, s, "no such file", "must not reach the filesystem")
})
}
}
// TestRest_LoadPictureDefenseHeaders saves a real PNG via the standard upload handler
// and asserts that GET /api/v1/picture/{user}/{id} carries the layered defense headers
// (strict CSP, nosniff, Content-Disposition with filename) and that the strict ETag
// matcher does not 304 on a substring-of-the-real-etag (the pre-fix matcher would).
func TestRest_LoadPictureDefenseHeaders(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
// upload a real PNG via /api/v1/picture
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)
fileWriter, err := bodyWriter.CreateFormFile("file", "picture.png")
require.NoError(t, err)
_, err = io.Copy(fileWriter, gopherPNG())
require.NoError(t, err)
contentType := bodyWriter.FormDataContentType()
require.NoError(t, bodyWriter.Close())
client := http.Client{}
defer client.CloseIdleConnections()
req, err := http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/picture?site=remark42", ts.URL), bodyBuf)
require.NoError(t, err)
req.Header.Add("Content-Type", contentType)
req.Header.Add("X-JWT", devToken)
resp, err := client.Do(req)
require.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
m := map[string]string{}
require.NoError(t, json.Unmarshal(body, &m))
require.NotEmpty(t, m["id"])
// fetch the picture and assert defense headers
resp, err = http.Get(fmt.Sprintf("%s/api/v1/picture/%s", ts.URL, m["id"]))
require.NoError(t, err)
defer resp.Body.Close()
require.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "default-src 'none'; sandbox; frame-ancestors 'none'",
resp.Header.Get("Content-Security-Policy"))
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, `inline; filename="image"`, resp.Header.Get("Content-Disposition"))
assert.Equal(t, "image/png", resp.Header.Get("Content-Type"))
realEtag := resp.Header.Get("Etag")
require.NotEmpty(t, realEtag)
// strict matcher: an If-None-Match value that CONTAINS the real etag as a substring
// but is not equal to it must NOT trigger 304. The pre-fix matcher used
// strings.Contains(header, etag) and would have returned true here.
require.True(t, len(realEtag) > 4)
substringMatch := "prefix-" + realEtag + "-suffix"
req2, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/picture/%s", ts.URL, m["id"]), http.NoBody)
require.NoError(t, err)
req2.Header.Set("If-None-Match", substringMatch)
resp2, err := client.Do(req2)
require.NoError(t, err)
defer resp2.Body.Close()
assert.Equal(t, http.StatusOK, resp2.StatusCode,
"strict etag matcher must NOT 304 when real etag appears only as a substring of If-None-Match; got %q vs real %q", substringMatch, realEtag)
// sanity: the exact real etag DOES validate
req3, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/picture/%s", ts.URL, m["id"]), http.NoBody)
require.NoError(t, err)
req3.Header.Set("If-None-Match", realEtag)
resp3, err := client.Do(req3)
require.NoError(t, err)
defer resp3.Body.Close()
assert.Equal(t, http.StatusNotModified, resp3.StatusCode, "exact etag must round-trip as 304")
}
// TestRest_LoadPictureRejectsNonImage proves the /picture/ handler rejects bytes that
// don't sniff as a real image — even when retrieved successfully from the image store.
// Uses a StoreMock so we can return arbitrary attacker bytes for a valid-looking id.
func TestRest_LoadPictureRejectsNonImage(t *testing.T) {
htmlBody := []byte("<html><body><script>alert(document.domain)</script></body></html>")
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return htmlBody, nil
}}
// minimal public struct on purpose: the reject path only exercises imageService.Load
// (other fields like dataService, cache, commentFormatter are not touched here).
p := &public{imageService: image.NewService(&imageStore, image.ServiceParams{})}
router := routegroup.New(http.NewServeMux())
router.HandleFunc("GET /api/v1/picture/{user}/{id}", p.loadPictureCtrl)
ts := httptest.NewServer(router)
defer ts.Close()
resp, err := http.Get(ts.URL + "/api/v1/picture/dev_user/abc.png")
require.NoError(t, err)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnsupportedMediaType, resp.StatusCode,
"non-image bytes must be rejected as 415")
assert.False(t, strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html"),
"reject response must not be text/html; got %q", resp.Header.Get("Content-Type"))
assert.NotContains(t, string(body), "<script>",
"attacker payload must not be echoed back")
assert.Equal(t, "no-store", resp.Header.Get("Cache-Control"),
"rejection path must not be cacheable")
// defense headers still present on the reject path
assert.Equal(t, "default-src 'none'; sandbox; frame-ancestors 'none'",
resp.Header.Get("Content-Security-Policy"))
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Equal(t, `inline; filename="image"`, resp.Header.Get("Content-Disposition"))
}
+126 -149
View File
@@ -68,6 +68,122 @@ func TestRest_FileServer(t *testing.T) {
_ = os.Remove(testHTMLFile)
}
// TestRest_FileServerStaticAssets covers the static file server behaviors that are
// sensitive to the router: the bare /web -> /web/ redirect, cache headers applied to
// served assets, 404 for missing files, and the directory-listing block.
func TestRest_FileServerStaticAssets(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
require.NoError(t, os.WriteFile(srv.WebRoot+"/asset-test.html", []byte("static body"), 0o600))
require.NoError(t, os.MkdirAll(srv.WebRoot+"/subdir-test", 0o700))
defer func() {
_ = os.Remove(srv.WebRoot + "/asset-test.html")
_ = os.RemoveAll(srv.WebRoot + "/subdir-test")
}()
noRedirect := http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
defer noRedirect.CloseIdleConnections()
t.Run("bare /web redirects to /web/", func(t *testing.T) {
resp, err := noRedirect.Get(ts.URL + "/web")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMovedPermanently, resp.StatusCode)
assert.Equal(t, "/web/", resp.Header.Get("Location"))
})
t.Run("serves an existing asset with cache headers", func(t *testing.T) {
resp, err := noRedirect.Get(ts.URL + "/web/asset-test.html")
require.NoError(t, err)
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "static body", string(body))
assert.NotEmpty(t, resp.Header.Get("Etag"), "cacheControl must set an Etag on served assets")
assert.Contains(t, resp.Header.Get("Cache-Control"), "max-age", "cacheControl must set max-age on served assets")
})
t.Run("missing asset returns 404", func(t *testing.T) {
_, code := get(t, ts.URL+"/web/does-not-exist.html")
assert.Equal(t, http.StatusNotFound, code)
})
t.Run("directory listing is blocked", func(t *testing.T) {
resp, err := noRedirect.Get(ts.URL + "/web/subdir-test/")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusNotFound, resp.StatusCode, "directory listings must be blocked")
})
}
// TestRest_RejectHeadOnDestructiveGET verifies that HEAD is blocked on the state-mutating
// GET routes (which stdlib http.ServeMux would otherwise route to the GET handler) while
// still being served for safe, read-only routes.
func TestRest_RejectHeadOnDestructiveGET(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
client := http.Client{}
defer client.CloseIdleConnections()
t.Run("HEAD is rejected on a destructive GET route", func(t *testing.T) {
req, err := http.NewRequest(http.MethodHead, ts.URL+"/api/v1/admin/deleteme?site=remark42", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode, "HEAD must not reach a state-mutating GET handler")
assert.Equal(t, "GET", resp.Header.Get("Allow"), "405 must carry an Allow header")
})
t.Run("HEAD is rejected on the email unsubscribe route", func(t *testing.T) {
// emailUnsubscribeCtrl deletes the user's email subscription on GET, so HEAD (which
// ServeMux would route to the GET handler) must be rejected before it runs
resp, err := client.Head(ts.URL + "/email/unsubscribe.html?site=remark42")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode, "HEAD must not reach the email-unsubscribe handler")
assert.Equal(t, "GET, POST", resp.Header.Get("Allow"), "Allow must list every method the resource supports")
})
t.Run("HEAD still works on a safe read-only route", func(t *testing.T) {
resp, err := client.Head(ts.URL + "/api/v1/config?site=remark42")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode, "HEAD must still be served for safe read-only routes")
})
t.Run("wrong method on a known route returns 405 with Allow", func(t *testing.T) {
// method-in-pattern is new under ServeMux; a wrong method on a known route must
// still yield 405 with the allowed methods advertised
resp, err := client.Post(ts.URL+"/api/v1/config?site=remark42", "application/json", http.NoBody)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Allow"), "GET", "405 must advertise the allowed methods")
})
}
// TestRest_AvatarMounts verifies both avatar mounts (root /avatar/ and /api/v1/avatar/)
// still route to the avatar handler after the chi Mount -> ServeMux Handle rewiring,
// rather than falling through to a router 404.
func TestRest_AvatarMounts(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
for _, path := range []string{"/api/v1/avatar/nonexistent.image", "/avatar/nonexistent.image"} {
t.Run(path, func(t *testing.T) {
body, code := get(t, ts.URL+path)
// the avatar handler responds (403 "can't load avatar"), not a router 404
assert.Equal(t, http.StatusForbidden, code, "avatar mount must reach the avatar handler")
assert.Contains(t, body, "can't load avatar", "request must reach the avatar handler, not a routing 404")
})
}
}
func TestRest_Shutdown(t *testing.T) {
srv := Rest{Authenticator: &auth.Service{}, ImageProxy: &proxy.Image{}}
done := make(chan bool)
@@ -193,28 +309,6 @@ func TestRest_RunAutocertModeHTTPOnly(t *testing.T) {
srv.Shutdown()
}
func TestRest_rejectAnonUser(t *testing.T) {
ts := httptest.NewServer(fakeAuth(rejectAnonUser(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprintln(w, "Hello")
}))))
defer ts.Close()
resp, err := http.Get(ts.URL)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "use not logged in")
resp, err = http.Get(ts.URL + "?fake_id=anonymous_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusForbidden, resp.StatusCode, "anon rejected")
resp, err = http.Get(ts.URL + "?fake_id=real_user123&fake_name=test")
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode, "real user")
}
func Test_URLKey(t *testing.T) {
tbl := []struct {
url string
@@ -284,43 +378,12 @@ func TestRest_parseError(t *testing.T) {
}
}
func TestRest_cacheControl(t *testing.T) {
tbl := []struct {
url string
version string
exp time.Duration
etag string
maxAge int
}{
{"http://example.com/foo", "v1", time.Hour, "b433be1ea19edaee9dc92ca4b895b6bdf3c058cb", 3600},
{"http://example.com/foo2", "v1", 10 * time.Hour, "6d8466aef3246c1057452561acddf7ad9d0d99e0", 36000},
{"http://example.com/foo", "v2", time.Hour, "481700c52aab0dfbca99f3ffc2a4fbb27884c114", 3600},
{"https://example.com/foo", "v2", time.Hour, "bebd4f1b87f474792c4e75e5affe31fbf67f5778", 3600},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", tt.url, http.NoBody)
w := httptest.NewRecorder()
h := cacheControl(tt.exp, tt.version)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
t.Logf("%+v", resp.Header)
assert.Equal(t, `"`+tt.etag+`"`, resp.Header.Get("Etag"))
assert.Equal(t, `max-age=`+strconv.Itoa(int(tt.exp.Seconds()))+", no-cache", resp.Header.Get("Cache-Control"))
})
}
}
func TestRest_frameAncestors(t *testing.T) {
ts, _, teardown := startupT(t, func(o *Rest) {
o.AllowedAncestors = []string{"'self'", "https://example.com"}
})
// Test case with frame-ancestors
// test case with frame-ancestors
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
@@ -329,7 +392,7 @@ func TestRest_frameAncestors(t *testing.T) {
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "frame-ancestors 'self' https://example.com;")
teardown()
// Test case without frame-ancestors
// test case without frame-ancestors
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.AllowedAncestors = []string{}
})
@@ -341,99 +404,9 @@ func TestRest_frameAncestors(t *testing.T) {
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "frame-ancestors *;")
}
// check CSP, img-src should be 'self' with proxy enabled and * without it
func TestRest_securityHeaders(t *testing.T) {
ts, _, teardown := startupT(t)
// with proxy disabled
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src *;")
teardown()
// check CSP with proxy enabled
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.ExternalImageProxy = true
})
defer teardown()
resp, err = client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src 'self';")
}
func TestRest_subscribersOnly(t *testing.T) {
paidSubUser := &token.User{}
paidSubUser.SetPaidSub(true)
tbl := []struct {
subsOnly bool
user token.User
setUser bool
status int
}{
{true, token.User{}, false, http.StatusUnauthorized},
{true, token.User{}, true, http.StatusForbidden},
{false, token.User{}, false, http.StatusOK},
{false, token.User{}, true, http.StatusOK},
{true, *paidSubUser, true, http.StatusOK},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", http.NoBody)
if tt.setUser {
req = token.SetUserInfo(req, tt.user)
}
w := httptest.NewRecorder()
h := subscribersOnly(tt.subsOnly)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
func Test_validEmailAuth(t *testing.T) {
tbl := []struct {
req string
status int
}{
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=site-with-dash_and_underscore-and.dot&address=umputun%example.com&user=someone", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someone+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=Евгений+Умпутун", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=12", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusForbidden},
{"/auth/email/login?site=remark42&address=umputun%example.com&user=someonelooong+loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong", http.StatusForbidden},
{"/auth/twitter/login?site=remark42&address=umputun%example.com&user=..blah+blah", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun%example.com", http.StatusOK},
{"/auth/email/login?site=remark42&address=umputun+example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=bad!site&address=umputun%example.com&user=someone", http.StatusForbidden},
{"/auth/email/login?site=loooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooongsite&address=umputun%example.com&user=someone", http.StatusForbidden},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com"+tt.req, http.NoBody)
w := httptest.NewRecorder()
h := validEmailAuth()(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
})
}
}
// randomPath pick a file or folder name which is not in use for sure
func randomPath(tempDir, basename, suffix string) (string, error) {
for i := 0; i < 10; i++ {
for range 10 {
fname := fmt.Sprintf("/%s/%s-%d%s", tempDir, basename, rand.Int31(), suffix)
fmt.Printf("fname %q", fname)
_, err := os.Stat(fname)
@@ -626,7 +599,11 @@ func addCommentGetCreatedTime(t *testing.T, c store.Comment, ts *httptest.Server
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", bytes.NewBuffer(b))
postURL := ts.URL + "/api/v1/comment"
if c.Locator.SiteID != "" {
postURL += "?site=" + c.Locator.SiteID
}
req, err := http.NewRequest("POST", postURL, bytes.NewBuffer(b))
require.NoError(t, err)
req.Header.Add("X-JWT", devToken)
resp, err := client.Do(req)
@@ -663,7 +640,7 @@ func requireAdminOnly(t *testing.T, req *http.Request) {
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
for range 10 {
port = 40000 + int(rand.Int31n(10000))
if ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port)); err == nil {
_ = ln.Close()
@@ -675,7 +652,7 @@ func chooseRandomUnusedPort() (port int) {
func waitForHTTPSServerStart(port int) {
// wait for up to 3 seconds for HTTPS server to start
for i := 0; i < 300; i++ {
for range 300 {
time.Sleep(time.Millisecond * 10)
conn, _ := net.DialTimeout("tcp", fmt.Sprintf("localhost:%d", port), time.Millisecond*10)
if conn != nil {
+3 -3
View File
@@ -56,7 +56,7 @@ func (s *rss) postCommentsCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
if _, err = w.Write(data); err != nil { //nolint:gosec // xml feed bytes from gorilla/feeds, not HTML
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
}
@@ -87,7 +87,7 @@ func (s *rss) siteCommentsCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
if _, err = w.Write(data); err != nil { //nolint:gosec // xml feed bytes from gorilla/feeds, not HTML
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
}
@@ -119,7 +119,7 @@ func (s *rss) repliesCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
if _, err = w.Write(data); err != nil { //nolint:gosec // xml feed bytes from gorilla/feeds, not HTML
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
}
+37 -18
View File
@@ -2,16 +2,16 @@ package api
import (
"crypto/tls"
"fmt"
"net/http"
"net/url"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
log "github.com/go-pkgz/lgr"
"golang.org/x/crypto/acme/autocert"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/routegroup"
"golang.org/x/crypto/acme/autocert"
)
// sslMode defines ssl mode for rest server
@@ -40,13 +40,13 @@ type SSLConfig struct {
// httpToHTTPSRouter creates new router which does redirect from http to https server
// with default middlewares. Used in 'static' ssl mode.
func (s *Rest) httpToHTTPSRouter() chi.Router {
log.Printf("[DEBUG] create https-to-http redirect routes")
router := chi.NewRouter()
router.Use(middleware.RealIP, R.Recoverer(log.Default()))
router.Use(middleware.Throttle(1000), middleware.Timeout(60*time.Second))
func (s *Rest) httpToHTTPSRouter() http.Handler {
log.Printf("[DEBUG] create http-to-https redirect routes")
router := routegroup.New(http.NewServeMux())
router.Use(R.Recoverer(log.Default()))
router.Use(R.Throttle(1000), R.Timeout(60*time.Second))
router.Handle("/*", s.redirectHandler())
router.Handle("/", s.redirectHandler())
return router
}
@@ -54,26 +54,45 @@ func (s *Rest) httpToHTTPSRouter() chi.Router {
// with default middlewares. This part is necessary to obtain certificate from LE.
// If it receives not a acme challenge it performs redirect to https server.
// Used in 'auto' ssl mode.
func (s *Rest) httpChallengeRouter(m *autocert.Manager) chi.Router {
func (s *Rest) httpChallengeRouter(m *autocert.Manager) http.Handler {
log.Printf("[DEBUG] create http-challenge routes")
router := chi.NewRouter()
router.Use(middleware.RealIP, R.Recoverer(log.Default()))
router.Use(middleware.Throttle(1000), middleware.Timeout(60*time.Second))
router := routegroup.New(http.NewServeMux())
router.Use(R.Recoverer(log.Default()))
router.Use(R.Throttle(1000), R.Timeout(60*time.Second))
router.Handle("/*", m.HTTPHandler(s.redirectHandler()))
router.Handle("/", m.HTTPHandler(s.redirectHandler()))
return router
}
func (s *Rest) redirectHandler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
newURL := s.RemarkURL + r.URL.Path
if r.URL.RawQuery != "" {
newURL += "?" + r.URL.RawQuery
newURL, err := s.redirectURL(r)
if err != nil {
log.Printf("[WARN] failed to build redirect URL, %s", err)
http.Error(w, "invalid redirect URL", http.StatusInternalServerError)
return
}
http.Redirect(w, r, newURL, http.StatusTemporaryRedirect)
})
}
func (s *Rest) redirectURL(r *http.Request) (string, error) {
baseURL, err := url.Parse(s.RemarkURL)
if err != nil {
return "", fmt.Errorf("parse remark URL: %w", err)
}
if baseURL.Scheme != "http" && baseURL.Scheme != "https" || baseURL.Host == "" {
return "", fmt.Errorf("remark URL must be absolute HTTP(S) URL")
}
basePath := strings.TrimRight(baseURL.Path, "/")
requestPath := "/" + strings.TrimLeft(r.URL.Path, "/")
baseURL.Path = basePath + requestPath
baseURL.RawQuery = r.URL.RawQuery
baseURL.Fragment = ""
return baseURL.String(), nil
}
func (s *Rest) makeAutocertManager() *autocert.Manager {
return &autocert.Manager{
Prompt: autocert.AcceptTOS,
+10
View File
@@ -40,6 +40,16 @@ func TestSSL_Redirect(t *testing.T) {
assert.Equal(t, "https://localhost:443/blah?param=1", resp.Header.Get("Location"))
}
func TestSSL_RedirectURLKeepsConfiguredHost(t *testing.T) {
rest := Rest{RemarkURL: "https://localhost:443/base"}
req, err := http.NewRequest("GET", "http://example.com//evil.test/path?next=//evil.test", http.NoBody)
require.NoError(t, err)
redirectURL, err := rest.redirectURL(req)
require.NoError(t, err)
assert.Equal(t, "https://localhost:443/base/evil.test/path?next=//evil.test", redirectURL)
}
func TestSSL_ACME_HTTPChallengeRouter(t *testing.T) {
rest := Rest{
RemarkURL: "https://localhost:443",
+1 -1
View File
@@ -51,7 +51,7 @@ type errTmplData struct {
// error code is not included in render as it is intended for UI developers and not for the users
func SendErrorHTML(w http.ResponseWriter, r *http.Request, httpStatusCode int, err error, details string, errCode int) {
// MustExecute behaves like template.Execute, but panics if an error occurs.
MustExecute := func(tmpl *template.Template, wr io.Writer, data interface{}) {
MustExecute := func(tmpl *template.Template, wr io.Writer, data any) {
if err = tmpl.Execute(wr, data); err != nil {
panic(err)
}
+76
View File
@@ -0,0 +1,76 @@
package rest
import (
"fmt"
"net/http"
"strings"
)
// StrictImageCSP is the strictest default-deny Content-Security-Policy used both by
// image-serving handlers (/api/v1/img, /api/v1/picture/{user}/{id}) and by the api-wide
// apiCSPMiddleware (covering all /api/v1/* responses — JSON, XML/RSS, images). The name
// keeps the "image" prefix for historical reasons; the policy itself is generic and
// suitable for any non-document API response.
//
// Re-setting the same value inside the image handlers (after the middleware already set
// it) is intentional defense-in-depth: if the middleware ever stops applying (route
// refactor, mount point change), the handlers still emit the header.
const StrictImageCSP = "default-src 'none'; sandbox; frame-ancestors 'none'"
// SafeImgContentType returns the sniffed content type for provided bytes if and only
// if it is in the strict allowlist of image formats safe to serve from a same-origin
// proxy endpoint: image/png, image/jpeg, image/gif, image/webp, image/bmp, image/x-icon.
// Anything else — HTML, XML, SVG, plain text, application/octet-stream, or any future
// image format the stdlib sniffer may learn (e.g. AVIF, HEIC, JXL, TIFF) — is rejected.
// SVG would also be rejected as it sniffs as text/xml or text/plain, never image/svg+xml.
// The previous behavior silently mapped application/octet-stream to image/* and is gone.
func SafeImgContentType(img []byte) (string, error) {
contentType := http.DetectContentType(img)
base, _, _ := strings.Cut(contentType, ";")
base = strings.TrimSpace(base)
switch base {
case "image/png", "image/jpeg", "image/gif", "image/webp", "image/bmp", "image/x-icon":
return base, nil
}
return "", fmt.Errorf("non-image content type %q", contentType)
}
// SetImageDefenseHeaders applies the layered defense headers shared by every response
// from image-serving endpoints (success, 304, or error). Each header survives content-type
// validation regressions, browser sniffing, and top-level navigation:
// - Content-Security-Policy: strict, with sandbox — blocks inline scripts and event handlers
// - X-Content-Type-Options: nosniff — prevents browsers from MIME-overriding the declared type
// - Content-Disposition: inline; filename="image" — frames the response as a file, not a document
//
// CSP is duplicated by apiCSPMiddleware for /api/v1/* — re-setting the same value here is
// harmless and provides defense-in-depth if the middleware is bypassed or moved. The other
// two headers (nosniff, Content-Disposition with filename) are image-specific and not set
// by the middleware.
func SetImageDefenseHeaders(w http.ResponseWriter) {
w.Header().Set("Content-Security-Policy", StrictImageCSP)
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Content-Disposition", `inline; filename="image"`)
}
// EtagMatches reports whether If-None-Match header value contains the given etag.
// Handles the * wildcard, comma-separated etag lists with the W/ weak-validator prefix.
// NOTE: This is intentionally a simple splitter — it does not handle opaque-tags that
// contain commas (allowed by RFC 7232 but never emitted by this codebase, whose etag
// format is `"v2:<base64-url>"` or `"<user>/<xid>"`). If the etag format ever changes
// to include comma-bearing values, revisit this parser.
// Replaces a substring search that could match unrelated entries (e.g. an etag that
// happens to be a prefix of another).
func EtagMatches(header, etag string) bool {
header = strings.TrimSpace(header)
if header == "*" {
return true
}
for tag := range strings.SplitSeq(header, ",") {
tag = strings.TrimSpace(tag)
tag = strings.TrimPrefix(tag, "W/")
if tag == etag {
return true
}
}
return false
}
+103
View File
@@ -0,0 +1,103 @@
package rest
import (
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestEtagMatches covers the strict If-None-Match parser that replaced a substring
// search prone to false positives (etag "abc" being matched inside "fooabc").
func TestEtagMatches(t *testing.T) {
tbl := []struct {
name string
header string
etag string
want bool
}{
{"exact match", `"v2:abc"`, `"v2:abc"`, true},
{"comma-separated, second matches", `"x", "v2:abc"`, `"v2:abc"`, true},
{"weak validator prefix", `W/"v2:abc"`, `"v2:abc"`, true},
{"wildcard matches anything", `*`, `"v2:abc"`, true},
{"leading/trailing whitespace", ` "v2:abc" `, `"v2:abc"`, true},
{"substring not enough", `"v2:abcdef"`, `"v2:abc"`, false},
{"prefix-only mismatch", `"v2:ab"`, `"v2:abc"`, false},
{"pre-fix etag no longer matches v2", `"abc"`, `"v2:abc"`, false},
{"empty header", ``, `"v2:abc"`, false},
{"different etag", `"v2:xyz"`, `"v2:abc"`, false},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, EtagMatches(tt.header, tt.etag))
})
}
}
func TestSetImageDefenseHeaders(t *testing.T) {
w := httptest.NewRecorder()
SetImageDefenseHeaders(w)
assert.Equal(t, StrictImageCSP, w.Header().Get("Content-Security-Policy"))
assert.Equal(t, "nosniff", w.Header().Get("X-Content-Type-Options"))
assert.Equal(t, `inline; filename="image"`, w.Header().Get("Content-Disposition"))
}
// TestSafeImgContentType exercises the strict allowlist. The previous behavior
// (HasPrefix "image/" with an explicit image/svg+xml carve-out) is gone — the
// allowlist is the source of truth, and the explicit svg branch was dead code
// because http.DetectContentType never returns image/svg+xml (real SVG bodies
// sniff as text/xml or text/plain depending on whether they carry an XML decl,
// so they are rejected implicitly by not matching the allowlist).
func TestSafeImgContentType(t *testing.T) {
// minimal magic-byte bodies — verified via http.DetectContentType to produce
// the expected image/* result without needing testdata files for every format
pngMagic := []byte("\x89PNG\r\n\x1a\n")
jpegMagic := []byte("\xff\xd8\xff\xe0\x00\x10JFIF\x00")
gifBytes := []byte("GIF89a")
webpBytes := []byte("RIFF\x00\x00\x00\x00WEBPVP8 ")
bmpBytes := []byte("BM\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")
icoBytes := []byte("\x00\x00\x01\x00\x01\x00")
// SVG with XML decl sniffs as text/xml — rejected because it's not in the allowlist
svgWithXMLDecl := []byte(`<?xml version="1.0"?><svg xmlns="http://www.w3.org/2000/svg" onload="alert(1)"></svg>`)
// SVG without XML decl sniffs as text/plain — also rejected
svgPlain := []byte(`<svg xmlns="http://www.w3.org/2000/svg" width="10"></svg>`)
tbl := []struct {
name string
body []byte
wantCT string
wantErr bool
}{
{name: "nil rejected", body: nil, wantErr: true},
{name: "empty rejected", body: []byte{}, wantErr: true},
{name: "png magic accepted", body: pngMagic, wantCT: "image/png"},
{name: "jpeg magic accepted", body: jpegMagic, wantCT: "image/jpeg"},
{name: "gif accepted", body: gifBytes, wantCT: "image/gif"},
{name: "webp accepted", body: webpBytes, wantCT: "image/webp"},
{name: "bmp accepted", body: bmpBytes, wantCT: "image/bmp"},
{name: "ico accepted", body: icoBytes, wantCT: "image/x-icon"},
{name: "html doc rejected", body: []byte(`<!DOCTYPE html><html></html>`), wantErr: true},
{name: "html fragment rejected", body: []byte(`<body><img></body>`), wantErr: true},
{name: "plain text rejected", body: []byte("hello world"), wantErr: true},
{name: "octet-stream rejected", body: []byte{0x00, 0x01, 0x02, 0x03, 0x04}, wantErr: true},
{name: "svg with xml decl rejected (sniffs as text/xml)", body: svgWithXMLDecl, wantErr: true},
{name: "svg without xml decl rejected (sniffs as text/plain)", body: svgPlain, wantErr: true},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
got, err := SafeImgContentType(tt.body)
if tt.wantErr {
require.Error(t, err)
assert.Empty(t, got)
assert.Contains(t, err.Error(), "non-image content type")
return
}
require.NoError(t, err)
assert.Equal(t, tt.wantCT, got)
// returned type must never carry a charset suffix (the strip code path)
assert.NotContains(t, got, ";")
})
}
}
+104 -25
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/base64"
"errors"
"fmt"
"io"
"net/http"
@@ -15,9 +16,15 @@ import (
"github.com/go-pkgz/repeater/v2"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/safehttp"
"github.com/umputun/remark42/backend/app/store/image"
)
// errInvalidUpstreamContentType is returned by downloadImage when the upstream's
// Content-Type header is not image/*. The handler checks via errors.Is to convert
// it into a 400 (input rejected) instead of the generic 404 (fetch failed).
var errInvalidUpstreamContentType = errors.New("invalid upstream content type")
// Image extracts image src from comment's html and provides proxy for them
// this is needed to keep remark42 running behind of HTTPS serve all images via https
type Image struct {
@@ -27,6 +34,11 @@ type Image struct {
CacheExternal bool
Timeout time.Duration
ImageService *image.Service
// Transport, if non-nil, is used as-is for outbound image fetches and is the
// caller's responsibility to make SSRF-safe. When nil, safehttp.Transport()
// is installed, which blocks dialing any private/reserved IP and resolves
// hostnames to defeat DNS rebinding.
Transport http.RoundTripper
}
// Convert img src links to proxied links depends on enabled options
@@ -78,31 +90,67 @@ func (p Image) replace(commentHTML string, imgs []string) string {
return commentHTML
}
// etagVersionPrefix is the security-version tag bumped whenever cached responses for the
// same src need to be invalidated. Pre-fix responses were served as text/html and cached
// by browsers/proxies under ETag `"<base64(src)>"`; the prefix invalidates those validators
// so revalidating clients get a fresh 200 instead of letting the cached HTML 304.
//
// LIMITATION: with the 30-day max-age below, browsers serve pre-fix bytes from their
// local cache without contacting the server until that TTL expires or the cache is
// evicted under memory pressure. The prefix only helps clients that revalidate during
// the cached lifetime (Ctrl+R, intermediaries, post-expiry use). Operators running a
// CDN/edge cache in front of remark42 should purge /api/v1/img after deploy. The
// realistic exposure is narrow: cache carryover only affects users who navigated
// top-level to an attacker URL pre-fix and still have that URL cached — the normal
// <img> embed path cached text/html but never executed it.
const etagVersionPrefix = "v2:"
// Handler returns http handler respond to proxied request
func (p Image) Handler(w http.ResponseWriter, r *http.Request) {
src, err := base64.URLEncoding.DecodeString(r.URL.Query().Get("src"))
rest.SetImageDefenseHeaders(w)
srcParam := r.URL.Query().Get("src")
src, err := base64.URLEncoding.DecodeString(srcParam)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't decode image url", rest.ErrDecode)
sendImageProxyError(w, r, http.StatusBadRequest, err, "can't decode image url", rest.ErrDecode)
return
}
imgURL := string(src)
var img []byte
imgID, err := image.CachedImgID(imgURL)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't parse image url "+imgURL, rest.ErrAssetNotFound)
sendImageProxyError(w, r, http.StatusBadRequest, fmt.Errorf("invalid image url"), "can't parse image url", rest.ErrAssetNotFound)
return
}
// compute the current-version etag once. We don't set it as a response header yet
// because error paths below must NOT inherit it — otherwise transient failures
// (4xx) would get cached alongside the 30-day Cache-Control of the success path.
// The etag (and Cache-Control) are set only on the 304 short-circuit and the
// validated 200 path.
etag := `"` + etagVersionPrefix + srcParam + `"`
// short-circuit revalidation before any cache lookup or upstream fetch: a matching
// current-version If-None-Match means the client already has bytes from a prior
// successful (post-fix, validated) 200, so a bodyless 304 is safe and avoids
// upstream DoS amplification on hot comment pages without CacheExternal.
if match := r.Header.Get("If-None-Match"); match != "" && rest.EtagMatches(match, etag) {
w.Header().Set("Etag", etag)
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
w.WriteHeader(http.StatusNotModified)
return
}
// try to load from cache for case it was saved when CacheExternal was enabled
img, _ = p.ImageService.Load(imgID)
img, _ := p.ImageService.Load(imgID)
if img == nil {
img, err = p.downloadImage(context.Background(), imgURL)
img, err = p.downloadImage(r.Context(), imgURL)
if err != nil {
if strings.Contains(err.Error(), "invalid content type") {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid content type", rest.ErrImgNotFound)
log.Printf("[WARN] failed to download image: %v", err)
if errors.Is(err, errInvalidUpstreamContentType) {
sendImageProxyError(w, r, http.StatusBadRequest, fmt.Errorf("invalid content type"), "invalid content type", rest.ErrImgNotFound)
return
}
rest.SendErrorJSON(w, r, http.StatusNotFound, err, "can't get image "+imgURL, rest.ErrAssetNotFound)
sendImageProxyError(w, r, http.StatusNotFound, fmt.Errorf("failed to fetch"), "can't get image", rest.ErrAssetNotFound)
return
}
if p.CacheExternal {
@@ -110,24 +158,37 @@ func (p Image) Handler(w http.ResponseWriter, r *http.Request) {
}
}
// enforce client-side caching
etag := `"` + r.URL.Query().Get("src") + `"`
w.Header().Set("Etag", etag)
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
if match := r.Header.Get("If-None-Match"); match != "" {
if strings.Contains(match, etag) {
w.WriteHeader(http.StatusNotModified)
return
}
// validate body bytes are actually an image — never trust upstream Content-Type or cache
contentType, err := rest.SafeImgContentType(img)
if err != nil {
log.Printf("[WARN] rejecting non-image content from %s: %v", imgURL, err)
sendImageProxyError(w, r, http.StatusUnsupportedMediaType, err, "invalid image content", rest.ErrImgNotFound)
return
}
w.Header().Add("Content-Type", p.ImageService.ImgContentType(img))
// success path: long-lived client cache with etag for cheap revalidation. 30-day
// TTL keeps the proxy efficient for hot pages; when clients DO revalidate
// (Ctrl+R, intermediaries, post-expiry), the versioned etag ensures pre-fix
// poisoned validators don't match and a fresh validated 200 is returned. See
// etagVersionPrefix godoc for the limitation on browser-local caches.
w.Header().Set("Etag", etag)
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
w.Header().Set("Content-Type", contentType)
_, err = io.Copy(w, bytes.NewReader(img))
if err != nil {
log.Printf("[WARN] can't copy image stream, %s", err)
}
}
// sendImageProxyError writes a no-store error response so a transient failure (4xx)
// cannot inherit the success path's 30-day Cache-Control or the versioned ETag, which
// would otherwise pin the error in the browser/intermediary cache for that TTL.
// Defense headers from SetImageDefenseHeaders at the top of the handler survive.
func sendImageProxyError(w http.ResponseWriter, r *http.Request, status int, err error, details string, errCode int) {
w.Header().Set("Cache-Control", "no-store")
rest.SendErrorJSON(w, r, status, err, details, errCode)
}
// cache image from provided Reader using given ID
func (p Image) cacheImage(r io.Reader, imgID string) {
err := p.ImageService.SaveWithID(imgID, r)
@@ -148,16 +209,26 @@ func (p Image) downloadImage(ctx context.Context, imgURL string) ([]byte, error)
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
client := http.Client{Timeout: 30 * time.Second}
transport := p.Transport
if transport == nil {
transport = safehttp.Transport()
}
client := http.Client{
Timeout: 30 * time.Second,
Transport: transport,
}
defer client.CloseIdleConnections()
var resp *http.Response
err := repeater.NewFixed(5, time.Second).Do(ctx, func() error {
var e error
req, e := http.NewRequest("GET", imgURL, http.NoBody)
// SSRF safety: client.Transport is safehttp.Transport() when p.Transport is nil
// (see Image.Transport contract above); when caller supplies a transport they
// own SSRF safety for that path.
req, e := http.NewRequest("GET", imgURL, http.NoBody) //nolint:gosec // see comment above
if e != nil {
return fmt.Errorf("failed to make request for %s: %w", imgURL, e)
}
resp, e = client.Do(req.WithContext(ctx)) //nolint:bodyclose // need a refactor to fix that
resp, e = client.Do(req.WithContext(ctx)) //nolint:bodyclose,gosec // body closed in defer; transport contract above
return e
})
if err != nil {
@@ -171,12 +242,20 @@ func (p Image) downloadImage(ctx context.Context, imgURL string) ([]byte, error)
contentType := resp.Header.Get("Content-Type")
if !strings.HasPrefix(contentType, "image/") {
return nil, fmt.Errorf("invalid content type %s", contentType)
return nil, fmt.Errorf("%w: %s", errInvalidUpstreamContentType, contentType)
}
imgData, err := io.ReadAll(resp.Body)
maxSize := 5 * 1024 * 1024 // 5MB default
if p.ImageService != nil && p.ImageService.MaxSize > 0 {
maxSize = p.ImageService.MaxSize
}
lr := io.LimitReader(resp.Body, int64(maxSize)+1)
imgData, err := io.ReadAll(lr)
if err != nil {
return nil, fmt.Errorf("unable to read image body")
return nil, fmt.Errorf("unable to read image body: %w", err)
}
if len(imgData) > maxSize {
return nil, fmt.Errorf("image is too large")
}
return imgData, nil
}
+523 -25
View File
@@ -100,6 +100,7 @@ func TestImage_Routes(t *testing.T) {
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
@@ -150,6 +151,7 @@ func TestImage_DisabledCachingAndHTTP2HTTPS(t *testing.T) {
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
@@ -183,6 +185,7 @@ func TestImage_RoutesCachingImage(t *testing.T) {
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{MaxSize: 1500}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
@@ -207,34 +210,59 @@ func TestImage_RoutesCachingImage(t *testing.T) {
}
func TestImage_RoutesUsingCachedImage(t *testing.T) {
// In order to validate that cached data used cache "will return" some other data from what http server would
testImage := []byte(fmt.Sprintf("%256s", "X"))
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return testImage, nil
}}
img := Image{
CacheExternal: true,
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
}
t.Run("cached image is served", func(t *testing.T) {
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return gopherPNGBytes(), nil
}}
img := Image{
CacheExternal: true,
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
httpSrv := imgHTTPTestsServer(t)
defer httpSrv.Close()
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
httpSrv := imgHTTPTestsServer(t)
defer httpSrv.Close()
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/img1.png"))
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/img1.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "image/png", resp.Header.Get("Content-Type"))
assert.Equal(t, 1, len(imageStore.LoadCalls()))
})
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "256", resp.Header["Content-Length"][0])
assert.Equal(t, "text/plain; charset=utf-8", resp.Header["Content-Type"][0],
"if you save text you receive text/plain in response, that's only fair option you got")
t.Run("non-image cached bytes are rejected (cache poisoning defense)", func(t *testing.T) {
nonImage := fmt.Appendf(nil, "%256s", "X")
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return nonImage, nil
}}
img := Image{
CacheExternal: true,
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
}
assert.Equal(t, 1, len(imageStore.LoadCalls()))
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
httpSrv := imgHTTPTestsServer(t)
defer httpSrv.Close()
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/img1.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
body, _ := io.ReadAll(resp.Body)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnsupportedMediaType, resp.StatusCode,
"non-image bytes from cache must be rejected, not served as text/plain (XSS defense)")
assert.False(t, strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html"),
"reject response must not be text/html; got %q", resp.Header.Get("Content-Type"))
assert.NotContains(t, string(body), "XXXXX", "non-image bytes must not be echoed back")
})
}
func TestImage_RoutesTimedOut(t *testing.T) {
@@ -246,6 +274,7 @@ func TestImage_RoutesTimedOut(t *testing.T) {
RoutePath: "/api/v1/proxy",
Timeout: 50 * time.Millisecond,
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
@@ -262,7 +291,8 @@ func TestImage_RoutesTimedOut(t *testing.T) {
assert.NoError(t, resp.Body.Close())
require.NoError(t, err)
t.Log(string(b))
assert.Contains(t, string(b), "deadline exceeded")
assert.Contains(t, string(b), "failed to fetch")
assert.NotContains(t, string(b), "deadline exceeded", "should not leak transport details")
assert.Equal(t, 1, len(imageStore.LoadCalls()))
}
@@ -304,6 +334,474 @@ func TestImage_ConvertCachingMode(t *testing.T) {
assert.Equal(t, `<img src="https://remark42.com/img?src=aHR0cDovL3JhZGlvLXQuY29tL2ltZzMucG5n"/> xyz <img src="https://remark42.com/img?src=aHR0cDovL2ltYWdlcy5wZXhlbHMuY29tLzY3NjM2L2ltZzQuanBlZw==">`, r)
}
func TestImage_PrivateIPBlocking(t *testing.T) {
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
HTTP2HTTPS: true,
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
Timeout: 100 * time.Millisecond,
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
// no Transport override — uses SSRF-safe transport
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
tbl := []struct {
name string
url string
}{
{"loopback", "http://127.0.0.1/image.png"},
{"rfc1918 10.x", "http://10.0.0.1/image.png"},
{"rfc1918 172.16.x", "http://172.16.0.1/image.png"},
{"rfc1918 192.168.x", "http://192.168.1.1/image.png"},
{"link-local", "http://169.254.1.1/image.png"},
{"ipv6 loopback", "http://[::1]/image.png"},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(tt.url))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, resp.Body.Close())
require.NoError(t, err)
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
assert.NotContains(t, string(b), "private address", "should not leak private IP check details")
assert.Contains(t, string(b), "failed to fetch")
})
}
}
func TestImage_ErrorSanitization(t *testing.T) {
// server that immediately closes connections to simulate transport errors
httpSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
hj, ok := w.(http.Hijacker)
if !ok {
w.WriteHeader(http.StatusInternalServerError)
return
}
conn, _, _ := hj.Hijack()
conn.Close() // forcefully close to trigger transport error
}))
defer httpSrv.Close()
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
Timeout: 2 * time.Second,
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, resp.Body.Close())
require.NoError(t, err)
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
assert.Contains(t, string(b), "failed to fetch")
assert.NotContains(t, string(b), "EOF", "should not leak transport details")
assert.NotContains(t, string(b), "connection", "should not leak transport details")
}
func TestImage_ResponseSizeLimit(t *testing.T) {
// create a test server that returns a large image
largeImg := make([]byte, 2000)
for i := range largeImg {
largeImg[i] = 0xFF
}
httpSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(largeImg)
}))
defer httpSrv.Close()
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{MaxSize: 1000}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/big-image.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, resp.Body.Close())
require.NoError(t, err)
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
assert.Contains(t, string(b), "failed to fetch")
}
// TestImage_ContentTypeHandling covers both the rock-solid acceptance of legitimate
// images and the rejection of content-type-spoofing payloads (the XSS vector where
// upstream lies about Content-Type and the proxy serves attacker HTML back from the
// remark42 origin). Every response — accept or reject — must carry the layered
// defense headers (strict CSP, nosniff, Content-Disposition: inline).
//
// The defense must not depend on the upstream Content-Type header: each row controls
// it independently of the body so the matrix exercises attackers who flip the upstream
// header on the fly, and polyglot bodies where image magic bytes prefix HTML payloads.
func TestImage_ContentTypeHandling(t *testing.T) {
htmlBody := []byte("<html><body><script>alert(document.domain)</script></body></html>")
// polyglot: real PNG magic + trailing HTML. Sniffs as image/png, must be served
// as image/png so the browser renders as image (broken or otherwise) — never as HTML.
polyglot := append(append([]byte{}, gopherPNGBytes()...), []byte("<script>alert(1)</script>")...)
tbl := []struct {
name string
upstreamCT string // Content-Type header the upstream sends
body []byte
accept bool // true: legitimate image, served back; false: attack, rejected
wantCT string // exact Content-Type if accept
payloadMarker string // attack substring that must NOT appear in the response body
}{
// legitimate
{name: "real png", upstreamCT: "image/png", body: gopherPNGBytes(), accept: true, wantCT: "image/png"},
// upstream lies — body is HTML, header varies. All must be rejected at body-sniff.
{name: "html body claimed as image/png", upstreamCT: "image/png", body: htmlBody, payloadMarker: "<script>"},
{name: "html body claimed as image/jpeg", upstreamCT: "image/jpeg", body: htmlBody, payloadMarker: "<script>"},
{name: "html body claimed as image/gif", upstreamCT: "image/gif", body: htmlBody, payloadMarker: "<script>"},
// upstream claims svg+xml; body still sniffs as text/html (the stdlib sniffer
// never returns image/svg+xml, see rest.SafeImgContentType godoc).
{name: "html body upstream claims image/svg+xml", upstreamCT: "image/svg+xml", body: htmlBody, payloadMarker: "<script>"},
{name: "html body claimed as image/webp", upstreamCT: "image/webp", body: htmlBody, payloadMarker: "<script>"},
// svg payloads — even if upstream claims a valid image format, the sniffer sees XML/text and we must reject
{
name: "svg with xml declaration and onload",
upstreamCT: "image/png",
body: []byte(`<?xml version="1.0"?><svg xmlns="http://www.w3.org/2000/svg" onload="alert(1)"></svg>`),
payloadMarker: "onload",
},
{
name: "html fragment without doctype",
upstreamCT: "image/png",
body: []byte(`<body><img src=x onerror=alert(1)></body>`),
payloadMarker: "onerror",
},
// polyglot — image magic + appended HTML. Sniffs as image/png so we accept and serve as image/png.
// Safety comes from the response headers (Content-Type: image/png + X-Content-Type-Options: nosniff),
// not from body filtering: the bytes round-trip verbatim by design (assertion below). The browser
// cannot execute the trailing HTML when the response type is image/png with nosniff.
{name: "polyglot png+html served as png", upstreamCT: "image/png", body: polyglot, accept: true, wantCT: "image/png"},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", tt.upstreamCT)
_, _ = w.Write(tt.body)
}))
defer upstream.Close()
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
encodedURL := base64.URLEncoding.EncodeToString([]byte(upstream.URL + "/logo.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedURL)
require.NoError(t, err)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
// every response — accept or reject — must carry the defense headers
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Contains(t, resp.Header.Get("Content-Disposition"), "inline")
csp := resp.Header.Get("Content-Security-Policy")
assert.Contains(t, csp, "default-src 'none'", "strict CSP missing")
assert.Contains(t, csp, "sandbox", "CSP sandbox missing")
if tt.accept {
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, tt.wantCT, resp.Header.Get("Content-Type"))
assert.Equal(t, tt.body, body, "body bytes must round-trip")
assert.Contains(t, resp.Header.Get("Cache-Control"), "max-age=2592000",
"validated success path carries the 30-day TTL")
assert.True(t, strings.HasPrefix(resp.Header.Get("Etag"), `"v2:`),
"validated success path carries the versioned etag")
return
}
// reject path
assert.GreaterOrEqual(t, resp.StatusCode, 400, "must reject non-image content")
// reject responses must NOT inherit the success path's long-lived cache
// headers — a transient 4xx would otherwise be pinned in browser/intermediary
// caches alongside the versioned etag for 30 days.
assert.Contains(t, resp.Header.Get("Cache-Control"), "no-store",
"reject path must set Cache-Control: no-store; got %q", resp.Header.Get("Cache-Control"))
assert.NotContains(t, resp.Header.Get("Cache-Control"), "max-age=2592000",
"reject path must not carry the success-path 30-day TTL")
assert.Empty(t, resp.Header.Get("Etag"),
"reject path must not carry the versioned etag (would pin the failure in cache)")
ct := resp.Header.Get("Content-Type")
assert.False(t, strings.HasPrefix(ct, "text/html"),
"reject response must not be text/html; got %q", ct)
assert.NotContains(t, string(body), tt.payloadMarker,
"reject response must not echo attack payload; got body=%q", string(body))
})
}
}
// TestEtagMatches lives in the rest package alongside the shared EtagMatches helper
// (see backend/app/rest/image_headers_test.go). The proxy handler delegates to it.
// TestImage_ContentTypeHandling_CacheHit exercises the cache-hit branch of the handler:
// the StoreMock returns attacker bytes directly, so the upstream is never contacted.
// Without the body-sniff at serve time, pre-fix code would have echoed cached HTML as
// text/html. After the fix the same content-type defense applies on the cache path.
func TestImage_ContentTypeHandling_CacheHit(t *testing.T) {
htmlBody := []byte("<html><body><script>alert(document.domain)</script></body></html>")
polyglot := append(append([]byte{}, gopherPNGBytes()...), []byte("<script>alert(1)</script>")...)
tbl := []struct {
name string
cached []byte
accept bool
wantCT string
payloadMarker string
}{
{name: "html in cache claimed as image/png", cached: htmlBody, payloadMarker: "<script>"},
{name: "polyglot in cache served as png", cached: polyglot, accept: true, wantCT: "image/png"},
}
for _, tt := range tbl {
t.Run(tt.name, func(t *testing.T) {
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return tt.cached, nil
}}
img := Image{
CacheExternal: true,
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
// no Transport — cache hit must not reach upstream
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
encodedURL := base64.URLEncoding.EncodeToString([]byte("https://attacker.example.com/logo.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedURL)
require.NoError(t, err)
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, 1, len(imageStore.LoadCalls()), "served from cache")
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Contains(t, resp.Header.Get("Content-Disposition"), "inline")
assert.Contains(t, resp.Header.Get("Content-Security-Policy"),
"default-src 'none'; sandbox; frame-ancestors 'none'")
if tt.accept {
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, tt.wantCT, resp.Header.Get("Content-Type"))
return
}
assert.GreaterOrEqual(t, resp.StatusCode, 400, "must reject non-image cached content")
assert.False(t, strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html"),
"reject response must not be text/html; got %q", resp.Header.Get("Content-Type"))
assert.NotContains(t, string(body), tt.payloadMarker,
"reject response must not echo cached attack payload")
})
}
}
// TestImage_EtagVersioned proves browser/proxy caches with pre-fix etags (the
// unversioned base64 of src that used to be served alongside text/html bodies)
// no longer satisfy revalidation: the server returns a fresh 200 with image
// content instead of 304-ing the poisoned cached entry. The 30-day Cache-Control
// max-age is unchanged — local browser caches still serving pre-fix bytes within
// their TTL are not reached; the prefix only helps clients that revalidate during
// the cached lifetime (Ctrl+R, intermediaries, post-expiry). See etagVersionPrefix
// godoc for the tradeoff.
func TestImage_EtagVersioned(t *testing.T) {
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
httpSrv := imgHTTPTestsServer(t)
defer httpSrv.Close()
srcRaw := httpSrv.URL + "/image/img1.png"
encodedSrc := base64.URLEncoding.EncodeToString([]byte(srcRaw))
preFixEtag := `"` + encodedSrc + `"` // what a pre-fix browser would have cached
req, err := http.NewRequest("GET", ts.URL+"/?src="+encodedSrc, http.NoBody)
require.NoError(t, err)
req.Header.Set("If-None-Match", preFixEtag)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode,
"pre-fix etag must NOT validate as 304 — old cached text/html must be replaced")
assert.Equal(t, "image/png", resp.Header.Get("Content-Type"))
assert.NotEqual(t, preFixEtag, resp.Header.Get("Etag"), "new etag must differ from pre-fix")
assert.True(t, strings.HasPrefix(resp.Header.Get("Etag"), `"v2:`), "new etag must carry the version prefix")
cc := resp.Header.Get("Cache-Control")
assert.Contains(t, cc, "max-age=2592000", "success path keeps 30-day TTL for cache efficiency")
// sanity: the NEW etag round-trips as 304 when sent back
loadsBefore := len(imageStore.LoadCalls())
req2, err := http.NewRequest("GET", ts.URL+"/?src="+encodedSrc, http.NoBody)
require.NoError(t, err)
req2.Header.Set("If-None-Match", resp.Header.Get("Etag"))
resp2, err := http.DefaultClient.Do(req2)
require.NoError(t, err)
defer resp2.Body.Close()
assert.Equal(t, http.StatusNotModified, resp2.StatusCode, "new etag must validate against itself")
body, _ := io.ReadAll(resp2.Body)
assert.Empty(t, body, "304 must have no body")
// 304 path must skip the store lookup entirely — revalidation must not amplify load
assert.Equal(t, loadsBefore, len(imageStore.LoadCalls()),
"revalidation 304 must not trigger any store Load (avoids upstream DoS amplification)")
// 304 path must still carry the layered defense headers
assert.Equal(t, "nosniff", resp2.Header.Get("X-Content-Type-Options"))
assert.Contains(t, resp2.Header.Get("Content-Disposition"), "inline")
assert.Contains(t, resp2.Header.Get("Content-Security-Policy"), "default-src 'none'")
assert.Contains(t, resp2.Header.Get("Content-Security-Policy"), "sandbox")
}
// TestImage_RevalidationSkipsIO proves that a matching current-version If-None-Match
// short-circuits before any cache lookup or upstream fetch. With no Transport and no
// upstream server reachable, the only way this test can pass with 304 is if Load is
// never called and downloadImage is never attempted. This closes the DoS amplification
// where every reuse on a hot comment page would otherwise re-hit the upstream when
// CacheExternal is false.
func TestImage_RevalidationSkipsIO(t *testing.T) {
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
t.Fatal("Load must not be called on the revalidation short-circuit path")
return nil, nil
}}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
// no Transport — any downloadImage attempt would also fail
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
encodedSrc := base64.URLEncoding.EncodeToString([]byte("https://example.com/whatever.png"))
currentEtag := `"v2:` + encodedSrc + `"`
req, err := http.NewRequest("GET", ts.URL+"/?src="+encodedSrc, http.NoBody)
require.NoError(t, err)
req.Header.Set("If-None-Match", currentEtag)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusNotModified, resp.StatusCode,
"matching current-version etag must short-circuit to 304 without I/O")
assert.Equal(t, 0, len(imageStore.LoadCalls()),
"revalidation must not trigger store Load")
body, _ := io.ReadAll(resp.Body)
assert.Empty(t, body, "304 must have no body")
// defense headers must still be set on the short-circuit path
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Contains(t, resp.Header.Get("Content-Disposition"), "inline")
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "default-src 'none'")
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "sandbox")
assert.Equal(t, currentEtag, resp.Header.Get("Etag"))
assert.Contains(t, resp.Header.Get("Cache-Control"), "max-age=2592000")
}
// TestImage_PerRequestRevalidation proves the defense holds when upstream flips its
// response body between requests (give a real PNG once, HTML next time, etc.). Each
// proxy response is independently validated against the body actually returned, so
// trust never accumulates and an earlier "good" response cannot grant the next one a
// free pass.
func TestImage_PerRequestRevalidation(t *testing.T) {
htmlBody := []byte("<html><script>alert(1)</script></html>")
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/png") // always lie consistently
switch r.URL.Path {
case "/png":
_, _ = w.Write(gopherPNGBytes())
case "/html":
_, _ = w.Write(htmlBody)
}
}))
defer upstream.Close()
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
ImageService: image.NewService(&imageStore, image.ServiceParams{}),
Transport: http.DefaultTransport,
}
ts := httptest.NewServer(http.HandlerFunc(img.Handler))
defer ts.Close()
// alternate calls: PNG, HTML, PNG, HTML — each must be judged on its own bytes.
type step struct {
path string
wantStatus int
wantCT string // prefix match
}
steps := []step{
{path: "/png", wantStatus: http.StatusOK, wantCT: "image/png"},
{path: "/html", wantStatus: http.StatusUnsupportedMediaType, wantCT: "application/json"},
{path: "/png", wantStatus: http.StatusOK, wantCT: "image/png"},
{path: "/html", wantStatus: http.StatusUnsupportedMediaType, wantCT: "application/json"},
}
for i, s := range steps {
t.Run(fmt.Sprintf("step_%d_%s", i, s.path), func(t *testing.T) {
encodedURL := base64.URLEncoding.EncodeToString([]byte(upstream.URL + s.path))
resp, err := http.Get(ts.URL + "/?src=" + encodedURL)
require.NoError(t, err)
body, _ := io.ReadAll(resp.Body)
require.NoError(t, resp.Body.Close())
assert.Equal(t, s.wantStatus, resp.StatusCode)
assert.True(t, strings.HasPrefix(resp.Header.Get("Content-Type"), s.wantCT),
"expected Content-Type prefix %q, got %q", s.wantCT, resp.Header.Get("Content-Type"))
assert.False(t, strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html"),
"must never serve text/html under any flip")
assert.NotContains(t, string(body), "<script>",
"attacker payload must never appear in response body")
// every response must still carry the defense headers
assert.Equal(t, "nosniff", resp.Header.Get("X-Content-Type-Options"))
assert.Contains(t, resp.Header.Get("Content-Disposition"), "inline")
assert.Contains(t, resp.Header.Get("Content-Security-Policy"),
"default-src 'none'; sandbox; frame-ancestors 'none'")
})
}
}
func imgHTTPTestsServer(t *testing.T) *httptest.Server {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/image/img1.png" {
+1 -1
View File
@@ -56,7 +56,7 @@ func SetUserInfo(r *http.Request, user store.User) *http.Request {
Picture: user.Picture,
IP: user.IP,
Audience: user.SiteID,
Attributes: map[string]interface{}{
Attributes: map[string]any{
"blocked": user.Blocked,
"verified": user.Verified,
},
+86
View File
@@ -0,0 +1,86 @@
// Package safehttp provides HTTP transports hardened against SSRF: outbound
// connections are dialed using a pre-resolved IP, with a check that all
// resolved IPs sit outside private/reserved ranges. This blocks both naive
// SSRF (private IP literals in user-supplied URLs) and DNS rebinding.
package safehttp
import (
"context"
"fmt"
"net"
"net/http"
"time"
)
// Transport returns an *http.Transport whose DialContext refuses any address
// that resolves to a private/reserved IP, choosing the IP itself for the dial
// to defeat DNS rebinding (an attacker cannot have the resolver hand back a
// public IP at the check and a private one at the connect).
//
// The returned transport is a clone of http.DefaultTransport with only
// DialContext overridden, preserving Proxy, HTTP/2, idle/keep-alive and
// TLS handshake timeouts that bare &http.Transport{} would lose.
func Transport() *http.Transport {
dialer := &net.Dialer{Timeout: 30 * time.Second}
t := http.DefaultTransport.(*http.Transport).Clone()
t.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
host, port, err := net.SplitHostPort(addr)
if err != nil {
return nil, fmt.Errorf("invalid address %s: %w", addr, err)
}
ips, err := net.DefaultResolver.LookupIPAddr(ctx, host)
if err != nil {
return nil, fmt.Errorf("can't resolve host %s: %w", host, err)
}
if len(ips) == 0 {
return nil, fmt.Errorf("no IP addresses resolved for host %s", host)
}
for _, ip := range ips {
if IsPrivateIP(ip.IP) {
return nil, fmt.Errorf("access to private address is not allowed")
}
}
var lastErr error
for _, ip := range ips {
conn, dialErr := dialer.DialContext(ctx, network, net.JoinHostPort(ip.String(), port))
if dialErr == nil {
return conn, nil
}
lastErr = dialErr
}
return nil, fmt.Errorf("can't connect to %s: %w", host, lastErr)
}
return t
}
// privateCIDRs holds pre-parsed private/reserved CIDR blocks.
var privateCIDRs = func() []*net.IPNet {
cidrs := []string{
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
"100.64.0.0/10", "127.0.0.0/8", "169.254.0.0/16",
"::1/128", "fc00::/7", "fe80::/10",
}
blocks := make([]*net.IPNet, 0, len(cidrs))
for _, cidr := range cidrs {
_, block, _ := net.ParseCIDR(cidr)
blocks = append(blocks, block)
}
return blocks
}()
// IsPrivateIP reports whether ip falls in any private, loopback, link-local,
// CGNAT, or reserved range — including IPv4 and IPv6 unspecified addresses.
func IsPrivateIP(ip net.IP) bool {
if ip.IsUnspecified() {
return true
}
for _, block := range privateCIDRs {
if block.Contains(ip) {
return true
}
}
return false
}
+90
View File
@@ -0,0 +1,90 @@
package safehttp
import (
"context"
"net"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIsPrivateIP(t *testing.T) {
tbl := []struct {
ip string
private bool
}{
{"127.0.0.1", true},
{"10.0.0.1", true},
{"10.255.255.255", true},
{"172.16.0.1", true},
{"172.31.255.255", true},
{"192.168.0.1", true},
{"192.168.255.255", true},
{"169.254.1.1", true},
{"100.64.0.1", true},
{"100.127.255.255", true},
{"::1", true},
{"fc00::1", true},
{"fe80::1", true},
{"0.0.0.0", true},
{"::", true},
{"8.8.8.8", false},
{"203.0.113.1", false},
{"1.1.1.1", false},
{"2001:db8::1", false},
}
for _, tt := range tbl {
t.Run(tt.ip, func(t *testing.T) {
ip := net.ParseIP(tt.ip)
require.NotNil(t, ip)
assert.Equal(t, tt.private, IsPrivateIP(ip))
})
}
}
func TestTransport_BlocksPrivate(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
client := &http.Client{Transport: Transport(), Timeout: 2 * time.Second}
resp, err := client.Get(srv.URL) // httptest.NewServer binds 127.0.0.1
if resp != nil {
_ = resp.Body.Close()
}
require.Error(t, err, "private address must be refused")
assert.Contains(t, err.Error(), "access to private address is not allowed")
}
func TestTransport_AllowsPublic(t *testing.T) {
tr := Transport()
// the policy check must reject the loopback literal
_, err := tr.DialContext(context.Background(), "tcp", "127.0.0.1:1")
require.Error(t, err)
assert.Contains(t, err.Error(), "access to private address is not allowed")
// public IP literal passes the policy check; bound the dial with a tight context
// so the test does not depend on real-world routing of TEST-NET-3 (203.0.113.0/24).
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err = tr.DialContext(ctx, "tcp", "203.0.113.1:1")
require.Error(t, err)
assert.NotContains(t, err.Error(), "access to private address is not allowed")
}
func TestTransport_PreservesDefaultTransportSettings(t *testing.T) {
def := http.DefaultTransport.(*http.Transport)
tr := Transport()
assert.NotNil(t, tr.Proxy, "Proxy must be inherited from http.DefaultTransport")
assert.Equal(t, def.ForceAttemptHTTP2, tr.ForceAttemptHTTP2, "ForceAttemptHTTP2")
assert.Equal(t, def.MaxIdleConns, tr.MaxIdleConns, "MaxIdleConns")
assert.Equal(t, def.IdleConnTimeout, tr.IdleConnTimeout, "IdleConnTimeout")
assert.Equal(t, def.TLSHandshakeTimeout, tr.TLSHandshakeTimeout, "TLSHandshakeTimeout")
assert.Equal(t, def.ExpectContinueTimeout, tr.ExpectContinueTimeout, "ExpectContinueTimeout")
}
+3 -3
View File
@@ -50,8 +50,8 @@ type PostInfo struct {
CountLeft int `json:"count_left"` // used only with returning search results limited by number, otherwise zero
LastComment string `json:"last_comment,omitempty"` // used only with returning search results limited by number
ReadOnly bool `json:"read_only,omitempty" bson:"read_only,omitempty"` // can be attached to site-wide comments but won't be set then
FirstTS time.Time `json:"first_time,omitempty" bson:"first_time,omitempty"`
LastTS time.Time `json:"last_time,omitempty" bson:"last_time,omitempty"`
FirstTS time.Time `json:"first_time" bson:"first_time,omitempty"`
LastTS time.Time `json:"last_time" bson:"last_time,omitempty"`
}
// BlockedUser holds id and ts for blocked user
@@ -157,7 +157,7 @@ func (c *Comment) Snippet(limit int) string {
break
}
}
// Don't add a space if comment is just a one single word which has been truncated.
// don't add a space if comment is just a one single word which has been truncated.
if len(snippet) == limit {
return string(snippet) + "..."
}
+3 -3
View File
@@ -120,7 +120,7 @@ func TestComment_PrepareUntrusted(t *testing.T) {
Score: 10,
Pin: true,
Deleted: true,
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.Local),
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.UTC),
Votes: map[string]bool{"uu": true},
Controversy: 123,
Imported: true,
@@ -150,7 +150,7 @@ func TestComment_SetDeleted(t *testing.T) {
Locator: Locator{SiteID: "site", URL: "url"},
Score: 10,
Deleted: false,
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.Local),
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.UTC),
Votes: map[string]bool{"uu": true},
Pin: true,
}
@@ -177,7 +177,7 @@ func TestComment_SetDeletedHard(t *testing.T) {
Locator: Locator{SiteID: "site", URL: "url"},
Score: 10,
Deleted: false,
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.Local),
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.UTC),
Votes: map[string]bool{"uu": true},
Pin: true,
}
+13 -16
View File
@@ -3,6 +3,7 @@ package engine
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
@@ -10,6 +11,7 @@ import (
log "github.com/go-pkgz/lgr"
"github.com/hashicorp/go-multierror"
bolt "go.etcd.io/bbolt"
berrors "go.etcd.io/bbolt/errors"
"github.com/umputun/remark42/backend/app/store"
)
@@ -346,13 +348,13 @@ func (b *BoltDB) Info(req InfoRequest) ([]store.PostInfo, error) {
// ListFlags get list of flagged keys, like blocked & verified user
// works for full locator (post flags) or with userID
func (b *BoltDB) ListFlags(req FlagRequest) (res []interface{}, err error) {
func (b *BoltDB) ListFlags(req FlagRequest) (res []any, err error) {
bdb, e := b.db(req.Locator.SiteID)
if e != nil {
return nil, e
}
res = []interface{}{}
res = []any{}
switch req.Flag {
case Verified:
err = bdb.View(func(tx *bolt.Tx) error {
@@ -893,31 +895,26 @@ func (b *BoltDB) deleteUser(bdb *bolt.DB, siteID, userID string, mode store.Dele
// delete collected comments
for _, ci := range comments {
if e := b.deleteComment(bdb, ci.locator, ci.commentID, mode); e != nil {
return fmt.Errorf("failed to delete comment %+v: %w", ci, err)
return fmt.Errorf("failed to delete comment %+v: %w", ci, e)
}
}
// delete user bucket in hard mode
// delete the user's bucket in hard mode. A user who only logged in but never commented has
// no per-user bucket, so tolerate ErrBucketNotFound; the top-level users bucket is created
// by NewBoltDB and is always present.
if mode == store.HardDelete {
err = bdb.Update(func(tx *bolt.Tx) error {
usersBkt := tx.Bucket([]byte(userBucketName))
if usersBkt != nil {
if e := usersBkt.DeleteBucket([]byte(userID)); e != nil {
return fmt.Errorf("failed to delete user bucket for %s: %w", userID, err)
}
if e := usersBkt.DeleteBucket([]byte(userID)); e != nil && !errors.Is(e, berrors.ErrBucketNotFound) {
return fmt.Errorf("failed to delete user bucket for %s: %w", userID, e)
}
return nil
})
if err != nil {
return fmt.Errorf("can't delete user meta: %w", err)
}
}
if len(comments) == 0 {
return fmt.Errorf("unknown user %s", userID)
}
return b.deleteUserDetail(bdb, userID, AllUserDetails)
}
@@ -957,7 +954,7 @@ func (b *BoltDB) getUserBucket(tx *bolt.Tx, userID string) (*bolt.Bucket, error)
}
// save marshaled value to key for bucket. Should run in update tx
func (b *BoltDB) save(bkt *bolt.Bucket, key string, value interface{}) (err error) {
func (b *BoltDB) save(bkt *bolt.Bucket, key string, value any) (err error) {
if value == nil {
return fmt.Errorf("can't save nil value for %s", key)
}
@@ -972,7 +969,7 @@ func (b *BoltDB) save(bkt *bolt.Bucket, key string, value interface{}) (err erro
}
// load and unmarshal json value by key from bucket. Should run in view tx
func (b *BoltDB) load(bkt *bolt.Bucket, key string, res interface{}) error {
func (b *BoltDB) load(bkt *bolt.Bucket, key string, res any) error {
value := bkt.Get([]byte(key))
if value == nil {
return fmt.Errorf("no value for %s", key)
@@ -1027,7 +1024,7 @@ func (b *BoltDB) db(siteID string) (*bolt.DB, error) {
// makeRef creates reference combining url and comment id
func (b *BoltDB) makeRef(comment store.Comment) []byte {
return []byte(fmt.Sprintf("%s!!%s", comment.Locator.URL, comment.ID))
return fmt.Appendf(nil, "%s!!%s", comment.Locator.URL, comment.ID)
}
// parseRef gets parts of reference
+112 -52
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"os"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -48,7 +49,7 @@ func TestBoltDB_CreateFailedReadOnly(t *testing.T) {
comment := store.Comment{
ID: "id-ro",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/ro", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -147,20 +148,20 @@ func TestBoltDB_FindLastSince(t *testing.T) {
var b, teardown = prep(t)
defer teardown()
ts := time.Date(2017, 12, 20, 15, 18, 21, 0, time.Local)
ts := time.Date(2017, 12, 20, 15, 18, 21, 0, time.UTC)
req := FindRequest{Locator: store.Locator{SiteID: "radio-t"}, Sort: "-time", Since: ts}
res, err := b.Find(req)
assert.NoError(t, err)
require.Equal(t, 2, len(res))
assert.Equal(t, "some text2", res[0].Text)
req.Since = time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local)
req.Since = time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC)
res, err = b.Find(req)
assert.NoError(t, err)
require.Equal(t, 1, len(res))
assert.Equal(t, "some text2", res[0].Text)
req.Since = time.Date(2017, 12, 20, 16, 18, 22, 0, time.Local)
req.Since = time.Date(2017, 12, 20, 16, 18, 22, 0, time.UTC)
res, err = b.Find(req)
assert.NoError(t, err)
assert.Equal(t, 0, len(res))
@@ -170,20 +171,20 @@ func TestBoltDB_FindInPostSince(t *testing.T) {
var b, teardown = prep(t)
defer teardown()
ts := time.Date(2017, 12, 20, 15, 18, 21, 0, time.Local)
ts := time.Date(2017, 12, 20, 15, 18, 21, 0, time.UTC)
req := FindRequest{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, Sort: "-time", Since: ts}
res, err := b.Find(req)
assert.NoError(t, err)
require.Equal(t, 2, len(res))
assert.Equal(t, "some text2", res[0].Text)
req.Since = time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local)
req.Since = time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC)
res, err = b.Find(req)
assert.NoError(t, err)
require.Equal(t, 1, len(res))
assert.Equal(t, "some text2", res[0].Text)
req.Since = time.Date(2017, 12, 20, 16, 18, 22, 0, time.Local)
req.Since = time.Date(2017, 12, 20, 16, 18, 22, 0, time.UTC)
res, err = b.Find(req)
assert.NoError(t, err)
assert.Equal(t, 0, len(res))
@@ -236,10 +237,10 @@ func TestBoltDB_FindForUserPagination(t *testing.T) {
}
// write 200 comments
for i := 0; i < 200; i++ {
for i := range 200 {
c.ID = fmt.Sprintf("id-%d", i)
c.Text = fmt.Sprintf("text #%d", i)
c.Timestamp = time.Date(2017, 12, 20, 15, 18, i, 0, time.Local)
c.Timestamp = time.Date(2017, 12, 20, 15, 18, i, 0, time.UTC)
_, err = b.Create(c)
require.NoError(t, err)
}
@@ -324,13 +325,13 @@ func TestBoltDB_InfoPost(t *testing.T) {
b, teardown := prep(t) // two comments for https://radio-t.com
defer teardown()
ts := func(minute int) time.Time { return time.Date(2017, 12, 20, 15, 18, minute, 0, time.Local) }
ts := func(minute int) time.Time { return time.Date(2017, 12, 20, 15, 18, minute, 0, time.UTC) }
// add one more for https://radio-t.com/2
comment := store.Comment{
ID: "12345",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 24, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 24, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -379,14 +380,14 @@ func TestBoltDB_InfoList(t *testing.T) {
comment := store.Comment{
ID: "12345",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Create(comment)
assert.NoError(t, err)
ts := func(sec int) time.Time { return time.Date(2017, 12, 20, 15, 18, sec, 0, time.Local) }
ts := func(sec int) time.Time { return time.Date(2017, 12, 20, 15, 18, sec, 0, time.UTC) }
req := InfoRequest{Locator: store.Locator{SiteID: "radio-t"}}
res, err := b.Info(req)
@@ -540,7 +541,7 @@ func TestBolt_FlagListVerified(t *testing.T) {
b, teardown := prep(t)
defer teardown()
toIDs := func(inp []interface{}) (res []string) {
toIDs := func(inp []any) (res []string) {
res = make([]string, len(inp))
for i, v := range inp {
vv, ok := v.(string)
@@ -571,47 +572,49 @@ func TestBolt_FlagListVerified(t *testing.T) {
}
func TestBolt_FlagListBlocked(t *testing.T) {
b, teardown := prep(t)
defer teardown()
synctest.Test(t, func(t *testing.T) {
b, teardown := prep(t)
defer teardown()
setBlocked := func(site, user string, status FlagStatus, ttl time.Duration) error {
req := FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: site}, UserID: user, Update: status, TTL: ttl}
_, err := b.Flag(req)
return err
}
toBlocked := func(inp []interface{}) (res []store.BlockedUser) {
res = make([]store.BlockedUser, len(inp))
for i, v := range inp {
vv, ok := v.(store.BlockedUser)
require.True(t, ok)
res[i] = vv
setBlocked := func(site, user string, status FlagStatus, ttl time.Duration) error {
req := FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: site}, UserID: user, Update: status, TTL: ttl}
_, err := b.Flag(req)
return err
}
return res
}
assert.NoError(t, setBlocked("radio-t", "user1", FlagTrue, 0))
assert.NoError(t, setBlocked("radio-t", "user2", FlagTrue, 150*time.Millisecond))
assert.NoError(t, setBlocked("radio-t", "user3", FlagFalse, 0))
vv, err := b.ListFlags(FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
toBlocked := func(inp []any) (res []store.BlockedUser) {
res = make([]store.BlockedUser, len(inp))
for i, v := range inp {
vv, ok := v.(store.BlockedUser)
require.True(t, ok)
res[i] = vv
}
return res
}
assert.NoError(t, setBlocked("radio-t", "user1", FlagTrue, 0))
assert.NoError(t, setBlocked("radio-t", "user2", FlagTrue, 150*time.Millisecond))
assert.NoError(t, setBlocked("radio-t", "user3", FlagFalse, 0))
blockedList := toBlocked(vv)
require.Equal(t, 2, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
assert.Equal(t, "user2", blockedList[1].ID)
t.Logf("%+v", blockedList)
vv, err := b.ListFlags(FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
// check block expiration
time.Sleep(150 * time.Millisecond)
vv, err = b.ListFlags(FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
blockedList = toBlocked(vv)
require.Equal(t, 1, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
blockedList := toBlocked(vv)
require.Equal(t, 2, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
assert.Equal(t, "user2", blockedList[1].ID)
t.Logf("%+v", blockedList)
_, err = b.ListFlags(FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: "bad"}})
assert.EqualError(t, err, `site "bad" not found`)
// check block expiration
time.Sleep(150 * time.Millisecond)
vv, err = b.ListFlags(FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: "radio-t"}})
assert.NoError(t, err)
blockedList = toBlocked(vv)
require.Equal(t, 1, len(blockedList))
assert.Equal(t, "user1", blockedList[0].ID)
_, err = b.ListFlags(FlagRequest{Flag: Blocked, Locator: store.Locator{SiteID: "bad"}})
assert.EqualError(t, err, `site "bad" not found`)
})
}
func TestBoltDB_UserDetail(t *testing.T) {
@@ -840,6 +843,63 @@ func TestBoltAdmin_DeleteUserHard(t *testing.T) {
assert.EqualError(t, err, `site "radio-t-bad" not found`)
}
// TestBoltAdmin_DeleteUserHard_NoComments covers hard-deleting a user who has no comments
// (and therefore no user bucket) — e.g. one who only logged in. This must succeed rather than
// fail on the missing bucket, and must still remove any stored user details.
func TestBoltAdmin_DeleteUserHard_NoComments(t *testing.T) {
b, teardown := prep(t)
defer teardown()
t.Run("login-only user with a detail but no comments", func(t *testing.T) {
const userID = "login-only-user"
loc := store.Locator{SiteID: "radio-t"}
// user logged in and has a stored detail, but never commented (no user bucket)
_, err := b.UserDetail(UserDetailRequest{Locator: loc, UserID: userID, Detail: UserEmail, Update: "user@example.com"})
require.NoError(t, err)
err = b.Delete(DeleteRequest{Locator: loc, UserID: userID, DeleteMode: store.HardDelete})
require.NoError(t, err, "hard delete must not fail on a missing user bucket")
details, err := b.UserDetail(UserDetailRequest{Locator: loc, UserID: userID, Detail: UserEmail})
require.NoError(t, err)
assert.Empty(t, details, "stored user detail must be removed on hard delete")
})
t.Run("unknown user is a no-op", func(t *testing.T) {
err := b.Delete(DeleteRequest{Locator: store.Locator{SiteID: "radio-t"}, UserID: "never-seen-user", DeleteMode: store.HardDelete})
assert.NoError(t, err, "hard-deleting an unknown user must not error")
})
}
// TestBoltAdmin_DeleteUserSoft_NoComments covers soft-deleting a user with no comments. As with the
// hard path (and the existing soft path for users who do have comments) it cleans stored user
// details and is a no-op for an unknown user.
func TestBoltAdmin_DeleteUserSoft_NoComments(t *testing.T) {
b, teardown := prep(t)
defer teardown()
t.Run("login-only user with a detail but no comments", func(t *testing.T) {
const userID = "login-only-soft"
loc := store.Locator{SiteID: "radio-t"}
_, err := b.UserDetail(UserDetailRequest{Locator: loc, UserID: userID, Detail: UserEmail, Update: "user@example.com"})
require.NoError(t, err)
err = b.Delete(DeleteRequest{Locator: loc, UserID: userID, DeleteMode: store.SoftDelete})
require.NoError(t, err)
details, err := b.UserDetail(UserDetailRequest{Locator: loc, UserID: userID, Detail: UserEmail})
require.NoError(t, err)
assert.Empty(t, details, "soft delete cleans stored user details, consistent with the has-comments path")
})
t.Run("unknown user is a no-op", func(t *testing.T) {
err := b.Delete(DeleteRequest{Locator: store.Locator{SiteID: "radio-t"}, UserID: "never-seen-soft", DeleteMode: store.SoftDelete})
assert.NoError(t, err, "soft-deleting an unknown user must not error")
})
}
func TestBoltAdmin_DeleteUserSoft(t *testing.T) {
b, teardown := prep(t)
defer teardown()
@@ -890,7 +950,7 @@ func TestBoltDB_ref(t *testing.T) {
comment := store.Comment{
ID: "12345",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -929,7 +989,7 @@ func prep(t *testing.T) (b *BoltDB, teardown func()) {
comment := store.Comment{
ID: "id-1",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -939,7 +999,7 @@ func prep(t *testing.T) (b *BoltDB, teardown func()) {
comment = store.Comment{
ID: "id-2",
Text: "some text2",
Timestamp: time.Date(2017, 12, 20, 15, 18, 23, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 23, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
+2 -2
View File
@@ -24,7 +24,7 @@ type Interface interface {
Count(req FindRequest) (int, error) // get count for post or user
Delete(req DeleteRequest) error // Delete post(s), user, comment, user details, or everything
Flag(req FlagRequest) (bool, error) // set and get flags
ListFlags(req FlagRequest) ([]interface{}, error) // get list of flagged keys, like blocked & verified user
ListFlags(req FlagRequest) ([]any, error) // get list of flagged keys, like blocked & verified user
// UserDetail sets or gets single detail value, or gets all details for requested site
// Returns list even for single entry request is a compromise in order to have both single detail getting and setting
@@ -45,7 +45,7 @@ type FindRequest struct {
Locator store.Locator `json:"locator"` // lack of URL means site operation
UserID string `json:"user_id,omitempty"` // presence of UserID treated as user-related find
Sort string `json:"sort,omitempty"` // sort order with +/-field syntax
Since time.Time `json:"since,omitempty"` // time limit for found results
Since time.Time `json:"since"` // time limit for found results
Limit int `json:"limit,omitempty"`
Skip int `json:"skip,omitempty"`
}
+2 -1
View File
@@ -4,8 +4,9 @@
package engine
import (
store "github.com/umputun/remark42/backend/app/store"
"sync"
store "github.com/umputun/remark42/backend/app/store"
)
// Ensure, that InterfaceMock does implement Interface.
+4 -4
View File
@@ -11,10 +11,10 @@ import (
func TestEngine_sortComments(t *testing.T) {
cc := []store.Comment{
{ID: "1", Score: 5, Controversy: 1, Timestamp: time.Date(2018, 2, 5, 10, 1, 0, 0, time.Local)},
{ID: "2", Score: 4, Controversy: 2, Timestamp: time.Date(2018, 2, 5, 10, 2, 0, 0, time.Local)},
{ID: "3", Score: 6, Controversy: 3, Timestamp: time.Date(2018, 2, 5, 10, 3, 0, 0, time.Local)},
{ID: "4", Score: 6, Controversy: 1, Timestamp: time.Date(2018, 2, 5, 10, 4, 0, 0, time.Local)},
{ID: "1", Score: 5, Controversy: 1, Timestamp: time.Date(2018, 2, 5, 10, 1, 0, 0, time.UTC)},
{ID: "2", Score: 4, Controversy: 2, Timestamp: time.Date(2018, 2, 5, 10, 2, 0, 0, time.UTC)},
{ID: "3", Score: 6, Controversy: 3, Timestamp: time.Date(2018, 2, 5, 10, 3, 0, 0, time.UTC)},
{ID: "4", Score: 6, Controversy: 1, Timestamp: time.Date(2018, 2, 5, 10, 4, 0, 0, time.UTC)},
}
SortComments(cc, "+time")
+5 -5
View File
@@ -71,24 +71,24 @@ func (r *RPC) Flag(req FlagRequest) (status bool, err error) {
return status, err
}
func unmarshalString(data []byte) ([]interface{}, error) {
func unmarshalString(data []byte) ([]any, error) {
var strings []string
if err := json.Unmarshal(data, &strings); err != nil {
return nil, err
}
list := make([]interface{}, 0, len(strings))
list := make([]any, 0, len(strings))
for _, w := range strings {
list = append(list, w)
}
return list, nil
}
func unmarshalBlockedUser(data []byte) ([]interface{}, error) {
func unmarshalBlockedUser(data []byte) ([]any, error) {
var blockedUsers []store.BlockedUser
if err := json.Unmarshal(data, &blockedUsers); err != nil {
return nil, err
}
list := make([]interface{}, 0, len(blockedUsers))
list := make([]any, 0, len(blockedUsers))
for _, w := range blockedUsers {
list = append(list, w)
}
@@ -96,7 +96,7 @@ func unmarshalBlockedUser(data []byte) ([]interface{}, error) {
}
// ListFlags get list of flagged keys, like blocked & verified user
func (r *RPC) ListFlags(req FlagRequest) ([]interface{}, error) {
func (r *RPC) ListFlags(req FlagRequest) ([]any, error) {
resp, err := r.Call("store.list_flags", req)
if err != nil {
return nil, err
+1 -1
View File
@@ -79,7 +79,7 @@ func TestFormatter_FormatComment(t *testing.T) {
Score: 10,
Pin: true,
Deleted: true,
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.Local),
Timestamp: time.Date(2018, 1, 1, 9, 30, 0, 0, time.UTC),
Votes: map[string]bool{"uu": true},
}
+44 -40
View File
@@ -5,6 +5,7 @@ import (
"os"
"path"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -89,50 +90,53 @@ func TestBoltStore_LoadAfterDelete(t *testing.T) {
}
func TestBoltStore_Cleanup(t *testing.T) {
svc, teardown := prepareBoltImageStorageTest(t)
defer teardown()
synctest.Test(t, func(t *testing.T) {
svc, teardown := prepareBoltImageStorageTest(t)
defer teardown()
save := func(file string) (id string) {
err := svc.Save(file, gopherPNGBytes())
save := func(file string) (id string) {
err := svc.Save(file, gopherPNGBytes())
require.NoError(t, err)
checkBoltImgData(t, svc.db, imagesStagedBktName, file, func(data []byte) error {
require.NotNil(t, data)
assert.Equal(t, 1462, len(data))
return nil
})
return file
}
// save 3 images to staging
img1 := save("blah_ff1.png")
img1ts := time.Now()
time.Sleep(100 * time.Millisecond)
img2 := save("blah_ff2.png")
time.Sleep(100 * time.Millisecond)
img3 := save("blah_ff3.png")
// Cleanup check is `age > ttl` (strict), so pick a ttl strictly less than img1's age
err := svc.Cleanup(context.Background(), time.Since(img1ts)-time.Millisecond)
assert.NoError(t, err)
assertBoltImgNil(t, svc.db, imagesStagedBktName, img1)
assertBoltImgNil(t, svc.db, imagesBktName, img1)
assertBoltImgNotNil(t, svc.db, imagesStagedBktName, img2)
assertBoltImgNotNil(t, svc.db, imagesStagedBktName, img3)
err = svc.Commit(img3)
require.NoError(t, err)
checkBoltImgData(t, svc.db, imagesStagedBktName, file, func(data []byte) error {
require.NotNil(t, data)
assert.Equal(t, 1462, len(data))
return nil
})
return file
}
// reset the time to cleanup
err = svc.ResetCleanupTimer(img2)
require.NoError(t, err)
err = svc.Cleanup(context.Background(), time.Millisecond*100)
assert.NoError(t, err)
// save 3 images to staging
img1 := save("blah_ff1.png")
img1ts := time.Now()
time.Sleep(100 * time.Millisecond)
img2 := save("blah_ff2.png")
time.Sleep(100 * time.Millisecond)
img3 := save("blah_ff3.png")
err := svc.Cleanup(context.Background(), time.Since(img1ts)) // clean first images
assert.NoError(t, err)
assertBoltImgNil(t, svc.db, imagesStagedBktName, img1)
assertBoltImgNil(t, svc.db, imagesBktName, img1)
assertBoltImgNotNil(t, svc.db, imagesStagedBktName, img2)
assertBoltImgNotNil(t, svc.db, imagesStagedBktName, img3)
err = svc.Commit(img3)
require.NoError(t, err)
// reset the time to cleanup
err = svc.ResetCleanupTimer(img2)
require.NoError(t, err)
err = svc.Cleanup(context.Background(), time.Millisecond*100)
assert.NoError(t, err)
assertBoltImgNotNil(t, svc.db, imagesStagedBktName, img2)
assertBoltImgNil(t, svc.db, imagesBktName, img2)
assertBoltImgNotNil(t, svc.db, imagesBktName, img3)
assert.NoError(t, err)
assertBoltImgNotNil(t, svc.db, imagesStagedBktName, img2)
assertBoltImgNil(t, svc.db, imagesBktName, img2)
assertBoltImgNotNil(t, svc.db, imagesBktName, img3)
assert.NoError(t, err)
})
}
func TestBolt_Info(t *testing.T) {
+4 -4
View File
@@ -37,11 +37,11 @@ type FileSystem struct {
func (f *FileSystem) Save(id string, img []byte) error {
dst := f.location(f.Staging, id)
if err := os.MkdirAll(path.Dir(dst), 0o700); err != nil {
if err := os.MkdirAll(path.Dir(dst), 0o700); err != nil { //nolint:gosec // id is server-generated hash via image.Service (Save / SaveWithID); dst computed via f.location
return fmt.Errorf("can't make image directory: %w", err)
}
if err := os.WriteFile(dst, img, 0o600); err != nil {
if err := os.WriteFile(dst, img, 0o600); err != nil { //nolint:gosec // same as MkdirAll above
return fmt.Errorf("can't write image file with id %s: %w", id, err)
}
@@ -147,8 +147,8 @@ func (f *FileSystem) Cleanup(_ context.Context, ttl time.Duration) error {
age := time.Since(info.ModTime())
if age > (ttl + 100*time.Millisecond) { // delay cleanup triggering to allow commit
log.Printf("[INFO] remove staging image %s, age %v", fpath, age)
rmErr := os.Remove(fpath)
_ = os.Remove(path.Dir(fpath)) // try to remove directory
rmErr := os.Remove(fpath) //nolint:gosec // staging dir is server-only, no untrusted symlinks land here
_ = os.Remove(path.Dir(fpath)) //nolint:gosec // same staging dir
return rmErr
}
return nil
+1 -1
View File
@@ -190,7 +190,7 @@ func TestFsStore_location(t *testing.T) {
}
svc := FileSystem{Location: "/tmp", Partitions: 10}
for i := 0; i < 1000; i++ {
for range 1000 {
v := randomID(rand.Intn(64))
location := svc.location("/tmp", v)
elems := strings.Split(location, "/")
+69 -35
View File
@@ -13,9 +13,8 @@ import (
"encoding/base64"
"fmt"
"image"
// support gif and jpeg images decoding
_ "image/gif"
_ "image/jpeg"
_ "image/gif" // register gif decoder
_ "image/jpeg" // register jpeg decoder
"image/png"
"io"
"net/http"
@@ -31,6 +30,7 @@ import (
"github.com/hashicorp/go-multierror"
"github.com/rs/xid"
"golang.org/x/image/draw"
_ "golang.org/x/image/webp" // register webp decoder so DecodeConfig accepts what readAndValidateImage allows
)
// Service wraps Store with common functions needed for any store implementation
@@ -110,9 +110,7 @@ func (s *Service) Submit(idsFn func() []string) {
s.once.Do(func() {
log.Printf("[DEBUG] image submitter activated")
s.submitCh = make(chan submitReq, submitQueueSize)
s.wg.Add(1)
go func() {
defer s.wg.Done()
s.wg.Go(func() {
for req := range s.submitCh {
// wait for EditDuration expiration with emergency pass on term
for atomic.LoadInt32(&s.term) == 0 && time.Since(req.TS) <= s.EditDuration {
@@ -126,7 +124,7 @@ func (s *Service) Submit(idsFn func() []string) {
atomic.AddInt32(&s.submitCount, -1)
}
log.Printf("[INFO] image submitter terminated")
}()
})
})
atomic.AddInt32(&s.submitCount, 1)
@@ -154,6 +152,12 @@ func (s *Service) ExtractNonProxiedPictures(commentHTML string) (ids []string) {
// Cleanup runs periodic cleanup with 1.5*ServiceParams.EditDuration. Blocking loop, should be called inside of goroutine by consumer
func (s *Service) Cleanup(ctx context.Context) {
if s.EditDuration <= 0 {
log.Printf("[INFO] pictures cleanup disabled, edit duration is %v", s.EditDuration)
<-ctx.Done()
return
}
cleanupTTL := s.EditDuration * 15 / 10 // cleanup images older than 1.5 * EditDuration
log.Printf("[INFO] start pictures cleanup, staging ttl=%v", cleanupTTL)
@@ -233,16 +237,6 @@ func (s *Service) SaveWithID(id string, r io.Reader) error {
return s.store.Save(id, img)
}
// ImgContentType returns content type for provided image
func (s *Service) ImgContentType(img []byte) string {
contentType := http.DetectContentType(img)
if contentType == "application/octet-stream" {
// replace generic fallback with one which make sense in our scenario
return "image/*"
}
return contentType
}
// returns list of image IDs from the comment html, including proxied images if includeProxied is true
func (s *Service) extractImageIDs(commentHTML string, includeProxied bool) (ids []string) {
doc, err := goquery.NewDocumentFromReader(strings.NewReader(commentHTML))
@@ -280,6 +274,13 @@ func (s *Service) extractImageIDs(commentHTML string, includeProxied bool) (ids
return ids
}
// maxImagePixels caps the declared pixel count of an image before any raster decode
// is allowed. Without this, a tiny compressed "decompression bomb" image declaring
// e.g. 65535x65535 px would force image.Decode to allocate gigabytes of pixel memory
// and OOM the service on a single comment upload. 16 MP covers any realistic image
// (~4096x4096) while keeping peak allocation bounded.
const maxImagePixels = 16 * 1024 * 1024
// prepareImage calls readAndValidateImage and resize on provided image.
func (s *Service) prepareImage(r io.Reader) ([]byte, error) {
data, err := readAndValidateImage(r, s.MaxSize)
@@ -287,32 +288,59 @@ func (s *Service) prepareImage(r io.Reader) ([]byte, error) {
return nil, fmt.Errorf("can't load image: %w", err)
}
data = resize(data, s.MaxWidth, s.MaxHeight)
return data, nil
resized := resize(data, s.MaxWidth, s.MaxHeight)
if resized == nil {
return nil, fmt.Errorf("image rejected: malformed or exceeds %d-pixel safe limit", maxImagePixels)
}
return resized, nil
}
// resize an image of supported format (PNG, JPG, GIF) to the size of "limit" px of
// the biggest side (width or height) preserving aspect ratio.
// Returns original data if resizing is not needed or failed.
// If resized the result will be for png format
// resize validates an image and, if needed, re-encodes it to fit within the given
// pixel limits preserving aspect ratio. Returns nil for malformed input or for
// declared dimensions exceeding maxImagePixels so attacker payloads (decompression
// bombs) never reach the store. With limit <= 0 or when the image already fits, the
// original bytes are returned verbatim so animated GIFs and other multi-frame formats
// round-trip without being flattened to one frame.
//
// Validation uses image.DecodeConfig (cheap — declares dimensions, allocates nothing)
// before any full image.Decode, so a 100 KB compressed image declaring 65535x65535 px
// is rejected without ever materializing the raster.
func resize(data []byte, limitW, limitH int) []byte {
if data == nil || limitW <= 0 || limitH <= 0 {
return data
if len(data) == 0 {
return nil
}
src, _, err := image.Decode(bytes.NewBuffer(data))
// validate format and dimensions without allocating pixel memory.
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
if err != nil {
log.Printf("[WARN] can't decode image, %s", err)
return data
log.Printf("[WARN] can't decode image config, %s", err)
return nil
}
// multiply in int64 — on 32-bit builds (GOARCH=386, 32-bit arm) the int
// product of two 16-bit-or-larger dimensions can overflow and wrap below
// maxImagePixels, bypassing the cap.
if cfg.Width <= 0 || cfg.Height <= 0 || int64(cfg.Width)*int64(cfg.Height) > int64(maxImagePixels) {
log.Printf("[WARN] image dimensions %dx%d exceed safe limit", cfg.Width, cfg.Height)
return nil
}
// dimensions are bounded — full decode is now safe to allocate. Decode also
// validates the raster body: a header that DecodeConfig accepts but with a
// corrupt or truncated payload would slip through if we returned early on the
// no-resize path without ever touching the pixels. Decode unconditionally,
// then either return the original bytes (no resize needed, multi-frame intact)
// or the re-encoded result.
src, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
log.Printf("[WARN] can't decode image after dim-check, %s", err)
return nil
}
bounds := src.Bounds()
w, h := bounds.Dx(), bounds.Dy()
if w <= limitW && h <= limitH || w <= 0 || h <= 0 {
log.Printf("[DEBUG] resizing image is smaller that the limit or has 0 size")
if limitW <= 0 || limitH <= 0 || (cfg.Width <= limitW && cfg.Height <= limitH) {
return data
}
w, h := src.Bounds().Dx(), src.Bounds().Dy()
newW, newH := getProportionalSizes(w, h, limitW, limitH)
m := image.NewRGBA(image.Rect(0, 0, newW, newH))
draw.CatmullRom.Scale(m, m.Bounds(), src, src.Bounds(), draw.Src, nil)
@@ -320,7 +348,7 @@ func resize(data []byte, limitW, limitH int) []byte {
var out bytes.Buffer
if err = png.Encode(&out, m); err != nil {
log.Printf("[WARN] can't encode resized image to png, %s", err)
return data
return data // fall back to the validated original
}
return out.Bytes()
}
@@ -361,8 +389,14 @@ func readAndValidateImage(r io.Reader, maxSize int) ([]byte, error) {
return nil, fmt.Errorf("file is too large (limit=%d)", maxSize)
}
// read header first, needs it to check if data is valid png/gif/jpeg
if !isValidImage(data[:512]) {
// read header first to check the format. http.DetectContentType inspects up
// to the first 512 bytes, but a smaller body is fine — pass the whole slice
// rather than panicking on a fixed-size sub-slice.
header := data
if len(header) > 512 {
header = header[:512]
}
if !isValidImage(header) {
return nil, fmt.Errorf("file format not allowed")
}
+135 -57
View File
@@ -11,6 +11,7 @@ import (
"strconv"
"strings"
"testing"
"testing/synctest"
"time"
"github.com/stretchr/testify/assert"
@@ -56,10 +57,10 @@ func TestService_ResizeJpeg(t *testing.T) {
img, err := readAndValidateImage(fh, 32000)
assert.NoError(t, err)
assert.Equal(t, 16756, len(img))
assert.InDelta(t, 16756, len(img), 100)
img = resize(img, 400, 300)
assert.Equal(t, 10918, len(img))
assert.InDelta(t, 10913, len(img), 100)
}
func TestService_SaveTooLarge(t *testing.T) {
@@ -125,37 +126,41 @@ func TestService_ExtractPictures(t *testing.T) {
}
func TestService_Cleanup(t *testing.T) {
store := StoreMock{
CleanupFunc: func(context.Context, time.Duration) error {
return nil
},
}
synctest.Test(t, func(t *testing.T) {
store := StoreMock{
CleanupFunc: func(context.Context, time.Duration) error {
return nil
},
}
svc := NewService(&store, ServiceParams{EditDuration: 20 * time.Millisecond})
// cancel context after 2.1 cleanup TTLs
ctx, cancel := context.WithTimeout(context.Background(), svc.EditDuration/100*15*21)
defer cancel()
svc.Cleanup(ctx)
assert.Equal(t, 2, len(store.CleanupCalls()))
svc := NewService(&store, ServiceParams{EditDuration: 20 * time.Millisecond})
// cancel context after 2.1 cleanup TTLs
ctx, cancel := context.WithTimeout(context.Background(), svc.EditDuration/100*15*21)
defer cancel()
svc.Cleanup(ctx)
assert.Equal(t, 2, len(store.CleanupCalls()))
})
}
func TestService_Submit(t *testing.T) {
store := StoreMock{
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
svc := NewService(&store, ServiceParams{ImageAPI: "/blah/", EditDuration: time.Millisecond * 100})
svc.Submit(func() []string { return []string{"id1", "id2", "id3"} })
assert.Equal(t, 3, len(store.ResetCleanupTimerCalls()))
err := svc.Commit(func() []string { return []string{"id4", "id5"} })
assert.NoError(t, err)
svc.Submit(func() []string { return []string{"id6", "id7"} })
assert.Equal(t, 5, len(store.ResetCleanupTimerCalls()))
svc.Submit(nil)
assert.Equal(t, 2, len(store.CommitCalls()))
time.Sleep(time.Millisecond * 175)
assert.Equal(t, 7, len(store.CommitCalls()))
svc.Close(context.TODO())
synctest.Test(t, func(t *testing.T) {
store := StoreMock{
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
svc := NewService(&store, ServiceParams{ImageAPI: "/blah/", EditDuration: time.Millisecond * 100})
svc.Submit(func() []string { return []string{"id1", "id2", "id3"} })
assert.Equal(t, 3, len(store.ResetCleanupTimerCalls()))
err := svc.Commit(func() []string { return []string{"id4", "id5"} })
assert.NoError(t, err)
svc.Submit(func() []string { return []string{"id6", "id7"} })
assert.Equal(t, 5, len(store.ResetCleanupTimerCalls()))
svc.Submit(nil)
assert.Equal(t, 2, len(store.CommitCalls()))
time.Sleep(time.Millisecond * 175)
assert.Equal(t, 7, len(store.CommitCalls()))
svc.Close(context.TODO())
})
}
func TestService_Close(t *testing.T) {
@@ -173,21 +178,23 @@ func TestService_Close(t *testing.T) {
}
func TestService_SubmitDelay(t *testing.T) {
store := StoreMock{
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error {
return nil
},
}
svc := NewService(&store, ServiceParams{EditDuration: 20 * time.Millisecond})
svc.Submit(func() []string { return []string{"id1", "id2", "id3"} })
time.Sleep(150 * time.Millisecond) // let first batch to pass TTL
svc.Submit(func() []string { return []string{"id4", "id5"} })
svc.Submit(nil)
assert.Equal(t, 5, len(store.ResetCleanupTimerCalls()))
assert.Equal(t, 3, len(store.CommitCalls()))
svc.Close(context.TODO())
assert.Equal(t, 5, len(store.CommitCalls()))
synctest.Test(t, func(t *testing.T) {
store := StoreMock{
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error {
return nil
},
}
svc := NewService(&store, ServiceParams{EditDuration: 20 * time.Millisecond})
svc.Submit(func() []string { return []string{"id1", "id2", "id3"} })
time.Sleep(150 * time.Millisecond) // let first batch to pass TTL
svc.Submit(func() []string { return []string{"id4", "id5"} })
svc.Submit(nil)
assert.Equal(t, 5, len(store.ResetCleanupTimerCalls()))
assert.Equal(t, 3, len(store.CommitCalls()))
svc.Close(context.TODO())
assert.Equal(t, 5, len(store.CommitCalls()))
})
}
func TestService_Info(t *testing.T) {
@@ -203,19 +210,17 @@ func TestService_Info(t *testing.T) {
}
func TestService_resize(t *testing.T) {
// reader is nil
resized := resize(nil, 100, 100)
assert.Nil(t, resized)
t.Run("empty data returns nil", func(t *testing.T) {
assert.Nil(t, resize(nil, 100, 100))
assert.Nil(t, resize([]byte{}, 100, 100))
})
// negative limit error
resized = resize([]byte("some picture bin data"), -1, -1)
require.NotNil(t, resized)
assert.Equal(t, resized, []byte("some picture bin data"))
// decode error
resized = resize([]byte("invalid image content"), 100, 100)
assert.NotNil(t, resized)
assert.Equal(t, resized, []byte("invalid image content"))
t.Run("non-image bytes are refused", func(t *testing.T) {
// previously resize would fall back to the raw bytes on decode failure, letting
// attacker-controlled non-image content reach the store. After hardening, refuse.
assert.Nil(t, resize([]byte("some picture bin data"), -1, -1))
assert.Nil(t, resize([]byte("invalid image content"), 100, 100))
})
cases := []struct {
file string
@@ -230,7 +235,7 @@ func TestService_resize(t *testing.T) {
require.NoError(t, err, "can't open test file %s", c.file)
// no need for resize, image dimensions are smaller than resize limit
resized = resize(img, 800, 800)
resized := resize(img, 800, 800)
assert.NotNil(t, resized, "file %s", c.file)
assert.Equal(t, resized, img)
@@ -246,6 +251,76 @@ func TestService_resize(t *testing.T) {
}
}
// TestService_SaveWithIDShortPayload guards readAndValidateImage from panicking
// on a body shorter than 512 bytes — historically it sliced data[:512] without
// a bounds check, which would panic before any decode-bomb defense could fire.
func TestService_SaveWithIDShortPayload(t *testing.T) {
short := []byte("not an image")
svc := Service{ServiceParams: ServiceParams{ImageAPI: "/blah/", MaxSize: 1500, MaxWidth: 32, MaxHeight: 32}}
err := svc.SaveWithID("test_id", bytes.NewReader(short))
require.Error(t, err, "short non-image body must return an error, not panic")
assert.Contains(t, err.Error(), "file format not allowed")
}
// TestService_SaveWithIDWebP confirms that WebP — listed as an allowed format
// in readAndValidateImage — still round-trips through prepareImage now that
// resize() runs image.DecodeConfig. Without registering the WebP decoder, a
// legitimate WebP upload would fail DecodeConfig and prepareImage would error.
func TestService_SaveWithIDWebP(t *testing.T) {
webp, err := os.ReadFile("testdata/pixel.webp")
require.NoError(t, err)
// sanity: the fixture must be a well-formed 1x1 WebP that DecodeConfig accepts.
cfg, format, err := image.DecodeConfig(bytes.NewReader(webp))
require.NoError(t, err)
require.Equal(t, "webp", format)
require.Equal(t, 1, cfg.Width)
require.Equal(t, 1, cfg.Height)
store := StoreMock{SaveFunc: func(string, []byte) error { return nil }}
svc := Service{store: &store, ServiceParams: ServiceParams{MaxSize: 1500}}
err = svc.SaveWithID("webp_id", bytes.NewReader(webp))
require.NoError(t, err, "valid WebP must round-trip through SaveWithID")
assert.Equal(t, 1, len(store.SaveCalls()))
assert.Equal(t, webp, store.SaveCalls()[0].Img, "no-resize path must return bytes verbatim")
}
// TestService_resizeRejectsDecompressionBomb verifies the dimension-cap defense.
// Builds a tiny GIF that declares 65535x65535 (4 gigapixels) in its logical-screen
// header — image.DecodeConfig reads the dimensions, the int64 product overflows
// any 32-bit int wrap, and resize must refuse before image.Decode allocates ~17 GB
// of pixel memory.
func TestService_resizeRejectsDecompressionBomb(t *testing.T) {
// minimal GIF87a header with 65535x65535 logical screen, no global color table.
// Bytes 6-7 are the little-endian width, 8-9 are the little-endian height.
bomb := []byte{
'G', 'I', 'F', '8', '7', 'a',
0xFF, 0xFF,
0xFF, 0xFF,
0x00,
0x00,
0x00,
0x3B,
}
cfg, _, err := image.DecodeConfig(bytes.NewReader(bomb))
require.NoError(t, err, "bomb header must decode at the config level")
assert.Equal(t, 65535, cfg.Width)
assert.Equal(t, 65535, cfg.Height)
assert.Nil(t, resize(bomb, 100, 100), "resize must refuse oversized dimensions before raster decode")
assert.Nil(t, resize(bomb, 0, 0), "even with no-resize limits, oversized dims must be refused")
// integration-level: SaveWithID must reject the same bomb without panicking
// or allocating gigabytes of raster memory.
store := StoreMock{SaveFunc: func(string, []byte) error { return nil }}
svc := Service{store: &store, ServiceParams: ServiceParams{MaxSize: 1500}}
err = svc.SaveWithID("bomb_id", bytes.NewReader(bomb))
require.Error(t, err, "SaveWithID must reject decompression bomb")
assert.Equal(t, 0, len(store.SaveCalls()), "rejected bomb must not be stored")
}
func TestGetProportionalSizes(t *testing.T) {
tbl := []struct {
inpW, inpH int
@@ -284,3 +359,6 @@ func TestService_DoubleClose(*testing.T) {
// second call should not result in panic
svc.Close(context.TODO())
}
// TestSafeImgContentType now lives in the rest package alongside the SafeImgContentType
// helper itself (see backend/app/rest/image_headers_test.go).
Binary file not shown.

After

Width:  |  Height:  |  Size: 38 B

@@ -1,6 +1,7 @@
package service
import (
"slices"
"strings"
"unicode"
"unicode/utf8"
@@ -47,12 +48,7 @@ func (m *RestrictedWordsMatcher) Match(siteID, text string) bool {
tokens := m.tokenize(text)
trie := newWildcardTrie(restrictedWords...)
for _, token := range tokens {
if trie.check(token) {
return true
}
}
return false
return slices.ContainsFunc(tokens, trie.check)
}
func (m *RestrictedWordsMatcher) tokenize(text string) []string {
+9 -14
View File
@@ -62,7 +62,7 @@ type UserMetaData struct {
Until time.Time `json:"until"`
} `json:"blocked"`
Verified bool `json:"verified"`
Details engine.UserDetailEntry `json:"details,omitempty"`
Details engine.UserDetailEntry `json:"details"`
}
// PostMetaData keeps info about post flags
@@ -552,9 +552,9 @@ func (s *DataStore) HasReplies(comment store.Comment) bool {
for _, c := range comments {
if c.ParentID != "" && !c.Deleted {
if c.ParentID == comment.ID {
// When this code is reached, key "comment.ID" is not in cache.
// Calling cache.Get on it will put it in cache with 5 minutes TTL.
// We call it with empty struct as value as we care about keys and not values.
// when this code is reached, key "comment.ID" is not in cache.
// calling cache.Get on it will put it in cache with 5 minutes TTL.
// we call it with empty struct as value as we care about keys and not values.
_, _ = s.repliesCache.Get(comment.ID, func() (struct{}, error) { return struct{}{}, nil })
return true
}
@@ -611,7 +611,7 @@ func (s *DataStore) SetTitle(locator store.Locator, commentID string) (comment s
// set title, overwrite the current one
title, e := s.TitleExtractor.Get(comment.Locator.URL)
if e != nil {
return comment, err
return comment, e
}
comment.PostTitle = title
comment.Locator = locator
@@ -676,12 +676,7 @@ func (s *DataStore) IsAdmin(siteID, userID string) bool {
log.Printf("[WARN] can't get admins for %s, %v", siteID, err)
return false
}
for _, a := range admins {
if a == userID {
return true
}
}
return false
return slices.Contains(admins, userID)
}
// IsReadOnly checks if post read-only
@@ -743,7 +738,7 @@ func (s *DataStore) SetBlock(siteID, userID string, status bool, ttl time.Durati
func (s *DataStore) BlockedUsers(siteID string) (res []store.BlockedUser, err error) {
blocked, e := s.Engine.ListFlags(engine.FlagRequest{Locator: store.Locator{SiteID: siteID}, Flag: engine.Blocked})
if e != nil {
return nil, fmt.Errorf("can't get list of blocked users for %s: %w", siteID, err)
return nil, fmt.Errorf("can't get list of blocked users for %s: %w", siteID, e)
}
for _, v := range blocked {
res = append(res, v.(store.BlockedUser))
@@ -801,7 +796,7 @@ func (s *DataStore) Delete(locator store.Locator, commentID string, mode store.D
idsFn := func() []string { // get IDs of all images from the same URL to verify if image from deleted comment was reused
comments, e := s.Engine.Find(engine.FindRequest{Locator: locator})
if e != nil {
log.Printf("[WARN] can't get comments %s text for deleted comment image check, %v", comment.ID, err)
log.Printf("[WARN] can't get comments %s text for deleted comment image check, %v", comment.ID, e)
return nil
}
var imgIDs = []string{}
@@ -822,7 +817,7 @@ func (s *DataStore) Delete(locator store.Locator, commentID string, mode store.D
}
}
}
log.Printf("[ERROR] commentImgIDs: %v, pageImgIDs: %v", commentImgIDs, pageImgIDs)
log.Printf("[DEBUG] commentImgIDs: %v, pageImgIDs: %v", commentImgIDs, pageImgIDs)
req := engine.DeleteRequest{Locator: locator, CommentID: commentID, DeleteMode: mode}
return s.Engine.Delete(req)
+311 -312
View File
@@ -14,6 +14,7 @@ import (
"sync"
"sync/atomic"
"testing"
"testing/synctest"
"time"
"github.com/go-pkgz/lgr"
@@ -184,7 +185,7 @@ func TestService_Put(t *testing.T) {
Text: "test text",
User: store.User{ID: "user2", Name: "user name 2"},
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
}
_, err := b.Create(comment)
require.NoError(t, err)
@@ -196,7 +197,7 @@ func TestService_Put(t *testing.T) {
Text: "new text",
User: store.User{ID: "user3", Name: "user name 3"},
Locator: store.Locator{URL: "https://example.com", SiteID: "example"},
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
}
err = b.Put(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, updatedComment)
@@ -212,7 +213,7 @@ func TestService_Put(t *testing.T) {
assert.Equal(t, "user name 2", got.User.Name, "should be unaltered")
assert.Equal(t, "https://radio-t.com", got.Locator.URL, "should be unaltered")
assert.Equal(t, "radio-t", got.Locator.SiteID, "should be unaltered")
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local), got.Timestamp, "should be unaltered")
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC), got.Timestamp, "should be unaltered")
}
@@ -438,13 +439,11 @@ func TestService_VoteAggressive(t *testing.T) {
// crazy vote +1 as user1
var wg sync.WaitGroup
for i := 0; i < 1000; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for range 1000 {
wg.Go(func() {
_, _ = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: res[0].ID,
UserID: "user1", Val: true})
}()
})
}
wg.Wait()
res, err = b.Last("radio-t", 0, time.Time{}, store.User{ID: "user1"})
@@ -458,14 +457,12 @@ func TestService_VoteAggressive(t *testing.T) {
assert.Equal(t, 0, len(res[0].VotedIPs), "vote ips hidden")
// random +1/-1 result should be [0..2]
for i := 0; i < 100; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for range 100 {
wg.Go(func() {
val := rand.Intn(2) > 0
_, _ = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: res[0].ID,
UserID: "user1", Val: val})
}()
})
}
wg.Wait()
res, err = b.Last("radio-t", 0, time.Time{}, store.User{})
@@ -492,14 +489,11 @@ func TestService_VoteConcurrent(t *testing.T) {
// concurrent vote +1 as multiple users for the same comment
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
wg.Add(1)
ii := i
go func() {
defer wg.Done()
for i := range 100 {
wg.Go(func() {
_, _ = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: res[0].ID,
UserID: fmt.Sprintf("user1-%d", ii), Val: true})
}()
UserID: fmt.Sprintf("user1-%d", i), Val: true})
})
}
wg.Wait()
res, err = b.Last("radio-t", 0, time.Time{}, store.User{})
@@ -602,7 +596,7 @@ func TestService_RestrictedWords(t *testing.T) {
ID: "c-1",
ParentID: "id-1",
Text: "restricted word",
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name 2"},
}
@@ -637,12 +631,12 @@ func TestDataStore_AdminStoreErrors(t *testing.T) {
ID: "c-1",
ParentID: "id-1",
Text: "restricted word",
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name 2"},
}
// Key call error
// key call error
id, err := b.Create(comment)
assert.ErrorContainsf(t, err, "mock key err", "should fail with mock error")
assert.Empty(t, id)
@@ -650,7 +644,7 @@ func TestDataStore_AdminStoreErrors(t *testing.T) {
assert.Equal(t, len(as.EnabledCalls()), 0)
assert.Equal(t, len(as.OnEventCalls()), 0)
// Enabled call error
// enabled call error
badKey = false
id, err = b.Create(comment)
assert.ErrorContains(t, err, "mock enabled err", "should fail with mock error")
@@ -676,7 +670,7 @@ func TestDataStore_AdminStoreErrors(t *testing.T) {
assert.Equal(t, len(as.EnabledCalls()), 3)
assert.Equal(t, len(as.OnEventCalls()), 2)
// Admins error
// admins error
isAdmin := b.IsAdmin("radio-t", "user2")
assert.False(t, isAdmin)
assert.Equal(t, len(as.AdminsCalls()), 1)
@@ -723,34 +717,36 @@ func TestService_VoteSameIP(t *testing.T) {
}
func TestService_VoteSameIPWithDuration(t *testing.T) {
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: admin.NewStaticKeyStore("secret 123"),
MaxVotes: -1}
b.RestrictSameIPVotes.Enabled = true
b.RestrictSameIPVotes.Duration = 500 * time.Millisecond
synctest.Test(t, func(t *testing.T) {
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: admin.NewStaticKeyStore("secret 123"),
MaxVotes: -1}
b.RestrictSameIPVotes.Enabled = true
b.RestrictSameIPVotes.Duration = 500 * time.Millisecond
c, err := b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user2", UserIP: "123", Val: true})
assert.NoError(t, err)
assert.Equal(t, 1, c.Score, "should have 1 score")
c, err := b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user2", UserIP: "123", Val: true})
assert.NoError(t, err)
assert.Equal(t, 1, c.Score, "should have 1 score")
c, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user3", UserIP: "123", Val: true})
assert.EqualError(t, err, "the same ip cce61be6e0a692420ae0de31dceca179123c3b8a already voted for id-2")
assert.Equal(t, 1, c.Score, "still have 1 score")
c, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user3", UserIP: "123", Val: true})
assert.EqualError(t, err, "the same ip cce61be6e0a692420ae0de31dceca179123c3b8a already voted for id-2")
assert.Equal(t, 1, c.Score, "still have 1 score")
c, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user4", UserIP: "12345", Val: true})
assert.NoError(t, err)
assert.Equal(t, 2, c.Score, "have 2 score")
c, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user4", UserIP: "12345", Val: true})
assert.NoError(t, err)
assert.Equal(t, 2, c.Score, "have 2 score")
time.Sleep(501 * time.Millisecond)
time.Sleep(501 * time.Millisecond)
c, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user3", UserIP: "123", Val: true})
assert.NoError(t, err)
assert.Equal(t, 3, c.Score, "have 3 score")
c, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "id-2",
UserID: "user3", UserIP: "123", Val: true})
assert.NoError(t, err)
assert.Equal(t, 3, c.Score, "have 3 score")
})
}
func TestService_Controversy(t *testing.T) {
@@ -864,8 +860,6 @@ func TestService_EditCommentDurationFailed(t *testing.T) {
require.Equal(t, 2, len(res))
assert.Nil(t, res[0].Edit)
time.Sleep(time.Second)
_, err = b.EditComment(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, res[0].ID,
EditRequest{Orig: "yyy", Text: "xxx", Summary: "my edit"})
assert.Error(t, err)
@@ -887,7 +881,7 @@ func TestService_EditCommentReplyFailed(t *testing.T) {
ID: "123456",
ParentID: "id-1",
Text: "some text",
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name 2"},
}
@@ -911,8 +905,6 @@ func TestService_EditCommentAdmin(t *testing.T) {
require.Equal(t, 2, len(res))
assert.Nil(t, res[0].Edit)
time.Sleep(time.Second)
_, err = b.EditComment(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, res[0].ID,
EditRequest{Orig: "yyy", Text: "xxx", Summary: "my edit", Admin: true})
assert.NoError(t, err)
@@ -959,7 +951,7 @@ func TestService_Counts(t *testing.T) {
comment := store.Comment{
ID: "123456",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -1130,7 +1122,7 @@ func TestService_HasReplies(t *testing.T) {
comment := store.Comment{
ID: "id-1",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -1140,7 +1132,7 @@ func TestService_HasReplies(t *testing.T) {
ID: "c-1",
ParentID: "id-1",
Text: "some text",
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name 2"},
}
@@ -1188,87 +1180,89 @@ func TestService_HasReplies(t *testing.T) {
}
func TestService_UserReplies(t *testing.T) {
// two comments for https://radio-t.com, no reply
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng,
AdminStore: admin.NewStaticStore("secret 123", nil, []string{"user2"}, "user@email.com")}
synctest.Test(t, func(t *testing.T) {
// two comments for https://radio-t.com, no reply
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng,
AdminStore: admin.NewStaticStore("secret 123", nil, []string{"user2"}, "user@email.com")}
c1 := store.Comment{
ID: "comment-id-1",
Text: "test 123",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u1", Name: "developer one u1"},
}
c2 := store.Comment{
ID: "comment-id-2",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u2", Name: "developer one u2"},
}
c3 := store.Comment{
ID: "comment-id-3",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u2", Name: "developer one u3"},
}
c4 := store.Comment{
ID: "comment-id-4",
ParentID: "",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u4", Name: "developer one u4"},
}
c5 := store.Comment{
ID: "comment-id-5",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u2", Name: "developer one u2"},
}
c1 := store.Comment{
ID: "comment-id-1",
Text: "test 123",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u1", Name: "developer one u1"},
}
c2 := store.Comment{
ID: "comment-id-2",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u2", Name: "developer one u2"},
}
c3 := store.Comment{
ID: "comment-id-3",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u2", Name: "developer one u3"},
}
c4 := store.Comment{
ID: "comment-id-4",
ParentID: "",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u4", Name: "developer one u4"},
}
c5 := store.Comment{
ID: "comment-id-5",
ParentID: "comment-id-1",
Text: "xyz test",
Locator: store.Locator{URL: "https://radio-t.com/blah10", SiteID: "radio-t"},
User: store.User{ID: "u2", Name: "developer one u2"},
}
_, err := b.Create(c1)
require.NoError(t, err)
_, err = b.Create(c2)
require.NoError(t, err)
_, err = b.Create(c3)
require.NoError(t, err)
_, err = b.Create(c4)
require.NoError(t, err)
// small sleeps give each Create a unique nanosecond timestamp under synctest's fake clock,
// since Bolt keys the "last" bucket by comment timestamp
_, err := b.Create(c1)
require.NoError(t, err)
time.Sleep(time.Nanosecond)
_, err = b.Create(c2)
require.NoError(t, err)
time.Sleep(time.Nanosecond)
_, err = b.Create(c3)
require.NoError(t, err)
time.Sleep(time.Nanosecond)
_, err = b.Create(c4)
require.NoError(t, err)
time.Sleep(100 * time.Millisecond)
_, err = b.Create(c5)
require.NoError(t, err)
time.Sleep(100 * time.Millisecond)
_, err = b.Create(c5)
require.NoError(t, err)
cc, u, err := b.UserReplies("radio-t", "u1", 10, time.Hour)
assert.NoError(t, err)
require.Equal(t, 3, len(cc), "3 replies to u1")
assert.Equal(t, "developer one u1", u)
cc, u, err := b.UserReplies("radio-t", "u1", 10, time.Hour)
assert.NoError(t, err)
require.Equal(t, 3, len(cc), "3 replies to u1")
assert.Equal(t, "developer one u1", u)
// mutex to prevent multiple b.UserReplies calls resulting in data race
l := sync.Mutex{}
assert.Eventually(t, func() bool {
l.Lock()
defer l.Unlock()
// advance fake clock so c2 and c3 (created 100ms before c5) age past the 299ms window,
// leaving only c5 as a recent reply
time.Sleep(200 * time.Millisecond)
cc, u, err = b.UserReplies("radio-t", "u1", 10, time.Millisecond*299)
require.NoError(t, err)
require.Equal(t, "developer one u1", u)
return len(cc) == 1
}, 300*time.Millisecond, 30*time.Millisecond, "1 reply to u1 in the last 300ms")
require.Equal(t, 1, len(cc), "1 reply to u1 in the last 299ms")
l.Lock()
defer l.Unlock()
cc, u, err = b.UserReplies("radio-t", "u2", 10, time.Hour)
assert.NoError(t, err)
assert.Equal(t, 0, len(cc), "0 replies to u2")
assert.Equal(t, "developer one u2", u)
cc, u, err = b.UserReplies("radio-t", "u2", 10, time.Hour)
assert.NoError(t, err)
assert.Equal(t, 0, len(cc), "0 replies to u2")
assert.Equal(t, "developer one u2", u)
cc, u, err = b.UserReplies("radio-t", "uxxx", 10, time.Hour)
assert.NoError(t, err)
assert.Equal(t, 0, len(cc), "0 replies to uxxx")
assert.Equal(t, "", u)
cc, u, err = b.UserReplies("radio-t", "uxxx", 10, time.Hour)
assert.NoError(t, err)
assert.Equal(t, 0, len(cc), "0 replies to uxxx")
assert.Equal(t, "", u)
})
}
func TestService_Find(t *testing.T) {
@@ -1286,7 +1280,7 @@ func TestService_Find(t *testing.T) {
comment := store.Comment{
ID: "123456",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
Score: 1,
@@ -1322,7 +1316,7 @@ func TestService_FindSince(t *testing.T) {
assert.Equal(t, "id-1", res[0].ID)
res, err = b.FindSince(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "time", store.User{},
time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local))
time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC))
require.NoError(t, err)
require.Equal(t, 1, len(res))
assert.Equal(t, "id-2", res[0].ID)
@@ -1339,7 +1333,7 @@ func TestService_Info(t *testing.T) {
comment := store.Comment{
ID: "123456xyz",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/another", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
}
@@ -1365,7 +1359,6 @@ func TestService_Info(t *testing.T) {
assert.True(t, info.LastTS.After(info.FirstTS))
firstTS := info.FirstTS
time.Sleep(1 * time.Second) // make post RO in 1sec
info, err = b.Info(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, 1)
require.NoError(t, err)
assert.Equal(t, "https://radio-t.com", info.URL)
@@ -1408,75 +1401,77 @@ func TestService_Delete(t *testing.T) {
func TestService_deleteImagesOnCommentDelete(t *testing.T) {
lgr.Setup(lgr.Debug, lgr.CallerFile, lgr.CallerFunc)
mockStore := image.StoreMock{
DeleteFunc: func(string) error { return nil },
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 50 * time.Millisecond,
ImageAPI: "/images/dev/",
ProxyAPI: "/non_existent",
})
defer imgSvc.Close(context.TODO())
synctest.Test(t, func(t *testing.T) {
mockStore := image.StoreMock{
DeleteFunc: func(string) error { return nil },
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 50 * time.Millisecond,
ImageAPI: "/images/dev/",
ProxyAPI: "/non_existent",
})
defer imgSvc.Close(context.TODO())
// two comments for https://radio-t.com
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 50 * time.Millisecond,
AdminStore: admin.NewStaticKeyStore("secret 123"), ImageService: imgSvc}
// two comments for https://radio-t.com
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 50 * time.Millisecond,
AdminStore: admin.NewStaticKeyStore("secret 123"), ImageService: imgSvc}
c := store.Comment{
ID: "id-22",
Text: `some text <img src="/images/dev/pic1.png"/> xx <img src="/images/dev/pic2.png"/>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
// reply to the first comment with one new image and one existing one
c = store.Comment{
ID: "id-23",
ParentID: "id-22",
Text: `some text <img src="/images/dev/pic2.png"/> xx <img src="/images/dev/pic3.png"/>`,
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err = b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
c := store.Comment{
ID: "id-22",
Text: `some text <img src="/images/dev/pic1.png"/> xx <img src="/images/dev/pic2.png"/>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
// reply to the first comment with one new image and one existing one
c = store.Comment{
ID: "id-23",
ParentID: "id-22",
Text: `some text <img src="/images/dev/pic2.png"/> xx <img src="/images/dev/pic3.png"/>`,
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err = b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
// verify that images are in staging store
assert.Equal(t, 4, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[1].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[2].ID)
assert.Equal(t, "dev/pic3.png", mockStore.ResetCleanupTimerCalls()[3].ID)
time.Sleep(b.EditDuration + 100*time.Millisecond)
// verify that they got into the main store
assert.Equal(t, 4, len(mockStore.CommitCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.CommitCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[1].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[2].ID)
assert.Equal(t, "dev/pic3.png", mockStore.CommitCalls()[3].ID)
// verify that images are in staging store
assert.Equal(t, 4, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[1].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[2].ID)
assert.Equal(t, "dev/pic3.png", mockStore.ResetCleanupTimerCalls()[3].ID)
time.Sleep(b.EditDuration + 100*time.Millisecond)
// verify that they got into the main store
assert.Equal(t, 4, len(mockStore.CommitCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.CommitCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[1].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[2].ID)
assert.Equal(t, "dev/pic3.png", mockStore.CommitCalls()[3].ID)
// delete the first comment
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "id-22", store.SoftDelete)
assert.NoError(t, err)
// verify that images are deleted from the main store
assert.Equal(t, 1, len(mockStore.DeleteCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.DeleteCalls()[0].ID)
// delete the first comment
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "id-22", store.SoftDelete)
assert.NoError(t, err)
// verify that images are deleted from the main store
assert.Equal(t, 1, len(mockStore.DeleteCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.DeleteCalls()[0].ID)
// delete the second comment
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "id-23", store.SoftDelete)
assert.NoError(t, err)
// verify that images are deleted from the main store
assert.Equal(t, 3, len(mockStore.DeleteCalls()))
assert.Equal(t, "dev/pic2.png", mockStore.DeleteCalls()[1].ID)
assert.Equal(t, "dev/pic3.png", mockStore.DeleteCalls()[2].ID)
// delete the second comment
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "id-23", store.SoftDelete)
assert.NoError(t, err)
// verify that images are deleted from the main store
assert.Equal(t, 3, len(mockStore.DeleteCalls()))
assert.Equal(t, "dev/pic2.png", mockStore.DeleteCalls()[1].ID)
assert.Equal(t, "dev/pic3.png", mockStore.DeleteCalls()[2].ID)
})
}
// DeleteUser removes all comments from user
@@ -1491,7 +1486,7 @@ func TestService_DeleteUser(t *testing.T) {
comment := store.Comment{
ID: "123456xyz",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
}
@@ -1521,7 +1516,7 @@ func TestService_List(t *testing.T) {
// add one more for user2
comment := store.Comment{
ID: "id-3",
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Text: `some text, <a href="http://radio-t.com">link</a>`,
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
@@ -1534,13 +1529,13 @@ func TestService_List(t *testing.T) {
require.Equal(t, 2, len(res), "2 posts")
assert.Equal(t, "https://radio-t.com/2", res[0].URL)
assert.Equal(t, 1, res[0].Count)
assert.Equal(t, time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local), res[0].FirstTS)
assert.Equal(t, time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local), res[0].LastTS)
assert.Equal(t, time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC), res[0].FirstTS)
assert.Equal(t, time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC), res[0].LastTS)
assert.Equal(t, "https://radio-t.com", res[1].URL)
assert.Equal(t, 2, res[1].Count)
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local), res[1].FirstTS)
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 23, 0, time.Local), res[1].LastTS)
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC), res[1].FirstTS)
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 23, 0, time.UTC), res[1].LastTS)
}
func TestService_Count(t *testing.T) {
@@ -1553,7 +1548,7 @@ func TestService_Count(t *testing.T) {
// add one more for user2
comment := store.Comment{
ID: "id-3",
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Text: `some text, <a href="http://radio-t.com">link</a>`,
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
@@ -1584,7 +1579,7 @@ func TestService_UserComments(t *testing.T) {
// add one more for user2
comment := store.Comment{
ID: "id-3",
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Text: `some text, <a href="http://radio-t.com">link</a>`,
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
@@ -1609,7 +1604,7 @@ func TestService_UserCount(t *testing.T) {
// add one more for user2
comment := store.Comment{
ID: "id-3",
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Text: `some text, <a href="http://radio-t.com">link</a>`,
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
@@ -1639,7 +1634,7 @@ func TestService_DeleteAll(t *testing.T) {
// add one more for user2
comment := store.Comment{
ID: "id-3",
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.UTC),
Text: `some text, <a href="http://radio-t.com">link</a>`,
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
@@ -1658,129 +1653,133 @@ func TestService_DeleteAll(t *testing.T) {
func TestService_submitImages(t *testing.T) {
lgr.Setup(lgr.Debug, lgr.CallerFile, lgr.CallerFunc)
mockStore := image.StoreMock{
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 50 * time.Millisecond,
ImageAPI: "/images/dev/",
ProxyAPI: "/non_existent",
})
defer imgSvc.Close(context.TODO())
synctest.Test(t, func(t *testing.T) {
mockStore := image.StoreMock{
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 50 * time.Millisecond,
ImageAPI: "/images/dev/",
ProxyAPI: "/non_existent",
})
defer imgSvc.Close(context.TODO())
// two comments for https://radio-t.com
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 50 * time.Millisecond,
AdminStore: admin.NewStaticKeyStore("secret 123"), ImageService: imgSvc}
// two comments for https://radio-t.com
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 50 * time.Millisecond,
AdminStore: admin.NewStaticKeyStore("secret 123"), ImageService: imgSvc}
c := store.Comment{
ID: "id-22",
Text: `some text <img src="/images/dev/pic1.png"/> xx <img src="/images/dev/pic2.png"/>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
c := store.Comment{
ID: "id-22",
Text: `some text <img src="/images/dev/pic1.png"/> xx <img src="/images/dev/pic2.png"/>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
assert.Equal(t, 2, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[1].ID)
time.Sleep(b.EditDuration + 100*time.Millisecond)
assert.Equal(t, 2, len(mockStore.CommitCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.CommitCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[1].ID)
b.submitImages(c)
assert.Equal(t, 2, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[1].ID)
time.Sleep(b.EditDuration + 100*time.Millisecond)
assert.Equal(t, 2, len(mockStore.CommitCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.CommitCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[1].ID)
})
}
func TestService_ResubmitStagingImages(t *testing.T) {
mockStore := image.StoreMock{
InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{FirstStagingImageTS: time.Time{}.Add(time.Second)}, nil
},
CommitFunc: func(string) error {
return nil
},
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 10 * time.Millisecond,
ImageAPI: "http://127.0.0.1:8080/api/v1/picture/",
ProxyAPI: "http://127.0.0.1:8080/api/v1/img",
})
defer imgSvc.Close(context.TODO())
synctest.Test(t, func(t *testing.T) {
mockStore := image.StoreMock{
InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{FirstStagingImageTS: time.Time{}.Add(time.Second)}, nil
},
CommitFunc: func(string) error {
return nil
},
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 10 * time.Millisecond,
ImageAPI: "http://127.0.0.1:8080/api/v1/picture/",
ProxyAPI: "http://127.0.0.1:8080/api/v1/img",
})
defer imgSvc.Close(context.TODO())
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 10 * time.Millisecond, ImageService: imgSvc}
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 10 * time.Millisecond, ImageService: imgSvc}
// create comment with three images without preparing it properly
comment := store.Comment{
ID: "id-0",
Text: `<img src="http://127.0.0.1:8080/api/v1/picture/dev_user/bqf122eq9r8ad657n3ng" alt="startrails_01.jpg"><br/>
// create comment with three images without preparing it properly
comment := store.Comment{
ID: "id-0",
Text: `<img src="http://127.0.0.1:8080/api/v1/picture/dev_user/bqf122eq9r8ad657n3ng" alt="startrails_01.jpg"><br/>
<img src="http://127.0.0.1:8080/api/v1/picture/dev_user/bqf321eq9r8ad657n3ng" alt="cat.png"><br/>
<img src="http://127.0.0.1:8080/api/v1/img?src=aHR0cHM6Ly9ob21lcGFnZXMuY2FlLndpc2MuZWR1L35lY2U1MzMvaW1hZ2VzL2JvYXQucG5n" alt="cat.png"><br/>
<img src="https://homepages.cae.wisc.edu/~ece533/images/boat.png" alt="boat.png">`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(comment)
require.NoError(t, err)
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(comment)
require.NoError(t, err)
// resubmit single comment with three images, of which two are in staging storage
err = b.ResubmitStagingImages([]string{"radio-t"})
assert.NoError(t, err)
// resubmit single comment with three images, of which two are in staging storage
err = b.ResubmitStagingImages([]string{"radio-t"})
assert.NoError(t, err)
// wait for Submit goroutine to commit image
time.Sleep(b.EditDuration + time.Millisecond*100)
// wait for Submit goroutine to commit image
time.Sleep(b.EditDuration + time.Millisecond*100)
assert.Equal(t, 1, len(mockStore.InfoCalls()))
assert.Equal(t, 3, len(mockStore.CommitCalls()))
assert.Equal(t, 1, len(mockStore.InfoCalls()))
assert.Equal(t, 3, len(mockStore.CommitCalls()))
// empty answer
mockStoreEmpty := image.StoreMock{InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{FirstStagingImageTS: time.Time{}}, nil
}}
imgSvcEmpty := image.NewService(&mockStoreEmpty,
image.ServiceParams{
EditDuration: 10 * time.Millisecond,
ImageAPI: "http://127.0.0.1:8080/api/v1/picture/",
})
defer imgSvcEmpty.Close(context.TODO())
bEmpty := DataStore{Engine: eng, EditDuration: 10 * time.Millisecond, ImageService: imgSvcEmpty}
// empty answer
mockStoreEmpty := image.StoreMock{InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{FirstStagingImageTS: time.Time{}}, nil
}}
imgSvcEmpty := image.NewService(&mockStoreEmpty,
image.ServiceParams{
EditDuration: 10 * time.Millisecond,
ImageAPI: "http://127.0.0.1:8080/api/v1/picture/",
})
defer imgSvcEmpty.Close(context.TODO())
bEmpty := DataStore{Engine: eng, EditDuration: 10 * time.Millisecond, ImageService: imgSvcEmpty}
// resubmit receive empty timestamp and should do nothing )
err = bEmpty.ResubmitStagingImages([]string{"radio-t", "non_existent"})
assert.NoError(t, err)
// resubmit receive empty timestamp and should do nothing )
err = bEmpty.ResubmitStagingImages([]string{"radio-t", "non_existent"})
assert.NoError(t, err)
assert.Equal(t, 1, len(mockStore.InfoCalls()))
assert.Equal(t, 1, len(mockStore.InfoCalls()))
// error from image storage
mockStoreError := image.StoreMock{InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{}, fmt.Errorf("mock_err")
}}
imgSvcError := image.NewService(&mockStoreError,
image.ServiceParams{
EditDuration: 10 * time.Millisecond,
ImageAPI: "http://127.0.0.1:8080/api/v1/picture/",
})
defer imgSvcError.Close(context.TODO())
bError := DataStore{Engine: eng, EditDuration: 10 * time.Millisecond, ImageService: imgSvcError}
// error from image storage
mockStoreError := image.StoreMock{InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{}, fmt.Errorf("mock_err")
}}
imgSvcError := image.NewService(&mockStoreError,
image.ServiceParams{
EditDuration: 10 * time.Millisecond,
ImageAPI: "http://127.0.0.1:8080/api/v1/picture/",
})
defer imgSvcError.Close(context.TODO())
bError := DataStore{Engine: eng, EditDuration: 10 * time.Millisecond, ImageService: imgSvcError}
// resubmit will receive error from image storage and should return it
err = bError.ResubmitStagingImages([]string{"radio-t"})
assert.EqualError(t, err, "mock_err")
// resubmit will receive error from image storage and should return it
err = bError.ResubmitStagingImages([]string{"radio-t"})
assert.EqualError(t, err, "mock_err")
assert.Equal(t, 1, len(mockStore.InfoCalls()))
assert.Equal(t, 3, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev_user/bqf122eq9r8ad657n3ng", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev_user/bqf321eq9r8ad657n3ng", mockStore.ResetCleanupTimerCalls()[1].ID)
assert.Equal(t, "cached_images/12318fbd4c55e9d177b8b5ae197bc89c5afd8e07-a41fcb00643f28d700504256ec81cbf2e1aac53e", mockStore.ResetCleanupTimerCalls()[2].ID)
assert.Equal(t, 1, len(mockStore.InfoCalls()))
assert.Equal(t, 3, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev_user/bqf122eq9r8ad657n3ng", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev_user/bqf321eq9r8ad657n3ng", mockStore.ResetCleanupTimerCalls()[1].ID)
assert.Equal(t, "cached_images/12318fbd4c55e9d177b8b5ae197bc89c5afd8e07-a41fcb00643f28d700504256ec81cbf2e1aac53e", mockStore.ResetCleanupTimerCalls()[2].ID)
})
}
func TestService_ResubmitStagingImages_EngineError(t *testing.T) {
@@ -1808,7 +1807,7 @@ func TestService_ResubmitStagingImages_EngineError(t *testing.T) {
site2Req := engine.FindRequest{Locator: store.Locator{SiteID: "site2", URL: ""}, Sort: "time", Since: time.Time{}.Add(time.Second)}
b := DataStore{Engine: &engineMock, EditDuration: 10 * time.Millisecond, ImageService: imgSvc}
// One call without error and one with error
// one call without error and one with error
err := b.ResubmitStagingImages([]string{"site1", "site2"})
assert.Error(t, err)
assert.Contains(t, err.Error(), "problem finding comments for site site2: mockError")
@@ -1892,7 +1891,7 @@ func Benchmark_ServiceCreate(b *testing.B) {
comment := store.Comment{
ID: "id-" + strconv.Itoa(i),
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -1939,7 +1938,7 @@ func prepStoreEngine(t *testing.T) (e engine.Interface, teardown func()) {
comment := store.Comment{
ID: "id-1",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
@@ -1949,7 +1948,7 @@ func prepStoreEngine(t *testing.T) (e engine.Interface, teardown func()) {
comment = store.Comment{
ID: "id-2",
Text: "some text2",
Timestamp: time.Date(2017, 12, 20, 15, 18, 23, 0, time.Local),
Timestamp: time.Date(2017, 12, 20, 15, 18, 23, 0, time.UTC),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
+49 -10
View File
@@ -14,6 +14,8 @@ import (
"github.com/go-pkgz/syncs"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/safehttp"
)
func TestTitle_GetTitle(t *testing.T) {
@@ -61,7 +63,7 @@ func TestTitle_Get(t *testing.T) {
_, err = ex.Get(ts.URL + "/bad")
require.Error(t, err)
for i := 0; i < 100; i++ {
for range 100 {
r, err := ex.Get(ts.URL + "/good")
require.NoError(t, err)
assert.Equal(t, "blah 123", r)
@@ -70,9 +72,9 @@ func TestTitle_Get(t *testing.T) {
}
func TestTitle_GetConcurrent(t *testing.T) {
body := ""
for n := 0; n < 1000; n++ {
body += "something something blah blah\n"
var body strings.Builder
for range 1000 {
body.WriteString("something something blah blah\n")
}
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{"127.0.0.1"})
defer ex.Close()
@@ -80,7 +82,7 @@ func TestTitle_GetConcurrent(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.String(), "/good") {
atomic.AddInt32(&hits, 1)
_, err := fmt.Fprintf(w, "<html><title>blah 123 %s</title><body>%s</body></html>", r.URL.String(), body)
_, err := fmt.Fprintf(w, "<html><title>blah 123 %s</title><body>%s</body></html>", r.URL.String(), body.String())
assert.NoError(t, err)
return
}
@@ -90,12 +92,11 @@ func TestTitle_GetConcurrent(t *testing.T) {
g := syncs.NewSizedGroup(10)
for i := 0; i < 100; i++ {
ii := i
for i := range 100 {
g.Go(func(_ context.Context) {
title, err := ex.Get(ts.URL + "/good/" + strconv.Itoa(ii))
title, err := ex.Get(ts.URL + "/good/" + strconv.Itoa(i))
require.NoError(t, err)
assert.Equal(t, "blah 123 "+"/good/"+strconv.Itoa(ii), title)
assert.Equal(t, "blah 123 "+"/good/"+strconv.Itoa(i), title)
})
}
g.Wait()
@@ -115,7 +116,7 @@ func TestTitle_GetFailed(t *testing.T) {
_, err := ex.Get(ts.URL + "/bad")
require.Error(t, err)
for i := 0; i < 100; i++ {
for range 100 {
r, err := ex.Get(ts.URL + "/bad")
require.NoError(t, err)
assert.Equal(t, "", r)
@@ -129,3 +130,41 @@ func TestTitle_DoubleClosed(t *testing.T) {
// second call should not result in panic
assert.NoError(t, ex.Close())
}
// TestTitle_GetBlocksPrivateIPViaSafeTransport reproduces the SSRF in TitleExtractor.
// In production (cmd/server.go) the TitleExtractor receives the comment's Locator.URL
// straight from the user JSON body. The domain allowlist alone is not enough — a
// hostname suffix-matching an allowed domain can resolve to a private IP (DNS rebinding)
// or an attacker can list 127.0.0.1 directly when AllowedHosts is empty.
//
// The fix is to wrap the http.Client with safehttp.Transport at construction time,
// matching what the image proxy already does. This test asserts the safehttp transport
// is honored by the title fetcher: even though "127.0.0.1" is in the allowed-domains
// list, the dialer refuses to connect to a private address.
//
// As a control, the second sub-test shows the same setup WITHOUT safehttp.Transport
// happily fetches the page — demonstrating the original vulnerability.
func TestTitle_GetBlocksPrivateIPViaSafeTransport(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`<html><title>secret</title></html>`))
}))
defer ts.Close()
t.Run("with safehttp transport: blocked", func(t *testing.T) {
client := http.Client{Timeout: 2 * time.Second, Transport: safehttp.Transport()}
ex := NewTitleExtractor(client, []string{"127.0.0.1"})
defer ex.Close()
_, err := ex.Get(ts.URL)
require.Error(t, err)
assert.Contains(t, err.Error(), "access to private address is not allowed")
})
t.Run("control: default transport leaks", func(t *testing.T) {
client := http.Client{Timeout: 2 * time.Second} // no safehttp.Transport — vulnerable
ex := NewTitleExtractor(client, []string{"127.0.0.1"})
defer ex.Close()
title, err := ex.Get(ts.URL)
require.NoError(t, err, "without safehttp.Transport the SSRF succeeds — this is the bug")
assert.Equal(t, "secret", title)
})
}
+9 -9
View File
@@ -185,7 +185,7 @@ func (t *Tree) limit(limit int, offsetID string) {
}
}
if start == len(t.Nodes) { // If the start index is beyond the available nodes, clear the nodes
if start == len(t.Nodes) { // if the start index is beyond the available nodes, clear the nodes
t.Nodes = []*Node{}
return
}
@@ -198,28 +198,28 @@ func (t *Tree) limit(limit int, offsetID string) {
return
}
// Traverse and limit the number of top-level nodes, including their replies
// traverse and limit the number of top-level nodes, including their replies
limitedNodes := []*Node{}
commentsCount := 0
for _, node := range t.Nodes {
repliesCount := countReplies(node) + 1 // Count this node and its replies
repliesCount := countReplies(node) + 1 // count this node and its replies
// If the limit is already reached or exceeded, calculate countLeft and move to the next node
// if the limit is already reached or exceeded, calculate countLeft and move to the next node
if commentsCount >= limit {
t.countLeft += repliesCount
continue
}
// Check if we just exceeded the limit and there are already some nodes in the list,
// check if we just exceeded the limit and there are already some nodes in the list,
// as otherwise we would have to return the first node with all its replies even if it exceeds the limit.
if commentsCount+repliesCount >= limit && len(limitedNodes) > 0 {
t.countLeft += repliesCount
commentsCount = limit // Adjust commentsCount to stop checking limit for the next nodes
commentsCount = limit // adjust commentsCount to stop checking limit for the next nodes
continue
}
// Add the node and its replies to the list
// add the node and its replies to the list
limitedNodes = append(limitedNodes, node)
commentsCount += repliesCount
}
@@ -232,8 +232,8 @@ func (t *Tree) limit(limit int, offsetID string) {
func countReplies(node *Node) int {
count := 0
for _, reply := range node.Replies {
count++ // Count the reply itself
count += countReplies(reply) // Recursively count its replies
count++ // count the reply itself
count += countReplies(reply) // recursively count its replies
}
return count
}
+24 -26
View File
@@ -1,23 +1,22 @@
module github.com/umputun/remark42/backend
go 1.25
go 1.25.0
require (
github.com/Depado/bfchroma/v2 v2.0.0
github.com/PuerkitoBio/goquery v1.11.0
github.com/alecthomas/chroma/v2 v2.21.1
github.com/PuerkitoBio/goquery v1.12.0
github.com/alecthomas/chroma/v2 v2.27.0
github.com/didip/tollbooth/v8 v8.0.1
github.com/go-chi/chi/v5 v5.2.3
github.com/go-chi/cors v1.2.2
github.com/go-pkgz/auth/v2 v2.1.1
github.com/go-pkgz/auth/v2 v2.1.5
github.com/go-pkgz/jrpc v0.4.0
github.com/go-pkgz/lcw/v2 v2.0.0
github.com/go-pkgz/lgr v0.12.1
github.com/go-pkgz/lgr v0.12.3
github.com/go-pkgz/notify v1.3.0
github.com/go-pkgz/repeater/v2 v2.2.0
github.com/go-pkgz/rest v1.20.6
github.com/go-pkgz/rest v1.22.0
github.com/go-pkgz/routegroup v1.6.0
github.com/go-pkgz/syncs v1.3.2
github.com/golang-jwt/jwt/v5 v5.3.0
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/gorilla/feeds v1.2.0
github.com/hashicorp/go-multierror v1.1.1
@@ -28,46 +27,45 @@ require (
github.com/russross/blackfriday/v2 v2.1.0
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
github.com/stretchr/testify v1.11.1
go.etcd.io/bbolt v1.4.3
go.etcd.io/bbolt v1.5.0
go.uber.org/goleak v1.3.0
golang.org/x/crypto v0.46.0
golang.org/x/image v0.34.0
golang.org/x/net v0.48.0
golang.org/x/crypto v0.53.0
golang.org/x/image v0.43.0
golang.org/x/net v0.56.0
golang.org/x/oauth2 v0.36.0
)
require (
cloud.google.com/go/compute/metadata v0.9.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/andybalholm/cascadia v1.3.4 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dghubble/oauth1 v0.7.3 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/dlclark/regexp2/v2 v2.2.2 // indirect
github.com/go-oauth2/oauth2/v4 v4.5.4 // indirect
github.com/go-pkgz/email v0.6.0 // indirect
github.com/go-pkgz/expirable-cache/v3 v3.1.0 // indirect
github.com/go-pkgz/repeater v1.2.0 // indirect
github.com/go-pkgz/routegroup v1.6.0 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/montanaflynn/stats v0.7.1 // indirect
github.com/klauspost/compress v1.18.7 // indirect
github.com/montanaflynn/stats v0.9.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/redis/go-redis/v9 v9.17.2 // indirect
github.com/redis/go-redis/v9 v9.21.0 // indirect
github.com/rrivera/identicon v0.0.0-20240116195454-d5ba35832c0d // indirect
github.com/slack-go/slack v0.17.3 // indirect
github.com/slack-go/slack v0.27.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.2.0 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.mongodb.org/mongo-driver v1.17.6 // indirect
golang.org/x/oauth2 v0.34.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/text v0.32.0 // indirect
go.mongodb.org/mongo-driver v1.17.9 // indirect
go.uber.org/atomic v1.11.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.46.0 // indirect
golang.org/x/text v0.38.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
+48 -90
View File
@@ -2,14 +2,14 @@ cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdB
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
github.com/Depado/bfchroma/v2 v2.0.0 h1:IRpN9BPkNwEpR6w1ectIcNWOuhDSLx+8f1pn83fzxx8=
github.com/Depado/bfchroma/v2 v2.0.0/go.mod h1:wFwW/Pw8Tnd0irzgO9Zxtxgzp3aPS8qBWlyadxujxmw=
github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw=
github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ=
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
github.com/ajg/form v1.5.1 h1:t9c7v8JUKu/XxOGBU0yjNpaMloxGEJhUkqFRq0ibGeU=
github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY=
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/chroma/v2 v2.21.1 h1:FaSDrp6N+3pphkNKU6HPCiYLgm8dbe5UXIXcoBhZSWA=
github.com/alecthomas/chroma/v2 v2.21.1/go.mod h1:NqVhfBR0lte5Ouh3DcthuUCTUpDC9cxBOfyMbMQPs3o=
github.com/alecthomas/chroma/v2 v2.27.0 h1:FodwmyOBgJULFYmDqibcp9pvfDLWdtPRh9v/r5BXYZs=
github.com/alecthomas/chroma/v2 v2.27.0/go.mod h1:NjJ3ciIgrqBNeIkWZ4e46nseoLDslxU1LmfCoL+wcY8=
github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 h1:uvdUDbHQHO85qeSydJtItA4T55Pw6BtAejd0APRJOCE=
@@ -18,8 +18,8 @@ github.com/alicebob/miniredis/v2 v2.31.1 h1:7XAt0uUg3DtwEKW5ZAGa+K7FZV2DdKQo5K/6
github.com/alicebob/miniredis/v2 v2.31.1/go.mod h1:UB/T2Uztp7MlFSDakaX1sTXUv5CASoprx0wulRT6HBg=
github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY=
github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/andybalholm/cascadia v1.3.4 h1:vM2lgh0Vru9Vwyfm4cQqWP2HHMW0u0+2PAW7Q38Qufg=
github.com/andybalholm/cascadia v1.3.4/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
@@ -32,24 +32,18 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dghubble/oauth1 v0.7.3 h1:EkEM/zMDMp3zOsX2DC/ZQ2vnEX3ELK0/l9kb+vs4ptE=
github.com/dghubble/oauth1 v0.7.3/go.mod h1:oxTe+az9NSMIucDPDCCtzJGsPhciJV33xocHfcR2sVY=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/didip/tollbooth/v8 v8.0.1 h1:VAAapTo1t4Bn6bbpcHjuovwoa9u3JH++wgjbpWv+rB8=
github.com/didip/tollbooth/v8 v8.0.1/go.mod h1:oEd9l+ep373d7DmvKLc0a5gasPOev2mTewi6KPQBGJ4=
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dlclark/regexp2/v2 v2.2.2 h1:MYWvNYw8okuqNhwTYO587EZMiDruVa2vhV6fsGpfya0=
github.com/dlclark/regexp2/v2 v2.2.2/go.mod h1:avUrQvPaLz2DrFNHJF0taWAFFX2C1GMSSoeiqFjcBmU=
github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/gavv/httpexpect v2.0.0+incompatible h1:1X9kcRshkSKEjNJJxX9Y9mQ5BRfbxU5kORdjhlA1yX8=
github.com/gavv/httpexpect v2.0.0+incompatible/go.mod h1:x+9tiU1YnrOvnB725RkpoLv1M62hOWzwo5OXotisrKc=
github.com/go-chi/chi/v5 v5.2.3 h1:WQIt9uxdsAbgIYgid+BpYc+liqQZGMHRaUwp0JUcvdE=
github.com/go-chi/chi/v5 v5.2.3/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE=
github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
github.com/go-oauth2/oauth2/v4 v4.5.4 h1:YjI0tmGW8oxVhn9QSBIxlr641QugWrJY5UWa6XmLcW0=
github.com/go-oauth2/oauth2/v4 v4.5.4/go.mod h1:BXiOY+QZtZy2ewbsGk2B5P8TWmtz/Rf7ES5ZttQFxfQ=
github.com/go-pkgz/auth/v2 v2.1.1 h1:CBH3Z6ovLT51Nx9TkBcu2L8Dd/xwL6CgJcgqUnC2isQ=
github.com/go-pkgz/auth/v2 v2.1.1/go.mod h1:9LwzESczjMavmXNZo1XhYpfYdKWtoCbXt/ZIi0GTvF0=
github.com/go-pkgz/auth/v2 v2.1.5 h1:CFL7XxRMNPga0S0YCnAnlvO61OHHEYvVEGrIZXuA98Y=
github.com/go-pkgz/auth/v2 v2.1.5/go.mod h1:IvxxhJIrwd1hKqFwQgBF9i+sMTmGfzAw66wmhw1zfJc=
github.com/go-pkgz/email v0.6.0 h1:snZnXldjeF4PgKSjnx9Fa25mtOgFpAOEeWvnQvrxjLE=
github.com/go-pkgz/email v0.6.0/go.mod h1:+wgi4x7S33IuCzfcCM5euN0GwQG6XvO/PBLxrNffYLI=
github.com/go-pkgz/expirable-cache/v3 v3.1.0 h1:s05P851/O6QJ6Mc+7o2bh9aGtD3romB1SxDTXifdoqc=
@@ -58,27 +52,26 @@ github.com/go-pkgz/jrpc v0.4.0 h1:oD7xiGrzDkndkuCjeHGugQXxbggLSV7O1QmHhoc5pYY=
github.com/go-pkgz/jrpc v0.4.0/go.mod h1:JFoY3bRjRyx4M3CbEVDFQStMB1m2gmQ7OjqFK7q3kOo=
github.com/go-pkgz/lcw/v2 v2.0.0 h1:gTwXpiJBhQeA1rXuqkRuLcV79uATFna8CckH8ZBBrH0=
github.com/go-pkgz/lcw/v2 v2.0.0/go.mod h1:yxJHOn+IbQBQHxUqkCtMrbGjIfdYcsBAZcVCBaL1Va8=
github.com/go-pkgz/lgr v0.12.1 h1:8GVfG2rSARq3Eaj5PP158rtBR2LHVGkwioIkQBGbvKg=
github.com/go-pkgz/lgr v0.12.1/go.mod h1:A4AxjOthFVFK6jRnVYMeusno5SeDAxcLVHd0kI/lN/Y=
github.com/go-pkgz/lgr v0.12.3 h1:QDug7kRkEsuQtruT9fNF5PVT2kZUqCDPc4GmsgS3fP8=
github.com/go-pkgz/lgr v0.12.3/go.mod h1:lpCDgVvCIxBHZp8+sGCj9MPctIzKZyZ3QdE19ddqd54=
github.com/go-pkgz/notify v1.3.0 h1:YxF/ThEoCetdcoghWdyeqaBpCkZ8mvyve7HXbCAOzYU=
github.com/go-pkgz/notify v1.3.0/go.mod h1:qdfi5OsViKlIFPryIOaINHTOtS9GFhOYXPqJmAMlaGU=
github.com/go-pkgz/repeater v1.2.0 h1:oJFvjyKdTDd5RCzpzxlzYIZFFj6Zfl17rE1aUfu6UjQ=
github.com/go-pkgz/repeater v1.2.0/go.mod h1:vypP6xamA53MFmafnGUucqOmALKk36xgKu2hSG73LHM=
github.com/go-pkgz/repeater/v2 v2.2.0 h1:8nZR/NaknmLfx2YMHbr78u9OL4Xj+8+romm9dz4FpMg=
github.com/go-pkgz/repeater/v2 v2.2.0/go.mod h1:RgX5vUbLKq7PV82QUDP5pFbQS1os4Z+U9XzKymK23A8=
github.com/go-pkgz/rest v1.20.6 h1:O/IhQ3I2cS4bJYvL1TLcy63w2OcXZTTBG3R+wTIqPS4=
github.com/go-pkgz/rest v1.20.6/go.mod h1:NY+MX1is2kJckJt+nHDNovS/5j9jmF4yQuSno4qg7XU=
github.com/go-pkgz/rest v1.22.0 h1:d3XFKlmAGBiU9MQER9/n46iXpyUr8IQUtfjU8JlqkkY=
github.com/go-pkgz/rest v1.22.0/go.mod h1:+AHzjHazq7Z3Tk/kRWOhbbAz/YZlUV40feC1Hf4NtbE=
github.com/go-pkgz/routegroup v1.6.0 h1:44XHZgF6JIIldRlv+zjg6SygULASmjifnfIQjwCT0e4=
github.com/go-pkgz/routegroup v1.6.0/go.mod h1:Pmu04fhgWhRtBMIJ8HXppnnzOPjnL/IEPBIdO2zmeqg=
github.com/go-pkgz/syncs v1.3.2 h1:gmioASlJNy3gNosPlgvWOM2QP0Hdjzn2u+/sUShgd8E=
github.com/go-pkgz/syncs v1.3.2/go.mod h1:qjgzpp7OpuhDf7BWsW/FHCu9DLjE32NPy6/vXAXT/Cw=
github.com/go-test/deep v1.1.1 h1:0r/53hagsehfO4bzD2Pgr/+RgHqhmf+k1Bpse2cTu1U=
github.com/go-test/deep v1.1.1/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/golang-jwt/jwt/v5 v5.3.0 h1:pv4AsKCKKZuqlgs5sUmn4x8UlGa0kEVt/puTpKx9vvo=
github.com/golang-jwt/jwt/v5 v5.3.0/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk=
@@ -108,8 +101,10 @@ github.com/jessevdk/go-flags v1.6.1 h1:Cvu5U8UGrLay1rZfv/zP7iLpSHGUZ/Ou68T0iX1bB
github.com/jessevdk/go-flags v1.6.1/go.mod h1:Mk8T1hIAWpOiJiHa9rJASDK2UGWji0EuPGBnNLMooyc=
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw=
github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -118,14 +113,14 @@ github.com/kyokomi/emoji/v2 v2.2.13 h1:GhTfQa67venUUvmleTNFnb+bi7S3aocF7ZCXU9fSO
github.com/kyokomi/emoji/v2 v2.2.13/go.mod h1:JUcn42DTdsXJo1SWanHh4HKDEyPaR5CqkmoirZZP9qE=
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
github.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=
github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE=
github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
github.com/montanaflynn/stats v0.9.0 h1:tsBJ0RXwph9BmAuFoCmqGv6e8xa0MENQ8m0ptKq29mQ=
github.com/montanaflynn/stats v0.9.0/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow=
github.com/moul/http2curl v1.0.0 h1:dRMWoAtb+ePxMlLkrCbAqh4TlPHXvoGUSQ323/9Zahs=
github.com/moul/http2curl v1.0.0/go.mod h1:8UbvGypXm98wA/IqH45anm5Y2Z6ep6O31QGOAZ3H0fQ=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI=
github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370=
github.com/redis/go-redis/v9 v9.21.0 h1:FPBE4hhbAke+TLmcY3WkpbDffJEomdqPn3HYiqAtL9E=
github.com/redis/go-redis/v9 v9.21.0/go.mod h1:v/M13XI1PVCDcm01VtPFOADfZtHf8YW3baQf57KlIkA=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rrivera/identicon v0.0.0-20240116195454-d5ba35832c0d h1:l3+2LWCbVxn5itfvXAfH9n4YL9jh8l1g5zcncbIc1cs=
@@ -138,8 +133,8 @@ github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e h1:MRM5ITcdelLK2j1vwZ3Je0FKVCfqOLp5zO6trqMLYs0=
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e/go.mod h1:XV66xRDqSt+GTGFMVlhk3ULuV0y9ZmzeVGR4mloJI3M=
github.com/slack-go/slack v0.17.3 h1:zV5qO3Q+WJAQ/XwbGfNFrRMaJ5T/naqaonyPV/1TP4g=
github.com/slack-go/slack v0.17.3/go.mod h1:X+UqOufi3LYQHDnMG1vxf0J8asC6+WllXrVrhl8/Prk=
github.com/slack-go/slack v0.27.0 h1:VWOpUzOK6UAPCCQlFxl79jhv8a/b+GOSJMnWziDJ8B8=
github.com/slack-go/slack v0.27.0/go.mod h1:UEe+jmo9WLlwHB04qsOrTDvqM7Aa4rQL3O5wF3n0hx4=
github.com/smartystreets/assertions v1.1.0 h1:MkTeG1DMwsrdH7QtLXy5W+fUxWq+vmb6cLmyJ7aRtF0=
github.com/smartystreets/assertions v1.1.0/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo=
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
@@ -189,89 +184,52 @@ github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82/go.mod h1:lgjkn3NuSvDf
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
go.mongodb.org/mongo-driver v1.17.6 h1:87JUG1wZfWsr6rIz3ZmpH90rL5tea7O3IHuSwHUpsss=
go.mongodb.org/mongo-driver v1.17.6/go.mod h1:Hy04i7O2kC4RS06ZrhPRqj/u4DTYkFDAAccj+rVKqgQ=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.etcd.io/bbolt v1.5.0 h1:S7GAl7Fxv12yohbwFfIbQCGDWbQbtDGPET4P/bD4lxU=
go.etcd.io/bbolt v1.5.0/go.mod h1:mkltfYE5aUHQxUct9N9V+Kp7aSjFqjgrhcXIS70Lrdk=
go.mongodb.org/mongo-driver v1.17.9 h1:IexDdCuuNJ3BHrELgBlyaH9p60JXAvdzWR128q+U5tU=
go.mongodb.org/mongo-driver v1.17.9/go.mod h1:LlOhpH5NUEfhxcAwG0UEkMqwYcc4JU18gtCdGudk/tQ=
go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/image v0.34.0 h1:33gCkyw9hmwbZJeZkct8XyR11yH889EQt/QH4VmXMn8=
golang.org/x/image v0.34.0/go.mod h1:2RNFBZRB+vnwwFil8GkMdRvrJOFd1AzdZI6vOY+eJVU=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/image v0.43.0 h1:FLxcP4ec2350nTfOC8ysKtqYSIFbk/QGjw1ZHNP4tsY=
golang.org/x/image v0.43.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+2
View File
@@ -24,6 +24,7 @@ Syntax-wise, it is as close as possible to jQuery, with the same function names
Required Go version:
* Starting with version `v1.12.0` of goquery, Go 1.25+ is required due to its dependencies.
* Starting with version `v1.11.0` of goquery, Go 1.24+ is required due to its dependencies.
* Starting with version `v1.10.0` of goquery, Go 1.23+ is required due to the use of function-based iterators.
* For `v1.9.0` of goquery, Go 1.18+ is required due to the use of generics.
@@ -47,6 +48,7 @@ Ongoing goquery development is tested on the latest 2 versions of Go.
**Note that goquery's API is now stable, and will not break.**
* **2026-03-15 (v1.12.0)** : Update `go.mod` dependencies, add go1.26 to the test matrix, **goquery now requires Go version 1.25+**.
* **2025-11-16 (v1.11.0)** : Update `go.mod` dependencies, add go1.25 to the test matrix, **goquery now requires Go version 1.24+**.
* **2025-04-11 (v1.10.3)** : Update `go.mod` dependencies, small optimization (thanks [@myxzlpltk](https://github.com/myxzlpltk)).
* **2025-02-13 (v1.10.2)** : Update `go.mod` dependencies, add go1.24 to the test matrix.
+111 -78
View File
@@ -1,89 +1,122 @@
version: "2"
run:
tests: true
output:
print-issued-lines: false
show-stats: false
formats:
text:
print-issued-lines: false
colors: true
linters:
enable-all: true
default: all
disable:
- lll
- gocyclo
- dupl
- gochecknoglobals
- funlen
- godox
- wsl
- gocognit
- nolintlint
- testpackage
- godot
- nestif
- paralleltest
- nlreturn
- cyclop
- gci
- gofumpt
- errorlint
- exhaustive
- wrapcheck
- stylecheck
- thelper
- nonamedreturns
- revive
- dupword
- exhaustruct
- varnamelen
- forcetypeassert
- ireturn
- maintidx
- govet
- testableexamples
- musttag
- prealloc
- dupl
- godoclint
- cyclop
- depguard
- goconst
- perfsprint
- dupword
- err113
- errname
- errorlint
- exhaustruct
- forbidigo
- forcetypeassert
- funlen
- gochecknoglobals
- gocognit
- gocritic
- gocyclo
- godot
- godox
- gomoddirectives
- ireturn
- lll
- maintidx
- mnd
- nakedret
- nestif
- nilnil
- nlreturn
- nolintlint
- nonamedreturns
- paralleltest
- perfsprint
- predeclared
- recvcheck
- tenv
- err113
linters-settings:
gocyclo:
min-complexity: 10
dupl:
threshold: 100
goconst:
min-len: 8
min-occurrences: 3
forbidigo:
#forbid:
# - (Must)?NewLexer$
exclude_godoc_examples: false
- revive
- testpackage
- varnamelen
- wastedassign
- whitespace
- wsl
- wsl_v5
- funcorder
- noinlineerr
- tagalign
- goconst
- gochecknoinits
- durationcheck
- embeddedstructfieldcheck
- wrapcheck
- gomodguard
settings:
dupl:
threshold: 100
exhaustive:
default-signifies-exhaustive: true
goconst:
min-len: 8
min-occurrences: 3
gocyclo:
min-complexity: 10
wrapcheck:
report-internal-errors: false
ignore-package-globs:
- github.com/alecthomas/errors
exclusions:
generated: lax
rules:
- path: (.+)\.go$
text: "^(G104|G204|G307|G304):"
- path: (.+)\.go$
text: Error return value of .(.*\.Help|.*\.MarkFlagRequired|(os\.)?std(out|err)\..*|.*Close|.*Flush|os\.Remove(All)?|.*printf?|os\.(Un)?Setenv). is not checked
- path: (.+)\.go$
text: exported method `(.*\.MarshalJSON|.*\.UnmarshalJSON|.*\.EntityURN|.*\.GoString|.*\.Pos)` should have comment or be unexported
- path: (.+)\.go$
text: uses unkeyed fields
- path: (.+)\.go$
text: declaration of "err" shadows declaration
- path: (.+)\.go$
text: bad syntax for struct tag key
- path: (.+)\.go$
text: bad syntax for struct tag pair
- path: (.+)\.go$
text: ^ST1012
- path: (.+)\.go$
text: log/slog.Logger.*must not be called
- path: (.+)_test\.go$
text: error returned from external package is unwrapped
- linters: [staticcheck]
text: QF1008
- text: "Error return value of `.*.Write` is not checked"
linters: [errcheck]
path: (.+)_test\.go$
paths:
- third_party$
- builtin$
- examples$
issues:
exclude-dirs:
- _examples
max-per-linter: 0
max-same: 0
exclude-use-default: false
exclude:
# Captured by errcheck.
- '^(G104|G204):'
# Very commonly not checked.
- 'Error return value of .(.*\.Help|.*\.MarkFlagRequired|(os\.)?std(out|err)\..*|.*Close|.*Flush|os\.Remove(All)?|.*printf?|os\.(Un)?Setenv). is not checked'
- 'exported method (.*\.MarshalJSON|.*\.UnmarshalJSON|.*\.EntityURN|.*\.GoString|.*\.Pos) should have comment or be unexported'
- 'composite literal uses unkeyed fields'
- 'declaration of "err" shadows declaration'
- 'should not use dot imports'
- 'Potential file inclusion via variable'
- 'should have comment or be unexported'
- 'comment on exported var .* should be of the form'
- 'at least one file in a package should have a package comment'
- 'string literal contains the Unicode'
- 'methods on the same type should have the same receiver name'
- '_TokenType_name should be _TokenTypeName'
- '`_TokenType_map` should be `_TokenTypeMap`'
- 'rewrite if-else to switch statement'
max-issues-per-linter: 0
max-same-issues: 0
formatters:
enable:
- gofmt
- goimports
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
+11
View File
@@ -0,0 +1,11 @@
Chroma is a syntax highlighting library, tool and web playground for Go. It is based on Pygments and includes importers for it, so most of the same concepts from Pygments apply to Chroma.
This project is written in Go, uses Hermit to manage tooling, and Just for helper commands. Helper tooling is primarily in ./_tools.
Language definitions are XML files defined in ./lexers/embedded/*.xml.
Styles/themes are defined in ./styles/*.xml.
The CLI can be run with `chroma`.
The web playground can be run with `chromad --csrf-key=moo`. It blocks, so should generally be run in the background. It also does not hot reload, so has to be manually restarted. The playground has two modes - for local development it uses the server itself to render, while for production running `just chromad` will compile ./cmd/libchromawasm into a WASM module that is bundled into `chromad`.
+93
View File
@@ -0,0 +1,93 @@
let version = exec("git describe --tags --dirty --always") | trim
# TinyGo's installation root; used to source `wasm_exec.js`.
let tinygoroot = exec("tinygo env TINYGOROOT") | trim
# Generate tokentype_enumer.go from types.go via `//go:generate`.
tokentype = go.generate {
package = "."
inputs = ["types.go"]
outputs = ["tokentype_enumer.go"]
}
# Regenerate the lexer table in README.md by invoking the host `chroma` binary.
# GOOS/GOARCH are cleared so cross-compile env vars don't break the local run.
protected readme = exec {
command = "./table.py"
inputs = ["table.py", "lexers/**/*.go", "lexers/**/*.xml"]
output = "README.md"
}
# Format frontend JS sources in place. Runs as a sub-step of `index-min-js`,
# so bundling always sees formatted sources.
format-js = exec {
command = "biome format --write cmd/chromad/static/index.js cmd/chromad/static/chroma.js"
inputs = ["biome.js", "cmd/chromad/static/index.js", "cmd/chromad/static/chroma.js"]
}
# Copy TinyGo's wasm_exec.js into the chromad static assets.
wasm-exec = exec {
command = "install -m644 '#{tinygoroot}/targets/wasm_exec.js' cmd/chromad/static/wasm_exec.js"
resolve = "sha256 '#{tinygoroot}/targets/wasm_exec.js'"
output = "cmd/chromad/static/wasm_exec.js"
}
# Build the chroma WASM module via tinygo (installed via hermit) for the
# smaller output binary.
chroma-wasm = exec {
command = "tinygo build -no-debug -target wasm -o cmd/chromad/static/chroma.wasm cmd/libchromawasm/main.go"
inputs = ["cmd/libchromawasm/**/*.go", "*.go", "lexers/**/*.go", "lexers/**/*.xml", "formatters/**/*.go", "styles/**/*.go"]
output = "cmd/chromad/static/chroma.wasm"
}
# Bundle and minify the frontend JS. Depends on `format-js` so the bundle
# always reflects formatted sources.
index-min-js = exec {
command = "esbuild --platform=browser --format=esm --bundle cmd/chromad/static/index.js --minify --external:./wasm_exec.js --outfile=cmd/chromad/static/index.min.js"
inputs = ["cmd/chromad/static/index.js", "cmd/chromad/static/chroma.js"]
output = "cmd/chromad/static/index.min.js"
depends_on = [format-js]
}
# Bundle and minify the frontend CSS.
index-min-css = exec {
command = "esbuild --bundle cmd/chromad/static/index.css --minify --outfile=cmd/chromad/static/index.min.css"
inputs = ["cmd/chromad/static/index.css", "cmd/chromad/static/bulma.css"]
output = "cmd/chromad/static/index.min.css"
}
# Build the chromad server binary. cmd/chromad is a separate Go module, so
# `dir` puts the build in there and `package = "."` resolves against that
# module. `output` stays project-root-relative; bit absolutises it before
# passing to `go build -o`. Defaults to linux/amd64 to match the deploy
# target; override with GOOS/GOARCH env vars for local builds.
chromad = go.exe {
dir = "cmd/chromad"
package = "."
output = "build/chromad"
flags = ["-ldflags", "-X 'main.version=#{version}'"]
goos = env("GOOS", "linux")
goarch = env("GOARCH", "amd64")
cgo = false
depends_on = [wasm-exec, chroma-wasm, index-min-js, index-min-css, test]
}
pre format-go = go.fmt {
package = "./..."
}
# Run Go tests.
test = go.test {
package = "./..."
}
# Deploy chromad to swapoff.org. Must be explicitly selected.
explicit upload = exec {
command = <<-EOF
scp #{chromad.path} root@swapoff.org:
ssh root@swapoff.org 'install -m755 ./chromad /srv/http/swapoff.org/bin && service chromad restart'
EOF
depends_on = [chromad]
}
target default = [test, chromad, readme, tokentype]
-24
View File
@@ -1,24 +0,0 @@
VERSION = %(git describe --tags --dirty --always)%
export CGOENABLED = 0
tokentype_enumer.go: types.go
build: go generate
# Regenerate the list of lexers in the README
README.md: lexers/*.go lexers/*/*.xml table.py
build: ./table.py
-clean
implicit %{1}%{2}.min.%{3}: **/*.{css,js}
build: esbuild --bundle %{IN} --minify --outfile=%{OUT}
implicit build/%{1}: cmd/*
cd cmd/%{1}
inputs: cmd/%{1}/**/* **/*.go
build: go build -ldflags="-X 'main.version=%{VERSION}'" -o ../../build/%{1} .
#upload: chromad
# build:
# scp chromad root@swapoff.org:
# ssh root@swapoff.org 'install -m755 ./chromad /srv/http/swapoff.org/bin && service chromad restart'
# touch upload
+99
View File
@@ -17,3 +17,102 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
// formatters/svg/font_liberation_mono.go
Digitized data copyright (c) 2010 Google Corporation
with Reserved Font Arimo, Tinos and Cousine.
Copyright (c) 2012 Red Hat, Inc.
with Reserved Font Name Liberation.
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://openfontlicense.org
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.
+4 -5
View File
@@ -1,13 +1,12 @@
# Multi-stage Dockerfile for chromad Go application using Hermit-managed tools
# Build stage
FROM ubuntu:24.04 AS builder
FROM ubuntu:26.04 AS builder
# Install system dependencies
RUN apt-get update && apt-get install -y \
curl \
git \
make \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
@@ -25,11 +24,11 @@ ENV CGO_ENABLED=0
ENV GOOS=linux
ENV GOARCH=amd64
# Build the application using make
RUN make build/chromad
# Build the application using just
RUN just chromad
# Runtime stage
FROM alpine:3.23 AS runtime
FROM alpine:3.24 AS runtime
# Install ca-certificates for HTTPS requests
RUN apk --no-cache add ca-certificates curl
+59
View File
@@ -0,0 +1,59 @@
set positional-arguments
set shell := ["bash", "-c"]
version := `git describe --tags --dirty --always`
export GOOS := env("GOOS", "linux")
export GOARCH := env("GOARCH", "amd64")
_help:
@just -l
# Generate README.md from lexer definitions
readme:
#!/usr/bin/env bash
GOOS= GOARCH= ./table.py
# Generate tokentype_string.go
tokentype-string:
go generate
# Format JavaScript files
format-js:
biome format --write cmd/chromad/static/index.js cmd/chromad/static/chroma.js
# Tidy Go modules
tidy:
find . -name 'go.mod' -execdir go mod tidy \;
# Build chromad binary
chromad: wasm-exec chroma-wasm
#!/usr/bin/env bash
rm -rf build
mk cmd/chromad/static/index.min.js : cmd/chromad/static/{index,chroma}.js -- \
esbuild --platform=browser --format=esm --bundle cmd/chromad/static/index.js --minify --external:./wasm_exec.js --outfile=cmd/chromad/static/index.min.js
mk cmd/chromad/static/index.min.css : cmd/chromad/static/index.css -- \
esbuild --bundle cmd/chromad/static/index.css --minify --outfile=cmd/chromad/static/index.min.css
cd cmd/chromad && CGOENABLED=0 go build -ldflags="-X 'main.version={{ version }}'" -o ../../build/chromad .
# Copy wasm_exec.js from TinyGo
wasm-exec:
#!/usr/bin/env bash
tinygoroot=$(tinygo env TINYGOROOT)
mk cmd/chromad/static/wasm_exec.js : "$tinygoroot/targets/wasm_exec.js" -- \
install -m644 "$tinygoroot/targets/wasm_exec.js" cmd/chromad/static/wasm_exec.js
# Build WASM binary
chroma-wasm:
#!/usr/bin/env bash
if type tinygo > /dev/null 2>&1; then
mk cmd/chromad/static/chroma.wasm : cmd/libchromawasm/main.go -- \
tinygo build -no-debug -target wasm -o cmd/chromad/static/chroma.wasm cmd/libchromawasm/main.go
else
mk cmd/chromad/static/chroma.wasm : cmd/libchromawasm/main.go -- \
GOOS=js GOARCH=wasm go build -o cmd/chromad/static/chroma.wasm cmd/libchromawasm/main.go
fi
# Upload chromad to server
upload: chromad
scp build/chromad root@swapoff.org:
ssh root@swapoff.org 'install -m755 ./chromad /srv/http/swapoff.org/bin && service chromad restart'
-42
View File
@@ -1,42 +0,0 @@
.PHONY: chromad upload all
VERSION ?= $(shell git describe --tags --dirty --always)
export GOOS ?= linux
export GOARCH ?= amd64
all: README.md tokentype_string.go
README.md: lexers/*.go lexers/embedded/*.xml
GOOS= GOARCH= ./table.py
tokentype_string.go: types.go
go generate
.PHONY: format-js
format-js:
biome format --write cmd/chromad/static/{index.js,chroma.js}
.PHONY: chromad
chromad: build/chromad
build/chromad: $(shell find cmd/chromad -name '*.go' -o -name '*.html' -o -name '*.css' -o -name '*.js') \
cmd/chromad/static/wasm_exec.js \
cmd/chromad/static/chroma.wasm
rm -rf build
esbuild --platform=node --bundle cmd/chromad/static/index.js --minify --outfile=cmd/chromad/static/index.min.js
esbuild --bundle cmd/chromad/static/index.css --minify --outfile=cmd/chromad/static/index.min.css
(export CGOENABLED=0 ; go build -C cmd/chromad -ldflags="-X 'main.version=$(VERSION)'" -o ../../build/chromad .)
cmd/chromad/static/wasm_exec.js: $(shell tinygo env TINYGOROOT)/targets/wasm_exec.js
install -m644 $< $@
cmd/chromad/static/chroma.wasm: $(shell git ls-files | grep '\.go|\.xml')
if type tinygo > /dev/null; then \
tinygo build -no-debug -target wasm -o $@ cmd/libchromawasm/main.go; \
else \
GOOS=js GOARCH=wasm go build -o $@ cmd/libchromawasm/main.go; \
fi
upload: build/chromad
scp build/chromad root@swapoff.org: && \
ssh root@swapoff.org 'install -m755 ./chromad /srv/http/swapoff.org/bin && service chromad restart'

Some files were not shown because too many files have changed in this diff Show More