Compare commits

...
260 Commits
Author SHA1 Message Date
Umputun 307e69e5c1 Bump dependencies
- chroma/v2: v2.20.0 → v2.21.1
- go-pkgz/auth/v2: v2.1.0 → v2.1.1
- go-pkgz/rest: v1.20.4 → v1.20.6
- golang.org/x/* packages to latest

Also exclude "meaningless package names" revive warning in linter config.
2025-12-24 01:48:14 -06:00
Dmitry VerkhoturovandUmputun d5b07d7670 Fix WriteHeader + RenderJSON causing wrong Content-Type header
Replace WriteHeader() + RenderJSON() pattern with EncodeJSON() which
properly sets Content-Type header before writing status code. The
previous pattern caused Content-Type to default to text/plain instead
of application/json, breaking frontend JSON parsing.

Fixes #1979
2025-12-16 13:01:19 -06:00
Dmitry VerkhoturovandGitHub 41b75eba08 Merge pull request #1977 from umputun/docker-native-arm64-runners
Improve GitHub Actions workflows security and performance
2025-12-08 14:46:13 -06:00
UmputunandGitHub 4aa8362de1 Merge pull request #1976 from umputun/ci-native-arm64-runners
Migrate Docker builds to native GitHub ARM64 runners
2025-12-07 16:01:57 -06:00
Umputun dc168f69bf Fix dual-registry Docker builds and drop armv7
- Remove armv7 platform (eliminates QEMU emulation bottleneck)
- Build separately to ghcr.io and DockerHub with distinct digests
- Create registry-specific manifests from corresponding digests
- Update expected digest count from 3 to 2 per registry
2025-12-07 15:58:35 -06:00
Dmitry Verkhoturov 2a4a1591fa Migrate Docker builds to native GitHub ARM64 runners
Replace QEMU emulation with GitHub's native ARM64 runners for faster builds:
- Use ubuntu-24.04-arm for ARM builds instead of QEMU emulation
- Split build job into matrix for parallel platform builds
- Add digest-based workflow for multi-arch manifest creation
- Keep build-test on ARM64 runner for faster PR verification
2025-12-07 21:39:59 +00:00
Dmitry VerkhoturovandUmputun bc612ddf81 Remove redundant frame ancestors log from middleware
The log message was printed on every request when allowed ancestors
were configured, creating unnecessary noise in the logs.
2025-12-04 11:15:06 -06:00
Dmitry VerkhoturovandUmputun 9132a6158b Add display name format to EMAIL_FROM examples in docs
Show users they can use "Display Name"<email@example.com> format
for AUTH_EMAIL_FROM and NOTIFY_EMAIL_FROM settings.
2025-12-04 11:14:32 -06:00
Dmitry VerkhoturovandUmputun 658bf307b1 Add mailto: to CSP frame-src directive
Allow mailto links on the deleteme page to work without being blocked
by Content Security Policy.
2025-12-04 11:13:50 -06:00
Fredrik AppelrosandUmputun c6efcc56a9 Fix deleteme feature
The last step of the data deletion feature–when the admin user visits the link sent by the user requesting to delete their data–was broken due to the deleteme.js script not being loaded.
2025-12-04 11:13:50 -06:00
dependabot[bot]andUmputun bdee00b662 Bump nanoid from 3.3.7 to 3.3.8 in /site
Bumps [nanoid](https://github.com/ai/nanoid) from 3.3.7 to 3.3.8.
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/3.3.7...3.3.8)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-04 11:13:02 -06:00
kumudiaorongandUmputun 3013d03be2 fix: Modify the example to avoid misunderstanding 2025-12-04 11:12:32 -06:00
4a2bb5eda8 #1833 - Toolbar buttons are stuck to the main comment form (#1948)
* 1833 - Toolbar buttons are stuck to the main comment form

* Add readonly and JSDoc to CommentForm textareaId properties

Improve code quality based on review feedback: mark textareaId as
readonly since it should never change after construction, and add
JSDoc to static textareaCounter explaining its purpose.

---------

Co-authored-by: Dmitry Verkhoturov <paskal.07@gmail.com>
2025-12-04 11:12:05 -06:00
d01b738741 Implement function to prune string keeping HTML closing tags (#1870)
* Implement function to prune string keeping HTML closing tags

Fixes #1587

* change const name

remove unneeded comment

* move pruneHTML to separated file

* move const back to telegram.go

* Add unit tests for string array manipulation and HTML pruning

Introduce comprehensive test cases for stringArr methods (Push, Pop, Unshift, Shift, String) to ensure correct behavior and state management. Additionally, add tests for HTML pruning functions (pruneHTML, pruneStringToWord) to validate handling of length constraints and formatting scenarios.

* Improve behavior

* Fix pruneHTML to count visible text only, add parent text pruning

- Fix bug where HTML tags were counted toward the character limit
  instead of only visible text content
- Add pruning for parent comment text in Telegram notifications
- Simplify pruneStringToWord using strings.LastIndex
- Remove unused stringArr type and its tests
- Consolidate and simplify test cases

---------

Co-authored-by: Umputun <umputun@gmail.com>
Co-authored-by: Dmitry Verkhoturov <paskal.07@gmail.com>
2025-12-04 11:11:10 -06:00
Dmitry VerkhoturovandGitHub 564e8ff316 Update go dependencies (#1972) 2025-12-03 19:47:01 -06:00
dependabot[bot]andUmputun b451142790 Bump js-yaml from 3.14.1 to 3.14.2 in /site
Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 3.14.1 to 3.14.2.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/3.14.1...3.14.2)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 3.14.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 18:18:10 -06:00
dependabot[bot]andUmputun 6d8a0c783b Bump the go-modules-updates group across 1 directory with 11 updates
Bumps the go-modules-updates group with 11 updates in the /backend directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.2` | `1.11.0` |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.15.0` | `2.20.0` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.2.1` | `5.2.3` |
| [github.com/go-chi/cors](https://github.com/go-chi/cors) | `1.2.1` | `1.2.2` |
| [github.com/go-pkgz/jrpc](https://github.com/go-pkgz/jrpc) | `0.3.1` | `0.4.0` |
| [github.com/go-pkgz/lgr](https://github.com/go-pkgz/lgr) | `0.12.0` | `0.12.1` |
| [github.com/go-pkgz/rest](https://github.com/go-pkgz/rest) | `1.20.3` | `1.20.4` |
| [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) | `5.2.2` | `5.3.0` |
| [github.com/stretchr/testify](https://github.com/stretchr/testify) | `1.10.0` | `1.11.1` |
| [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt) | `1.4.0` | `1.4.3` |
| [golang.org/x/image](https://github.com/golang/image) | `0.26.0` | `0.33.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.10.2 to 1.11.0
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.2...v1.11.0)

Updates `github.com/alecthomas/chroma/v2` from 2.15.0 to 2.20.0
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.15.0...v2.20.0)

Updates `github.com/go-chi/chi/v5` from 5.2.1 to 5.2.3
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.1...v5.2.3)

Updates `github.com/go-chi/cors` from 1.2.1 to 1.2.2
- [Release notes](https://github.com/go-chi/cors/releases)
- [Commits](https://github.com/go-chi/cors/compare/v1.2.1...v1.2.2)

Updates `github.com/go-pkgz/jrpc` from 0.3.1 to 0.4.0
- [Release notes](https://github.com/go-pkgz/jrpc/releases)
- [Commits](https://github.com/go-pkgz/jrpc/compare/v0.3.1...v0.4.0)

Updates `github.com/go-pkgz/lgr` from 0.12.0 to 0.12.1
- [Release notes](https://github.com/go-pkgz/lgr/releases)
- [Commits](https://github.com/go-pkgz/lgr/compare/v0.12.0...v0.12.1)

Updates `github.com/go-pkgz/rest` from 1.20.3 to 1.20.4
- [Release notes](https://github.com/go-pkgz/rest/releases)
- [Commits](https://github.com/go-pkgz/rest/compare/v1.20.3...v1.20.4)

Updates `github.com/golang-jwt/jwt/v5` from 5.2.2 to 5.3.0
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.2...v5.3.0)

Updates `github.com/stretchr/testify` from 1.10.0 to 1.11.1
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.10.0...v1.11.1)

Updates `go.etcd.io/bbolt` from 1.4.0 to 1.4.3
- [Release notes](https://github.com/etcd-io/bbolt/releases)
- [Commits](https://github.com/etcd-io/bbolt/compare/v1.4.0...v1.4.3)

Updates `golang.org/x/image` from 0.26.0 to 0.33.0
- [Commits](https://github.com/golang/image/compare/v0.26.0...v0.33.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-version: 1.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-version: 2.20.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-version: 5.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/cors
  dependency-version: 1.2.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/jrpc
  dependency-version: 0.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/lgr
  dependency-version: 0.12.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/rest
  dependency-version: 1.20.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-version: 5.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/stretchr/testify
  dependency-version: 1.11.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: go.etcd.io/bbolt
  dependency-version: 1.4.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-version: 0.33.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 18:17:46 -06:00
notmalicikandGitHub d925584af8 Romanian language added (#1962) 2025-12-03 12:57:07 -06:00
dependabot[bot]andUmputun 22ee7a06d5 Bump playwright from 1.25.0 to 1.55.1 in /frontend
Bumps [playwright](https://github.com/microsoft/playwright) from 1.25.0 to 1.55.1.
- [Release notes](https://github.com/microsoft/playwright/releases)
- [Commits](https://github.com/microsoft/playwright/compare/v1.25.0...v1.55.1)

---
updated-dependencies:
- dependency-name: playwright
  dependency-version: 1.55.1
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:44:51 -06:00
dependabot[bot]andUmputun a311ff7b38 Bump the github-actions-updates group across 1 directory with 6 updates
Bumps the github-actions-updates group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [actions/checkout](https://github.com/actions/checkout) | `4` | `5` |
| [actions/setup-go](https://github.com/actions/setup-go) | `5` | `6` |
| [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) | `6` | `8` |
| [actions/setup-node](https://github.com/actions/setup-node) | `4` | `6` |
| [pnpm/action-setup](https://github.com/pnpm/action-setup) | `4.1.0` | `4.2.0` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `5` |



Updates `actions/checkout` from 4 to 5
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

Updates `actions/setup-go` from 5 to 6
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v6)

Updates `golangci/golangci-lint-action` from 6 to 8
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v6...v8)

Updates `actions/setup-node` from 4 to 6
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v6)

Updates `pnpm/action-setup` from 4.1.0 to 4.2.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v4.1.0...v4.2.0)

Updates `actions/upload-artifact` from 4 to 5
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/setup-go
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: golangci/golangci-lint-action
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/setup-node
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-version: 4.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:43:50 -06:00
ClaudeandUmputun 88257931ca Update Google and X (Twitter) logos to latest brand versions
- Update Google logo to use gradient version per new brand guidelines
- Replace Twitter bird logo with X logo
- Add light/dark variants for X logo (like Apple and GitHub)
- Update oauth.consts.ts to use new X logo variants

Fixes #1957
2025-12-03 11:43:25 -06:00
dependabot[bot]andUmputun 588ec169ff Bump golang.org/x/crypto from 0.37.0 to 0.45.0 in /backend
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.37.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.37.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:43:03 -06:00
dependabot[bot]andUmputun d9efa765d1 Bump golang.org/x/crypto in /backend/_example/memory_store
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.37.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.37.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-03 11:42:34 -06:00
Dmitry VerkhoturovandGitHub baf0db1947 Get rid of github.com/go-chi/render use (#1919)
Replace go-chi/render with go-pkgz/rest for JSON responses and custom
helpers for HTML/plain text responses.

Key changes:
- Replace render.JSON/render.Status with rest.RenderJSON and explicit
  w.WriteHeader() calls
- Replace render.DecodeJSON with json.NewDecoder().Decode()
- Add SendErrorJSON helper that sets Content-Type header before
  WriteHeader (required since rest.RenderJSON can't set headers after
  WriteHeader is called)
- Add HTMLResponse and PlainTextResponse helpers

Fix export double-execution in migrator.go:
The original code called Export twice - once to io.Discard to check for
errors, then again to actually write. This was wasteful and had a race
condition risk. Now file mode buffers to memory first for atomic
success/failure, while stream mode writes directly with proper error
handling.
2025-12-03 11:41:29 -06:00
UmputunandGitHub 34ea4c3e83 Migrate golangci-lint to v2 and update Go version (#1965)
* migrate golangci-lint to v2 and update go version

- migrated .golangci.yml to version 2 format
- updated go.mod from 1.23.0 to 1.24
- removed deprecated run.timeout configuration

* update to go 1.25 and baseimage v1.17.0

- updated go.mod to go 1.25
- updated Dockerfile to use buildgo-v1.17.0 (go 1.25.0)
- updated Dockerfile to use app-v1.17.0

* fix flaky tests with proper synchronization

- use assert.Eventually instead of fixed sleep in TestService_Many
- wait for webhook before shutdown in TestMain_WithWebhook
- fixes race conditions exposed by Go 1.25 scheduler changes

* fix data race in MockDest.closed field

- add IsClosed() method with proper locking
- add locking to String() method
- use IsClosed() in tests instead of direct field access
- fixes race condition detected by go test -race

* update example go.mod to go 1.25

* update golangci-lint to v2.6.0 for go 1.25 support

* fix linter issue and update CLAUDE.md

- merge conditional assignment in example accessor/data.go
- add reminder in CLAUDE.md to always test and lint examples before committing

* update example Dockerfile to baseimage v1.17.0 for go 1.25
2025-11-02 14:17:39 -06:00
UmputunandGitHub 8112f445f4 Merge pull request #1947 from schnerring/add-backup-docs
docs: improve backup instructions; closes #1613
2025-10-01 02:02:46 -05:00
UmputunandGitHub bd7bbe629f Merge pull request #1953 from diosfera/master
Add Macedonian translation
2025-10-01 02:01:43 -05:00
diosfera ed8b3c314d Delete package-lock.json file 2025-09-21 20:07:12 +02:00
diosfera 51ec396691 Add Macedonian translation 2025-09-17 06:29:51 +02:00
Michael Schnerring ca8ab66812 docs: improve backup instructions; closes #1613 2025-09-04 22:20:18 +02:00
UmputunandGitHub eaa64bac45 Merge pull request #1929 from umputun/fix/auth-send-jwt-header
Fix login persistence with AUTH_SEND_JWT_HEADER enabled
2025-07-06 17:55:09 -05:00
UmputunandGitHub abdca907a7 Merge branch 'master' into fix/auth-send-jwt-header 2025-07-06 15:39:12 -05:00
UmputunandGitHub a9c8cf51a0 Merge pull request #1937 from umputun/docs/telegram-group-notifications 2025-06-16 00:43:37 -05:00
Dmitry Verkhoturov d829a57061 docs: update Telegram configuration with group notification details
Add information about sending notifications to users, groups, and channels.
Document that public group usernames can be used without @ symbol as IDs.

Addresses discussion #1756
2025-06-16 05:19:50 +01:00
UmputunandGitHub 1dffb2f16e Merge pull request #1936 from talentedunicorn/patch-1
Fixed typo
2025-06-06 12:12:56 -05:00
EzeandGitHub 9e2a1da0df Fixed typo 2025-06-07 01:11:29 +08:00
UmputunandGitHub 6fbaa806f0 Merge pull request #1934 from up9cloud/master
Fix typo (rootDissapear should be rootDisappear)
2025-06-06 10:46:29 -05:00
HsüanandGitHub 58acca4dcb Fix profile.spec.tsx typo 2025-06-04 15:18:46 -07:00
HsüanandGitHub 22d21df22b Fix profile.ts typo 2025-06-04 15:17:40 -07:00
Dmitry Verkhoturov ddb490bbc1 Fix login persistence with AUTH_SEND_JWT_HEADER enabled
With AUTH_SEND_JWT_HEADER=true, frontend now properly handles JWT authentication:
- Store JWT token in client-side cookie named 'JWT'
- Extract and store XSRF token from JWT payload
- Set Secure flag automatically when on HTTPS connection
- Update documentation to clarify this behavior

This fixes an issue where login state would be lost after page reload
when using header-based JWT authentication.
2025-04-29 23:45:55 +01:00
UmputunandGitHub 242499787e Merge pull request #1930 from umputun/paskal/update-gopkgz
Update go-pkgz, system modules
2025-04-29 02:31:12 -05:00
Dmitry Verkhoturov fd0799384f Update go-pkgz, system modules
This brings stricter check for auth provider names, slog support.
2025-04-29 08:24:53 +01:00
UmputunandGitHub 61dbf6b1f2 Merge pull request #1915 from umputun/dependabot/go_modules/backend/github.com/redis/go-redis/v9-9.7.3
Bump github.com/redis/go-redis/v9 from 9.7.0 to 9.7.3 in /backend
2025-03-24 21:51:56 -05:00
UmputunandGitHub 4a3c73db31 Merge pull request #1911 from umputun/dependabot/npm_and_yarn/site/prismjs-1.30.0
Bump prismjs from 1.29.0 to 1.30.0 in /site
2025-03-24 21:51:02 -05:00
UmputunandGitHub df510360e6 Merge pull request #1909 from umputun/dependabot/github_actions/github-actions-updates-e0e9667351
Bump pnpm/action-setup from 4.0.0 to 4.1.0 in the github-actions-updates group
2025-03-24 21:50:47 -05:00
UmputunandGitHub a3309516c3 Merge pull request #1916 from umputun/dependabot/go_modules/backend/github.com/golang-jwt/jwt/v5-5.2.2
Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 in /backend
2025-03-24 21:50:33 -05:00
UmputunandGitHub 5e30dc86e1 Merge pull request #1917 from umputun/paskal/golangci_lint_v2
Automatic fix of errors reported by golangci-lint v2
2025-03-24 21:50:20 -05:00
dependabot[bot]andGitHub bbcba5487e Bump github.com/redis/go-redis/v9 from 9.7.0 to 9.7.3 in /backend
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.7.0 to 9.7.3.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/CHANGELOG.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.7.0...v9.7.3)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-25 02:49:28 +00:00
UmputunandGitHub c32e5efdc2 Merge pull request #1918 from umputun/paskal/add_systemd
Add install instructions for setting up Remark42 as a systemd service
2025-03-24 21:48:27 -05:00
UmputunandGitHub af139a7f4c Merge pull request #1910 from umputun/dependabot/go_modules/backend/go-modules-updates-e61953c257
Bump the go-modules-updates group in /backend with 8 updates
2025-03-24 21:48:05 -05:00
dependabot[bot]andDmitry V 89221ff2bc Bump the go-modules-updates group in /backend with 8 updates
Bumps the go-modules-updates group in /backend with 8 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.1` | `1.10.2` |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.14.0` | `2.15.0` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.2.0` | `5.2.1` |
| [github.com/go-pkgz/jrpc](https://github.com/go-pkgz/jrpc) | `0.3.0` | `0.3.1` |
| [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt) | `1.3.11` | `1.4.0` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.31.0` | `0.33.0` |
| [golang.org/x/image](https://github.com/golang/image) | `0.23.0` | `0.25.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.33.0` | `0.35.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.10.1 to 1.10.2
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.1...v1.10.2)

Updates `github.com/alecthomas/chroma/v2` from 2.14.0 to 2.15.0
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Changelog](https://github.com/alecthomas/chroma/blob/master/.goreleaser.yml)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.14.0...v2.15.0)

Updates `github.com/go-chi/chi/v5` from 5.2.0 to 5.2.1
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.2.0...v5.2.1)

Updates `github.com/go-pkgz/jrpc` from 0.3.0 to 0.3.1
- [Release notes](https://github.com/go-pkgz/jrpc/releases)
- [Commits](https://github.com/go-pkgz/jrpc/compare/v0.3.0...v0.3.1)

Updates `go.etcd.io/bbolt` from 1.3.11 to 1.4.0
- [Release notes](https://github.com/etcd-io/bbolt/releases)
- [Commits](https://github.com/etcd-io/bbolt/compare/v1.3.11...v1.4.0)

Updates `golang.org/x/crypto` from 0.31.0 to 0.33.0
- [Commits](https://github.com/golang/crypto/compare/v0.31.0...v0.33.0)

Updates `golang.org/x/image` from 0.23.0 to 0.25.0
- [Commits](https://github.com/golang/image/compare/v0.23.0...v0.25.0)

Updates `golang.org/x/net` from 0.33.0 to 0.35.0
- [Commits](https://github.com/golang/net/compare/v0.33.0...v0.35.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/jrpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: go.etcd.io/bbolt
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-24 23:13:46 +01:00
Dmitry Verkhoturov 528242c1ca Add install instructions for setting up Remark42 as a systemd service 2025-03-24 23:09:28 +01:00
Dmitry Verkhoturov edfc5b9d76 Automatic fix of errors reported by golangci-lint v2
- Use strings.ReplaceAll
- Remove redundant internal structure names
2025-03-24 22:46:43 +01:00
dependabot[bot]andGitHub 7e944bfe0d Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 in /backend
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) from 5.2.1 to 5.2.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-21 22:26:36 +00:00
UmputunandGitHub e6afc58b34 Merge pull request #1914 from umputun/paskal/add_system_requirements 2025-03-16 14:28:22 -05:00
Dmitry Verkhoturov f49b878eb4 Add system requirements section to installation guide 2025-03-16 19:26:42 +00:00
UmputunandGitHub 5fe843de71 Merge pull request #1912 from umputun/dependabot/go_modules/backend/golang.org/x/net-0.36.0
Bump golang.org/x/net from 0.33.0 to 0.36.0 in /backend
2025-03-12 20:30:34 -05:00
dependabot[bot]andDmitry Verkhoturov 6c9ade9062 Bump golang.org/x/net from 0.33.0 to 0.36.0 in /backend
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.33.0 to 0.36.0.
- [Commits](https://github.com/golang/net/compare/v0.33.0...v0.36.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-12 22:58:15 +00:00
dependabot[bot]andGitHub ed3c104ba4 Bump prismjs from 1.29.0 to 1.30.0 in /site
Bumps [prismjs](https://github.com/PrismJS/prism) from 1.29.0 to 1.30.0.
- [Release notes](https://github.com/PrismJS/prism/releases)
- [Changelog](https://github.com/PrismJS/prism/blob/master/CHANGELOG.md)
- [Commits](https://github.com/PrismJS/prism/compare/v1.29.0...v1.30.0)

---
updated-dependencies:
- dependency-name: prismjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 22:24:16 +00:00
dependabot[bot]andGitHub a90b4296c6 Bump pnpm/action-setup in the github-actions-updates group
Bumps the github-actions-updates group with 1 update: [pnpm/action-setup](https://github.com/pnpm/action-setup).


Updates `pnpm/action-setup` from 4.0.0 to 4.1.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v4.0.0...v4.1.0)

---
updated-dependencies:
- dependency-name: pnpm/action-setup
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-09 18:01:26 +00:00
UmputunandGitHub 0ebe893125 Merge pull request #1908 from umputun/paskal/disable_npm_updates
Disable dependabot automatic PRs for NPM modules
2025-03-09 13:00:21 -05:00
Dmitry Verkhoturov e32ca020c0 Disable dependabot automatic PRs for NPM modules
These have to be updated manually
because there are always breaking changes.
2025-03-09 17:17:20 +01:00
UmputunandGitHub 558350c546 Merge pull request #1897 from umputun/paskal/claude
Add first version of the CLAUDE.md
2025-03-01 17:19:34 -06:00
UmputunandGitHub c50a5d6245 Merge pull request #1907 from umputun/paskal/get_rid_of_dockerhub
Migrate Docker images from Docker Hub to GitHub Container Registry
2025-03-01 17:19:01 -06:00
Dmitry Verkhoturov bb650d7526 Migrate Docker images from Docker Hub to GitHub Container Registry
This commit replaces all references to `umputun/remark42` Docker images
on Docker Hub with `ghcr.io/umputun/remark42` from the GitHub
Container Registry. It updates various Docker Compose files,
documentation, and the Makefile to use the new image location.
It also updates the kubernetes example to use the latest version.

Docker Hub is going to kill free pulls for too long by now.
2025-03-01 23:15:09 +00:00
Dmitry Verkhoturov 551212db82 Add first version of the CLAUDE.md 2025-02-25 22:05:25 +00:00
dependabot[bot]andUmputun 2e00002413 Bump the go-modules-updates group in /backend with 5 updates
Bumps the go-modules-updates group in /backend with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.0` | `1.10.1` |
| [github.com/go-chi/chi/v5](https://github.com/go-chi/chi) | `5.1.0` | `5.2.0` |
| [github.com/go-pkgz/rest](https://github.com/go-pkgz/rest) | `1.19.0` | `1.20.2` |
| [golang.org/x/image](https://github.com/golang/image) | `0.22.0` | `0.23.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.31.0` | `0.33.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.10.0 to 1.10.1
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.0...v1.10.1)

Updates `github.com/go-chi/chi/v5` from 5.1.0 to 5.2.0
- [Release notes](https://github.com/go-chi/chi/releases)
- [Changelog](https://github.com/go-chi/chi/blob/master/CHANGELOG.md)
- [Commits](https://github.com/go-chi/chi/compare/v5.1.0...v5.2.0)

Updates `github.com/go-pkgz/rest` from 1.19.0 to 1.20.2
- [Release notes](https://github.com/go-pkgz/rest/releases)
- [Commits](https://github.com/go-pkgz/rest/compare/v1.19.0...v1.20.2)

Updates `golang.org/x/image` from 0.22.0 to 0.23.0
- [Commits](https://github.com/golang/image/compare/v0.22.0...v0.23.0)

Updates `golang.org/x/net` from 0.31.0 to 0.33.0
- [Commits](https://github.com/golang/net/compare/v0.31.0...v0.33.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: github.com/go-chi/chi/v5
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/rest
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-02 14:02:42 -06:00
UmputunandGitHub 81f70aa287 Merge pull request #1875 from codiflow/codiflow-patch-1 2024-12-28 20:44:53 -06:00
Christian 🦄andDmitry Verkhoturov f104e3c775 Update de.json
Updated Telegram bot description
2024-12-29 03:43:57 +01:00
UmputunandGitHub be076d03e9 Merge pull request #1878 from cubismod/webhook-params
Docs update describing available variables for webhook templating
2024-12-21 11:26:57 -06:00
Ryan e1166a48cf add a table describing additional variables available for use with webhook templating 2024-12-20 21:08:15 -06:00
Umputun 3f0789fd90 mod tidy for memory store example 2024-12-17 23:39:29 -06:00
UmputunandGitHub 518ae79556 Merge pull request #1869 from aliksend/aliksend/fix-importing-anonymous-comments-from-commento
Straightforward fix for importing anonymous comments from commento
2024-12-17 16:04:38 -06:00
UmputunandGitHub 3c55238bdd Merge pull request #1874 from umputun/dependabot/go_modules/backend/golang.org/x/crypto-0.31.0
Bump golang.org/x/crypto from 0.29.0 to 0.31.0 in /backend
2024-12-17 13:55:45 -06:00
Alik Send 556b95a655 Fix tests. Add test for anonymous comment 2024-12-16 23:16:26 -06:00
Alik Send bfef15f05f Straightforward fix for importing anonymous comments from commento
Fixes #1821
2024-12-16 23:16:26 -06:00
UmputunandGitHub c3ba55ba43 Merge pull request #1699 from umputun/paskal/comments_pagination
add pagination to GET /api/v1/find endpoint
2024-12-11 20:17:23 -06:00
dependabot[bot]andGitHub 0af8b2eab0 Bump golang.org/x/crypto from 0.29.0 to 0.31.0 in /backend
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.29.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.29.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-11 22:23:01 +00:00
Dmitry Verkhoturov 82a0888c42 add pagination to GET /api/v1/find endpoint
`format=tree` pagination provides top-level comments with all replies
and returns the last top-level comment as `last_comment` to be used
as `offset` for the next page. If comments and replies overflow
the limit, the one stepping out of the limit will not be returned.
 If the first comment and its replies after the given offset overflow
 the limit, it will be returned with all the replies.

`format=plain` pagination works by providing all comments and returning
the last comment as `last_comment` to be used as `offset`
for the next page.
2024-12-10 14:26:55 +00:00
UmputunandGitHub d5162d3fe6 Merge pull request #1758 from umputun/paskal/jwt_v5
Update to go-pkgz/auth/v2 and golang-jwt/jwt/v5
2024-12-09 19:53:00 -06:00
Dmitry Verkhoturov e61a46efff Improve error message for checking claims.Audience 2024-12-10 00:45:04 +00:00
Dmitry Verkhoturov f473105c52 add tests for jwt5 multiple auds and improve existing tests 2024-12-09 01:54:04 +00:00
Dmitry Verkhoturov c2d386230c vendor new modules 2024-12-09 01:54:03 +00:00
Dmitry Verkhoturov e3ad01b555 migrate to go-pkgz/auth/v2 2024-12-09 01:54:03 +00:00
Dmitry Verkhoturov 8d9e55c33c update github.com/golang-jwt/jwt to v5 2024-12-09 01:54:03 +00:00
UmputunandGitHub 6402ef9cae Merge pull request #1871 from umputun/paskal/improve_tests
Improve tests
2024-12-08 19:53:15 -06:00
Dmitry Verkhoturov 4ed48dd85c Improve tests 2024-12-09 01:45:57 +00:00
UmputunandGitHub 9628312b5d Merge pull request #1865 from umputun/dependabot/go_modules/backend/go-modules-updates-81f599025a 2024-12-01 22:32:10 -06:00
UmputunandGitHub c65c2b395d Merge branch 'master' into dependabot/go_modules/backend/go-modules-updates-81f599025a 2024-12-01 22:31:58 -06:00
UmputunandGitHub 27671c50c4 Merge pull request #1866 from umputun/dependabot/npm_and_yarn/site/npm-modules-updates-for-tests-87ff55c30d 2024-12-01 22:31:40 -06:00
UmputunandGitHub 56ac1bb841 Merge branch 'master' into dependabot/npm_and_yarn/site/npm-modules-updates-for-tests-87ff55c30d 2024-12-01 22:31:28 -06:00
dependabot[bot]andDmitry Verkhoturov 0aadf0ba86 Bump the go-modules-updates group in /backend with 3 updates
Bumps the go-modules-updates group in /backend with 3 updates: [github.com/stretchr/testify](https://github.com/stretchr/testify), [golang.org/x/crypto](https://github.com/golang/crypto) and [golang.org/x/net](https://github.com/golang/net).

Updates `github.com/stretchr/testify` from 1.9.0 to 1.10.0
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.9.0...v1.10.0)

Updates `golang.org/x/crypto` from 0.27.0 to 0.29.0
- [Commits](https://github.com/golang/crypto/compare/v0.27.0...v0.29.0)

Updates `golang.org/x/net` from 0.29.0 to 0.31.0
- [Commits](https://github.com/golang/net/compare/v0.29.0...v0.31.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-02 04:24:48 +00:00
UmputunandGitHub 34c667b83a Merge pull request #1862 from umputun/paskal/improve_server_test 2024-12-01 22:24:11 -06:00
dependabot[bot]andDmitry Verkhoturov f2e5758b6c Bump the npm-modules-updates-for-tests group in /site with 2 updates
Bumps the npm-modules-updates-for-tests group in /site with 2 updates: [prettier](https://github.com/prettier/prettier) and [tailwindcss](https://github.com/tailwindlabs/tailwindcss).

Updates `prettier` from 3.3.3 to 3.4.1
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.3.3...3.4.1)

Updates `tailwindcss` from 3.4.14 to 3.4.15
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/v3.4.15/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/compare/v3.4.14...v3.4.15)

---
updated-dependencies:
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-modules-updates-for-tests
- dependency-name: tailwindcss
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-02 04:03:21 +00:00
Dmitry Verkhoturov cd7f616596 Fix server test configuration
Previously, tmp boltdb was always created in directory 8080.
2024-12-02 03:36:32 +00:00
UmputunandGitHub a0c412ee1e Merge pull request #1861 from umputun/paskal/example_dependabot 2024-12-01 21:33:33 -06:00
Dmitry VerkhoturovandGitHub 15f5b7dde5 Stop updating examples via dependabot
Examples should be updated alongside the backend directory and it makes no sense to have separate update PRs for it.
2024-12-02 03:04:46 +00:00
UmputunandGitHub a561588117 Merge pull request #1859 from umputun/dependabot/github_actions/github-actions-updates-d109cd9e8b
Bump codecov/codecov-action from 4 to 5 in the github-actions-updates group
2024-12-01 21:01:52 -06:00
dependabot[bot]andGitHub ac36dccd19 Bump codecov/codecov-action in the github-actions-updates group
Bumps the github-actions-updates group with 1 update: [codecov/codecov-action](https://github.com/codecov/codecov-action).


Updates `codecov/codecov-action` from 4 to 5
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-12-01 11:33:06 +00:00
UmputunandGitHub afdac27651 Merge pull request #1853 from umputun/paskal/docker-compose
Update docker-compose.yml with example for local reverse proxy
2024-11-19 18:59:10 -06:00
Dmitry VerkhoturovandGitHub 995c4963fb Update docker-compose.yml with example for local reverse proxy
In discussion #1852 I realised that we don't have example with reverse proxy running outside of docker, this commit fixes that.
2024-11-20 00:57:46 +00:00
UmputunandGitHub 16ff2690f0 Merge pull request #1824 from mgkbadola/discord-oauth
feat: Discord OAuth support
2024-11-18 14:57:31 -06:00
Mrigank BadolaandGitHub d0dc1131ea Merge branch 'master' into discord-oauth 2024-11-12 21:24:24 +05:30
Dmitry VerkhoturovandUmputun d5e3602e54 Fix fetch module import so that latest release version would be shown on website 2024-11-10 18:30:48 -06:00
Dmitry VerkhoturovandUmputun a4772bd9de Do not fail if frontend pre-commit hook fails 2024-11-10 18:12:43 -06:00
Dmitry VerkhoturovandUmputun d23b7003c6 Downgrade eleventy to fix the website 2024-11-10 18:05:51 -06:00
Dmitry VerkhoturovandUmputun 3646c4a871 Drop linux/arm/v7 docker image from site as it doesn't build well 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun 29fc63f116 Clarify steps names for site deployment 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun 8c59bab921 Add nvmrc to /site, update yarn.lock 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun c42511d5a1 Fix cache-dependency-path for actions/setup-go caching to work, fix CI files path 2024-11-10 17:25:16 -06:00
Dmitry VerkhoturovandUmputun 9b5f6ee2c5 Remove deprecated set-output function usage in GitHub actions
The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2024-11-10 17:25:16 -06:00
dependabot[bot]andUmputun b5579152cb Bump the npm-modules-updates-for-tests group across 1 directory with 12 updates
Bumps the npm-modules-updates-for-tests group with 12 updates in the /frontend/packages/api directory:

| Package | From | To |
| --- | --- | --- |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `18.19.64` | `18.19.64` |
| [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin) | `5.62.0` | `8.13.0` |
| [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) | `5.62.0` | `8.13.0` |
| [@vitest/coverage-c8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-c8) | `0.22.1` | `0.33.0` |
| [eslint-config-prettier](https://github.com/prettier/eslint-config-prettier) | `8.10.0` | `9.1.0` |
| [eslint-plugin-prettier](https://github.com/prettier/eslint-plugin-prettier) | `4.2.1` | `5.2.1` |
| [jsdom](https://github.com/jsdom/jsdom) | `20.0.3` | `25.0.1` |
| [msw](https://github.com/mswjs/msw) | `0.44.2` | `1.3.5` |
| [prettier](https://github.com/prettier/prettier) | `2.8.8` | `3.3.3` |
| [typescript](https://github.com/microsoft/TypeScript) | `4.7.4` | `4.9.5` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `0.22.1` | `2.1.4` |

Updates `@types/node` from 18.19.64 to 22.9.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `@typescript-eslint/eslint-plugin` from 5.62.0 to 8.13.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.13.0/packages/eslint-plugin)

Updates `@typescript-eslint/parser` from 5.62.0 to 8.13.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.13.0/packages/parser)

Updates `@vitest/coverage-c8` from 0.22.1 to 0.33.0
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v0.33.0/packages/coverage-c8)

Updates `eslint-config-prettier` from 8.10.0 to 9.1.0
- [Changelog](https://github.com/prettier/eslint-config-prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/eslint-config-prettier/compare/v8.10.0...v9.1.0)

Updates `eslint-plugin-prettier` from 4.2.1 to 5.2.1
- [Release notes](https://github.com/prettier/eslint-plugin-prettier/releases)
- [Changelog](https://github.com/prettier/eslint-plugin-prettier/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prettier/eslint-plugin-prettier/compare/v4.2.1...v5.2.1)

Updates `jsdom` from 20.0.3 to 25.0.1
- [Release notes](https://github.com/jsdom/jsdom/releases)
- [Changelog](https://github.com/jsdom/jsdom/blob/main/Changelog.md)
- [Commits](https://github.com/jsdom/jsdom/compare/20.0.3...25.0.1)

Updates `msw` from 0.44.2 to 2.6.3
- [Release notes](https://github.com/mswjs/msw/releases)
- [Changelog](https://github.com/mswjs/msw/blob/main/CHANGELOG.md)
- [Commits](https://github.com/mswjs/msw/compare/v0.44.2...v2.6.3)

Updates `prettier` from 2.8.8 to 3.3.3
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/2.8.8...3.3.3)

Updates `typescript` from 4.9.5 to 5.6.3
- [Release notes](https://github.com/microsoft/TypeScript/releases)
- [Changelog](https://github.com/microsoft/TypeScript/blob/main/azure-pipelines.release.yml)
- [Commits](https://github.com/microsoft/TypeScript/compare/v4.9.5...v5.6.3)

Updates `vitest` from 0.22.1 to 2.1.4
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Commits](https://github.com/vitest-dev/vitest/commits/v2.1.4/packages/vitest)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@typescript-eslint/eslint-plugin"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@typescript-eslint/parser"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@vitest/coverage-c8"
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-modules-updates-for-tests
- dependency-name: eslint
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: eslint-config-prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: eslint-plugin-prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: jsdom
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: msw
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: typescript
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: vitest
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-10 14:00:52 -06:00
dependabot[bot]andUmputun 3c78a54be6 Bump the npm-modules-updates-for-tests group
Bumps the npm-modules-updates-for-tests group in /frontend/e2e with 3 updates: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node), [nanoid](https://github.com/ai/nanoid) and [typescript](https://github.com/microsoft/TypeScript).

Updates `@types/node` from 18.19.64 to 22.9.0
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `nanoid` from 4.0.2 to 5.0.8
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/4.0.2...5.0.8)

Updates `typescript` from 4.9.5 to 5.6.3
- [Release notes](https://github.com/microsoft/TypeScript/releases)
- [Changelog](https://github.com/microsoft/TypeScript/blob/main/azure-pipelines.release.yml)
- [Commits](https://github.com/microsoft/TypeScript/compare/v4.9.5...v5.6.3)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: nanoid
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: typescript
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-10 13:55:30 -06:00
Dmitry VerkhoturovandUmputun 3001b37812 Update pnpm from v7 to v8
That suppose to help with Dependabot security updates not updating lockfile.
2024-11-10 13:31:03 -06:00
Dmitry VerkhoturovandUmputun 63048cc798 Specify package manager version in frontend project configuration
This is necessary for Dependabot updates to work properly.
2024-11-10 12:42:07 -06:00
Mrigank Badola 1a27913404 chore: update go-pkgz/auth package 2024-11-10 12:40:07 +05:30
Mrigank Badola 4a39ceee8d sync: Merge branch 'master' of https://github.com/mgkbadola/remark42 into discord-oauth 2024-11-10 10:43:10 +05:30
dependabot[bot]andUmputun 1e659917d2 Bump the npm-modules-updates-for-tests group in /site with 7 updates
Bumps the npm-modules-updates-for-tests group in /site with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [@11ty/eleventy](https://github.com/11ty/eleventy) | `2.0.1` | `3.0.0` |
| [@tailwindcss/typography](https://github.com/tailwindlabs/tailwindcss-typography) | `0.5.10` | `0.5.15` |
| [date-fns](https://github.com/date-fns/date-fns) | `3.3.1` | `4.1.0` |
| [markdown-it](https://github.com/markdown-it/markdown-it) | `14.0.0` | `14.1.0` |
| [markdown-it-anchor](https://github.com/valeriangalliat/markdown-it-anchor) | `8.6.7` | `9.2.0` |
| [prettier](https://github.com/prettier/prettier) | `2.8.8` | `3.3.3` |
| [tailwindcss](https://github.com/tailwindlabs/tailwindcss) | `3.4.1` | `3.4.14` |


Updates `@11ty/eleventy` from 2.0.1 to 3.0.0
- [Release notes](https://github.com/11ty/eleventy/releases)
- [Changelog](https://github.com/11ty/eleventy/blob/main/docs/release-instructions.md)
- [Commits](https://github.com/11ty/eleventy/compare/v2.0.1...v3.0.0)

Updates `@tailwindcss/typography` from 0.5.10 to 0.5.15
- [Release notes](https://github.com/tailwindlabs/tailwindcss-typography/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss-typography/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss-typography/compare/v0.5.10...v0.5.15)

Updates `date-fns` from 3.3.1 to 4.1.0
- [Release notes](https://github.com/date-fns/date-fns/releases)
- [Changelog](https://github.com/date-fns/date-fns/blob/main/CHANGELOG.md)
- [Commits](https://github.com/date-fns/date-fns/compare/v3.3.1...v4.1.0)

Updates `markdown-it` from 14.0.0 to 14.1.0
- [Changelog](https://github.com/markdown-it/markdown-it/blob/master/CHANGELOG.md)
- [Commits](https://github.com/markdown-it/markdown-it/compare/14.0.0...14.1.0)

Updates `markdown-it-anchor` from 8.6.7 to 9.2.0
- [Release notes](https://github.com/valeriangalliat/markdown-it-anchor/releases)
- [Changelog](https://github.com/valeriangalliat/markdown-it-anchor/blob/master/CHANGELOG.md)
- [Commits](https://github.com/valeriangalliat/markdown-it-anchor/compare/v8.6.7...v9.2.0)

Updates `prettier` from 2.8.8 to 3.3.3
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/2.8.8...3.3.3)

Updates `tailwindcss` from 3.4.1 to 3.4.14
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/v3.4.14/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/compare/v3.4.1...v3.4.14)

---
updated-dependencies:
- dependency-name: "@11ty/eleventy"
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: "@tailwindcss/typography"
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-modules-updates-for-tests
- dependency-name: date-fns
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: markdown-it
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm-modules-updates-for-tests
- dependency-name: markdown-it-anchor
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: prettier
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: npm-modules-updates-for-tests
- dependency-name: tailwindcss
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: npm-modules-updates-for-tests
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-08 17:54:33 -06:00
daniyalandUmputun 5c586e12bd feat: add "fa" locale 2024-11-08 16:38:09 -06:00
Dmitry VerkhoturovandUmputun 73ca4a1b6d Capitalize "AS" keyword in Dockerfile FROM instructions 2024-11-08 16:27:10 -06:00
Dmitry VerkhoturovandUmputun 7604548035 Update go to 1.23, golangci-lint to 1.61, baseimage to 1.14 2024-11-08 16:19:58 -06:00
dependabot[bot]andUmputun 7529aa7e17 Bump the go-modules-updates group in /backend with 9 updates
Bumps the go-modules-updates group in /backend with 9 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.9.2` | `1.10.0` |
| [github.com/alecthomas/chroma/v2](https://github.com/alecthomas/chroma) | `2.13.0` | `2.14.0` |
| [github.com/go-pkgz/notify](https://github.com/go-pkgz/notify) | `1.1.1` | `1.2.0` |
| [github.com/go-pkgz/repeater](https://github.com/go-pkgz/repeater) | `1.1.3` | `1.2.0` |
| [github.com/rs/xid](https://github.com/rs/xid) | `1.5.0` | `1.6.0` |
| [go.etcd.io/bbolt](https://github.com/etcd-io/bbolt) | `1.3.10` | `1.3.11` |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.25.0` | `0.27.0` |
| [golang.org/x/image](https://github.com/golang/image) | `0.18.0` | `0.22.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.27.0` | `0.29.0` |

Updates `github.com/PuerkitoBio/goquery` from 1.9.2 to 1.10.0
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.9.2...v1.10.0)

Updates `github.com/alecthomas/chroma/v2` from 2.13.0 to 2.14.0
- [Release notes](https://github.com/alecthomas/chroma/releases)
- [Changelog](https://github.com/alecthomas/chroma/blob/master/.goreleaser.yml)
- [Commits](https://github.com/alecthomas/chroma/compare/v2.13.0...v2.14.0)

Updates `github.com/go-pkgz/notify` from 1.1.1 to 1.2.0
- [Release notes](https://github.com/go-pkgz/notify/releases)
- [Commits](https://github.com/go-pkgz/notify/compare/v1.1.1...v1.2.0)

Updates `github.com/go-pkgz/repeater` from 1.1.3 to 1.2.0
- [Release notes](https://github.com/go-pkgz/repeater/releases)
- [Commits](https://github.com/go-pkgz/repeater/compare/v1.1.3...v1.2.0)

Updates `github.com/rs/xid` from 1.5.0 to 1.6.0
- [Release notes](https://github.com/rs/xid/releases)
- [Commits](https://github.com/rs/xid/compare/v1.5.0...v1.6.0)

Updates `go.etcd.io/bbolt` from 1.3.10 to 1.3.11
- [Release notes](https://github.com/etcd-io/bbolt/releases)
- [Commits](https://github.com/etcd-io/bbolt/compare/v1.3.10...v1.3.11)

Updates `golang.org/x/crypto` from 0.25.0 to 0.27.0
- [Commits](https://github.com/golang/crypto/compare/v0.25.0...v0.27.0)

Updates `golang.org/x/image` from 0.18.0 to 0.22.0
- [Commits](https://github.com/golang/image/compare/v0.18.0...v0.22.0)

Updates `golang.org/x/net` from 0.27.0 to 0.29.0
- [Commits](https://github.com/golang/net/compare/v0.27.0...v0.29.0)

---
updated-dependencies:
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/alecthomas/chroma/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/notify
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/go-pkgz/repeater
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: github.com/rs/xid
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: go.etcd.io/bbolt
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/image
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-modules-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-08 16:19:58 -06:00
Dmitry VerkhoturovandUmputun c1a447b873 unpin bugfix js version, use codecov github action 2024-11-08 15:56:59 -06:00
dependabot[bot]andUmputun 3c110eb0ec Bump the github-actions-updates group with 5 updates
Bumps the github-actions-updates group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) | `3` | `6` |
| [pnpm/action-setup](https://github.com/pnpm/action-setup) | `2.4.0` | `4.0.0` |
| [actions/cache](https://github.com/actions/cache) | `3` | `4` |
| [andresz1/size-limit-action](https://github.com/andresz1/size-limit-action) | `7313b26c76b3666c1dc41e2ca05370e201a9b7de` | `94bc357df29c36c8f8d50ea497c3e225c3c95d1d` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `2` | `4` |


Updates `golangci/golangci-lint-action` from 3 to 6
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3...v6)

Updates `pnpm/action-setup` from 2.4.0 to 4.0.0
- [Release notes](https://github.com/pnpm/action-setup/releases)
- [Commits](https://github.com/pnpm/action-setup/compare/v2.4.0...v4.0.0)

Updates `actions/cache` from 3 to 4
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

Updates `andresz1/size-limit-action` from 7313b26c76b3666c1dc41e2ca05370e201a9b7de to 94bc357df29c36c8f8d50ea497c3e225c3c95d1d
- [Release notes](https://github.com/andresz1/size-limit-action/releases)
- [Commits](https://github.com/andresz1/size-limit-action/compare/7313b26c76b3666c1dc41e2ca05370e201a9b7de...94bc357df29c36c8f8d50ea497c3e225c3c95d1d)

Updates `actions/upload-artifact` from 2 to 4
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v2...v4)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: pnpm/action-setup
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
- dependency-name: andresz1/size-limit-action
  dependency-type: direct:production
  dependency-group: github-actions-updates
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-updates
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-08 15:56:59 -06:00
Dmitry VerkhoturovandUmputun d7494d5392 add dependabot updates for Go, TypeScript, GitHub Actions 2024-11-07 19:13:55 -06:00
W.T. ChangandUmputun c1048e95b3 fix: missing comma in example frontend configuration 2024-10-27 01:58:27 -05:00
Dmitry VerkhoturovandUmputun 5b6d8de807 Deprecate Twitter OAuth and remove from feature list 2024-10-20 17:18:14 -05:00
Dmitry VerkhoturovandUmputun dd2cff6a13 Clarify ALLOWED_HOSTS description and usage
This clarifies that the parameter sets CSP 'frame-ancestors'
to limit hosts allowed to embed comments. The commit also improves
the documentation on how to use ALLOWED_HOSTS with AUTH_SAME_SITE
for different setup scenarios.

We might want to change AUTH_SAME_SITE to `strong` in v2.0 as it works
on the subdomain of the same site as well as current Lax option.
2024-10-20 17:17:35 -05:00
Dmitry VerkhoturovandUmputun 68fe6eb55f Update docker-compose command to docker compose
Compose is part of docker command for years and during local development
 on Mac I have to change makefile locally for commands to work.
2024-10-20 17:17:10 -05:00
Dmitry VerkhoturovandUmputun 4d5f9f269b Add missing AUTH_APPLE_KID env variable cleanup 2024-10-20 16:06:50 -05:00
Dmitry VerkhoturovandUmputun 6140d82eb2 Fix CSP img-src directive to allow everything without proxy
Change the default img-src value to "*" and sets it to "'self'" when
image proxy is enabled. The previous state was inversion of this logic
which was wrong.
2024-10-20 15:55:51 -05:00
Mrigank Badola cff261c15b chore: add discord in server test file 2024-10-20 11:47:47 +05:30
Mrigank Badola 18ea40582a sync: Merge remote-tracking branch 'origin' into discord-oauth 2024-10-20 11:46:33 +05:30
Dmitry VerkhoturovandUmputun f9d4837567 Add Content-Security-Policy and Permissions-Policy headers
`Content-Security-Policy` now restricts resource loading and execution
to enhance security:
  - `default-src 'none'`: Disallow all resource loading by default.
  - `base-uri 'none'`: Prevents the use of `<base>` tag to change the
  base URL for relative URLs.
  - `form-action 'none'`: Disallows form submissions.
  - `connect-src 'self'`: Restricts the origins that can be connected to
   (via XHR, WebSockets, etc.) to the same origin.
  - `frame-src 'self'`: Restricts the origins that can be embedded using
   `<frame>` and `<iframe>` to the same origin (for `/web/` demo
    endpoint).
  - `frame-ancestors %s;`: Specifies the origins that are allowed to
  embed this content in a frame. If no specific origins are allowed, it
  defaults to `*` (any origin). This enhances security by controlling
  which sites can embed your content.
  - `img-src 'self'`: Allows images to be loaded only from the same
  origin. If `imageProxyEnabled` is true, allows images from any origin
  (`*`).
  - `script-src 'self' 'unsafe-inline'`: Allows scripts to be loaded and
   executed only from the same origin and allows inline scripts.
  - `style-src 'self' 'unsafe-inline'`: Allows styles to be loaded and
  applied only from the same origin and allows inline styles.
  - `font-src data:`: Allows fonts to be loaded from data URIs.
  - `object-src 'none'`: Disallows the use of `<object>`, `<embed>`, and
   `<applet>` tags.

`Permissions-Policy` now restricts the use of certain browser features
which we don't use to enhance user privacy and security:
  - `accelerometer=()`: Disables the use of the accelerometer sensor.
  - `autoplay=()`: Disables automatic playback of media.
  - `camera=()`: Disables the use of the camera.
  - `cross-origin-isolated=()`: Disallows the page from being treated as
   cross-origin isolated.
  - `display-capture=()`: Disables the ability to capture the display.
  - `encrypted-media=()`: Disables the use of Encrypted Media Extensions
  .
  - `fullscreen=()`: Disables the ability to use fullscreen mode.
  - `geolocation=()`: Disables the use of geolocation.
  - `gyroscope=()`: Disables the use of the gyroscope sensor.
  - `keyboard-map=()`: Disables the use of the keyboard map.
  - `magnetometer=()`: Disables the use of the magnetometer sensor.
  - `microphone=()`: Disables the use of the microphone.
  - `midi=()`: Disables the use of the MIDI API.
  - `payment=()`: Disables the Payment Request API.
  - `picture-in-picture=()`: Disables the use of Picture-in-Picture mode
  .
  - `publickey-credentials-get=()`: Disables the use of the Web
  Authentication API.
  - `screen-wake-lock=()`: Disables the ability to prevent the screen
  from dimming.
  - `sync-xhr=()`: Disables synchronous XMLHttpRequest.
  - `usb=()`: Disables the use of the USB API.
  - `xr-spatial-tracking=()`: Disables the use of spatial tracking in
  WebXR.
  - `clipboard-read=()`: Disables the ability to read from the clipboard
  .
  - `clipboard-write=()`: Disables the ability to write to the clipboard
  .
  - `gamepad=()`: Disables the use of the Gamepad API.
  - `hid=()`: Disables the use of the Human Interface Device API.
  - `idle-detection=()`: Disables the ability to detect idle state.
  - `interest-cohort=()`: Disables the use of interest cohort tracking.
  - `serial=()`: Disables the use of the Serial API.
  - `unload=()`: Disables the ability to use the `beforeunload` and
  `unload` events.
  - `window-management=()`: Disables the ability to use window
  management APIs.
2024-10-15 17:53:12 -05:00
Mrigank Badola e5ae07b1c2 chore: add cid and csec in readmes and backend docker compose yml 2024-10-09 09:45:10 +05:30
Mrigank Badola 4fbb3b59be chore: add changes in accordance to paskal/discord_poc 2024-10-09 08:57:47 +05:30
Dmitry VerkhoturovandUmputun 9fb3014229 Detect proper avatar type to return instead of returning image/* 2024-09-22 14:37:29 -05:00
dependabot[bot]andUmputun 2a9b29dd53 Bump micromatch from 4.0.5 to 4.0.8 in /site
Bumps [micromatch](https://github.com/micromatch/micromatch) from 4.0.5 to 4.0.8.
- [Release notes](https://github.com/micromatch/micromatch/releases)
- [Changelog](https://github.com/micromatch/micromatch/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/micromatch/compare/4.0.5...4.0.8)

---
updated-dependencies:
- dependency-name: micromatch
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-20 22:31:53 -05:00
Ivan BandUmputun 872b818323 docs: fix frontend paths in translation guide (frontend/{path} -> frontend/apps/remark42/{path}) 2024-09-19 19:06:06 -05:00
Dmitry VerkhoturovandUmputun 4a7bee1d98 Fix restrictions for anonymous usernames
Tested on remark42 demo to clarify what works and what doesn't.
2024-09-19 19:05:43 -05:00
dependabot[bot]andUmputun cbe793fb42 Bump path-to-regexp from 6.2.1 to 6.3.0 in /site
Bumps [path-to-regexp](https://github.com/pillarjs/path-to-regexp) from 6.2.1 to 6.3.0.
- [Release notes](https://github.com/pillarjs/path-to-regexp/releases)
- [Changelog](https://github.com/pillarjs/path-to-regexp/blob/master/History.md)
- [Commits](https://github.com/pillarjs/path-to-regexp/compare/v6.2.1...v6.3.0)

---
updated-dependencies:
- dependency-name: path-to-regexp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-19 19:05:22 -05:00
Dmitry VerkhoturovandUmputun cbf9a82a92 Move gcc installation before backend files addition
This allows to not reinstall apk files when backend files change and the
new image is built.
2024-09-19 19:05:01 -05:00
Dmitry VerkhoturovandUmputun 6cd5c45a6c Fix problem with logout button
The logout auth endpoint was returning no response body and type
application/json which is not valid, this commit changes it to return
plain/text instead which makes it valid.
2024-09-19 19:04:21 -05:00
adueckandUmputun 0bc85a6ff6 added RTL support 2024-08-24 18:08:09 -05:00
Tomy HsiehandUmputun 88bf4b7d70 feat: Update CLI help message 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun 26c5425646 📖 docs: restore numbering 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun 15d2ab9644 🕺🏻 style: Format 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun 50c56cb771 📖 docs: Update apple integration docs 2024-08-23 19:19:59 -05:00
Tomy HsiehandUmputun e65f71b958 🛠 fix: Fix sign in with apple integration 2024-08-23 19:19:59 -05:00
Dmitry VerkhoturovandUmputun a9b439602b update go modules 2024-07-30 20:23:33 -05:00
Dmitry VerkhoturovandUmputun d2027f5241 switch playwright (e2e) to latest stable version 2024-07-30 20:23:06 -05:00
Dmitry VerkhoturovandUmputun 95966f6407 add escaping of comment text in webhook default JSON template 2024-07-01 23:41:50 -05:00
Umputun 8df986e70a Add content type check for images endpoint
A check in image proxy for validating content type of requested images added. Modified the related tests to accommodate these changes.
2024-07-01 14:47:25 -05:00
dependabot[bot]andUmputun 71a6d0b385 Bump pug from 3.0.2 to 3.0.3 in /site
Bumps [pug](https://github.com/pugjs/pug) from 3.0.2 to 3.0.3.
- [Release notes](https://github.com/pugjs/pug/releases)
- [Commits](https://github.com/pugjs/pug/compare/pug@3.0.2...pug@3.0.3)

---
updated-dependencies:
- dependency-name: pug
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:12:14 -05:00
dependabot[bot]andUmputun 974d4aaf55 Bump ejs from 3.1.9 to 3.1.10 in /site
Bumps [ejs](https://github.com/mde/ejs) from 3.1.9 to 3.1.10.
- [Release notes](https://github.com/mde/ejs/releases)
- [Commits](https://github.com/mde/ejs/compare/v3.1.9...v3.1.10)

---
updated-dependencies:
- dependency-name: ejs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:11:10 -05:00
dependabot[bot]andUmputun dc8d7d46cb Bump braces from 3.0.2 to 3.0.3 in /site
Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3.
- [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/braces/compare/3.0.2...3.0.3)

---
updated-dependencies:
- dependency-name: braces
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:10:47 -05:00
dependabot[bot]andUmputun c4ace9fc0c Bump ws from 8.16.0 to 8.17.1 in /site
Bumps [ws](https://github.com/websockets/ws) from 8.16.0 to 8.17.1.
- [Release notes](https://github.com/websockets/ws/releases)
- [Commits](https://github.com/websockets/ws/compare/8.16.0...8.17.1)

---
updated-dependencies:
- dependency-name: ws
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-06-30 12:10:26 -05:00
Dmitry VerkhoturovandUmputun 16b07ded66 fix golangci-lint reported unused parameters in example module 2024-06-30 12:07:06 -05:00
Tomy HsiehandUmputun c04705947a 📖 docs: Update README 2024-06-22 12:04:49 -05:00
Dmitry VerkhoturovandUmputun eadd65e247 update docker images, clarify comments 2024-06-05 13:05:42 -05:00
Xin LiandUmputun 4428f79046 SubscribeByEmailForm: Ensure onInput and onClick props are typed correctly.
Resolves #1771
2024-06-04 11:06:14 -05:00
Armen MkrtchyanandUmputun bad6af87f7 Update .golangci.yml
Sorted linters alphabetically, removed duplicates
2024-05-30 11:30:19 -05:00
Edward NavarroandUmputun f7ba43e5f1 Complete and update Spanish translations 2024-05-27 15:22:09 -05:00
Dmitry VerkhoturovandUmputun 661f042cb4 pin golangci-lint version to latest available, fix reported errors 2024-05-09 22:32:00 -05:00
Pavel FrancírekandUmputun 877765cda2 Update cs.json - typos
Only typo corrections.
2024-04-11 11:37:58 -05:00
Dmitry VerkhoturovandUmputun 4bb0017060 update go modules 2024-04-10 19:33:20 -05:00
Dmitry VerkhoturovandUmputun e0423b8683 fix type for value for refresh token cache
It was set to string by mistake, proper type is token.Claims.
2024-03-22 04:16:29 -05:00
Dmitry VerkhoturovandUmputun 5a781693aa hide delete button for non-admin users after edit period expires 2024-03-17 16:47:28 -05:00
Dmitry VerkhoturovandUmputun e5743185b0 collect /find Info for tree and plain types consistently
MakeTree calculated Info locally for historical reasons,
and the results were consistent with the dataService.Info call
but calculated differently.

That change fixes that, ensuring that Info is requested
in the same manner.
2024-03-16 12:49:45 -05:00
dependabot[bot]andUmputun 1510aec17c Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /backend
Bumps google.golang.org/protobuf from 1.32.0 to 1.33.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-03-13 18:33:05 -05:00
Dmitry VerkhoturovandUmputun 01837b69e5 fix readonly status, deleted count for plain /find request 2024-03-04 15:44:19 -06:00
Dmitry VerkhoturovandUmputun d02099844e apply prettier to site files 2024-02-25 18:27:06 -06:00
Dmitry VerkhoturovandUmputun 6fcfaa12b7 update site dependencies 2024-02-25 18:27:06 -06:00
Dmitry VerkhoturovandUmputun 6269c19881 add more tests for GET /find endpoint 2024-02-21 10:43:28 -06:00
Dmitry VerkhoturovandUmputun 1313dee829 update to lcw v2 with generic types 2024-02-20 14:15:35 -06:00
Dmitry VerkhoturovandUmputun 3210de8f7b update go modules 2024-02-20 12:23:16 -06:00
Dmitry VerkhoturovandUmputun 532573fb34 fix problems reported by golangci-lint 2024-02-20 12:10:39 -06:00
Konstantsin KryuleniaandUmputun e1173bbcad use specific pnpm version 2024-02-03 12:09:57 -06:00
Konstantsin KryuleniaandUmputun e748951182 skip lint jpeg files 2024-02-03 12:09:57 -06:00
Konstantsin KryuleniaandUmputun df8670752a copy example image to public folder 2024-02-03 12:09:57 -06:00
Dmitry VerkhoturovandUmputun 654250f033 don't load kitten picture from third party site 2024-02-03 12:09:57 -06:00
Dmitry VerkhoturovandUmputun 0050c65596 add meaningful error for lack of auth on import, remap and backup
Previously, the error printed was just the following:

error response "401 Unauthorized", Unauthorized"

New error:

error response "401 Unauthorized", ensure you have set ADMIN_PASSWD
and provided it to the command you're running: Unauthorized
2024-01-28 12:16:24 -06:00
Dmitry VerkhoturovandUmputun 02db7a917d fix lack of error on file site export
Previously, status 200 was set for file export, which is used
for backup, which resulted in an inability to set an error status code
in case of a problem with file generation.

After this change, status code 200 would be written automatically by Go
before we start writing the response's body.
2024-01-28 12:16:24 -06:00
Dmitry VerkhoturovandUmputun 81c30e01f8 cleanup images from deleted comments
Previously, images were deleted only from comments deleted
before EditDuration expiration. After this change, any deletion
of the comment deletes images if they are not used elsewhere
in comments under the same page.
2024-01-20 13:29:06 -06:00
Paul MineevandUmputun 82c617806d chore: remove theme from comment content 2024-01-18 02:56:10 -06:00
Paul MineevandUmputun e043dc2ac3 fix: break long words in code tag, combine all styles in one file, move styles to correct place 2024-01-18 02:56:10 -06:00
Dmitry VerkhoturovandUmputun cbd73865bd update go modules, update go-pkgz/auth to latest commit 2024-01-11 15:57:39 -06:00
Dmitry VerkhoturovandUmputun 884b5685eb update docker images and github CI actions 2024-01-11 01:26:30 -06:00
NavyStackandUmputun 3f14651653 fix: add missing Japanese translation 2024-01-08 11:23:03 -06:00
NavyStackandUmputun 310b797679 fix: add missing Japanese translation 2024-01-08 11:23:03 -06:00
NavyStackandUmputun 0594565143 FIX: Character escaping 2024-01-08 11:23:03 -06:00
NavyStackandUmputun d4c153662b update: Korean translation 2024-01-08 11:23:03 -06:00
NavyStackandUmputun f64b0b8831 fix: Korean missing strings 2024-01-08 11:23:03 -06:00
Vladimir DandUmputun 94893b77dc bump deps 2023-12-26 11:47:09 -06:00
Vladimir DandUmputun 30f46efa5b TLS InsecureSkipVerify option 2023-12-26 11:47:09 -06:00
Vladimir DandUmputun e0904603c6 go-pkgz/auth and go-pkgz/email modules updated, bump deps 2023-12-26 11:47:09 -06:00
Dmitry VerkhoturovandUmputun d143932924 add MIN_COMMENT_SIZE parameter 2023-12-02 12:16:26 -06:00
Dmitry Verkhoturov dcc7613409 allow disabling fancy HTML formatting
It might be necessary if the comments should preserve
original quotes instead of replacing them with angled ones.
2023-11-26 09:13:53 +01:00
Dmitry VerkhoturovandUmputun d04d2097f8 fix Commento import URL
Previously, it was not using the domain
and relying on another export format.
2023-11-20 10:59:05 -06:00
Dmitry Verkhoturov ce678bf967 fix Commento top-level comments import
Previously, top-level comments were incorrectly assigned
parent comment id "root", which made them non-root,
so they are not returned when requested
in the `/find?format=tree` API call.

To fix the previously imported comments, please export all your comments
and replace `"pid":"root"` with `"pid":""` and then re-import them.
2023-11-18 20:30:18 +01:00
Dmitry VerkhoturovandUmputun cd481d401d add tests for admin Store and DataService 2023-11-04 12:49:40 -05:00
Dmitry VerkhoturovandUmputun 618c267370 combine multiple post info in DataStore.Info instead of returning first
Previously, only the first one was returned for site-wide requests,
and now all returned information will be correctly aggregated,
and the PostInfo.URL and PostInfo.ReadOnly parameters will be dropped.
2023-11-04 12:49:40 -05:00
Dmitry VerkhoturovandUmputun 307866f7f5 simplify BoltDB.Info code
The new code does the same as the old one but doesn't call the checkFlag
in case ReadOnly is already set based on age.
2023-11-04 12:46:38 -05:00
Dmitry VerkhoturovandUmputun 19e1616129 allow title extraction only from full match of AllowedHosts
Previously, we extracted the second-level domain,
but it doesn't make sense for a list of domains defined explicitly
to display the comments.
2023-11-04 12:45:19 -05:00
dependabot[bot]andUmputun c6506b8905 Bump luxon from 2.3.0 to 2.5.2 in /site
Bumps [luxon](https://github.com/moment/luxon) from 2.3.0 to 2.5.2.
- [Release notes](https://github.com/moment/luxon/releases)
- [Changelog](https://github.com/moment/luxon/blob/master/CHANGELOG.md)
- [Commits](https://github.com/moment/luxon/compare/2.3.0...2.5.2)

---
updated-dependencies:
- dependency-name: luxon
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-28 03:15:10 -05:00
Paul MineevandUmputun 676ae77456 update links styles 2023-10-28 01:56:04 -05:00
Paul MineevandUmputun b93fc48b73 fix types on error message extraction 2023-10-28 01:45:03 -05:00
Paul MineevandUmputun 4be664e78d fix styles around admin controls 2023-10-28 01:36:17 -05:00
Paul MineevandUmputun 62aaa35287 fix types on error message extraction 2023-10-28 01:35:58 -05:00
Dmitry VerkhoturovandUmputun 69b18d3536 fix wrapped errors checks
As errors can be wrapped in recent versions of Go, the proper way
to check the error types are `errors.As` and `errors.Is`.
2023-10-10 23:36:04 -05:00
Dmitry VerkhoturovandUmputun efceed6f68 limit TitleExtractor to allow only Remark42 whitelisted domains
Allowed domains consist of `REMARK_URL` second-level domain (or whole IP in case it's IP like `127.0.0.1`) and `ALLOWED_HOSTS`. That is needed to prevent Remark42 from asking arbitrary servers and storing the page title as the comment.PostTitle.

Previous behaviour allowed the caller of the API to create a comment
with an arbitrary URL and learn the title of the page, which might be
accessible to the server Remark42 is installed on but not to the user
outside that network (CWE-918).
2023-10-10 23:34:35 -05:00
Dmitry VerkhoturovandUmputun f4358173c7 limit TitleExtractor to allow only Remark42 whitelisted domains
Allowed domains consist of `REMARK_URL` second-level domain (or whole IP in case it's IP like `127.0.0.1`) and `ALLOWED_HOSTS`. That is needed to prevent Remark42 from asking arbitrary servers and storing the page title as the comment.PostTitle.

Previous behaviour allowed the caller of the API to create a comment
with an arbitrary URL and learn the title of the page, which might be
accessible to the server Remark42 is installed on but not to the user
outside that network (CWE-918).
2023-10-10 23:34:35 -05:00
Dmitry VerkhoturovandUmputun 7a71d47556 remove all HTML tags from comment title and username
Previously, we stripped unsafe HTML tags but left some,
but it's not expected to have a link in a title or username,
so the new behaviour is stripping everything.
2023-10-10 12:41:26 -05:00
Dmitry VerkhoturovandUmputun 41d27e2a7f clarify titles for frontend and backend configuration doc 2023-10-10 11:51:26 -05:00
Dmitry VerkhoturovandUmputun 10e4686f1a reproduce report of CWE-918 from #1677 2023-10-10 11:43:56 -05:00
sharief007andUmputun eba447319d Extract error msg from API response for preview. 2023-10-08 12:53:04 -05:00
Dmitry VerkhoturovandUmputun 40a0d7ca62 update Telegram notifications instructions
Remove outdated information about functionality which was already implemented.
2023-09-12 20:45:25 -05:00
Dzung DoandUmputun c9b6f9272f Update vi.json 2023-09-04 01:35:57 -05:00
Umputun 1f2500f16f switch to stable, updated auth 2023-08-21 11:20:24 -05:00
Umputun 4b855ceddd mod tidy for mem_store example 2023-08-20 19:37:39 -05:00
Umputun e30d4da455 update auth lib to master #1660
adopt tests for the mandatory provider check

fix leftover test for the server
2023-08-20 19:37:39 -05:00
Umputun 26e6e57949 add cmd/var to ignore 2023-08-07 13:33:04 -05:00
Umputun b572966bc4 remove cmd/var 2023-08-07 13:32:44 -05:00
Umputun bbfa4f1043 update base images for backend docker build 2023-08-07 13:12:42 -05:00
Umputun 9ad4f0b75e lint: remove deprecated rand.Seed from tests 2023-08-07 13:09:38 -05:00
Umputun 2093f4ece2 update go to 1.20, bump deps 2023-08-07 13:03:16 -05:00
Alexander GusmanandUmputun 7bc7703dc2 build: use pnpm instead of npm to install deps for size checks 2023-07-24 12:29:55 -05:00
Alexander GusmanandUmputun 0ed7452e77 feat(ui): telegram user subscriptions 2023-07-24 12:29:55 -05:00
Alexander GusmanandUmputun 366cc19c1b fix(rest): correct status code for telegram check token 2023-07-24 12:29:55 -05:00
Dmitry VerkhoturovandUmputun c72f30eabb remove proxied images from sanity check
Previously, proxied and local images were checked for presence in the
storage before previewing or posting the comment. That logic resulted in
 an inability to post with an image when a proxy for images is enabled,
 as proxied images are not downloaded to disk before the first time
 someone loads them, which could only happen after the user either
 previews or posts the message.

After this change, preview and post only checks the local images'
presence and ignore the proxied ones.
2023-07-23 12:10:52 -05:00
Justin HawkinsandGitHub 235f0dade0 Update documentation to support Caddy V2 (#1657)
* Update index.md

Change Caddy example config to the current V2 config.

* Simplify config

* Include legacy config as well
2023-07-20 21:12:11 -05:00
Dmitry VerkhoturovandUmputun 9c718cbc5f clarify SharedSecret usage by comment
It's not used aside from `server` but it was decided to keep it in place
for backwards compatibility in PR #1544.
2023-07-16 20:50:03 -05:00
Dmitry VerkhoturovandUmputun 02de92afc7 update Google auth setup instructions
Update based on my experience going through them.
2023-07-16 18:14:38 -05:00
Yury KotovandUmputun add01455fb Fix snippet generation
1) Current implementation simply removes the last word, without truncating up to limit length.
2) In case if even the first word (magnet link or some base64?) is too long don't add extra space.
2023-07-04 13:32:37 -05:00
EugeneandGitHub 497f3ce47f Skip confirmation step on email subscription (#1646)
* Email subscription params in request body

* Email subscription params in request body

fix tests

* Skip confirm step on email sub

When user logged in with the same email he tries to subscribe

* Skip confirm step on email sub

set autoConfirm param to make it work

* Update size-limit

* Handle 409: already subscribed

* refactor: prevStep to justSubscribed

prevStep is not used anywhere else and because of it influences output text (haveSubscribed), have changed it to more intuitive justSubscribed variable

* Test case for http error 409
2023-06-29 14:15:59 -05:00
LoneExileandUmputun 64188e5713 docs: added Astro w/ React/Preact Components Integration 2023-06-29 13:21:42 -05:00
Eugene OrlovandUmputun 33a6d6da97 Email subscription params in request body
fix tests
2023-06-29 10:52:36 -05:00
Eugene OrlovandUmputun 6410e3be85 Email subscription params in request body 2023-06-29 10:52:36 -05:00
Dmitry VerkhoturovandUmputun 136d7e8215 switch from telegram_bot_username to telegram_notifications
Bot username is returned as an answer to subscribe request,
so knowing it in advance is unnecessary.
2023-06-29 10:52:00 -05:00
Eugene OrlovandUmputun d3fdd7b0d8 Fix color var usage 2023-06-27 22:16:38 -05:00
Eugene OrlovandUmputun 329fcc204c Update size-limit 2023-06-27 22:16:38 -05:00
Eugene OrlovandUmputun ba2c7894a8 Telegram QR styling
- 1/1 aspect ratio
- white border around (for easier scan)
2023-06-27 22:16:38 -05:00
Eugene OrlovandUmputun 07667c8881 fix: Gap between buttons and markdown tip 2023-06-12 09:44:04 -05:00
Eugene OrlovandUmputun 68504a70a0 [site] fix: anchor offset
use different header size on mobiles and desktops
2023-06-06 15:22:02 -05:00
Eugene OrlovandUmputun 32073b3d66 [site] fix: anchor offset
make it use tailwind styles
2023-06-06 15:22:02 -05:00
Eugene OrlovandUmputun d1c1664a38 [site] fix: anchor offset
when navigating through an anchor
2023-06-06 15:22:02 -05:00
Vasilii BlazhnovandUmputun 8cbcff98ec Fixed wrong markdown 2023-05-08 14:35:30 -05:00
Dmitry VerkhoturovandUmputun 26f82ad95c don't allow relative links in comments
(url) is a text inserted by default and never an intended URL.

That additional validation will ensure that users won't post relative
links because they are rarely intended.
2023-04-09 23:30:21 -05:00
Dmitry VerkhoturovandUmputun 1b90604b2d update go modules, fix Apple auth redirect
Previously the redirect after successful authentication didn't work.
2023-04-02 03:47:35 -05:00
Dmitry VerkhoturovandUmputun a03c002df4 bump golangci-lint to latest 2023-04-02 01:21:33 -05:00
Dmitry VerkhoturovandUmputun 1ce9415d34 ping pnmp version to 7
Previously version 8 was installed once it became available, breaking
the pipelines and docker build.
2023-04-01 14:45:44 -05:00
SimonHaasandUmputun cc842901b3 updated deprecated link 2023-03-17 10:59:45 -05:00
Paul MineevandUmputun 23d7e4cdbb fix indentation for apps/remark42 2023-03-10 17:46:41 -06:00
Dejavu MoeandUmputun b48f8fca31 Fix spell errors 2023-03-10 17:21:04 -06:00
DejavuMoeandUmputun a4da93326e Improve Simplified Chinese translation
modified:frontend/apps/remark42/app/locales/zh.json
2023-03-10 17:20:45 -06:00
konstantin krivleniaandUmputun 8bd5c0d163 #1605 fix loading th locale 2023-03-10 17:20:25 -06:00
DejavuMoeandUmputun 972ab87247 Update translation of the Simplified Chinese 2023-03-08 17:22:06 -06:00
dmitry.konchalenkovandUmputun d1ea664b41 Update translations 2023-02-28 11:52:29 -06:00
Dmitry Verkhoturov a55fadd53a make docker build fail on backend test failure
Previously, the commands were combined incorrectly, and the failure
of backend tests was ignored.
2023-02-05 21:11:12 +01:00
Dmitry VerkhoturovandUmputun c70a66a1c5 bump go modules 2023-01-21 13:30:44 -06:00
2339 changed files with 214581 additions and 122370 deletions
+2
View File
@@ -1,3 +1,5 @@
root = true
[*]
indent_style = tab
insert_final_newline = true
+72
View File
@@ -0,0 +1,72 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
groups:
"GitHub Actions updates":
patterns:
- "*"
- package-ecosystem: "gomod"
directory: "/backend"
schedule:
interval: "monthly"
groups:
"Go modules updates":
dependency-type: "production"
- package-ecosystem: "npm"
directory: "/frontend"
open-pull-requests-limit: 0
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
- package-ecosystem: "npm"
directory: "/frontend/packages/api"
open-pull-requests-limit: 0
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
- package-ecosystem: "npm"
directory: "/frontend/e2e"
open-pull-requests-limit: 0
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
- package-ecosystem: "npm"
directory: "/frontend/apps/remark42"
open-pull-requests-limit: 0
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
- package-ecosystem: "npm"
directory: "/site"
open-pull-requests-limit: 0
schedule:
interval: "monthly"
groups:
"NPM modules updates":
dependency-type: "production"
"NPM modules updates for tests":
dependency-type: "development"
+31 -17
View File
@@ -5,24 +5,29 @@ on:
branches:
tags:
paths:
- ".github/workflows/ci-test-backend.yml"
- ".github/workflows/ci-backend.yml"
- "backend/**"
- "!backend/scripts/**"
- "!**.md"
pull_request:
types: [opened, reopened]
paths:
- ".github/workflows/ci-test-backend.yml"
- ".github/workflows/ci-backend.yml"
- "backend/**"
- "!backend/scripts/**"
- "!**.md"
jobs:
test:
name: Test & Coverage
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: debug if needed
run: if [[ "$DEBUG" == "true" ]]; then env; fi
@@ -30,36 +35,45 @@ jobs:
DEBUG: ${{secrets.DEBUG}}
- name: install go
uses: actions/setup-go@v3
uses: actions/setup-go@v6
with:
go-version: 1.17
go-version: "1.25"
cache-dependency-path: backend
- name: install golangci-lint and goveralls
run: |
curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $GITHUB_WORKSPACE v1.49.0
go install github.com/mattn/goveralls@latest
- name: test and lint backend
- name: test and build backend
run: |
go test -race -timeout=60s -covermode=atomic -coverprofile=$GITHUB_WORKSPACE/profile.cov_tmp ./...
cat $GITHUB_WORKSPACE/profile.cov_tmp | grep -v "_mock.go" > $GITHUB_WORKSPACE/profile.cov
$GITHUB_WORKSPACE/golangci-lint --config ${GITHUB_WORKSPACE}/backend/.golangci.yml run --out-format=github-actions ./...
go build -race ./...
working-directory: backend/app
env:
TZ: "America/Chicago"
- name: test and lint examples
- name: test examples
run: |
go version
$GITHUB_WORKSPACE/golangci-lint version
go test -race ./...
$GITHUB_WORKSPACE/golangci-lint --config ${GITHUB_WORKSPACE}/backend/.golangci.yml run --out-format=github-actions ./...
go build -race ./...
working-directory: backend/_example/memory_store
env:
TZ: "America/Chicago"
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: "v2.6.0"
working-directory: backend/app
- name: golangci-lint on example directory
uses: golangci/golangci-lint-action@v9
with:
version: "v2.6.0"
args: --config ../../.golangci.yml
working-directory: backend/_example/memory_store
- name: submit coverage
run: goveralls -service="github" -coverprofile=$GITHUB_WORKSPACE/profile.cov
run: |
go install github.com/mattn/goveralls@latest
goveralls -service="github" -coverprofile=$GITHUB_WORKSPACE/profile.cov
working-directory: backend
env:
COVERALLS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+35 -70
View File
@@ -1,18 +1,6 @@
name: build
on:
push:
branches:
tags:
paths:
- ".github/workflows/ci-build.yml"
- "backend/**"
- "frontend/apps/**"
- ".dockerignore"
- "docker-init.sh"
- "Dockerfile"
- "!**.md"
- "!frontend/packages/**"
pull_request:
paths:
- ".github/workflows/ci-build.yml"
@@ -23,79 +11,56 @@ on:
- "Dockerfile"
- "!**.md"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build-images:
name: Build Docker images
name: Validate Docker build
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v3
- name: set up QEMU
uses: docker/setup-qemu-action@v2
- uses: actions/checkout@v6
with:
persist-credentials: false
- name: set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- name: available platforms
run: echo ${{ steps.buildx.outputs.platforms }}
- name: expose GitHub Actions cache
uses: actions/cache@v4
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
- name: build docker image without pushing (only outside master)
if: ${{ github.ref != 'refs/heads/master' }}
- name: free disk space
run: |
docker buildx build \
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
docker system prune -af
- name: build docker image without pushing
run: |
docker buildx build --load \
--cache-from type=local,src=/tmp/.buildx-cache \
--cache-to type=local,dest=/tmp/.buildx-cache-new,mode=max \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true \
--platform linux/amd64 .
- name: build example docker image without pushing (only outside master)
if: ${{ github.ref != 'refs/heads/master' }}
- name: build example docker image without pushing
run: |
docker buildx build \
docker buildx build --load \
--cache-from type=local,src=/tmp/.buildx-cache \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true \
--platform linux/amd64 -f backend/_example/memory_store/Dockerfile .
- name: build and deploy master image to ghcr.io and dockerhub
if: ${{ github.ref == 'refs/heads/master' }}
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
- name: rotate cache
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo "GITHUB_REF=${GITHUB_REF}, GITHUB_SHA=${GITHUB_SHA}, GIT_BRANCH=${ref}"
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
echo ${DOCKER_HUB_TOKEN} | docker login -u umputun --password-stdin
docker buildx build --push \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true --build-arg CI=github \
--build-arg GITHUB_SHA=${GITHUB_SHA} --build-arg GIT_BRANCH=${ref} --build-arg GITHUB_REF=${GITHUB_REF} \
--platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42:${ref} -t umputun/remark42:${ref} .
- name: deploy tagged (latest) to ghcr.io and dockerhub
if: ${{ startsWith(github.ref, 'refs/tags/') }}
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
DOCKER_HUB_TOKEN: ${{ secrets.DOCKER_HUB_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo "GITHUB_REF=${GITHUB_REF}, GITHUB_SHA=${GITHUB_SHA}, GIT_BRANCH=${ref}"
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
echo ${DOCKER_HUB_TOKEN} | docker login -u umputun --password-stdin
docker buildx build --push \
--build-arg SKIP_BACKEND_TEST=true --build-arg SKIP_FRONTEND_TEST=true --build-arg CI=github \
--build-arg GITHUB_SHA=${GITHUB_SHA} --build-arg GIT_BRANCH=${ref} --build-arg GITHUB_REF=${GITHUB_REF} \
--platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42:${ref} -t ghcr.io/umputun/remark42:latest \
-t umputun/remark42:${ref} -t umputun/remark42:latest .
- name: remote deployment to remark42.com from master
if: ${{ github.ref == 'refs/heads/master' }}
env:
UPDATER_KEY: ${{ secrets.UPDATER_KEY }}
run: curl -s https://jess.umputun.com/update/remark42-core/${UPDATER_KEY}
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache || true
+42 -31
View File
@@ -18,35 +18,38 @@ jobs:
type-check:
name: Type check
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [ 16 ]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -62,35 +65,38 @@ jobs:
lint:
name: Lint
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [ 16 ]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -106,35 +112,38 @@ jobs:
test:
name: Tests & Coverage
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [ 16 ]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -147,7 +156,9 @@ jobs:
run: pnpm coverage:api
working-directory: ./frontend
- name: Submit coverage
run: ${{ github.workspace }}/frontend/apps/remark42/node_modules/.bin/codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
working-directory: ./frontend
codecov_yml_path: ./frontend/apps/remark42/codecov.yml
+64 -44
View File
@@ -18,35 +18,38 @@ jobs:
translations-check:
name: Translations check
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [16]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -62,35 +65,38 @@ jobs:
type-check:
name: Type check
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [16]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -106,35 +112,38 @@ jobs:
lint:
name: Eslint & Stylelint
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [16]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -151,57 +160,66 @@ jobs:
name: Size limit
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
permissions:
contents: read
pull-requests: write
env:
CI_JOB_NUMBER: 1
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Check bundle size
uses: andresz1/size-limit-action@v1
uses: andresz1/size-limit-action@94bc357df29c36c8f8d50ea497c3e225c3c95d1d
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
directory: ./frontend/apps/remark42
package_manager: pnpm
test:
name: Tests & Coverage
runs-on: ubuntu-latest
permissions:
contents: read
strategy:
matrix:
node: [16.15.1]
node: [16]
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Install node
uses: actions/setup-node@v3
uses: actions/setup-node@v6
with:
node-version: ${{ matrix.node }}
- name: Install pnpm
uses: pnpm/action-setup@v2.0.1
uses: pnpm/action-setup@v4.2.0
id: pnpm-install
with:
version: 7
version: 8
run_install: false
- name: Get pnpm store directory
id: pnpm-cache
run: |
echo "::set-output name=pnpm_cache_dir::$(pnpm store path)"
run: echo "pnpm_cache_dir=$(pnpm store path)" >> $GITHUB_ENV
- name: Setup pnpm cache
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.pnpm_cache_dir }}
path: ${{ env.pnpm_cache_dir }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-pnpm-store-
@@ -214,7 +232,9 @@ jobs:
run: pnpm coverage
working-directory: ./frontend/apps/remark42
- name: Submit coverage
run: ${{ github.workspace }}/frontend/apps/remark42/node_modules/.bin/codecov
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
working-directory: ./frontend/apps/remark42
codecov_yml_path: ./frontend/apps/remark42/codecov.yml
+111 -38
View File
@@ -13,57 +13,130 @@ on:
- ".github/workflows/ci-site.yml"
- "site/**"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
name: Build
runs-on: ubuntu-latest
name: Build site image (${{ matrix.platform }})
if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/')
runs-on: ${{ matrix.runner }}
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
artifact: linux-amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
artifact: linux-arm64
steps:
- name: checkout
uses: actions/checkout@v3
- name: set up QEMU
uses: docker/setup-qemu-action@v2
uses: actions/checkout@v6
with:
persist-credentials: false
- name: set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
uses: docker/setup-buildx-action@v3
- name: available platforms
run: echo ${{ steps.buildx.outputs.platforms }}
- name: login to ghcr.io
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: build and deploy master image to ghcr.io and dockerhub
if: ${{ github.ref == 'refs/heads/master' }}
- name: build and push by digest
id: build
uses: docker/build-push-action@v6
with:
context: ./site
platforms: ${{ matrix.platform }}
cache-from: type=gha,scope=site-${{ matrix.platform }}
cache-to: type=gha,scope=site-${{ matrix.platform }},mode=max
outputs: type=image,name=ghcr.io/umputun/remark42-site,push-by-digest=true,name-canonical=true,push=true
- name: export digest
run: |
mkdir -p /tmp/digests
digest="${{ steps.build.outputs.digest }}"
touch "/tmp/digests/${digest#sha256:}"
- name: upload digest
uses: actions/upload-artifact@v5
with:
name: site-digests-${{ matrix.artifact }}
path: /tmp/digests/*
retention-days: 1
merge:
name: Create site multi-arch manifest
runs-on: ubuntu-latest
needs: build
permissions:
contents: read
packages: write
steps:
- name: download digests
uses: actions/download-artifact@v6
with:
path: /tmp/digests
pattern: site-digests-*
merge-multiple: true
- name: verify all digests present
run: |
expected=2
actual=$(find /tmp/digests -maxdepth 1 -type f | wc -l)
if [ "$actual" -ne "$expected" ]; then
echo "Expected $expected digests, found $actual"
ls -la /tmp/digests
exit 1
fi
echo "All $expected digests present"
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: login to ghcr.io
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: create manifest and push
working-directory: /tmp/digests
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
working-directory: ./site
GITHUB_REF: ${{ github.ref }}
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo GITHUB_REF - $ref
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
docker buildx build --push --no-cache --platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42-site:${ref} .
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
docker buildx imagetools create \
-t ghcr.io/umputun/remark42-site:${ref} \
-t ghcr.io/umputun/remark42-site:latest \
$(printf 'ghcr.io/umputun/remark42-site@sha256:%s ' *)
else
docker buildx imagetools create \
-t ghcr.io/umputun/remark42-site:${ref} \
$(printf 'ghcr.io/umputun/remark42-site@sha256:%s ' *)
fi
- name: deploy tagged (latest) to ghcr.io and dockerhub
if: ${{ startsWith(github.ref, 'refs/tags/') }}
env:
GITHUB_PACKAGE_TOKEN: ${{ secrets.PKG_TOKEN }}
USERNAME: ${{ github.actor }}
GITHUB_SHA: ${{ github.sha}}
GITHUB_REF: ${{ github.ref}}
working-directory: ./site
run: |
ref="$(echo ${GITHUB_REF} | cut -d'/' -f3)"
echo "GITHUB_REF=$ref, GITHUB_SHA=${GITHUB_SHA}"
echo ${GITHUB_PACKAGE_TOKEN} | docker login ghcr.io -u ${USERNAME} --password-stdin
docker buildx build --push --no-cache --platform linux/amd64,linux/arm/v7,linux/arm64 \
-t ghcr.io/umputun/remark42-site:${ref} -t ghcr.io/umputun/remark42-site:latest .
deploy:
name: Deploy site
runs-on: ubuntu-latest
needs: merge
if: github.ref == 'refs/heads/master'
- name: remote site deployment from master
if: ${{ github.ref == 'refs/heads/master' }}
steps:
- name: trigger deployment
env:
UPDATER_KEY: ${{ secrets.UPDATER_KEY }}
run: curl https://jess.umputun.com/update/remark42-site/${UPDATER_KEY}
run: curl -sf https://jess.umputun.com/update/remark42-site/${UPDATER_KEY}
+214
View File
@@ -0,0 +1,214 @@
name: docker
on:
workflow_run:
workflows: [backend]
types: [completed]
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: true
jobs:
build:
name: Build Docker image (${{ matrix.platform }})
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event != 'pull_request' &&
(github.event.workflow_run.head_branch == 'master' ||
startsWith(github.event.workflow_run.head_branch, 'v'))
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
artifact: linux-amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
artifact: linux-arm64
runs-on: ${{ matrix.runner }}
steps:
- name: checkout
uses: actions/checkout@v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
persist-credentials: false
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: login to ghcr.io
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: login to DockerHub
uses: docker/login-action@v3
with:
username: umputun
password: ${{ secrets.DOCKER_HUB_TOKEN }}
- name: free disk space
run: |
sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf /usr/local/share/boost
docker system prune -af
- name: build and push to ghcr.io by digest
id: build-ghcr
uses: docker/build-push-action@v6
with:
context: .
platforms: ${{ matrix.platform }}
cache-from: type=gha,scope=${{ matrix.platform }}
cache-to: type=gha,scope=${{ matrix.platform }},mode=max
build-args: |
SKIP_BACKEND_TEST=true
SKIP_FRONTEND_TEST=true
CI=github
GITHUB_SHA=${{ github.event.workflow_run.head_sha }}
GIT_BRANCH=${{ github.event.workflow_run.head_branch }}
GITHUB_REF=refs/heads/${{ github.event.workflow_run.head_branch }}
outputs: type=image,name=ghcr.io/umputun/remark42,push-by-digest=true,name-canonical=true,push=true
- name: build and push to DockerHub by digest
id: build-dockerhub
uses: docker/build-push-action@v6
with:
context: .
platforms: ${{ matrix.platform }}
cache-from: type=gha,scope=${{ matrix.platform }}
build-args: |
SKIP_BACKEND_TEST=true
SKIP_FRONTEND_TEST=true
CI=github
GITHUB_SHA=${{ github.event.workflow_run.head_sha }}
GIT_BRANCH=${{ github.event.workflow_run.head_branch }}
GITHUB_REF=refs/heads/${{ github.event.workflow_run.head_branch }}
outputs: type=image,name=umputun/remark42,push-by-digest=true,name-canonical=true,push=true
- name: export digests
run: |
mkdir -p /tmp/digests/ghcr /tmp/digests/dockerhub
digest_ghcr="${{ steps.build-ghcr.outputs.digest }}"
digest_dockerhub="${{ steps.build-dockerhub.outputs.digest }}"
touch "/tmp/digests/ghcr/${digest_ghcr#sha256:}"
touch "/tmp/digests/dockerhub/${digest_dockerhub#sha256:}"
- name: upload ghcr digest
uses: actions/upload-artifact@v5
with:
name: digests-ghcr-${{ matrix.artifact }}
path: /tmp/digests/ghcr/*
retention-days: 1
- name: upload dockerhub digest
uses: actions/upload-artifact@v5
with:
name: digests-dockerhub-${{ matrix.artifact }}
path: /tmp/digests/dockerhub/*
retention-days: 1
merge:
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: build
permissions:
contents: read
packages: write
steps:
- name: download ghcr digests
uses: actions/download-artifact@v6
with:
path: /tmp/digests/ghcr
pattern: digests-ghcr-*
merge-multiple: true
- name: download dockerhub digests
uses: actions/download-artifact@v6
with:
path: /tmp/digests/dockerhub
pattern: digests-dockerhub-*
merge-multiple: true
- name: verify all digests present
run: |
expected=2
for registry in ghcr dockerhub; do
actual=$(find /tmp/digests/$registry -maxdepth 1 -type f | wc -l)
if [ "$actual" -ne "$expected" ]; then
echo "Expected $expected digests for $registry, found $actual"
ls -la /tmp/digests/$registry
exit 1
fi
done
echo "All digests present for both registries"
- name: set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: login to ghcr.io
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.PKG_TOKEN }}
- name: login to DockerHub
uses: docker/login-action@v3
with:
username: umputun
password: ${{ secrets.DOCKER_HUB_TOKEN }}
- name: create ghcr.io manifest and push
working-directory: /tmp/digests/ghcr
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
if [[ "$HEAD_BRANCH" == v* ]]; then
docker buildx imagetools create \
-t ghcr.io/umputun/remark42:${HEAD_BRANCH} \
-t ghcr.io/umputun/remark42:latest \
$(printf 'ghcr.io/umputun/remark42@sha256:%s ' *)
else
docker buildx imagetools create \
-t ghcr.io/umputun/remark42:${HEAD_BRANCH} \
$(printf 'ghcr.io/umputun/remark42@sha256:%s ' *)
fi
- name: create DockerHub manifest and push
working-directory: /tmp/digests/dockerhub
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
if [[ "$HEAD_BRANCH" == v* ]]; then
docker buildx imagetools create \
-t umputun/remark42:${HEAD_BRANCH} \
-t umputun/remark42:latest \
$(printf 'umputun/remark42@sha256:%s ' *)
else
docker buildx imagetools create \
-t umputun/remark42:${HEAD_BRANCH} \
$(printf 'umputun/remark42@sha256:%s ' *)
fi
deploy:
name: Deploy to remark42.com
runs-on: ubuntu-latest
needs: merge
if: github.event.workflow_run.head_branch == 'master'
steps:
- name: trigger deployment
env:
UPDATER_KEY: ${{ secrets.UPDATER_KEY }}
run: curl -sf https://jess.umputun.com/update/remark42-core/${UPDATER_KEY}
+10 -2
View File
@@ -4,29 +4,37 @@ on:
push:
branches: [master]
paths:
- ".github/workflows/e2e-tests.yml"
- "frontend/apps/remark42/**"
- "frontend/e2e/**"
- "frontend/Dockerfile.e2e"
pull_request:
branches: [master]
paths:
- ".github/workflows/e2e-tests.yml"
- "frontend/apps/remark42/**"
- "frontend/e2e/**"
- "frontend/Dockerfile.e2e"
jobs:
tests:
name: Tests
timeout-minutes: 60
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v6
with:
persist-credentials: false
- name: Build & run containers
id: tests
run: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -f compose-e2e-test.yml up --build --quiet-pull --exit-code-from tests
- uses: actions/upload-artifact@v2
- uses: actions/upload-artifact@v5
if: always()
with:
name: playwright-report
+1
View File
@@ -25,3 +25,4 @@ compose-private.yml
/backend/_example/*/vendor
http-client.env.json
/playwright-report/
/backend/app/cmd/var
+36
View File
@@ -0,0 +1,36 @@
# Remark42 Development Guidelines
## Build/Test/Lint Commands
- **Backend**:
- Run server: `make rundev`
- Build: `make backend`
- Race test: `make race_test`
- **Backend Testing**:
- Run all tests: `cd backend/app && go test -timeout=60s -count 1 ./...`
- Run single test: `cd backend/app && go test -run TestName ./path/to/package`
- **IMPORTANT**: Run example tests: `cd backend/_example/memory_store && go test -race ./... && go build -race ./...`
- **Frontend**:
- Development: `cd frontend && pnpm dev:app`
- Tests: `cd frontend && pnpm test`
- **Lint**:
- Backend: `cd backend && golangci-lint run`
- **IMPORTANT**: Example lint: `cd backend/_example/memory_store && golangci-lint run --config ../../.golangci.yml`
- Frontend: `cd frontend && pnpm lint`
- **Before committing**: Always run tests and linter on both main backend AND examples
## Code Style
- **Backend**: Formatting with golangci-lint, strict error handling
- **Frontend**: TypeScript with ESLint, Stylelint and Prettier
- **Imports**: Group stdlib, external packages, then internal packages
- **CSS**: CSS Modules for new components (`component.module.css`)
## Key Backend Packages
- **Web/API**: `github.com/go-chi/chi/v5`, `github.com/go-pkgz/rest`
- **Auth**: `github.com/go-pkgz/auth/v2`
- **Logging**: `github.com/go-pkgz/lgr`
- **Testing**: `github.com/stretchr/testify`
- **Notifications**: `github.com/go-pkgz/notify`
## Repository Structure
- Backend: Go server using BoltDB for storage
- Frontend: Preact/Redux-based UI with iframe embedding
+9 -9
View File
@@ -1,4 +1,4 @@
FROM --platform=$BUILDPLATFORM node:16.15.1-alpine AS frontend-deps
FROM --platform=$BUILDPLATFORM node:16.20-alpine AS frontend-deps
ARG SKIP_FRONTEND_TEST
ARG SKIP_FRONTEND_BUILD
@@ -11,7 +11,7 @@ COPY ./frontend/apps/remark42/package.json /srv/frontend/apps/remark42/
RUN \
if [[ -z "$SKIP_FRONTEND_BUILD" || -z "$SKIP_FRONTEND_TEST" ]]; then \
apk add --no-cache --update git && \
npm i -g pnpm; \
npm i -g pnpm@8; \
fi
RUN --mount=type=cache,id=pnpm,target=/root/.pnpm-store/v3 \
@@ -45,7 +45,7 @@ RUN \
echo 'Skip frontend build'; \
fi
FROM umputun/baseimage:buildgo-v1.9.2 as build-backend
FROM umputun/baseimage:buildgo-v1.17.0 AS build-backend
ARG CI
ARG GITHUB_REF
@@ -54,15 +54,15 @@ ARG GIT_BRANCH
ARG SKIP_BACKEND_TEST
ARG BACKEND_TEST_TIMEOUT
# install gcc in order to be able to go test package with -race
RUN apk --no-cache add gcc libc-dev
ADD backend /build/backend
# to embed the frontend files statically into Remark42 binary
COPY --from=build-frontend /srv/frontend/apps/remark42/public/ /build/backend/app/cmd/web/
RUN find /build/backend/app/cmd/web/ -regex '.*\.\(html\|js\|mjs\)$' -print -exec sed -i "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
WORKDIR /build/backend
# install gcc in order to be able to go test package with -race
RUN apk --no-cache add gcc libc-dev
RUN echo go version: `go version`
# run tests
@@ -70,7 +70,7 @@ RUN \
cd app && \
if [ -z "$SKIP_BACKEND_TEST" ] ; then \
CGO_ENABLED=1 go test -race -p 1 -timeout="${BACKEND_TEST_TIMEOUT:-300s}" -covermode=atomic -coverprofile=/profile.cov_tmp ./... && \
cat /profile.cov_tmp | grep -v "_mock.go" > /profile.cov ; \
cat /profile.cov_tmp | grep -v "_mock.go" > /profile.cov && \
golangci-lint run --config ../.golangci.yml ./... ; \
else \
echo "skip backend tests and linter" \
@@ -81,7 +81,7 @@ RUN \
echo "version=$version" && \
go build -o remark42 -ldflags "-X main.revision=${version} -s -w" ./app
FROM umputun/baseimage:app-v1.9.2
FROM umputun/baseimage:app-v1.17.0
ARG GITHUB_SHA
@@ -89,7 +89,7 @@ LABEL org.opencontainers.image.authors="Umputun <umputun@gmail.com>" \
org.opencontainers.image.description="Remark42 comment engine" \
org.opencontainers.image.documentation="https://remark42.com/docs/getting-started/" \
org.opencontainers.image.licenses="MIT" \
org.opencontainers.image.source="https://github.com/umputun/remark42.git" \
org.opencontainers.image.source="https://github.com/umputun/remark42" \
org.opencontainers.image.title="Remark42" \
org.opencontainers.image.url="https://remark42.com/" \
org.opencontainers.image.revision="${GITHUB_SHA}"
+3 -3
View File
@@ -1,4 +1,4 @@
FROM node:16.15.1-alpine AS frontend-deps
FROM node:16-alpine AS frontend-deps
ENV CI=true
@@ -7,7 +7,7 @@ WORKDIR /srv/frontend
COPY ./frontend/package.json ./frontend/pnpm-lock.yaml ./frontend/pnpm-workspace.yaml /srv/frontend/
COPY ./frontend/apps/remark42/package.json /srv/frontend/apps/remark42/package.json
RUN apk add --no-cache --update git && npm i -g pnpm
RUN apk add --no-cache --update git && npm i -g pnpm@8
RUN --mount=type=cache,id=pnpm,target=/root/.pnpm-store/v3 pnpm i
FROM frontend-deps AS build-frontend
@@ -19,7 +19,7 @@ WORKDIR /srv/frontend/apps/remark42/
COPY ./frontend/apps/remark42/ /srv/frontend/apps/remark42/
RUN pnpm build
FROM umputun/baseimage:buildgo-v1.9.2 as build-backend
FROM umputun/baseimage:buildgo-v1.14.0 AS build-backend
ARG GITHUB_TOKEN
ARG GITHUB_REF
+5 -5
View File
@@ -11,7 +11,7 @@ bin:
docker rm -f remark42.bin
docker:
DOCKER_BUILDKIT=1 docker build -t umputun/remark42 --build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) \
DOCKER_BUILDKIT=1 docker build -t umputun/remark42 -t ghcr.io/umputun/remark42 --build-arg GITHUB_REF=$(GITHUB_REF) --build-arg GITHUB_SHA=$(GITHUB_SHA) \
--build-arg CI=true --build-arg SKIP_FRONTEND_TEST=true --build-arg SKIP_BACKEND_TEST=true .
dockerx:
@@ -39,15 +39,15 @@ race_test:
cd backend/app && go test -race -timeout=60s -count 1 ./...
backend:
docker-compose -f compose-dev-backend.yml build
docker compose -f compose-dev-backend.yml build
frontend:
docker-compose -f compose-dev-frontend.yml build
docker compose -f compose-dev-frontend.yml build
rundev:
SKIP_BACKEND_TEST=true SKIP_FRONTEND_TEST=true GITHUB_REF=$(GITHUB_REF) GITHUB_SHA=$(GITHUB_SHA) CI=true \
docker-compose -f compose-private.yml build
docker-compose -f compose-private.yml up
docker compose -f compose-private.yml build
docker compose -f compose-private.yml up
e2e:
docker compose -f compose-e2e-test.yml up --build --quiet-pull --exit-code-from tests
+1 -1
View File
@@ -2,7 +2,7 @@
Remark42 is a self-hosted, lightweight and simple (yet functional) comment engine, which doesn't spy on users. It can be embedded into blogs, articles, or any other place where readers add comments.
* Social login via Google, Twitter, Facebook, Microsoft, GitHub, Yandex, Patreon and Telegram
* Social login via Google, Facebook, Microsoft, GitHub, Apple, Yandex, Patreon, Discord and Telegram
* Login via email
* Optional anonymous access
* Multi-level nested comments with both tree and plain presentations
+62 -61
View File
@@ -1,68 +1,69 @@
run:
timeout: 5m
output:
format: tab
skip-dirs:
- vendor
linters-settings:
govet:
check-shadowing: true
maligned:
suggest-new: true
goconst:
min-len: 2
min-occurrences: 2
misspell:
locale: US
lll:
line-length: 140
gocritic:
enabled-tags:
- performance
- style
- experimental
disabled-checks:
- wrapperFunc
# TODO: feel free to remove these excludes and fix the code
- hugeParam
- rangeValCopy
version: "2"
linters:
default: none
enable:
- bodyclose
- megacheck
- revive
- govet
- unconvert
- gas
- gocyclo
- copyloopvar
- dupl
- gochecknoinits
- gocritic
- gocyclo
- gosec
- govet
- ineffassign
- misspell
- nakedret
- prealloc
- revive
- staticcheck
- unconvert
- unparam
- unused
- typecheck
- ineffassign
- stylecheck
- gochecknoinits
- exportloopref
- gocritic
- nakedret
- gosimple
- prealloc
fast: false
disable-all: true
issues:
exclude-rules:
- text: "at least one file in a package should have a package comment"
linters:
- stylecheck
- text: "package-comments: should have a package comment"
linters:
- revive
- path: _test\.go
linters:
- gosec
- dupl
exclude-use-default: false
settings:
goconst:
min-len: 2
min-occurrences: 2
gocritic:
disabled-checks:
- wrapperFunc
- hugeParam
- rangeValCopy
enabled-tags:
- performance
- style
- experimental
govet:
enable:
- shadow
lll:
line-length: 140
misspell:
locale: US
exclusions:
generated: lax
rules:
- linters:
- staticcheck
text: at least one file in a package should have a package comment
- linters:
- revive
text: 'package-comments: should have a package comment'
- linters:
- revive
text: 'var-naming: avoid meaningless package names'
- linters:
- dupl
- gosec
path: _test\.go
paths:
- vendor
- third_party$
- builtin$
- examples$
formatters:
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
+3 -3
View File
@@ -1,11 +1,11 @@
FROM umputun/baseimage:buildgo-v1.9.2 as build-backend
FROM umputun/baseimage:buildgo-v1.17.0 AS build-backend
ADD backend /build/backend
WORKDIR /build/backend/_example/memory_store
RUN go build -o /build/bin/memory_store -ldflags "-X main.revision=0.0.0 -s -w"
FROM umputun/baseimage:app-v1.9.2
FROM umputun/baseimage:app-v1.17.0
ARG GITHUB_SHA
@@ -13,7 +13,7 @@ LABEL org.opencontainers.image.authors="Umputun <umputun@gmail.com>" \
org.opencontainers.image.description="Remark42 comment engine example JRPC memory store" \
org.opencontainers.image.documentation="https://github.com/umputun/remark42/tree/master/backend/_example/memory_store" \
org.opencontainers.image.licenses="MIT" \
org.opencontainers.image.source="https://github.com/umputun/remark42.git" \
org.opencontainers.image.source="https://github.com/umputun/remark42" \
org.opencontainers.image.title="Remark42 JRPC example memory store" \
org.opencontainers.image.url="https://remark42.com/" \
org.opencontainers.image.revision="${GITHUB_SHA}"
+3 -3
View File
@@ -4,9 +4,9 @@
In order to run remark42 with memory_store copy provided `compose-dev-memstore.yml` to the root directory and run:
1. `docker-compose -f compose-dev-memstore.yml build`
1. `docker-compose -f compose-dev-memstore.yml up`
1. `docker compose -f compose-dev-memstore.yml build`
1. `docker compose -f compose-dev-memstore.yml up`
As usual, demo site will run on http://127.0.0.1:8080/web/
note: in order to work with the latest (current) version of master `go.mod` uses replacement directive for the backend package. In real-life usage `replace github.com/umputun/remark42/backend => ../../` should not be used.
note: in order to work with the latest (current) version of master `go.mod` uses replacement directive for the backend package. In real-life usage `replace github.com/umputun/remark42/backend => ../../` should not be used.
+23 -25
View File
@@ -293,17 +293,17 @@ func (m *MemData) UserDetail(req engine.UserDetailRequest) ([]engine.UserDetailE
defer m.mu.Unlock()
if req.Update == "" { // read detail value, no update requested
return m.getUserDetail(req)
return m.getUserDetail(req), nil
}
return m.setUserDetail(req)
return m.setUserDetail(req), nil
case engine.AllUserDetails:
// list of all details returned in case request is a read request
// (Update is not set) and does not have UserID or Detail set
if req.Update == "" && req.UserID == "" { // read list of all details
m.mu.Lock()
defer m.mu.Unlock()
return m.listDetails(req.Locator)
return m.listDetails(req.Locator), nil
}
return nil, fmt.Errorf("unsupported request with userdetail all")
default:
@@ -319,7 +319,8 @@ func (m *MemData) Delete(req engine.DeleteRequest) error {
switch {
case req.UserDetail != "": // delete user detail
return m.deleteUserDetail(req.Locator, req.UserID, req.UserDetail)
m.deleteUserDetail(req.Locator, req.UserID, req.UserDetail)
return nil
case req.Locator.URL != "" && req.CommentID != "" && req.UserDetail == "": // delete comment
return m.deleteComment(req.Locator, req.CommentID, req.DeleteMode)
@@ -332,7 +333,8 @@ func (m *MemData) Delete(req engine.DeleteRequest) error {
return e
}
}
return m.deleteUserDetail(req.Locator, req.UserID, engine.AllUserDetails)
m.deleteUserDetail(req.Locator, req.UserID, engine.AllUserDetails)
return nil
case req.Locator.SiteID != "" && req.Locator.URL == "" && req.CommentID == "" && req.UserID == "" && req.UserDetail == "": // delete site
if _, ok := m.posts[req.Locator.SiteID]; !ok {
@@ -389,10 +391,7 @@ func (m *MemData) checkFlag(req engine.FlagRequest) (val bool) {
func (m *MemData) setFlag(req engine.FlagRequest) (res bool, err error) {
status := false
if req.Update == engine.FlagTrue {
status = true
}
status := req.Update == engine.FlagTrue
switch req.Flag {
@@ -437,29 +436,29 @@ func (m *MemData) setFlag(req engine.FlagRequest) (res bool, err error) {
// getUserDetail returns UserDetailEntry with requested userDetail (omitting other details)
// as an only element of the slice.
func (m *MemData) getUserDetail(req engine.UserDetailRequest) ([]engine.UserDetailEntry, error) {
func (m *MemData) getUserDetail(req engine.UserDetailRequest) []engine.UserDetailEntry {
if meta, ok := m.metaUsers[req.UserID]; ok {
if meta.SiteID != req.Locator.SiteID {
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
switch req.Detail {
case engine.UserEmail:
return []engine.UserDetailEntry{{UserID: req.UserID, Email: meta.Details.Email}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Email: meta.Details.Email}}
case engine.UserTelegram:
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: meta.Details.Telegram}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: meta.Details.Telegram}}
}
}
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
// setUserDetail sets requested userDetail, returning complete updated UserDetailEntry as an onlyIps
// element of the slice in case of success
func (m *MemData) setUserDetail(req engine.UserDetailRequest) ([]engine.UserDetailEntry, error) {
func (m *MemData) setUserDetail(req engine.UserDetailRequest) []engine.UserDetailEntry {
var entry metaUser
if meta, ok := m.metaUsers[req.UserID]; ok {
if meta.SiteID != req.Locator.SiteID {
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
entry = meta
}
@@ -476,42 +475,42 @@ func (m *MemData) setUserDetail(req engine.UserDetailRequest) ([]engine.UserDeta
case engine.UserEmail:
entry.Details.Email = req.Update
m.metaUsers[req.UserID] = entry
return []engine.UserDetailEntry{{UserID: req.UserID, Email: req.Update}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Email: req.Update}}
case engine.UserTelegram:
entry.Details.Telegram = req.Update
m.metaUsers[req.UserID] = entry
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: req.Update}}, nil
return []engine.UserDetailEntry{{UserID: req.UserID, Telegram: req.Update}}
}
return []engine.UserDetailEntry{}, nil
return []engine.UserDetailEntry{}
}
// listDetails lists all available users details for given siteID
func (m *MemData) listDetails(loc store.Locator) ([]engine.UserDetailEntry, error) {
func (m *MemData) listDetails(loc store.Locator) []engine.UserDetailEntry {
var res []engine.UserDetailEntry
for _, u := range m.metaUsers {
if u.SiteID == loc.SiteID {
res = append(res, u.Details)
}
}
return res, nil
return res
}
// deleteUserDetail deletes requested UserDetail or whole UserDetailEntry,
// deletion of the absent entry doesn't produce error.
// Trying to delete user with wrong siteID doesn't to anything and doesn't produce error.
func (m *MemData) deleteUserDetail(locator store.Locator, userID string, userDetail engine.UserDetail) error {
func (m *MemData) deleteUserDetail(locator store.Locator, userID string, userDetail engine.UserDetail) {
var entry metaUser
if meta, ok := m.metaUsers[userID]; ok {
if meta.SiteID != locator.SiteID {
return nil
return
}
entry = meta
}
if entry == (metaUser{}) || entry.Details == (engine.UserDetailEntry{}) {
// absent entry means that we should not do anything
return nil
return
}
switch userDetail {
@@ -529,7 +528,6 @@ func (m *MemData) deleteUserDetail(locator store.Locator, userID string, userDet
}
m.metaUsers[userID] = entry
return nil
}
func (m *MemData) get(loc store.Locator, commentID string) (store.Comment, error) {
@@ -286,7 +286,7 @@ func TestMemData_CountUser(t *testing.T) {
func TestMemData_InfoPost(t *testing.T) {
b := prepMem(t)
ts := func(min int) time.Time { return time.Date(2017, 12, 20, 15, 18, min, 0, time.Local).In(time.UTC) }
ts := func(minute int) time.Time { return time.Date(2017, 12, 20, 15, 18, minute, 0, time.Local).In(time.UTC) }
// add one more for https://radio-t.com/2
comment := store.Comment{
@@ -547,12 +547,12 @@ func TestMemData_FlagListBlocked(t *testing.T) {
assert.NoError(t, err)
blockedList := toBlocked(vv)
var blockedIds = make([]string, len(blockedList))
var blockedIDs = make([]string, len(blockedList))
for i, x := range blockedList {
blockedIds[i] = x.ID
blockedIDs[i] = x.ID
}
require.Equal(t, 2, len(blockedList), b.metaUsers)
assert.ElementsMatch(t, []string{"user1", "user2"}, blockedIds)
assert.ElementsMatch(t, []string{"user1", "user2"}, blockedIDs)
t.Logf("%+v", blockedList)
// check block expiration
@@ -70,6 +70,17 @@ func (m *MemImage) Load(id string) ([]byte, error) {
return img, nil
}
// Delete image by ID
func (m *MemImage) Delete(id string) error {
m.mu.Lock()
// delete key from permanent and staging storage
delete(m.images, id)
delete(m.insertTime, id)
delete(m.imagesStaging, id)
m.mu.Unlock()
return nil
}
// Commit moves image from staging to permanent
func (m *MemImage) Commit(id string) error {
m.mu.RLock()
@@ -18,7 +18,7 @@ import (
)
// gopher png for test, from https://golang.org/src/image/png/example_test.go
const gopher = "iVBORw0KGgoAAAANSUhEUgAAAEsAAAA8CAAAAAALAhhPAAAFfUlEQVRYw62XeWwUVRzHf2" +
const rawGopher = "iVBORw0KGgoAAAANSUhEUgAAAEsAAAA8CAAAAAALAhhPAAAFfUlEQVRYw62XeWwUVRzHf2" +
"+OPbo9d7tsWyiyaZti6eWGAhISoIGKECEKCAiJJkYTiUgTMYSIosYYBBIUIxoSPIINEBDi2VhwkQrVsj1ESgu9doHWdrul7ba" +
"73WNm3vOPtsseM9MdwvvrzTs+8/t95ze/33sI5BqiabU6m9En8oNjduLnAEDLUsQXFF8tQ5oxK3vmnNmDSMtrncks9Hhtt" +
"/qeWZapHb1ha3UqYSWVl2ZmpWgaXMXGohQAvmeop3bjTRtv6SgaK/Pb9/bFzUrYslbFAmHPp+3WhAYdr+7GN/YnpN46Opv55VDs" +
@@ -38,7 +38,9 @@ const gopher = "iVBORw0KGgoAAAANSUhEUgAAAEsAAAA8CAAAAAALAhhPAAAFfUlEQVRYw62XeWwU
"1y98c3D27eppUjsZ6fql3jcd5rUe7+ZIlLNQny3Rd+E5Tct3WVhTM5RBCEdiEK0b6B+/ca2gYU393nFj/n1AygRQxPIUA043M42u85+z2S" +
"nssKrPl8Mx76NL3E6eXc3be7OD+H4WHbJkKI8AU8irbITQjZ+0hQcPEgId/Fn/pl9crKH02+5o2b9T/eMx7pKoskYgAAAABJRU5ErkJggg=="
func gopherPNG() io.Reader { return base64.NewDecoder(base64.StdEncoding, strings.NewReader(gopher)) }
func gopherPNG() io.Reader {
return base64.NewDecoder(base64.StdEncoding, strings.NewReader(rawGopher))
}
func TestMemImage_LoadAfterSave(t *testing.T) {
svc := NewMemImageStore()
@@ -57,7 +59,8 @@ func TestMemImage_LoadAfterSave(t *testing.T) {
assert.NoError(t, err)
assert.Equal(t, gopher, img)
svc.ResetCleanupTimer(id)
err = svc.ResetCleanupTimer(id)
assert.NoError(t, err)
err = svc.Commit(id)
assert.NoError(t, err)
@@ -70,6 +73,26 @@ func TestMemImage_LoadAfterSave(t *testing.T) {
assert.Equal(t, gopher, img)
}
func TestMemImage_LoadAfterDelete(t *testing.T) {
svc := NewMemImageStore()
gopher, err := io.ReadAll(gopherPNG())
assert.NoError(t, err)
id := "test_img"
err = svc.Save(id, gopher)
assert.NoError(t, err)
err = svc.Delete(id)
assert.NoError(t, err)
img, err := svc.Load(id)
assert.EqualError(t, err, "image test_img not found")
assert.Empty(t, img)
err = svc.ResetCleanupTimer(id)
assert.EqualError(t, err, "image test_img not found")
}
func TestMemImage_CommitFail(t *testing.T) {
svc := NewMemImageStore()
err := svc.Commit("test_id")
@@ -11,7 +11,7 @@ services:
args:
- SKIP_BACKEND_TEST=true
- SKIP_FRONTEND_TEST=true
image: umputun/remark42:dev
image: ghcr.io/umputun/remark42:dev
container_name: "remark42-dev"
hostname: "remark42-dev"
restart: always
+23 -25
View File
@@ -1,42 +1,40 @@
module github.com/umputun/remark42/memory_store
go 1.17
go 1.25
require (
github.com/go-pkgz/jrpc v0.3.0
github.com/go-pkgz/lgr v0.10.4
github.com/jessevdk/go-flags v1.5.0
github.com/stretchr/testify v1.8.1
github.com/umputun/remark42/backend v1.10.1
github.com/go-pkgz/jrpc v0.4.0
github.com/go-pkgz/lgr v0.12.1
github.com/jessevdk/go-flags v1.6.1
github.com/stretchr/testify v1.11.1
github.com/umputun/remark42/backend v1.1000.0
)
require (
github.com/Depado/bfchroma/v2 v2.0.0 // indirect
github.com/PuerkitoBio/goquery v1.8.0 // indirect
github.com/ajg/form v1.5.1 // indirect
github.com/alecthomas/chroma/v2 v2.3.0 // indirect
github.com/andybalholm/cascadia v1.3.1 // indirect
github.com/PuerkitoBio/goquery v1.11.0 // indirect
github.com/alecthomas/chroma/v2 v2.21.1 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/didip/tollbooth/v7 v7.0.1 // indirect
github.com/didip/tollbooth_chi v0.0.0-20220719025231-d662a7f6928f // indirect
github.com/dlclark/regexp2 v1.4.0 // indirect
github.com/go-chi/chi/v5 v5.0.7 // indirect
github.com/go-chi/render v1.0.2 // indirect
github.com/go-pkgz/expirable-cache v0.1.0 // indirect
github.com/go-pkgz/rest v1.16.0 // indirect
github.com/gorilla/css v1.0.0 // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/go-pkgz/rest v1.20.6 // indirect
github.com/go-pkgz/routegroup v1.6.0 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/microcosm-cc/bluemonday v1.0.20 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/microcosm-cc/bluemonday v1.0.27 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rs/xid v1.4.0 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
go.etcd.io/bbolt v1.3.6 // indirect
golang.org/x/image v0.0.0-20220617043117-41969df76e82 // indirect
golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b // indirect
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
go.etcd.io/bbolt v1.4.3 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/image v0.34.0 // indirect
golang.org/x/net v0.48.0 // indirect
golang.org/x/sys v0.39.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
replace github.com/umputun/remark42/backend => ../../
replace github.com/umputun/remark42/backend v1.1000.0 => ../../
+114 -70
View File
@@ -1,92 +1,136 @@
github.com/Depado/bfchroma/v2 v2.0.0 h1:IRpN9BPkNwEpR6w1ectIcNWOuhDSLx+8f1pn83fzxx8=
github.com/Depado/bfchroma/v2 v2.0.0/go.mod h1:wFwW/Pw8Tnd0irzgO9Zxtxgzp3aPS8qBWlyadxujxmw=
github.com/PuerkitoBio/goquery v1.8.0 h1:PJTF7AmFCFKk1N6V6jmKfrNH9tV5pNE6lZMkG0gta/U=
github.com/PuerkitoBio/goquery v1.8.0/go.mod h1:ypIiRMtY7COPGk+I/YbZLbxsxn9g5ejnI2HSMtkjZvI=
github.com/ajg/form v1.5.1 h1:t9c7v8JUKu/XxOGBU0yjNpaMloxGEJhUkqFRq0ibGeU=
github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY=
github.com/alecthomas/chroma/v2 v2.3.0 h1:83xfxrnjv8eK+Cf8qZDzNo3PPF9IbTWHs7z28GY6D0U=
github.com/alecthomas/chroma/v2 v2.3.0/go.mod h1:mZxeWZlxP2Dy+/8cBob2PYd8O2DwNAzave5AY7A2eQw=
github.com/alecthomas/repr v0.1.0 h1:ENn2e1+J3k09gyj2shc0dHr/yjaWSHRlrJ4DPMevDqE=
github.com/andybalholm/cascadia v1.3.1 h1:nhxRkql1kdYCc8Snf7D5/D3spOX+dBgjA6u8x004T2c=
github.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA=
github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw=
github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ=
github.com/alecthomas/assert/v2 v2.11.0 h1:2Q9r3ki8+JYXvGsDyBXwH3LcJ+WK5D0gc5E8vS6K3D0=
github.com/alecthomas/assert/v2 v2.11.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/chroma/v2 v2.21.1 h1:FaSDrp6N+3pphkNKU6HPCiYLgm8dbe5UXIXcoBhZSWA=
github.com/alecthomas/chroma/v2 v2.21.1/go.mod h1:NqVhfBR0lte5Ouh3DcthuUCTUpDC9cxBOfyMbMQPs3o=
github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/didip/tollbooth/v7 v7.0.0/go.mod h1:VZhDSGl5bDSPj4wPsih3PFa4Uh9Ghv8hgacaTm5PRT4=
github.com/didip/tollbooth/v7 v7.0.1 h1:TkT4sBKoQoHQFPf7blQ54iHrZiTDnr8TceU+MulVAog=
github.com/didip/tollbooth/v7 v7.0.1/go.mod h1:VZhDSGl5bDSPj4wPsih3PFa4Uh9Ghv8hgacaTm5PRT4=
github.com/didip/tollbooth_chi v0.0.0-20220719025231-d662a7f6928f h1:jtKwihcLmUC9BAhoJ9adCUqdSSZcOdH2KL7mPTUm2aw=
github.com/didip/tollbooth_chi v0.0.0-20220719025231-d662a7f6928f/go.mod h1:q9C80dnsuVRP2dAskjnXRNWdUJqtGgwG9wNrzt0019s=
github.com/dlclark/regexp2 v1.4.0 h1:F1rxgk7p4uKjwIQxBs9oAXe5CqrXlCduYEJvrF4u93E=
github.com/dlclark/regexp2 v1.4.0/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
github.com/go-chi/chi/v5 v5.0.7 h1:rDTPXLDHGATaeHvVlLcR4Qe0zftYethFucbjVQ1PxU8=
github.com/go-chi/chi/v5 v5.0.7/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
github.com/go-chi/render v1.0.1/go.mod h1:pq4Rr7HbnsdaeHagklXub+p6Wd16Af5l9koip1OvJns=
github.com/go-chi/render v1.0.2 h1:4ER/udB0+fMWB2Jlf15RV3F4A2FDuYi/9f+lFttR/Lg=
github.com/go-chi/render v1.0.2/go.mod h1:/gr3hVkmYR0YlEy3LxCuVRFzEu9Ruok+gFqbIofjao0=
github.com/go-pkgz/expirable-cache v0.1.0 h1:3bw0m8vlTK8qlwz5KXuygNBTkiKRTPrAGXU0Ej2AC1g=
github.com/go-pkgz/expirable-cache v0.1.0/go.mod h1:GTrEl0X+q0mPNqN6dtcQXksACnzCBQ5k/k1SwXJsZKs=
github.com/go-pkgz/jrpc v0.3.0 h1:Fls38KqPsHzvp0FWfivr6cGnncC+iFBodHBqvUPY+0U=
github.com/go-pkgz/jrpc v0.3.0/go.mod h1:MFtKs75JESiSqVicsQkgN2iDFFuCd3gVT1/vKiwRi00=
github.com/go-pkgz/lgr v0.10.4 h1:l7qyFjqEZgwRgaQQSEp6tve4A3OU80VrfzpvtEX8ngw=
github.com/go-pkgz/lgr v0.10.4/go.mod h1:CD0s1z6EFpIUplV067gitF77tn25JItzwHNKAPqeCF0=
github.com/go-pkgz/rest v1.15.6/go.mod h1:KUWAqbDteYGS/CiXftomQsKjtEOifXsJ36Ka0skYbmk=
github.com/go-pkgz/rest v1.16.0 h1:3An51XJXUq1XtICHvfAEp048ZCbUHBEga3g0WEu3zK0=
github.com/go-pkgz/rest v1.16.0/go.mod h1:HHlLOt02NJc2sgffXBF6hYVMcRo4Gz3vjg43zTzN7VM=
github.com/gorilla/css v1.0.0 h1:BQqNyPTi50JCFMTw/b67hByjMVXZRwGha6wxVGkeihY=
github.com/gorilla/css v1.0.0/go.mod h1:Dn721qIggHpt4+EFCcTLTU/vk5ySda2ReITrtgBl60c=
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/go-pkgz/jrpc v0.4.0 h1:oD7xiGrzDkndkuCjeHGugQXxbggLSV7O1QmHhoc5pYY=
github.com/go-pkgz/jrpc v0.4.0/go.mod h1:JFoY3bRjRyx4M3CbEVDFQStMB1m2gmQ7OjqFK7q3kOo=
github.com/go-pkgz/lgr v0.12.1 h1:8GVfG2rSARq3Eaj5PP158rtBR2LHVGkwioIkQBGbvKg=
github.com/go-pkgz/lgr v0.12.1/go.mod h1:A4AxjOthFVFK6jRnVYMeusno5SeDAxcLVHd0kI/lN/Y=
github.com/go-pkgz/rest v1.20.6 h1:O/IhQ3I2cS4bJYvL1TLcy63w2OcXZTTBG3R+wTIqPS4=
github.com/go-pkgz/rest v1.20.6/go.mod h1:NY+MX1is2kJckJt+nHDNovS/5j9jmF4yQuSno4qg7XU=
github.com/go-pkgz/routegroup v1.6.0 h1:44XHZgF6JIIldRlv+zjg6SygULASmjifnfIQjwCT0e4=
github.com/go-pkgz/routegroup v1.6.0/go.mod h1:Pmu04fhgWhRtBMIJ8HXppnnzOPjnL/IEPBIdO2zmeqg=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=
github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/jessevdk/go-flags v1.5.0 h1:1jKYvbxEjfUl0fmqTCOfonvskHHXMjBySTLW4y9LFvc=
github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/microcosm-cc/bluemonday v1.0.20 h1:flpzsq4KU3QIYAYGV/szUat7H+GPOXR0B2JU5A1Wp8Y=
github.com/microcosm-cc/bluemonday v1.0.20/go.mod h1:yfBmMi8mxvaZut3Yytv+jTXRY8mxyjJ0/kQBTElld50=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/jessevdk/go-flags v1.6.1 h1:Cvu5U8UGrLay1rZfv/zP7iLpSHGUZ/Ou68T0iX1bBK4=
github.com/jessevdk/go-flags v1.6.1/go.mod h1:Mk8T1hIAWpOiJiHa9rJASDK2UGWji0EuPGBnNLMooyc=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
github.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY=
github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
go.etcd.io/bbolt v1.3.6 h1:/ecaJf0sk1l4l6V4awd65v2C3ILy7MSj+s/x1ADCIMU=
go.etcd.io/bbolt v1.3.6/go.mod h1:qXsaaIqmgQH0T+OPdb99Bf+PKfBBQVAdyD6TY9G8XM4=
golang.org/x/image v0.0.0-20220617043117-41969df76e82 h1:KpZB5pUSBvrHltNEdK/tw0xlPeD13M6M6aGP32gKqiw=
golang.org/x/image v0.0.0-20220617043117-41969df76e82/go.mod h1:doUCurBvlfPMKfmIpRIywoHmhN3VyhnoFDbvIEWF4hY=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b h1:ZmngSVLe/wycRns9MKikG9OWIEjGcGAkacif7oYQaUY=
golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
golang.org/x/sys v0.0.0-20200923182605-d9f96fdee20d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/image v0.34.0 h1:33gCkyw9hmwbZJeZkct8XyR11yH889EQt/QH4VmXMn8=
golang.org/x/image v0.34.0/go.mod h1:2RNFBZRB+vnwwFil8GkMdRvrJOFd1AzdZI6vOY+eJVU=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 h1:WIoqL4EROvwiPdUtaip4VcDdpZ4kha7wBWZrbVKCIZg=
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+10 -1
View File
@@ -35,7 +35,6 @@ func (s *RPC) imgResetClnTimerHndl(id uint64, params json.RawMessage) (rr jrpc.R
}
err := s.img.ResetCleanupTimer(fileID)
return jrpc.EncodeResponse(id, nil, err)
}
func (s *RPC) imgLoadHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
@@ -47,6 +46,16 @@ func (s *RPC) imgLoadHndl(id uint64, params json.RawMessage) (rr jrpc.Response)
return jrpc.EncodeResponse(id, value, err)
}
func (s *RPC) imgDeleteHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
var fileID string
if err := json.Unmarshal(params, &fileID); err != nil {
return jrpc.Response{Error: err.Error()}
}
err := s.img.Delete(fileID)
return jrpc.EncodeResponse(id, nil, err)
}
func (s *RPC) imgCommitHndl(id uint64, params json.RawMessage) (rr jrpc.Response) {
var fileID string
if err := json.Unmarshal(params, &fileID); err != nil {
@@ -158,4 +158,9 @@ func TestRPC_imgInfoHndl(t *testing.T) {
info, err = ri.Info()
assert.NoError(t, err)
assert.False(t, info.FirstStagingImageTS.IsZero())
err = ri.Delete("test_img")
assert.NoError(t, err)
_, err = ri.Load("test_img")
assert.EqualError(t, err, "image test_img not found")
}
@@ -60,6 +60,7 @@ func (s *RPC) addHandlers() {
"save_with_id": s.imgSaveWithIDHndl,
"reset_cleanup_timer": s.imgResetClnTimerHndl,
"load": s.imgLoadHndl,
"delete": s.imgDeleteHndl,
"commit": s.imgCommitHndl,
"cleanup": s.imgCleanupHndl,
"info": s.imgInfoHndl,
+1 -1
View File
@@ -7,7 +7,7 @@ import (
log "github.com/go-pkgz/lgr"
bolt "go.etcd.io/bbolt"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/v2/avatar"
)
// AvatarCommand set of flags and command for avatar migration
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"os"
"testing"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/v2/avatar"
"github.com/jessevdk/go-flags"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
+28
View File
@@ -1,10 +1,12 @@
package cmd
import (
"encoding/base64"
"fmt"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/jessevdk/go-flags"
@@ -16,6 +18,10 @@ func TestBackup_Execute(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/export")
assert.Equal(t, "GET", r.Method)
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:secret", string(auth))
fmt.Fprint(w, "blah\nblah2\n12345678\n")
}))
defer ts.Close()
@@ -34,6 +40,28 @@ func TestBackup_Execute(t *testing.T) {
assert.Equal(t, "blah\nblah2\n12345678\n", string(data))
}
func TestBackup_ExecuteNoPassword(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/export")
assert.Equal(t, "GET", r.Method)
t.Logf("Authorization: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
require.Equal(t, "admin:", string(auth))
w.WriteHeader(http.StatusUnauthorized)
fmt.Fprint(w, "Unauthorized")
}))
defer ts.Close()
cmd := BackupCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p := flags.NewParser(&cmd, flags.Default)
_, err := p.ParseArgs([]string{"--site=remark", "--path=/tmp", "--file={{.SITE}}-test.export"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
}
func TestBackup_ExecuteFailedStatus(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/export")
+1 -1
View File
@@ -77,7 +77,7 @@ func (cc *CleanupCommand) procSpam(comments []store.Comment) int {
log.Printf("[WARN] can't remove comment, %v", err)
}
}
comment.Text = strings.Replace(comment.Text, "\n", " ", -1)
comment.Text = strings.ReplaceAll(comment.Text, "\n", " ")
log.Printf("[SPAM] %+v [%.0f%%]", comment, score)
}
}
+2 -3
View File
@@ -46,7 +46,6 @@ func TestCleanup_IsSpam(t *testing.T) {
}
for n, tt := range tbl {
tt := tt
checkName := fmt.Sprintf("check-%d-%s", n, tt.name)
t.Run(checkName, func(t *testing.T) {
c := store.Comment{ID: checkName, Text: tt.text, Score: tt.score}
@@ -195,7 +194,7 @@ func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
require.NoError(t, json.NewEncoder(w).Encode(commentsWithInfo))
})
r.HandleFunc("/api/v1/admin/comment/{id}", func(w http.ResponseWriter, r *http.Request) {
r.HandleFunc("/api/v1/admin/comment/{id}", func(_ http.ResponseWriter, r *http.Request) {
require.Equal(t, "DELETE", r.Method)
t.Log("delete ", r.URL.Path)
c.lock.Lock()
@@ -203,7 +202,7 @@ func cleanupRoutes(t *testing.T, r *chi.Mux, c *cleanedComments) {
c.lock.Unlock()
})
r.HandleFunc("/api/v1/admin/title/{id}", func(w http.ResponseWriter, r *http.Request) {
r.HandleFunc("/api/v1/admin/title/{id}", func(_ http.ResponseWriter, r *http.Request) {
require.Equal(t, "PUT", r.Method)
t.Log("title for ", r.URL.Path)
c.lock.Lock()
+3
View File
@@ -115,6 +115,9 @@ func responseError(resp *http.Response) error {
if e != nil {
body = []byte("")
}
if resp.StatusCode == http.StatusUnauthorized {
return fmt.Errorf("error response %q, ensure you have set ADMIN_PASSWD and provided it to the command you're running: %s", resp.Status, body)
}
return fmt.Errorf("error response %q, %s", resp.Status, body)
}
+42
View File
@@ -1,10 +1,12 @@
package cmd
import (
"encoding/base64"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
@@ -18,6 +20,10 @@ func TestImport_Execute(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
assert.Equal(t, "POST", r.Method)
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:secret", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "blah\nblah2\n12345678\n", string(body))
@@ -46,6 +52,42 @@ func TestImport_Execute(t *testing.T) {
assert.NoError(t, err)
}
func TestImport_ExecuteNoPassword(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
assert.Equal(t, "POST", r.Method)
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "blah\nblah2\n12345678\n", string(body))
w.WriteHeader(401)
fmt.Fprint(w, "Unauthorized")
}))
defer ts.Close()
cmd := ImportCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p := flags.NewParser(&cmd, flags.Default)
_, err := p.ParseArgs([]string{"--site=remark", "--file=testdata/import.txt"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
cmd = ImportCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p = flags.NewParser(&cmd, flags.Default)
_, err = p.ParseArgs([]string{"--site=remark", "--file=testdata/import.txt.gz"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
}
func TestImport_ExecuteFailed(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/import")
+35
View File
@@ -1,9 +1,12 @@
package cmd
import (
"encoding/base64"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/jessevdk/go-flags"
@@ -16,6 +19,10 @@ func TestRemap_Execute(t *testing.T) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/remap")
assert.Equal(t, "POST", r.Method)
assert.Equal(t, "remark", r.URL.Query().Get("site"))
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:secret", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "http://oldsite.com* https://newsite.com*\nhttp://oldsite.com/from-old-page/1 https://newsite.com/to-new-page/1", string(body))
@@ -33,3 +40,31 @@ func TestRemap_Execute(t *testing.T) {
err = cmd.Execute(nil)
assert.NoError(t, err)
}
func TestRemap_ExecuteNoPassword(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/api/v1/admin/remap")
assert.Equal(t, "POST", r.Method)
assert.Equal(t, "remark", r.URL.Query().Get("site"))
t.Logf("Authorization header: %+v", r.Header.Get("Authorization"))
auth, err := base64.StdEncoding.DecodeString(strings.Split(r.Header.Get("Authorization"), " ")[1])
require.NoError(t, err)
assert.Equal(t, "admin:", string(auth))
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
assert.Equal(t, "http://oldsite.com* https://newsite.com*\nhttp://oldsite.com/from-old-page/1 https://newsite.com/to-new-page/1", string(body))
w.WriteHeader(401)
fmt.Fprint(w, "Unauthorized")
}))
defer ts.Close()
cmd := RemapCommand{}
cmd.SetCommon(CommonOpts{RemarkURL: ts.URL})
p := flags.NewParser(&cmd, flags.Default)
_, err := p.ParseArgs([]string{"--site=remark", "--file=testdata/remap_urls.txt"})
require.NoError(t, err)
err = cmd.Execute(nil)
assert.EqualError(t, err, "error response \"401 Unauthorized\", ensure you have set ADMIN_PASSWD and provided it to the command you're running: Unauthorized")
}
+195 -129
View File
@@ -4,6 +4,7 @@ import (
"context"
"embed"
"fmt"
"net"
"net/http"
"net/url"
"os"
@@ -15,19 +16,19 @@ import (
"time"
"github.com/go-pkgz/jrpc"
"github.com/go-pkgz/lcw/eventbus"
"github.com/go-pkgz/lcw/v2/eventbus"
log "github.com/go-pkgz/lgr"
ntf "github.com/go-pkgz/notify"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/kyokomi/emoji/v2"
bolt "go.etcd.io/bbolt"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/provider"
"github.com/go-pkgz/auth/provider/sender"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
"github.com/go-pkgz/auth/v2/provider"
"github.com/go-pkgz/auth/v2/provider/sender"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
"github.com/umputun/remark42/backend/app/migrator"
"github.com/umputun/remark42/backend/app/notify"
@@ -58,34 +59,36 @@ type ServerCommand struct {
SSL SSLGroup `group:"ssl" namespace:"ssl" env-namespace:"SSL"`
ImageProxy ImageProxyGroup `group:"image-proxy" namespace:"image-proxy" env-namespace:"IMAGE_PROXY"`
Sites []string `long:"site" env:"SITE" default:"remark" description:"site names" env-delim:","`
AnonymousVote bool `long:"anon-vote" env:"ANON_VOTE" description:"enable anonymous votes (works only with VOTES_IP enabled)"`
AdminPasswd string `long:"admin-passwd" env:"ADMIN_PASSWD" default:"" description:"admin basic auth password"`
BackupLocation string `long:"backup" env:"BACKUP_PATH" default:"./var/backup" description:"backups location"`
MaxBackupFiles int `long:"max-back" env:"MAX_BACKUP_FILES" default:"10" description:"max backups to keep"`
LegacyImageProxy bool `long:"img-proxy" env:"IMG_PROXY" description:"[deprecated, use image-proxy.http2https] enable image proxy"`
MaxCommentSize int `long:"max-comment" env:"MAX_COMMENT_SIZE" default:"2048" description:"max comment size"`
MaxVotes int `long:"max-votes" env:"MAX_VOTES" default:"-1" description:"maximum number of votes per comment"`
RestrictVoteIP bool `long:"votes-ip" env:"VOTES_IP" description:"restrict votes from the same ip"`
DurationVoteIP time.Duration `long:"votes-ip-time" env:"VOTES_IP_TIME" default:"5m" description:"same ip vote duration"`
LowScore int `long:"low-score" env:"LOW_SCORE" default:"-5" description:"low score threshold"`
CriticalScore int `long:"critical-score" env:"CRITICAL_SCORE" default:"-10" description:"critical score threshold"`
PositiveScore bool `long:"positive-score" env:"POSITIVE_SCORE" description:"enable positive score only"`
ReadOnlyAge int `long:"read-age" env:"READONLY_AGE" default:"0" description:"read-only age of comments, days"`
EditDuration time.Duration `long:"edit-time" env:"EDIT_TIME" default:"5m" description:"edit window"`
AdminEdit bool `long:"admin-edit" env:"ADMIN_EDIT" description:"unlimited edit for admins"`
Port int `long:"port" env:"REMARK_PORT" default:"8080" description:"port"`
Address string `long:"address" env:"REMARK_ADDRESS" default:"" description:"listening address"`
WebRoot string `long:"web-root" env:"REMARK_WEB_ROOT" default:"./web" description:"web root directory"`
UpdateLimit float64 `long:"update-limit" env:"UPDATE_LIMIT" default:"0.5" description:"updates/sec limit"`
RestrictedWords []string `long:"restricted-words" env:"RESTRICTED_WORDS" description:"words prohibited to use in comments" env-delim:","`
RestrictedNames []string `long:"restricted-names" env:"RESTRICTED_NAMES" description:"names prohibited to use by user" env-delim:","`
EnableEmoji bool `long:"emoji" env:"EMOJI" description:"enable emoji"`
SimpleView bool `long:"simple-view" env:"SIMPLE_VIEW" description:"minimal comment editor mode"`
ProxyCORS bool `long:"proxy-cors" env:"PROXY_CORS" description:"disable internal CORS and delegate it to proxy"`
AllowedHosts []string `long:"allowed-hosts" env:"ALLOWED_HOSTS" description:"limit hosts/sources allowed to embed comments" env-delim:","`
SubscribersOnly bool `long:"subscribers-only" env:"SUBSCRIBERS_ONLY" description:"enable commenting only for Patreon subscribers"`
DisableSignature bool `long:"disable-signature" env:"DISABLE_SIGNATURE" description:"disable server signature in headers"`
Sites []string `long:"site" env:"SITE" default:"remark" description:"site names" env-delim:","`
AnonymousVote bool `long:"anon-vote" env:"ANON_VOTE" description:"enable anonymous votes (works only with VOTES_IP enabled)"`
AdminPasswd string `long:"admin-passwd" env:"ADMIN_PASSWD" default:"" description:"admin basic auth password"`
BackupLocation string `long:"backup" env:"BACKUP_PATH" default:"./var/backup" description:"backups location"`
MaxBackupFiles int `long:"max-back" env:"MAX_BACKUP_FILES" default:"10" description:"max backups to keep"`
LegacyImageProxy bool `long:"img-proxy" env:"IMG_PROXY" description:"[deprecated, use image-proxy.http2https] enable image proxy"`
MinCommentSize int `long:"min-comment" env:"MIN_COMMENT_SIZE" default:"0" description:"min comment size"`
MaxCommentSize int `long:"max-comment" env:"MAX_COMMENT_SIZE" default:"2048" description:"max comment size"`
MaxVotes int `long:"max-votes" env:"MAX_VOTES" default:"-1" description:"maximum number of votes per comment"`
RestrictVoteIP bool `long:"votes-ip" env:"VOTES_IP" description:"restrict votes from the same ip"`
DurationVoteIP time.Duration `long:"votes-ip-time" env:"VOTES_IP_TIME" default:"5m" description:"same ip vote duration"`
LowScore int `long:"low-score" env:"LOW_SCORE" default:"-5" description:"low score threshold"`
CriticalScore int `long:"critical-score" env:"CRITICAL_SCORE" default:"-10" description:"critical score threshold"`
PositiveScore bool `long:"positive-score" env:"POSITIVE_SCORE" description:"enable positive score only"`
ReadOnlyAge int `long:"read-age" env:"READONLY_AGE" default:"0" description:"read-only age of comments, days"`
EditDuration time.Duration `long:"edit-time" env:"EDIT_TIME" default:"5m" description:"edit window"`
AdminEdit bool `long:"admin-edit" env:"ADMIN_EDIT" description:"unlimited edit for admins"`
Port int `long:"port" env:"REMARK_PORT" default:"8080" description:"port"`
Address string `long:"address" env:"REMARK_ADDRESS" default:"" description:"listening address"`
WebRoot string `long:"web-root" env:"REMARK_WEB_ROOT" default:"./web" description:"web root directory"`
UpdateLimit float64 `long:"update-limit" env:"UPDATE_LIMIT" default:"0.5" description:"updates/sec limit"`
RestrictedWords []string `long:"restricted-words" env:"RESTRICTED_WORDS" description:"words prohibited to use in comments" env-delim:","`
RestrictedNames []string `long:"restricted-names" env:"RESTRICTED_NAMES" description:"names prohibited to use by user" env-delim:","`
EnableEmoji bool `long:"emoji" env:"EMOJI" description:"enable emoji"`
SimpleView bool `long:"simple-view" env:"SIMPLE_VIEW" description:"minimal comment editor mode"`
ProxyCORS bool `long:"proxy-cors" env:"PROXY_CORS" description:"disable internal CORS and delegate it to proxy"`
AllowedHosts []string `long:"allowed-hosts" env:"ALLOWED_HOSTS" description:"limit hosts/sources allowed to embed comments via CSP 'frame-ancestors''" env-delim:","`
SubscribersOnly bool `long:"subscribers-only" env:"SUBSCRIBERS_ONLY" description:"enable commenting only for Patreon subscribers"`
DisableSignature bool `long:"disable-signature" env:"DISABLE_SIGNATURE" description:"disable server signature in headers"`
DisableFancyTextFormatting bool `long:"disable-fancy-text-formatting" env:"DISABLE_FANCY_TEXT_FORMATTING" description:"disable fancy comments text formatting (replacement of quotes, dashes, fractions, etc)"`
Auth struct {
TTL struct {
@@ -93,7 +96,7 @@ type ServerCommand struct {
Cookie time.Duration `long:"cookie" env:"COOKIE" default:"200h" description:"auth cookie TTL"`
} `group:"ttl" namespace:"ttl" env-namespace:"TTL"`
SendJWTHeader bool `long:"send-jwt-header" env:"SEND_JWT_HEADER" description:"send JWT as a header instead of cookie"`
SendJWTHeader bool `long:"send-jwt-header" env:"SEND_JWT_HEADER" description:"send JWT as a header instead of server-set cookie; with this enabled, frontend stores the JWT in a client-side cookie (note: increases vulnerability to XSS attacks)"`
SameSite string `long:"same-site" env:"SAME_SITE" description:"set same site policy for cookies" choice:"default" choice:"none" choice:"lax" choice:"strict" default:"default"` // nolint
Apple AppleGroup `group:"apple" namespace:"apple" env-namespace:"APPLE" description:"Apple OAuth"`
@@ -102,8 +105,9 @@ type ServerCommand struct {
Facebook AuthGroup `group:"facebook" namespace:"facebook" env-namespace:"FACEBOOK" description:"Facebook OAuth"`
Microsoft AuthGroup `group:"microsoft" namespace:"microsoft" env-namespace:"MICROSOFT" description:"Microsoft OAuth"`
Yandex AuthGroup `group:"yandex" namespace:"yandex" env-namespace:"YANDEX" description:"Yandex OAuth"`
Twitter AuthGroup `group:"twitter" namespace:"twitter" env-namespace:"TWITTER" description:"Twitter OAuth"`
Twitter AuthGroup `group:"twitter" namespace:"twitter" env-namespace:"TWITTER" description:"[deprecated, doesn't work] Twitter OAuth"`
Patreon AuthGroup `group:"patreon" namespace:"patreon" env-namespace:"PATREON" description:"Patreon OAuth"`
Discord AuthGroup `group:"discord" namespace:"discord" env-namespace:"DISCORD" description:"Discord OAuth"`
Telegram bool `long:"telegram" env:"TELEGRAM" description:"Enable Telegram auth (using token from telegram.token)"`
Dev bool `long:"dev" env:"DEV" description:"enable dev (local) oauth2"`
Anonymous bool `long:"anon" env:"ANON" description:"enable anonymous login"`
@@ -136,7 +140,7 @@ type ImageProxyGroup struct {
// AppleGroup defines options for Apple auth params
type AppleGroup struct {
CID string `long:"cid" env:"CID" description:"Apple client ID"`
CID string `long:"cid" env:"CID" description:"Apple client ID (App ID or Services ID)"`
TID string `long:"tid" env:"TID" description:"Apple service ID"`
KID string `long:"kid" env:"KID" description:"Private key ID"`
PrivateKeyFilePath string `long:"private-key-filepath" env:"PRIVATE_KEY_FILEPATH" description:"Private key file location" default:"/srv/var/apple.p8"`
@@ -217,14 +221,15 @@ type TelegramGroup struct {
// SMTPGroup defines options for SMTP server connection, used in auth and notify modules
type SMTPGroup struct {
Host string `long:"host" env:"HOST" description:"SMTP host"`
Port int `long:"port" env:"PORT" description:"SMTP port"`
Username string `long:"username" env:"USERNAME" description:"SMTP user name"`
Password string `long:"password" env:"PASSWORD" description:"SMTP password"`
TLS bool `long:"tls" env:"TLS" description:"enable TLS"`
LoginAuth bool `long:"login_auth" env:"LOGIN_AUTH" description:"enable LOGIN auth instead of PLAIN"`
StartTLS bool `long:"starttls" env:"STARTTLS" description:"enable StartTLS"`
TimeOut time.Duration `long:"timeout" env:"TIMEOUT" default:"10s" description:"SMTP TCP connection timeout"`
Host string `long:"host" env:"HOST" description:"SMTP host"`
Port int `long:"port" env:"PORT" description:"SMTP port"`
Username string `long:"username" env:"USERNAME" description:"SMTP user name"`
Password string `long:"password" env:"PASSWORD" description:"SMTP password"`
TLS bool `long:"tls" env:"TLS" description:"enable TLS"`
InsecureSkipVerify bool `long:"insecure_skip_verify" env:"INSECURE_SKIP_VERIFY" description:"skip certificate verification"`
LoginAuth bool `long:"login_auth" env:"LOGIN_AUTH" description:"enable LOGIN auth instead of PLAIN"`
StartTLS bool `long:"starttls" env:"STARTTLS" description:"enable StartTLS"`
TimeOut time.Duration `long:"timeout" env:"TIMEOUT" default:"10s" description:"SMTP TCP connection timeout"`
}
// NotifyGroup defines options for notification
@@ -309,6 +314,7 @@ func (s *ServerCommand) Execute(_ []string) error {
log.Printf("[INFO] start server on port %s:%d", s.Address, s.Port)
resetEnv(
"SECRET",
"AUTH_APPLE_KID",
"AUTH_GOOGLE_CSEC",
"AUTH_GITHUB_CSEC",
"AUTH_FACEBOOK_CSEC",
@@ -316,6 +322,7 @@ func (s *ServerCommand) Execute(_ []string) error {
"AUTH_TWITTER_CSEC",
"AUTH_YANDEX_CSEC",
"AUTH_PATREON_CSEC",
"AUTH_DISCORD_CSEC",
"TELEGRAM_TOKEN",
"SMTP_PASSWORD",
"ADMIN_PASSWD",
@@ -402,6 +409,12 @@ func (s *ServerCommand) HandleDeprecatedFlags() (result []DeprecatedFlag) {
if s.Notify.Telegram.API != "https://api.telegram.org/bot" {
result = append(result, DeprecatedFlag{Old: "notify.telegram.api", Version: "1.9"})
}
if s.Auth.Twitter.CID != "" {
result = append(result, DeprecatedFlag{Old: "auth.twitter.cid", Version: "1.14"})
}
if s.Auth.Twitter.CSEC != "" {
result = append(result, DeprecatedFlag{Old: "auth.twitter.csec", Version: "1.14"})
}
return append(result, s.findDeprecatedFlagsCollisions()...)
}
@@ -500,11 +513,12 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
EditDuration: s.EditDuration,
AdminEdits: s.AdminEdit,
AdminStore: adminStore,
MinCommentSize: s.MinCommentSize,
MaxCommentSize: s.MaxCommentSize,
MaxVotes: s.MaxVotes,
PositiveScore: s.PositiveScore,
ImageService: imageService,
TitleExtractor: service.NewTitleExtractor(http.Client{Timeout: time.Second * 5}),
TitleExtractor: service.NewTitleExtractor(http.Client{Timeout: time.Second * 5}, s.getAllowedDomains()),
RestrictedWordsMatcher: service.NewRestrictedWordsMatcher(service.StaticRestrictedWordsLister{Words: s.RestrictedWords}),
}
dataService.RestrictSameIPVotes.Enabled = s.RestrictVoteIP
@@ -525,7 +539,7 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
authenticator := s.getAuthenticator(dataService, avatarStore, adminStore, authRefreshCache)
telegramAuth := s.makeTelegramAuth(authenticator) // telegram auth requires TelegramAPI listener which is constructed below
telegramService, telegramBotUsername := s.startTelegramAuthAndNotify(ctx, telegramAuth)
telegramService := s.startTelegramAuthAndNotify(ctx, telegramAuth)
err = s.addAuthProviders(authenticator)
if err != nil {
@@ -540,7 +554,7 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
Cache: loadingCache,
NativeImporter: &migrator.Native{DataStore: dataService},
DisqusImporter: &migrator.Disqus{DataStore: dataService},
WordPressImporter: &migrator.WordPress{DataStore: dataService},
WordPressImporter: &migrator.WordPress{DataStore: dataService, DisableFancyTextFormatting: s.DisableFancyTextFormatting},
CommentoImporter: &migrator.Commento{DataStore: dataService},
NativeExporter: &migrator.Native{DataStore: dataService},
URLMapperMaker: migrator.NewURLMapper,
@@ -575,33 +589,35 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
}
srv := &api.Rest{
Version: s.Revision,
DataService: dataService,
WebRoot: s.WebRoot,
WebFS: webFS,
RemarkURL: s.RemarkURL,
ImageProxy: imgProxy,
CommentFormatter: commentFormatter,
Migrator: migr,
ReadOnlyAge: s.ReadOnlyAge,
SharedSecret: s.SharedSecret,
Authenticator: authenticator,
Cache: loadingCache,
NotifyService: notifyService,
TelegramService: telegramService,
SSLConfig: sslConfig,
UpdateLimiter: s.UpdateLimit,
ImageService: imageService,
EmailNotifications: contains("email", s.Notify.Users),
TelegramBotUsername: telegramBotUsername,
EmojiEnabled: s.EnableEmoji,
AnonVote: s.AnonymousVote && s.RestrictVoteIP,
SimpleView: s.SimpleView,
ProxyCORS: s.ProxyCORS,
AllowedAncestors: s.AllowedHosts,
SendJWTHeader: s.Auth.SendJWTHeader,
SubscribersOnly: s.SubscribersOnly,
DisableSignature: s.DisableSignature,
Version: s.Revision,
DataService: dataService,
WebRoot: s.WebRoot,
WebFS: webFS,
RemarkURL: s.RemarkURL,
ImageProxy: imgProxy,
CommentFormatter: commentFormatter,
Migrator: migr,
ReadOnlyAge: s.ReadOnlyAge,
SharedSecret: s.SharedSecret,
Authenticator: authenticator,
Cache: loadingCache,
NotifyService: notifyService,
TelegramService: telegramService,
SSLConfig: sslConfig,
UpdateLimiter: s.UpdateLimit,
ImageService: imageService,
EmailNotifications: contains("email", s.Notify.Users),
TelegramNotifications: contains("telegram", s.Notify.Users) && telegramService != nil,
EmojiEnabled: s.EnableEmoji,
AnonVote: s.AnonymousVote && s.RestrictVoteIP,
SimpleView: s.SimpleView,
ProxyCORS: s.ProxyCORS,
AllowedAncestors: s.AllowedHosts,
SendJWTHeader: s.Auth.SendJWTHeader,
SubscribersOnly: s.SubscribersOnly,
DisableSignature: s.DisableSignature,
DisableFancyTextFormatting: s.DisableFancyTextFormatting,
ExternalImageProxy: s.ImageProxy.CacheExternal,
}
srv.ScoreThresholds.Low, srv.ScoreThresholds.Critical = s.LowScore, s.CriticalScore
@@ -633,6 +649,47 @@ func (s *ServerCommand) newServerApp(ctx context.Context) (*serverApp, error) {
}, nil
}
// Extract domains from s.AllowedHosts and second level domain from s.RemarkURL.
// It can be and IP like http://127.0.0.1 in which case we need to use whole IP as domain
// Beware, if s.RemarkURL is in third-level domain like https://example.co.uk, co.uk will be returned.
func (s *ServerCommand) getAllowedDomains() []string {
rawDomains := s.AllowedHosts
rawDomains = append(rawDomains, s.RemarkURL)
allowedDomains := []string{}
for _, rawURL := range rawDomains {
// case of 'self' AllowedHosts, which is not a valid rawURL name
if rawURL == "self" || rawURL == "'self'" || rawURL == "\"self\"" {
continue
}
// AllowedHosts usually don't have https:// prefix, so we're adding it just to make parsing below work the same way as for RemarkURL
if !strings.HasPrefix(rawURL, "http://") && !strings.HasPrefix(rawURL, "https://") {
rawURL = "https://" + rawURL
}
parsedURL, err := url.Parse(rawURL)
if err != nil {
log.Printf("[WARN] failed to parse URL %s for TitleExtract whitelist: %v", rawURL, err)
continue
}
domain := parsedURL.Hostname()
if domain == "" || // don't add empty domain as it will allow everything to be extracted
(len(strings.Split(domain, ".")) < 2 && // don't allow single-word domains like "com"
domain != "localhost") { // localhost is an exceptional single-word domain which is allowed
continue
}
// Only for RemarkURL if domain is not IP and has more than two levels, extract second level domain.
// For AllowedHosts we don't do this as they are exact list of domains which can host comments, but
// RemarkURL might be on a subdomain and we must allow parent domain to be used for TitleExtract.
if rawURL == s.RemarkURL && net.ParseIP(domain) == nil && len(strings.Split(domain, ".")) > 2 {
domain = strings.Join(strings.Split(domain, ".")[len(strings.Split(domain, "."))-2:], ".")
}
allowedDomains = append(allowedDomains, domain)
}
return allowedDomains
}
// Run all application objects
func (a *serverApp) run(ctx context.Context) error {
if a.AdminPasswd != "" {
@@ -822,27 +879,28 @@ func (s *ServerCommand) makeAdminStore() (admin.Store, error) {
func (s *ServerCommand) makeCache() (LoadingCache, error) {
log.Printf("[INFO] make cache, type=%s", s.Cache.Type)
o := cache.NewOpts[[]byte]()
switch s.Cache.Type {
case "redis_pub_sub":
redisPubSub, err := eventbus.NewRedisPubSub(s.Cache.RedisAddr, "remark42-cache")
if err != nil {
return nil, fmt.Errorf("cache backend initialization, redis PubSub initialisation: %w", err)
}
backend, err := cache.NewLruCache(cache.MaxCacheSize(s.Cache.Max.Size), cache.MaxValSize(s.Cache.Max.Value),
cache.MaxKeys(s.Cache.Max.Items), cache.EventBus(redisPubSub))
backend, err := cache.NewLruCache(o.MaxCacheSize(s.Cache.Max.Size), o.MaxValSize(s.Cache.Max.Value),
o.MaxKeys(s.Cache.Max.Items), o.EventBus(redisPubSub))
if err != nil {
return nil, fmt.Errorf("cache backend initialization: %w", err)
}
return cache.NewScache(backend), nil
return cache.NewScache[[]byte](backend), nil
case "mem":
backend, err := cache.NewLruCache(cache.MaxCacheSize(s.Cache.Max.Size), cache.MaxValSize(s.Cache.Max.Value),
cache.MaxKeys(s.Cache.Max.Items))
backend, err := cache.NewLruCache(o.MaxCacheSize(s.Cache.Max.Size), o.MaxValSize(s.Cache.Max.Value),
o.MaxKeys(s.Cache.Max.Items))
if err != nil {
return nil, fmt.Errorf("cache backend initialization: %w", err)
}
return cache.NewScache(backend), nil
return cache.NewScache[[]byte](backend), nil
case "none":
return cache.NewScache(&cache.Nop{}), nil
return cache.NewScache[[]byte](&cache.Nop[[]byte]{}), nil
}
return nil, fmt.Errorf("unsupported cache type %s", s.Cache.Type)
}
@@ -896,6 +954,10 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
authenticator.AddProvider("patreon", s.Auth.Patreon.CID, s.Auth.Patreon.CSEC)
providersCount++
}
if s.Auth.Discord.CID != "" && s.Auth.Discord.CSEC != "" {
authenticator.AddProvider("discord", s.Auth.Discord.CID, s.Auth.Discord.CSEC)
providersCount++
}
if s.Auth.Dev {
log.Print("[INFO] dev access enabled")
@@ -909,18 +971,19 @@ func (s *ServerCommand) addAuthProviders(authenticator *auth.Service) error {
if s.Auth.Email.Enable {
params := sender.EmailParams{
Host: s.SMTP.Host,
Port: s.SMTP.Port,
SMTPUserName: s.SMTP.Username,
SMTPPassword: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
StartTLS: s.SMTP.StartTLS,
LoginAuth: s.SMTP.LoginAuth,
TLS: s.SMTP.TLS,
Charset: "UTF-8",
From: s.Auth.Email.From,
Subject: s.Auth.Email.Subject,
ContentType: s.Auth.Email.ContentType,
Host: s.SMTP.Host,
Port: s.SMTP.Port,
SMTPUserName: s.SMTP.Username,
SMTPPassword: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
StartTLS: s.SMTP.StartTLS,
LoginAuth: s.SMTP.LoginAuth,
TLS: s.SMTP.TLS,
InsecureSkipVerify: s.SMTP.InsecureSkipVerify,
Charset: "UTF-8",
From: s.Auth.Email.From,
Subject: s.Auth.Email.Subject,
ContentType: s.Auth.Email.ContentType,
}
sndr := sender.NewEmailClient(params, log.Default())
tmpl, err := templates.Read(s.Auth.Email.MsgTemplate)
@@ -1046,10 +1109,10 @@ func (s *ServerCommand) makeNotifyDestinations(authenticator *auth.Service) ([]n
TokenGenFn: func(userID, email, site string) (string, error) {
claims := token.Claims{
Handshake: &token.Handshake{ID: userID + "::" + email},
StandardClaims: jwt.StandardClaims{
Audience: site,
ExpiresAt: time.Now().Add(100 * 365 * 24 * time.Hour).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{site},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(100 * 365 * 24 * time.Hour)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
Issuer: "remark42",
},
}
@@ -1064,16 +1127,17 @@ func (s *ServerCommand) makeNotifyDestinations(authenticator *auth.Service) ([]n
emailParams.AdminEmails = s.Admin.Shared.Email
}
smtpParams := ntf.SMTPParams{
Host: s.SMTP.Host,
Port: s.SMTP.Port,
TLS: s.SMTP.TLS,
StartTLS: s.SMTP.StartTLS,
LoginAuth: s.SMTP.LoginAuth,
Username: s.SMTP.Username,
Password: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
ContentType: "text/html",
Charset: "UTF-8",
Host: s.SMTP.Host,
Port: s.SMTP.Port,
TLS: s.SMTP.TLS,
StartTLS: s.SMTP.StartTLS,
InsecureSkipVerify: s.SMTP.InsecureSkipVerify,
LoginAuth: s.SMTP.LoginAuth,
Username: s.SMTP.Username,
Password: s.SMTP.Password,
TimeOut: s.SMTP.TimeOut,
ContentType: "text/html",
Charset: "UTF-8",
}
emailService, err := notify.NewEmail(emailParams, smtpParams)
if err != nil {
@@ -1151,10 +1215,16 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
if c.User == nil {
return c
}
c.User.SetAdmin(ds.IsAdmin(c.Audience, c.User.ID))
c.User.SetBoolAttr("blocked", ds.IsBlocked(c.Audience, c.User.ID))
// Audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
if len(c.Audience) != 1 {
return c
}
audience := c.Audience[0]
c.User.SetAdmin(ds.IsAdmin(audience, c.User.ID))
c.User.SetBoolAttr("blocked", ds.IsBlocked(audience, c.User.ID))
var err error
c.User.Email, err = ds.GetUserEmail(c.Audience, c.User.ID)
c.User.Email, err = ds.GetUserEmail(audience, c.User.ID)
if err != nil {
log.Printf("[WARN] can't read email for %s, %v", c.User.ID, err)
}
@@ -1174,7 +1244,7 @@ func (s *ServerCommand) getAuthenticator(ds *service.DataStore, avas avatar.Stor
return c
}),
AdminPasswd: s.AdminPasswd,
Validator: token.ValidatorFunc(func(token string, claims token.Claims) bool { // check on each auth call (in middleware)
Validator: token.ValidatorFunc(func(_ string, claims token.Claims) bool { // check on each auth call (in middleware)
if claims.User == nil {
return false
}
@@ -1211,20 +1281,15 @@ func (s *ServerCommand) parseSameSite(ss string) http.SameSite {
// startTelegramAuthAndNotify initializes telegram notify and auth Telegram Bot listen loop.
// Does nothing if telegram auth and notifications are disabled.
// Doesn't return telegram bot username if user notifications are disabled, as that is the way frontend knows they are enabled.
func (s *ServerCommand) startTelegramAuthAndNotify(ctx context.Context, telegramAuth providers.TGUpdatesReceiver) (tg *notify.Telegram, telegramBotUsername string) {
func (s *ServerCommand) startTelegramAuthAndNotify(ctx context.Context, telegramAuth providers.TGUpdatesReceiver) (tg *notify.Telegram) {
if !contains("telegram", s.Notify.Users) && !contains("telegram", s.Notify.Admins) && !s.Auth.Telegram {
return nil, ""
return nil
}
var err error
if tg, err = s.makeTelegramNotify(); err != nil {
log.Printf("[WARN] failed to make telegram notify service, %s", err)
return nil, ""
}
if contains("telegram", s.Notify.Users) {
telegramBotUsername = tg.GetBotUsername()
return nil
}
telegramReceivers := []providers.TGUpdatesReceiver{tg}
@@ -1234,7 +1299,7 @@ func (s *ServerCommand) startTelegramAuthAndNotify(ctx context.Context, telegram
// start bot messages receiver for both notify and auth services
go providers.DispatchTelegramUpdates(ctx, tg, telegramReceivers, time.Second*5)
return tg, telegramBotUsername
return tg
}
// splitAtCommas split s at commas, ignoring commas in strings.
@@ -1281,20 +1346,21 @@ func splitAtCommas(s string) []string {
// authRefreshCache used by authenticator to minimize repeatable token refreshes
type authRefreshCache struct {
cache.LoadingCache
cache.LoadingCache[token.Claims]
}
func newAuthRefreshCache() *authRefreshCache {
expirableCache, _ := cache.NewExpirableCache(cache.TTL(5 * time.Minute))
o := cache.NewOpts[token.Claims]()
expirableCache, _ := cache.NewExpirableCache(o.TTL(5 * time.Minute))
return &authRefreshCache{LoadingCache: expirableCache}
}
// Get implements cache getter with key converted to string
func (c *authRefreshCache) Get(key interface{}) (interface{}, bool) {
return c.LoadingCache.Peek(key.(string))
func (c *authRefreshCache) Get(key string) (token.Claims, bool) {
return c.Peek(key)
}
// Set implements cache setter with key converted to string
func (c *authRefreshCache) Set(key, value interface{}) {
_, _ = c.LoadingCache.Get(key.(string), func() (interface{}, error) { return value, nil })
func (c *authRefreshCache) Set(key string, value token.Claims) {
_, _ = c.LoadingCache.Get(key, func() (token.Claims, error) { return value, nil })
}
+90 -22
View File
@@ -15,8 +15,8 @@ import (
"testing"
"time"
"github.com/go-pkgz/auth/token"
"github.com/golang-jwt/jwt"
"github.com/go-pkgz/auth/v2/token"
"github.com/golang-jwt/jwt/v5"
"github.com/jessevdk/go-flags"
"go.uber.org/goleak"
@@ -79,7 +79,7 @@ func TestServerApp_DevMode(t *testing.T) {
waitForHTTPServerStart(port)
providers := app.restSrv.Authenticator.Providers()
require.Equal(t, 10+1, len(providers), "extra auth provider")
require.Equal(t, 11+1, len(providers), "extra auth provider")
assert.Equal(t, "dev", providers[len(providers)-2].Name(), "dev auth provider")
// send ping
resp, err := http.Get(fmt.Sprintf("http://localhost:%d/api/v1/ping", port))
@@ -107,7 +107,7 @@ func TestServerApp_AnonMode(t *testing.T) {
waitForHTTPServerStart(port)
providers := app.restSrv.Authenticator.Providers()
require.Equal(t, 10+1, len(providers), "extra auth provider for anon")
require.Equal(t, 11+1, len(providers), "extra auth provider for anon")
assert.Equal(t, "anonymous", providers[len(providers)-1].Name(), "anon auth provider")
client := http.Client{Timeout: 10 * time.Second}
@@ -250,7 +250,7 @@ func TestServerApp_WithSSL(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
@@ -457,6 +457,8 @@ func TestServerApp_DeprecatedArgs(t *testing.T) {
"--notify.telegram.token=abcd",
"--notify.telegram.timeout=3m",
"--notify.telegram.api=http://example.org",
"--auth.twitter.cid=123",
"--auth.twitter.csec=456",
}
assert.Empty(t, s.SMTP.Host)
assert.Empty(t, s.SMTP.Port)
@@ -482,6 +484,8 @@ func TestServerApp_DeprecatedArgs(t *testing.T) {
{Old: "notify.telegram.token", New: "telegram.token", Version: "1.9"},
{Old: "notify.telegram.timeout", New: "telegram.timeout", Version: "1.9"},
{Old: "notify.telegram.api", Version: "1.9"},
{Old: "auth.twitter.cid", Version: "1.14"},
{Old: "auth.twitter.csec", Version: "1.14"},
},
deprecatedFlags)
assert.Equal(t, "smtp.example.org", s.SMTP.Host)
@@ -605,14 +609,14 @@ func TestServerAuthHooks(t *testing.T) {
tkService.TokenDuration = time.Second
claims := token.Claims{
StandardClaims: jwt.StandardClaims{
Audience: "remark",
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark"},
Issuer: "remark",
ExpiresAt: time.Now().Add(time.Second).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Second)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
},
User: &token.User{
ID: "dev",
ID: "github_dev",
Name: "developer one",
},
}
@@ -633,10 +637,10 @@ func TestServerAuthHooks(t *testing.T) {
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusCreated, resp.StatusCode, "non-blocked user able to post")
// add comment with no-aud claim
claimsNoAud := claims
claimsNoAud.Audience = ""
tkNoAud, err := tkService.Token(claimsNoAud)
// try to add comment with no-aud claim
badClaimsNoAud := claims
badClaimsNoAud.Audience = jwt.ClaimStrings{""}
tkNoAud, err := tkService.Token(badClaimsNoAud)
require.NoError(t, err)
t.Logf("no-aud claims: %s", tkNoAud)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
@@ -651,14 +655,51 @@ func TestServerAuthHooks(t *testing.T) {
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "user without aud claim rejected, \n"+tkNoAud+"\n"+string(body))
// block user dev as admin
// try to add comment with multiple auds
badClaimsMultipleAud := claims
badClaimsMultipleAud.Audience = jwt.ClaimStrings{"remark", "second_aud"}
tkMultipleAuds, err := tkService.Token(badClaimsMultipleAud)
require.NoError(t, err)
t.Logf("multiple aud claims: %s", tkMultipleAuds)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tkMultipleAuds)
resp, err = client.Do(req)
require.NoError(t, err)
body, err = io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "user with multiple auds claim rejected, \n"+tkMultipleAuds+"\n"+string(body))
// try to add comment without user set
badClaimsNoUser := claims
badClaimsNoUser.Audience = jwt.ClaimStrings{"remark"}
badClaimsNoUser.User = nil
tkNoUser, err := tkService.Token(badClaimsNoUser)
require.NoError(t, err)
t.Logf("no user claims: %s", tkNoUser)
req, err = http.NewRequest("POST", fmt.Sprintf("http://localhost:%d/api/v1/comment", port),
strings.NewReader(`{"text": "test 123", "locator":{"url": "https://radio-t.com/p/2018/12/29/podcast-631/",
"site": "remark"}}`))
require.NoError(t, err)
req.Header.Set("X-JWT", tkNoUser)
resp, err = client.Do(req)
require.NoError(t, err)
body, err = io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode, "user without user information rejected, \n"+tkNoUser+"\n"+string(body))
// block user github_dev as admin
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("http://localhost:%d/api/v1/admin/user/dev?site=remark&block=1&ttl=10d", port), http.NoBody)
fmt.Sprintf("http://localhost:%d/api/v1/admin/user/github_dev?site=remark&block=1&ttl=10d", port), http.NoBody)
assert.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode, "user dev blocked")
assert.Equal(t, http.StatusOK, resp.StatusCode, "user github_dev blocked")
b, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
@@ -697,7 +738,6 @@ func TestServerCommand_parseSameSite(t *testing.T) {
cmd := ServerCommand{}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.res, cmd.parseSameSite(tt.inp))
})
@@ -726,6 +766,28 @@ func Test_splitAtCommas(t *testing.T) {
}
}
func Test_getAllowedDomains(t *testing.T) {
tbl := []struct {
s ServerCommand
allowedDomains []string
}{
// correct example, parsed and returned as allowed domain
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "https://remark42.example.org"}}, []string{"example.org"}},
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "http://remark42.example.org"}}, []string{"example.org"}},
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "http://localhost"}}, []string{"localhost"}},
// incorrect URLs, so Hostname is empty but returned list doesn't include empty string as it would allow any domain
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "bad hostname"}}, []string{}},
{ServerCommand{AllowedHosts: []string{}, CommonOpts: CommonOpts{RemarkURL: "not_a_hostname"}}, []string{}},
// test removal of 'self', multiple AllowedHosts. No deduplication is expected
{ServerCommand{AllowedHosts: []string{"'self'", "example.org", "test.example.org", "remark42.com"}, CommonOpts: CommonOpts{RemarkURL: "https://example.org"}}, []string{"example.org", "test.example.org", "remark42.com", "example.org"}},
}
for i, tt := range tbl {
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.allowedDomains, tt.s.getAllowedDomains())
})
}
}
func chooseRandomUnusedPort() (port int) {
for i := 0; i < 10; i++ {
port = 40000 + int(rand.Int31n(10000))
@@ -771,7 +833,6 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
_, err := p.ParseArgs([]string{"--admin-passwd=password", "--site=remark"})
require.NoError(t, err)
cmd.Avatar.FS.Path, cmd.Avatar.Type, cmd.BackupLocation, cmd.Image.FS.Path = "/tmp/remark42_test", "fs", "/tmp/remark42_test", "/tmp/remark42_test"
cmd.Store.Bolt.Path = fmt.Sprintf("/tmp/%d", cmd.Port)
cmd.Store.Bolt.Timeout = 10 * time.Second
cmd.Auth.Apple.CID, cmd.Auth.Apple.KID, cmd.Auth.Apple.TID = "cid", "kid", "tid"
cmd.Auth.Apple.PrivateKeyFilePath = "testdata/apple.p8"
@@ -782,6 +843,7 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
cmd.Auth.Microsoft.CSEC, cmd.Auth.Microsoft.CID = "csec", "cid"
cmd.Auth.Twitter.CSEC, cmd.Auth.Twitter.CID = "csec", "cid"
cmd.Auth.Patreon.CSEC, cmd.Auth.Patreon.CID = "csec", "cid"
cmd.Auth.Discord.CSEC, cmd.Auth.Discord.CID = "csec", "cid"
cmd.Auth.Telegram = true
cmd.Telegram.Token = "token"
cmd.Auth.Email.Enable = true
@@ -802,7 +864,10 @@ func prepServerApp(t *testing.T, fn func(o ServerCommand) ServerCommand) (*serve
cmd.RestrictedNames = []string{"umputun", "bobuk"}
cmd.emailMsgTemplatePath = "../../templates/email_reply.html.tmpl"
cmd.emailVerificationTemplatePath = "../../templates/email_confirmation_subscription.html.tmpl"
cmd = fn(cmd)
// as is uses port, call it after fn which could set it
cmd.Store.Bolt.Path = fmt.Sprintf("/tmp/%d", cmd.Port)
app, ctx, cancel := createAppFromCmd(t, cmd)
@@ -821,12 +886,15 @@ func createAppFromCmd(t *testing.T, cmd ServerCommand) (*serverApp, context.Cont
ctx, cancel := context.WithCancel(context.Background())
app, err := cmd.newServerApp(ctx)
require.NoError(t, err)
rand.Seed(time.Now().UnixNano())
return app, ctx, cancel
}
func TestMain(m *testing.M) {
// ignore is added only for GitHub Actions, can't reproduce locally
goleak.VerifyTestMain(m, goleak.IgnoreTopFunction("net/http.(*Server).Shutdown"))
goleak.VerifyTestMain(
m,
goleak.IgnoreTopFunction("net/http.(*Server).Shutdown"),
// this will be fixed in https://github.com/hashicorp/golang-lru/issues/159
goleak.IgnoreTopFunction("github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1"),
)
}
+7 -5
View File
@@ -1,6 +1,7 @@
package main
import (
"errors"
"fmt"
"os"
"os/signal"
@@ -23,7 +24,8 @@ type Opts struct {
CleanupCmd cmd.CleanupCommand `command:"cleanup"`
RemapCmd cmd.RemapCommand `command:"remap"`
RemarkURL string `long:"url" env:"REMARK_URL" required:"true" description:"url to remark"`
RemarkURL string `long:"url" env:"REMARK_URL" required:"true" description:"url to remark"`
// SharedSecret is only used in server command, but defined for all commands for historical reasons
SharedSecret string `long:"secret" env:"SECRET" required:"true" description:"the shared secret key used to sign JWT, should be a random, long, hard-to-guess string"`
Dbg bool `long:"dbg" env:"DEBUG" description:"debug mode"`
@@ -54,11 +56,11 @@ func main() {
}
if _, err := p.Parse(); err != nil {
if flagsErr, ok := err.(*flags.Error); ok && flagsErr.Type == flags.ErrHelp {
var flagsErr *flags.Error
if errors.As(err, &flagsErr) && flagsErr.Type == flags.ErrHelp {
os.Exit(0)
} else {
os.Exit(1)
}
os.Exit(1)
}
}
@@ -97,7 +99,7 @@ func getDump() string {
return string(stacktrace[:length])
}
// nolint:gochecknoinits // can't avoid it in this place
//nolint:gochecknoinits // can't avoid it in this place
func init() {
// catch SIGQUIT and print stack traces
sigChan := make(chan os.Signal, 1)
+11 -7
View File
@@ -64,7 +64,7 @@ func TestMain_WithWebhook(t *testing.T) {
defer os.RemoveAll(dir)
var webhookSent int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ts := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
atomic.StoreInt32(&webhookSent, 1)
assert.Equal(t, "application/json", r.Header.Get("Content-Type"))
@@ -98,9 +98,6 @@ func TestMain_WithWebhook(t *testing.T) {
finished := make(chan struct{})
go func() {
main()
assert.Eventually(t, func() bool {
return atomic.LoadInt32(&webhookSent) == int32(1)
}, time.Second, 100*time.Millisecond, "webhook was not sent")
close(finished)
}()
@@ -117,13 +114,18 @@ func TestMain_WithWebhook(t *testing.T) {
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusCreated, resp.StatusCode)
// wait for webhook to be sent before shutting down
assert.Eventually(t, func() bool {
return atomic.LoadInt32(&webhookSent) == int32(1)
}, time.Second, 100*time.Millisecond, "webhook was not sent")
}
func TestGetDump(t *testing.T) {
dump := getDump()
assert.True(t, strings.Contains(dump, "goroutine"))
assert.True(t, strings.Contains(dump, "[running]"))
assert.True(t, strings.Contains(dump, "backend/app/main.go"))
assert.Contains(t, dump, "goroutine")
assert.Contains(t, dump, "[running]")
assert.Contains(t, dump, "backend/app/main.go")
t.Logf("\n dump: %s", dump)
}
@@ -157,5 +159,7 @@ func TestMain(m *testing.M) {
m,
goleak.IgnoreTopFunction("github.com/umputun/remark42/backend/app.init.0.func1"),
goleak.IgnoreTopFunction("net/http.(*Server).Shutdown"),
// this will be fixed in https://github.com/hashicorp/golang-lru/issues/159
goleak.IgnoreTopFunction("github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1"),
)
}
+20 -2
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"io"
"net/url"
"time"
"github.com/umputun/remark42/backend/app/store"
@@ -100,6 +101,11 @@ func (d *Commento) convert(r io.Reader, siteID string) (ch chan store.Comment) {
}
}
usersMap["anonymous"] = store.User{
Name: "Anonymous",
ID: "commento_" + store.EncodeID("anonymous"),
}
for _, comment := range exportedData.Comments {
u, ok := usersMap[comment.CommenterHex]
if !ok {
@@ -110,16 +116,28 @@ func (d *Commento) convert(r io.Reader, siteID string) (ch chan store.Comment) {
continue
}
parentID := comment.ParentHex
// comments with ParentHex == "root" are top-level comments
if parentID == "root" {
parentID = ""
}
commentURL, e := url.JoinPath("https://", comment.Domain, comment.Path)
if e != nil {
log.Printf("[WARN] can't construct comment URL in commento import, %s", err.Error())
}
log.Printf("[ERROR] commentoURL: %s", commentURL)
c := store.Comment{
ID: comment.CommentHex,
Locator: store.Locator{
URL: comment.Path,
URL: commentURL,
SiteID: siteID,
},
User: u,
Text: comment.Markdown,
Timestamp: comment.CreationDate,
ParentID: comment.ParentHex,
ParentID: parentID,
Imported: true,
}
+16 -3
View File
@@ -27,11 +27,11 @@ func TestCommento_Import(t *testing.T) {
require.NoError(t, err)
size, err := d.Import(fh, "test")
assert.NoError(t, err)
assert.Equal(t, 2, size)
assert.Equal(t, 3, size)
last, err := dataStore.Last("test", 10, time.Time{}, adminUser)
assert.NoError(t, err)
require.Equal(t, 2, len(last), "2 comments imported")
require.Equal(t, 3, len(last), "3 comments imported")
t.Log(last[0])
@@ -44,11 +44,24 @@ func TestCommento_Import(t *testing.T) {
assert.Equal(t, "commento_35369aeb6ac5255de30410a0f86dc71eb9c6d0ca", c.User.ID)
assert.True(t, c.Imported)
c = last[2] // anonymous comment
assert.Equal(t, "Example comment created by user.", c.Text)
assert.Equal(t, "e7069a7dfcfaed43caf62300a9b0edb1c124ad79d0f5887c93649c15d7f69945", c.ID)
assert.Equal(t, "", c.ParentID)
assert.Equal(t, store.Locator{SiteID: "test", URL: "https://example.com/blog/post/2"}, c.Locator)
assert.Equal(t, "Anonymous", c.User.Name)
assert.Equal(t, "commento_0a92fab3230134cca6eadd9898325b9b2ae67998", c.User.ID)
assert.True(t, c.Imported)
posts, err := dataStore.List("test", 0, 0)
assert.NoError(t, err)
assert.Equal(t, 1, len(posts), "1 post")
assert.Equal(t, 2, len(posts), "2 posts")
count, err := dataStore.Count(store.Locator{SiteID: "test", URL: "https://example.com/blog/post/1"})
assert.NoError(t, err)
assert.Equal(t, 2, count)
count, err = dataStore.Count(store.Locator{SiteID: "test", URL: "https://example.com/blog/post/2"})
assert.NoError(t, err)
assert.Equal(t, 1, count)
}
+2 -2
View File
@@ -175,7 +175,7 @@ func (d *Disqus) convert(r io.Reader, siteID string) (ch chan store.Comment) {
func (*Disqus) cleanText(text string) string {
text = strings.TrimSpace(text)
text = strings.Replace(text, "\n", "", -1)
text = strings.Replace(text, "\t", "", -1)
text = strings.ReplaceAll(text, "\n", "")
text = strings.ReplaceAll(text, "\t", "")
return text
}
+2 -2
View File
@@ -77,11 +77,11 @@ func TestMigrator_ImportCommento(t *testing.T) {
Provider: "commento",
})
assert.NoError(t, err)
assert.Equal(t, 2, size)
assert.Equal(t, 3, size)
last, err := dataStore.Last("test", 10, time.Time{}, store.User{})
assert.NoError(t, err)
assert.Equal(t, 2, len(last), "2 comments imported")
assert.Equal(t, 3, len(last), "3 comments imported")
}
func TestMigrator_ImportNative(t *testing.T) {
+1 -1
View File
@@ -163,7 +163,7 @@ func TestNative_ImportManyWithError(t *testing.T) {
buf := &bytes.Buffer{}
buf.WriteString(`{"version":1, "users":[], "posts":[]}` + "\n")
for i := 0; i < 100; i++ {
buf.WriteString(fmt.Sprintf(goodRec, i))
fmt.Fprintf(buf, goodRec, i)
}
buf.WriteString("{}\n")
buf.WriteString("{}\n")
+3 -3
View File
@@ -4,7 +4,7 @@
{
"commentHex": "e7069a7dfcfaed43caf62300a9b0edb1c124ad79d0f5887c93649c15d7f69945",
"domain": "example.com",
"url": "https://example.com/blog/post/1",
"url": "/blog/post/2",
"commenterHex": "anonymous",
"markdown": "Example comment created by user.",
"html": "",
@@ -18,7 +18,7 @@
{
"commentHex": "7d77e39fcd813241d6281478cc8f21ab5f807d043c750bc1a936bc23b34fb854",
"domain": "example.com",
"url": "https://example.com/blog/post/1",
"url": "/blog/post/1",
"commenterHex": "a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"markdown": "Example 2 comment created by user.",
"html": "",
@@ -32,7 +32,7 @@
{
"commentHex": "ea5f7bcd6ac9bb7b657f7d0569831104e1bcf9c253d03c1e16bf9654c49a5ce9",
"domain": "example.com",
"url": "https://example.com/blog/post/1",
"url": "/blog/post/1",
"commenterHex": "bd1290ab5c858cf2a05903c2a9a61fd63399c6635db38cc6597002195e22e061",
"markdown": "Great reply!",
"html": "",
+3 -2
View File
@@ -16,7 +16,8 @@ const wpTimeLayout = "2006-01-02 15:04:05"
// WordPress implements Importer from WP xml
type WordPress struct {
DataStore Store
DataStore Store
DisableFancyTextFormatting bool
}
type wpItem struct {
@@ -138,7 +139,7 @@ func (w *WordPress) convert(r io.Reader, siteID string) chan store.Comment {
ParentID: comment.PID,
Imported: true,
}
commentsCh <- commentFormatter.Format(c)
commentsCh <- commentFormatter.Format(c, w.DisableFancyTextFormatting)
stats.inpComments++
if stats.inpComments%1000 == 0 {
log.Printf("[DEBUG] processed %d comments", stats.inpComments)
+15 -3
View File
@@ -24,7 +24,7 @@ func TestWordPress_Import(t *testing.T) {
dataStore := service.DataStore{Engine: b, AdminStore: admin.NewStaticStore("12345", nil, []string{}, "")}
defer dataStore.Close()
wp := WordPress{DataStore: &dataStore}
wp := WordPress{DataStore: &dataStore, DisableFancyTextFormatting: false}
size, err := wp.Import(strings.NewReader(xmlTestWP), siteID)
assert.NoError(t, err)
assert.Equal(t, 3, size)
@@ -41,7 +41,7 @@ func TestWordPress_Import(t *testing.T) {
assert.Equal(t, "e8b1e92bbcf5b9bb88472f9bdb82d1b8c7ed39d6", c.User.IP)
ts, _ := time.Parse(wpTimeLayout, "2010-08-18 15:19:14")
assert.Equal(t, ts, c.Timestamp)
assert.Equal(t, c.Text, "<p>Mekkatorque was over in that tent up to the right</p>\n")
assert.Equal(t, "<p>«Mekkatorque» was over in that tent up to the right</p>\n", c.Text)
assert.True(t, c.Imported)
posts, err := dataStore.List(siteID, 0, 0)
@@ -54,6 +54,18 @@ func TestWordPress_Import(t *testing.T) {
count, err := dataStore.Count(store.Locator{URL: "https://realmenweardress.es/2010/07/do-you-rp/", SiteID: siteID})
assert.NoError(t, err)
assert.Equal(t, 3, count)
// test with DisableFancyTextFormatting
wp = WordPress{DataStore: &dataStore, DisableFancyTextFormatting: true}
size, err = wp.Import(strings.NewReader(xmlTestWP), siteID)
assert.NoError(t, err)
assert.Equal(t, 3, size)
last, err = dataStore.Last(siteID, 10, time.Time{}, adminUser)
assert.NoError(t, err)
require.Equal(t, 3, len(last), "3 comments imported")
assert.Equal(t, "<p>&#34;Mekkatorque&#34; was over in that tent up to the right</p>\n", last[0].Text)
}
func TestWordPress_Convert(t *testing.T) {
@@ -247,7 +259,7 @@ var xmlTestWP = `
<wp:comment_author_IP><![CDATA[128.243.253.117]]></wp:comment_author_IP>
<wp:comment_date><![CDATA[2010-08-18 15:19:14]]></wp:comment_date>
<wp:comment_date_gmt><![CDATA[2010-08-18 15:19:14]]></wp:comment_date_gmt>
<wp:comment_content><![CDATA[Mekkatorque was over in that tent up to the right]]></wp:comment_content>
<wp:comment_content><![CDATA["Mekkatorque" was over in that tent up to the right]]></wp:comment_content>
<wp:comment_approved><![CDATA[1]]></wp:comment_approved>
<wp:comment_type><![CDATA[]]></wp:comment_type>
<wp:comment_parent>13</wp:comment_parent>
+3 -3
View File
@@ -10,7 +10,7 @@ import (
log "github.com/go-pkgz/lgr"
ntf "github.com/go-pkgz/notify"
"github.com/go-pkgz/repeater"
"github.com/go-pkgz/repeater/v2"
"github.com/hashicorp/go-multierror"
"github.com/umputun/remark42/backend/app/templates"
@@ -161,7 +161,7 @@ func (e *Email) buildAndSendMessage(ctx context.Context, req Request, email stri
return err
}
return repeater.NewDefault(5, time.Millisecond*250).Do(
return repeater.NewFixed(5, time.Millisecond*250).Do(
ctx,
func() error {
return e.Email.Send(
@@ -196,7 +196,7 @@ func (e *Email) SendVerification(ctx context.Context, req VerificationRequest) e
return err
}
return repeater.NewDefault(5, time.Millisecond*250).Do(
return repeater.NewFixed(5, time.Millisecond*250).Do(
ctx,
func() error {
return e.Email.Send(
+1 -2
View File
@@ -34,7 +34,7 @@ func TestEmailNew(t *testing.T) {
assert.NotNil(t, email, "email returned")
assert.NotNil(t, email.msgTmpl, "e.template is set")
assert.Equal(t, emailParams.From, email.EmailParams.From, "emailParams.From unchanged after creation")
assert.Equal(t, emailParams.From, email.From, "emailParams.From unchanged after creation")
if smtpParams.TimeOut == 0 {
assert.Equal(t, defaultEmailTimeout, email.TimeOut, "empty emailParams.TimeOut changed to default")
} else {
@@ -88,7 +88,6 @@ func Test_initTemplatesErr(t *testing.T) {
}
for _, d := range testSet {
d := d
t.Run(d.name, func(t *testing.T) {
e, err := NewEmail(d.emailParams, ntf.SMTPParams{})
require.Error(t, err)
+12 -1
View File
@@ -66,4 +66,15 @@ func (m *MockDest) GetVerify() []VerificationRequest {
return res
}
func (m *MockDest) String() string { return fmt.Sprintf("mock id=%d, closed=%v", m.id, m.closed) }
// IsClosed returns closed status safely
func (m *MockDest) IsClosed() bool {
m.lock.Lock()
defer m.lock.Unlock()
return m.closed
}
func (m *MockDest) String() string {
m.lock.Lock()
defer m.lock.Unlock()
return fmt.Sprintf("mock id=%d, closed=%v", m.id, m.closed)
}
+5 -3
View File
@@ -102,15 +102,17 @@ func TestService_Many(t *testing.T) {
time.Sleep(time.Millisecond * time.Duration(rand.Int31n(20)))
}
s.Close()
time.Sleep(time.Millisecond * 10)
// wait for destinations to close
assert.Eventually(t, func() bool { return d1.IsClosed() && d2.IsClosed() }, 100*time.Millisecond, 10*time.Millisecond)
assert.NotEqual(t, 10, len(d1.Get()), "some comments dropped from d1")
assert.NotEqual(t, 10, len(d1.GetVerify()), "some verifications dropped from d1")
assert.NotEqual(t, 10, len(d2.Get()), "some comments dropped from d2")
assert.NotEqual(t, 10, len(d2.GetVerify()), "some verifications dropped from d2")
assert.True(t, d1.closed)
assert.True(t, d2.closed)
assert.True(t, d1.IsClosed())
assert.True(t, d2.IsClosed())
assert.Equal(t, "mock id=1, closed=true", d1.String())
}
+77
View File
@@ -0,0 +1,77 @@
package notify
import (
"fmt"
"strings"
"golang.org/x/net/html"
)
// pruneHTML prunes string keeping HTML closing tags.
// maxLength applies to visible text only, not HTML tags.
func pruneHTML(htmlText string, maxLength int) string {
var result strings.Builder
var endTokens []string
visibleLen := 0
suffix := "..."
suffixLen := len(suffix)
tokenizer := html.NewTokenizer(strings.NewReader(htmlText))
for {
if tokenizer.Next() == html.ErrorToken {
return result.String()
}
token := tokenizer.Token()
switch token.Type {
case html.CommentToken, html.DoctypeToken:
continue
case html.StartTagToken:
endTokens = append([]string{fmt.Sprintf("</%s>", token.Data)}, endTokens...)
result.WriteString(token.String())
case html.EndTagToken:
if len(endTokens) > 0 {
endTokens = endTokens[1:]
}
result.WriteString(token.String())
case html.SelfClosingTagToken:
result.WriteString(token.String())
case html.TextToken:
text := token.String()
if visibleLen+len(text)+suffixLen > maxLength {
remaining := maxLength - visibleLen - suffixLen
text = pruneStringToWord(text, remaining)
result.WriteString(text)
result.WriteString(suffix)
for _, endTag := range endTokens {
result.WriteString(endTag)
}
return result.String()
}
visibleLen += len(text)
result.WriteString(text)
}
}
}
// pruneStringToWord prunes string to specified length respecting word boundaries
func pruneStringToWord(text string, maxLength int) string {
if maxLength <= 0 {
return ""
}
if len(text) <= maxLength {
return text
}
// find last space at or before maxLength to cut at word boundary
lastSpace := strings.LastIndex(text[:maxLength+1], " ")
if lastSpace <= 0 {
return ""
}
return text[:lastSpace]
}
+47
View File
@@ -0,0 +1,47 @@
package notify
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestPruneHTML(t *testing.T) {
tests := []struct {
name string
html string
maxLength int
expected string
}{
{"within limit", "<p>Hello</p>", 20, "<p>Hello</p>"},
{"exceeds limit", "<p>Hello world, this is a long text</p>", 15, "<p>Hello world,...</p>"},
{"nested tags", "<div><p>Hello world</p><p>More text</p></div>", 20, "<div><p>Hello world</p><p>More...</p></div>"},
{"html comment stripped", "<!-- comment --><p>Hello</p>", 20, "<p>Hello</p>"},
{"self-closing tag", "<p>Hello<br/>World</p>", 8, "<p>Hello<br/>...</p>"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, pruneHTML(tt.html, tt.maxLength))
})
}
}
func TestPruneStringToWord(t *testing.T) {
tests := []struct {
name string
text string
maxLength int
expected string
}{
{"within limit", "hello world", 15, "hello world"},
{"cut at word boundary", "hello world and more", 11, "hello world"},
{"zero length", "hello", 0, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, pruneStringToWord(tt.text, tt.maxLength))
})
}
}
+4 -2
View File
@@ -10,6 +10,8 @@ import (
"github.com/hashicorp/go-multierror"
)
const commentTextLengthLimit = 100
// TelegramParams contain settings for telegram notifications
type TelegramParams struct {
AdminChannelID string // unique identifier for the target chat or username of the target channel (in the format @channelusername)
@@ -85,10 +87,10 @@ func (t *Telegram) buildMessage(req Request) string {
msg += fmt.Sprintf(" -> <a href=%q>%s</a>", commentURLPrefix+req.parent.ID, ntf.EscapeTelegramText(req.parent.User.Name))
}
msg += fmt.Sprintf("\n\n%s", ntf.TelegramSupportedHTML(req.Comment.Text))
msg += fmt.Sprintf("\n\n%s", pruneHTML(ntf.TelegramSupportedHTML(req.Comment.Text), commentTextLengthLimit))
if req.Comment.ParentID != "" {
msg += fmt.Sprintf("\n\n\"<i>%s</i>\"", ntf.TelegramSupportedHTML(req.parent.Text))
msg += fmt.Sprintf("\n\n\"<i>%s</i>\"", pruneHTML(ntf.TelegramSupportedHTML(req.parent.Text), commentTextLengthLimit))
}
if req.Comment.PostTitle != "" {
+9
View File
@@ -53,6 +53,15 @@ some text
<b>Hello</b><i><b>World</b></i>`,
res)
// prune string keeping HTML closing tags
c = store.Comment{
Text: "<b>Lorem ipsum <i>dolor sit amet</i>, consectetur adipiscing <code>elit, sed do eiusmod tempor incididunt</code> ut labore et dolore magna aliqua.</b>",
}
res = tb.buildMessage(Request{Comment: c})
assert.Equal(t, `<a href="#remark42__comment-"></a>
<b>Lorem ipsum <i>dolor sit amet</i>, consectetur adipiscing <code>elit, sed do eiusmod tempor incididunt</code> ut...</b>`, res)
}
func TestTelegram_SendVerification(t *testing.T) {
+12 -2
View File
@@ -3,6 +3,7 @@ package notify
import (
"bytes"
"context"
"encoding/json"
"fmt"
"text/template"
"time"
@@ -12,7 +13,7 @@ import (
)
const (
webhookDefaultTemplate = `{"text": "{{.Text}}"}`
webhookDefaultTemplate = `{"text": {{.Text | escapeJSONString}}}`
)
// WebhookParams contain settings for webhook notifications
@@ -49,7 +50,7 @@ func NewWebhook(params WebhookParams) (*Webhook, error) {
params.Template = webhookDefaultTemplate
}
payloadTmpl, err := template.New("webhook").Parse(params.Template)
payloadTmpl, err := template.New("webhook").Funcs(template.FuncMap{"escapeJSONString": escapeJSONString}).Parse(params.Template)
if err != nil {
return nil, fmt.Errorf("unable to parse webhook template: %w", err)
}
@@ -82,3 +83,12 @@ func (w *Webhook) SendVerification(_ context.Context, _ VerificationRequest) err
func (w *Webhook) String() string {
return fmt.Sprintf("%s to %s", w.Webhook.String(), w.url)
}
// escapeJSONString escapes string for JSON
func escapeJSONString(s string) (string, error) {
b, err := json.Marshal(s)
if err != nil {
return "", err
}
return string(b), nil
}
+29
View File
@@ -2,6 +2,9 @@ package notify
import (
"context"
"io"
"net/http"
"net/http/httptest"
"testing"
"time"
@@ -34,6 +37,32 @@ func TestWebhook_NewWebhook(t *testing.T) {
assert.Contains(t, err.Error(), "unable to parse webhook template")
}
// https://github.com/umputun/remark42/issues/1791
func TestWebhook_ReceiveValidJSON(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
assert.Equal(t, r.URL.Path, "/webhook-notify")
assert.Equal(t, "POST", r.Method)
body, err := io.ReadAll(r.Body)
assert.NoError(t, err)
t.Log("received body", string(body))
assert.JSONEq(t, `{"text": "<p>testme</p>\n"}`, string(body))
}))
defer ts.Close()
wh, err := NewWebhook(WebhookParams{
URL: ts.URL + "/webhook-notify",
Headers: []string{"Content-Type:application/json,text/plain"},
})
assert.NoError(t, err)
assert.NotNil(t, wh)
f := store.NewCommentFormatter()
c := store.Comment{Text: f.FormatText("testme", false), ParentID: "1", ID: "999"}
err = wh.Send(context.Background(), Request{Comment: c})
assert.NoError(t, err)
}
func TestWebhook_Send(t *testing.T) {
wh, err := NewWebhook(WebhookParams{
URL: "bad-url",
+23 -21
View File
@@ -7,9 +7,8 @@ import (
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
"github.com/go-pkgz/auth"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
@@ -54,8 +53,7 @@ func (a *admin) deleteCommentCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.SiteID, locator.URL, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"id": id, "locator": locator})
R.RenderJSON(w, R.JSON{"id": id, "locator": locator})
}
// DELETE /user/{userid}?site=side-id - delete all user comments for requested userid
@@ -69,8 +67,7 @@ func (a *admin) deleteUserCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(siteID).Scopes(userID, siteID, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"user_id": userID, "site_id": siteID})
R.RenderJSON(w, R.JSON{"user_id": userID, "site_id": siteID})
}
// GET /user/{userid}?site=side-id - get user info for requested userid
@@ -84,8 +81,7 @@ func (a *admin) getUserInfoCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get user info", rest.ErrInternal)
return
}
render.Status(r, http.StatusOK)
render.JSON(w, r, ucomments[0].User)
R.RenderJSON(w, ucomments[0].User)
}
// GET /deleteme?token=jwt - delete all user comments and details by user's request. Gets info about deleted used from provided token
@@ -107,13 +103,21 @@ func (a *admin) deleteMeRequestCtrl(w http.ResponseWriter, r *http.Request) {
return
}
if err = a.dataService.DeleteUserDetail(claims.Audience, claims.User.ID, engine.AllUserDetails); err != nil {
// Audience is a slice but we set it to a single element, and situation when there is no audience or there are more than one is unexpected
if len(claims.Audience) != 1 {
rest.SendErrorJSON(w, r, http.StatusBadRequest, fmt.Errorf("bad request"), "can't process token, claims.Audience expected to be a single element but it's not", rest.ErrActionRejected)
return
}
audience := claims.Audience[0]
if err = a.dataService.DeleteUserDetail(audience, claims.User.ID, engine.AllUserDetails); err != nil {
code := parseError(err, rest.ErrInternal)
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete user details for user", code)
return
}
if err = a.dataService.DeleteUser(claims.Audience, claims.User.ID, store.HardDelete); err != nil {
if err = a.dataService.DeleteUser(audience, claims.User.ID, store.HardDelete); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete user", rest.ErrNoAccess)
return
}
@@ -126,9 +130,8 @@ func (a *admin) deleteMeRequestCtrl(w http.ResponseWriter, r *http.Request) {
}
}
a.cache.Flush(cache.Flusher(claims.Audience).Scopes(claims.Audience, claims.User.ID, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"user_id": claims.User.ID, "site_id": claims.Audience})
a.cache.Flush(cache.Flusher(audience).Scopes(audience, claims.User.ID, lastCommentsScope))
R.RenderJSON(w, R.JSON{"user_id": claims.User.ID, "site_id": claims.Audience})
}
// PUT /user/{userid}?site=side-id&block=1&ttl=7d - block or unblock user
@@ -156,7 +159,7 @@ func (a *admin) setBlockCtrl(w http.ResponseWriter, r *http.Request) {
}
}
a.cache.Flush(cache.Flusher(siteID).Scopes(userID, siteID, lastCommentsScope))
render.JSON(w, r, R.JSON{"user_id": userID, "site_id": siteID, "block": blockStatus})
R.RenderJSON(w, R.JSON{"user_id": userID, "site_id": siteID, "block": blockStatus})
}
// GET /blocked?site=siteID - list blocked users
@@ -167,7 +170,7 @@ func (a *admin) blockedUsersCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get blocked users", rest.ErrSiteNotFound)
return
}
render.JSON(w, r, users)
R.RenderJSON(w, users)
}
// PUT /readonly?site=siteID&url=post-url&ro=1 - set or reset read-only status for the post
@@ -194,7 +197,7 @@ func (a *admin) setReadOnlyCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL, locator.SiteID))
render.JSON(w, r, R.JSON{"locator": locator, "read-only": roStatus})
R.RenderJSON(w, R.JSON{"locator": locator, "read-only": roStatus})
}
// PUT /title/{id}?site=siteID&url=post-url - set comment PostTitle to page's title
@@ -210,8 +213,7 @@ func (a *admin) setTitleCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[INFO] set comment's title %s to %q", id, c.PostTitle)
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL, lastCommentsScope))
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"id": id, "locator": locator})
R.RenderJSON(w, R.JSON{"id": id, "locator": locator})
}
// PUT /verify?site=siteID&url=post-url&ro=1 - set or reset read-only status for the post
@@ -225,7 +227,7 @@ func (a *admin) setVerifyCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(siteID).Scopes(siteID, userID))
render.JSON(w, r, R.JSON{"user": userID, "verified": verifyStatus})
R.RenderJSON(w, R.JSON{"user": userID, "verified": verifyStatus})
}
// PUT /pin/{id}?site=siteID&url=post-url&pin=1
@@ -240,5 +242,5 @@ func (a *admin) setPinCtrl(w http.ResponseWriter, r *http.Request) {
return
}
a.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL))
render.JSON(w, r, R.JSON{"id": commentID, "locator": locator, "pin": pinStatus})
R.RenderJSON(w, R.JSON{"id": commentID, "locator": locator, "pin": pinStatus})
}
+55 -25
View File
@@ -13,10 +13,10 @@ import (
"testing"
"time"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -111,7 +111,7 @@ func TestAdmin_Title(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
srv.DataService.TitleExtractor = service.NewTitleExtractor(http.Client{Timeout: time.Second})
srv.DataService.TitleExtractor = service.NewTitleExtractor(http.Client{Timeout: time.Second}, []string{"127.0.0.1"})
tss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.String() == "/post1" {
_, err := w.Write([]byte("<html><title>post1 blah 123</title><body> 2222</body></html>"))
@@ -348,7 +348,7 @@ func TestAdmin_Block(t *testing.T) {
assert.Equal(t, "test test #1", comments.Comments[2].Text, "comment not removed and not cleared")
assert.False(t, comments.Comments[2].Deleted, "not deleted")
srv.pubRest.cache = cache.NewScache(cache.NewNopCache()) // TODO: with lru cache it won't be refreshed and invalidated for long
srv.pubRest.cache = cache.NewScache[[]byte](cache.NewNopCache[[]byte]()) // TODO: with lru cache it won't be refreshed and invalidated for long
// time
time.Sleep(50 * time.Millisecond)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah&sort=+time")
@@ -513,6 +513,7 @@ func TestAdmin_ReadOnlyNoComments(t *testing.T) {
_, err = srv.DataService.Info(store.Locator{SiteID: "remark42", URL: "https://radio-t.com/blah"}, 0)
assert.Error(t, err)
// test format "tree"
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah&format=tree")
assert.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
@@ -521,6 +522,16 @@ func TestAdmin_ReadOnlyNoComments(t *testing.T) {
assert.Equal(t, 0, len(comments.Comments), "should have 0 comments")
assert.True(t, comments.Info.ReadOnly)
t.Logf("%+v", comments)
// test format "plain"
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah")
assert.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
assert.NoError(t, err)
assert.Equal(t, 0, len(comments.Comments), "should have 0 comments")
assert.True(t, comments.Info.ReadOnly)
t.Logf("%+v", comments)
}
func TestAdmin_ReadOnlyWithAge(t *testing.T) {
@@ -697,12 +708,12 @@ func TestAdmin_DeleteMeRequest(t *testing.T) {
claims := token.Claims{
SessionOnly: true,
StandardClaims: jwt.StandardClaims{
Audience: "remark42",
Id: "1234567",
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "1234567",
Issuer: "remark42",
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: time.Now().Add(30 * time.Minute).Unix(),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user1",
@@ -743,9 +754,9 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
defer teardown()
c1 := store.Comment{Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user1 name", ID: "user1"}}
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user1 name", ID: "provider1_user1"}}
c2 := store.Comment{Text: "test test #2", ParentID: "p1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user2", ID: "user2"}}
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user2", ID: "provider1_user2"}}
_, err := srv.DataService.Create(c1)
assert.NoError(t, err)
@@ -766,15 +777,15 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
// try with bad auth
claims := token.Claims{
SessionOnly: true,
StandardClaims: jwt.StandardClaims{
Audience: "remark42",
Id: "1234567",
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ID: "provider1_1234567",
Issuer: "remark42",
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: time.Now().Add(30 * time.Minute).Unix(),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
},
User: &token.User{
ID: "user1",
ID: "provider1_user1",
Attributes: map[string]interface{}{
"delete_me": true,
},
@@ -792,9 +803,9 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
// try bad user
badClaims := claims
badClaims.User.ID = "no-such-id"
tkn, err = srv.Authenticator.TokenService().Token(badClaims)
badClaimsUser := claims
badClaimsUser.User.ID = "no-such-id"
tkn, err = srv.Authenticator.TokenService().Token(badClaimsUser)
assert.NoError(t, err)
req, err = http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
assert.NoError(t, err)
@@ -803,11 +814,12 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusBadRequest, resp.StatusCode, resp.Status)
badClaimsUser.User.ID = "provider1_user1"
// try without deleteme flag
badClaims2 := claims
badClaims2.User.SetBoolAttr("delete_me", false)
tkn, err = srv.Authenticator.TokenService().Token(badClaims2)
badClaimsWithoutDeleteMe := claims
badClaimsWithoutDeleteMe.User.SetBoolAttr("delete_me", false)
tkn, err = srv.Authenticator.TokenService().Token(badClaimsWithoutDeleteMe)
assert.NoError(t, err)
req, err = http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
assert.NoError(t, err)
@@ -818,7 +830,25 @@ func TestAdmin_DeleteMeRequestFailed(t *testing.T) {
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.True(t, strings.Contains(string(b), "can't use provided token"))
assert.Contains(t, string(b), "can't use provided token")
badClaimsWithoutDeleteMe.User.SetBoolAttr("delete_me", true)
// try with wrong audience
badClaimsMultipleAudience := claims
badClaimsMultipleAudience.Audience = jwt.ClaimStrings{"remark42", "something else"}
tkn, err = srv.Authenticator.TokenService().Token(badClaimsMultipleAudience)
assert.NoError(t, err)
req, err = http.NewRequest(http.MethodGet, fmt.Sprintf("%s/api/v1/admin/deleteme?token=%s", ts.URL, tkn), http.NoBody)
assert.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Contains(t, string(b), "can't process token, claims.Audience expected to be a single element but it's not")
badClaimsMultipleAudience.Audience = jwt.ClaimStrings{"remark42"}
}
func TestAdmin_GetUserInfo(t *testing.T) {
+30 -28
View File
@@ -1,17 +1,18 @@
package api
import (
"bytes"
"compress/gzip"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"sync"
"time"
"github.com/go-chi/render"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
@@ -58,8 +59,7 @@ func (m *Migrator) importCtrl(w http.ResponseWriter, r *http.Request) {
go m.runImport(siteID, r.URL.Query().Get("provider"), tmpfile) // import runs in background and sets busy flag for site
render.Status(r, http.StatusAccepted)
render.JSON(w, r, R.JSON{"status": "import request accepted"})
_ = R.EncodeJSON(w, http.StatusAccepted, R.JSON{"status": "import request accepted"})
}
// POST /import/form?secret=key&site=site-id&provider=disqus|remark|wordpress
@@ -93,8 +93,7 @@ func (m *Migrator) importFormCtrl(w http.ResponseWriter, r *http.Request) {
go m.runImport(siteID, r.URL.Query().Get("provider"), tmpfile) // import runs in background and sets busy flag for site
render.Status(r, http.StatusAccepted)
render.JSON(w, r, R.JSON{"status": "import request accepted"})
_ = R.EncodeJSON(w, http.StatusAccepted, R.JSON{"status": "import request accepted"})
}
// GET /wait?site=site-id
@@ -110,20 +109,16 @@ func (m *Migrator) waitCtrl(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(context.Background(), timeOut)
defer cancel()
for {
if !m.isBusy(siteID) {
break
}
for m.isBusy(siteID) {
select {
case <-ctx.Done():
render.Status(r, http.StatusGatewayTimeout)
render.JSON(w, r, R.JSON{"status": "timeout expired", "site_id": siteID})
_ = R.EncodeJSON(w, http.StatusGatewayTimeout, R.JSON{"status": "timeout expired", "site_id": siteID})
return
case <-time.After(100 * time.Millisecond):
}
}
render.Status(r, http.StatusOK)
render.JSON(w, r, R.JSON{"status": "completed", "site_id": siteID})
R.RenderJSON(w, R.JSON{"status": "completed", "site_id": siteID})
}
// GET /export?site=site-id&secret=12345&?mode=file|stream
@@ -131,24 +126,32 @@ func (m *Migrator) waitCtrl(w http.ResponseWriter, r *http.Request) {
func (m *Migrator) exportCtrl(w http.ResponseWriter, r *http.Request) {
siteID := r.URL.Query().Get("site")
var writer io.Writer = w
if r.URL.Query().Get("mode") == "file" {
// buffer to memory to handle errors before committing to response
var buf bytes.Buffer
gzWriter := gzip.NewWriter(&buf)
if _, err := m.NativeExporter.Export(gzWriter, siteID); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "export failed", rest.ErrInternal)
return
}
if err := gzWriter.Close(); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "export failed", rest.ErrInternal)
return
}
exportFile := fmt.Sprintf("%s-%s.json.gz", siteID, time.Now().Format("20060102"))
w.Header().Set("Content-Type", "application/gzip")
w.Header().Set("Content-Disposition", "attachment;filename="+exportFile)
w.WriteHeader(http.StatusOK)
gzWriter := gzip.NewWriter(w)
defer func() {
if e := gzWriter.Close(); e != nil {
log.Printf("[WARN] can't close gzip writer, %s", e)
}
}()
writer = gzWriter
w.Header().Set("Content-Length", strconv.Itoa(buf.Len()))
if _, err := io.Copy(w, &buf); err != nil {
log.Printf("[WARN] failed to write export response: %v", err)
}
return
}
if _, err := m.NativeExporter.Export(writer, siteID); err != nil {
// stream mode - write directly to response
if _, err := m.NativeExporter.Export(w, siteID); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "export failed", rest.ErrInternal)
return
}
}
@@ -163,7 +166,7 @@ func (m *Migrator) remapCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "remap failed, bad given rules", rest.ErrDecode)
return
}
defer r.Body.Close()
defer r.Body.Close() //nolint gosec // we don't care about response body
// start remap procedure with mapper
go func() {
@@ -204,8 +207,7 @@ func (m *Migrator) remapCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[DEBUG] convert request completed. site=%s, comments=%d", siteID, size)
}()
render.Status(r, http.StatusAccepted)
render.JSON(w, r, R.JSON{"status": "convert request accepted"})
_ = R.EncodeJSON(w, http.StatusAccepted, R.JSON{"status": "convert request accepted"})
}
// runImport reads from tmpfile and import for given siteID and provider
+164 -2
View File
@@ -9,6 +9,7 @@ import (
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
@@ -49,6 +50,22 @@ func TestMigrator_Import(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportForm(t *testing.T) {
@@ -84,13 +101,29 @@ func TestMigrator_ImportForm(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportFromWP(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(strings.Replace(xmlTestWP, "'", "`", -1))
r := strings.NewReader(strings.ReplaceAll(xmlTestWP, "'", "`"))
client := &http.Client{Timeout: 1 * time.Second}
defer client.CloseIdleConnections()
@@ -108,6 +141,22 @@ func TestMigrator_ImportFromWP(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://realmenweardress.es/2010/07/do-you-rp/")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 3, comments.Info.Count)
require.Equal(t, 3, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://realmenweardress.es/2010/07/do-you-rp/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 3, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments), "2 comments with 1 reply")
}
func TestMigrator_ImportFromCommento(t *testing.T) {
@@ -115,7 +164,7 @@ func TestMigrator_ImportFromCommento(t *testing.T) {
defer teardown()
r := strings.NewReader(`{"version":1,"comments":[{"commentHex":"7d77e39fcd813241d6281478cc8f21ab5f807d043c750bc1a936bc23b34fb854",
"domain":"example.com","url":"https://example.com/blog/post/1","commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"domain":"example.com","url":"/blog/post/1","commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"markdown":"Example content","html":"","parentHex":"root","score":0,"state":"approved","creationDate":"2021-03-17T12:09:47.722181Z",
"direction":0,"deleted":false}],"commenters":[{"commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"email":"somegreatmail@gmail.com","name":"User5276","link":"https://example.com/profile/257","photo":"https://secure.gravatar.com/avatar/8f279626d26175134b0d5c88648172f7",
@@ -137,6 +186,63 @@ func TestMigrator_ImportFromCommento(t *testing.T) {
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/blog/post/1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://example.com/blog/post/1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportFromCommentoJSON(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r, err := os.Open("testdata/commento.json")
require.NoError(t, err)
client := &http.Client{Timeout: 1 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=commento", r)
assert.NoError(t, err)
req.Header.Add("Content-Type", "application/json; charset=utf-8")
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 7, comments.Info.Count)
require.Equal(t, 7, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 7, comments.Info.Count)
require.Equal(t, 5, len(comments.Comments), "five comments with two replies")
}
func TestMigrator_ImportRejected(t *testing.T) {
@@ -197,6 +303,20 @@ func TestMigrator_ImportDouble(t *testing.T) {
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusConflict, resp.StatusCode)
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 50, comments.Info.Count)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 50, comments.Info.Count)
}
func TestMigrator_ImportWaitExpired(t *testing.T) {
@@ -236,6 +356,14 @@ func TestMigrator_ImportWaitExpired(t *testing.T) {
assert.Equal(t, http.StatusGatewayTimeout, resp.StatusCode)
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
}
func TestMigrator_Export(t *testing.T) {
@@ -263,6 +391,16 @@ func TestMigrator_Export(t *testing.T) {
require.Equal(t, http.StatusAccepted, resp.StatusCode)
waitForMigrationCompletion(t, ts)
// export wrong site, should result in error
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=file&site=test", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusInternalServerError, resp.StatusCode)
require.Equal(t, "application/json", resp.Header.Get("Content-Type"))
// check file mode
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=file&site=remark42", http.NoBody)
require.NoError(t, err)
@@ -339,6 +477,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo-another/")
@@ -347,6 +486,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
// we want remap urls to another domain - www.remark42.com
@@ -364,6 +504,16 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://www.remark42.com/demo/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://www.remark42.com/demo-another/")
@@ -372,6 +522,16 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://www.remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
// should find nothing from previous url
@@ -381,6 +541,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
@@ -388,6 +549,7 @@ func TestMigrator_Remap(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
}
func TestMigrator_RemapReject(t *testing.T) {
+122 -97
View File
@@ -15,14 +15,13 @@ import (
"sync"
"time"
"github.com/didip/tollbooth/v7"
"github.com/didip/tollbooth_chi"
"github.com/didip/tollbooth/v8"
"github.com/didip/tollbooth/v8/limiter"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/go-chi/cors"
"github.com/go-chi/render"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/go-pkgz/rest/logger"
@@ -59,26 +58,29 @@ type Rest struct {
Low int
Critical int
}
UpdateLimiter float64
EmailNotifications bool
TelegramBotUsername string
EmojiEnabled bool
SimpleView bool
ProxyCORS bool
SendJWTHeader bool
AllowedAncestors []string // sets Content-Security-Policy "frame-ancestors ..."
SubscribersOnly bool
DisableSignature bool // prevent signature from being added to headers
UpdateLimiter float64
EmailNotifications bool
TelegramNotifications bool
EmojiEnabled bool
SimpleView bool
ProxyCORS bool
SendJWTHeader bool
AllowedAncestors []string // sets Content-Security-Policy "frame-ancestors ..."
SubscribersOnly bool
DisableSignature bool // prevent signature from being added to headers
DisableFancyTextFormatting bool // disables SmartyPants in the comment text rendering of the posted comments
ExternalImageProxy bool
SSLConfig SSLConfig
httpsServer *http.Server
httpServer *http.Server
lock sync.Mutex
pubRest public
privRest private
adminRest admin
rssRest rss
pubRest public
privRest private
adminRest admin
rssRest rss
openRouteLimiter float64
}
// LoadingCache defines interface for caching
@@ -89,7 +91,7 @@ type LoadingCache interface {
}
const hardBodyLimit = 1024 * 64 // limit size of body
const openRouteLimiter = 10 // limit for open routes
const lastCommentsScope = "last"
type commentsWithInfo struct {
@@ -97,6 +99,11 @@ type commentsWithInfo struct {
Info store.PostInfo `json:"info,omitempty"`
}
type treeWithInfo struct {
*service.Tree
Info store.PostInfo `json:"info,omitempty"`
}
// Run the lister and request's router, activate rest server
func (s *Rest) Run(address string, port int) {
if address == "*" {
@@ -192,8 +199,13 @@ func (s *Rest) makeHTTPServer(address string, port int, router http.Handler) *ht
}
func (s *Rest) routes() chi.Router {
if s.openRouteLimiter == 0 {
// set the default open route limiter. Just a safety measure as it should be set by Run method anyway
s.openRouteLimiter = openRouteLimiter
}
router := chi.NewRouter()
router.Use(middleware.Throttle(1000), middleware.RealIP, R.Recoverer(log.Default()))
router.Use(securityHeadersMiddleware(s.ExternalImageProxy, s.AllowedAncestors))
if !s.DisableSignature {
router.Use(R.AppInfo("remark42", "umputun", s.Version))
}
@@ -215,11 +227,6 @@ func (s *Rest) routes() chi.Router {
router.Use(corsMiddleware.Handler)
}
if len(s.AllowedAncestors) > 0 {
log.Printf("[INFO] allowed from %+v only", s.AllowedAncestors)
router.Use(frameAncestors(s.AllowedAncestors))
}
ipFn := func(ip string) string { return store.HashValue(ip, s.SharedSecret)[:12] } // logger uses it for anonymization
logInfoWithBody := logger.New(logger.Log(log.Default()), logger.WithBody, logger.IPfn(ipFn), logger.Prefix("[INFO]")).Handler
@@ -227,14 +234,14 @@ func (s *Rest) routes() chi.Router {
router.Group(func(r chi.Router) {
r.Use(middleware.Timeout(5 * time.Second))
r.Use(logInfoWithBody, tollbooth_chi.LimitHandler(tollbooth.NewLimiter(2, nil)), middleware.NoCache)
r.Use(logInfoWithBody, rateLimiter(2), middleware.NoCache)
r.Use(validEmailAuth()) // reject suspicious email logins
r.Mount("/auth", authHandler)
})
router.Group(func(r chi.Router) {
r.Use(middleware.Timeout(5 * time.Second))
r.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(100, nil)))
r.Use(rateLimiter(100))
r.Mount("/avatar", avatarHandler)
})
@@ -244,14 +251,14 @@ func (s *Rest) routes() chi.Router {
router.Route("/api/v1", func(rapi chi.Router) {
rapi.Group(func(rava chi.Router) {
rava.Use(middleware.Timeout(5 * time.Second))
rava.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(100, nil)))
rava.Use(rateLimiter(100))
rava.Mount("/avatar", avatarHandler)
})
// open routes
rapi.Group(func(ropen chi.Router) {
ropen.Use(middleware.Timeout(30 * time.Second))
ropen.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
ropen.Use(rateLimiter(s.openRouteLimiter))
ropen.Use(authMiddleware.Trace, middleware.NoCache, logInfoWithBody)
ropen.Get("/config", s.configCtrl)
ropen.Get("/find", s.pubRest.findCommentsCtrl)
@@ -274,7 +281,7 @@ func (s *Rest) routes() chi.Router {
// open routes, cached
rapi.Group(func(ropen chi.Router) {
ropen.Use(middleware.Timeout(30 * time.Second))
ropen.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
ropen.Use(rateLimiter(10))
ropen.Use(authMiddleware.Trace, logInfoWithBody)
ropen.Get("/picture/{user}/{id}", s.pubRest.loadPictureCtrl)
ropen.Get("/qr/telegram", s.pubRest.telegramQrCtrl)
@@ -283,7 +290,7 @@ func (s *Rest) routes() chi.Router {
// protected routes, require auth
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(30 * time.Second))
rauth.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
rauth.Use(rateLimiter(10))
rauth.Use(authMiddleware.Auth, matchSiteID, middleware.NoCache, logInfoWithBody)
rauth.Get("/user", s.privRest.userInfoCtrl)
rauth.Get("/userdata", s.privRest.userAllDataCtrl)
@@ -292,7 +299,7 @@ func (s *Rest) routes() chi.Router {
// admin routes, require auth and admin users only
rapi.Route("/admin", func(radmin chi.Router) {
radmin.Use(middleware.Timeout(30 * time.Second))
radmin.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
radmin.Use(rateLimiter(10))
radmin.Use(authMiddleware.Auth, authMiddleware.AdminOnly, matchSiteID)
radmin.Use(middleware.NoCache, logInfoWithBody)
@@ -318,7 +325,7 @@ func (s *Rest) routes() chi.Router {
// protected routes, throttled to 10/s by default, controlled by external UpdateLimiter param
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(10 * time.Second))
rauth.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(s.updateLimiter(), nil)))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, matchSiteID, subscribersOnly(s.SubscribersOnly))
rauth.Use(middleware.NoCache, logInfoWithBody)
@@ -338,7 +345,7 @@ func (s *Rest) routes() chi.Router {
// protected routes, anonymous rejected
rapi.Group(func(rauth chi.Router) {
rauth.Use(middleware.Timeout(10 * time.Second))
rauth.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(s.updateLimiter(), nil)))
rauth.Use(rateLimiter(s.updateLimiter()))
rauth.Use(authMiddleware.Auth, rejectAnonUser, matchSiteID)
rauth.Use(logger.New(logger.Log(log.Default()), logger.Prefix("[DEBUG]"), logger.IPfn(ipFn)).Handler)
rauth.Post("/picture", s.privRest.savePictureCtrl)
@@ -348,7 +355,7 @@ func (s *Rest) routes() chi.Router {
// open routes on root level
router.Group(func(rroot chi.Router) {
rroot.Use(middleware.Timeout(10 * time.Second))
rroot.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(50, nil)))
rroot.Use(rateLimiter(50))
rroot.Get("/robots.txt", s.pubRest.robotsCtrl)
rroot.Get("/email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
rroot.Post("/email/unsubscribe.html", s.privRest.emailUnsubscribeCtrl)
@@ -369,16 +376,17 @@ func (s *Rest) controllerGroups() (public, private, admin, rss) {
}
privGrp := private{
dataService: s.DataService,
cache: s.Cache,
imageService: s.ImageService,
commentFormatter: s.CommentFormatter,
readOnlyAge: s.ReadOnlyAge,
authenticator: s.Authenticator,
notifyService: s.NotifyService,
telegramService: s.TelegramService,
remarkURL: s.RemarkURL,
anonVote: s.AnonVote,
dataService: s.DataService,
cache: s.Cache,
imageService: s.ImageService,
commentFormatter: s.CommentFormatter,
readOnlyAge: s.ReadOnlyAge,
authenticator: s.Authenticator,
notifyService: s.NotifyService,
telegramService: s.TelegramService,
remarkURL: s.RemarkURL,
anonVote: s.AnonVote,
disableFancyTextFormatting: s.DisableFancyTextFormatting,
}
admGrp := admin{
@@ -414,44 +422,46 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
emails, _ := s.DataService.AdminStore.Email(siteID)
cnf := struct {
Version string `json:"version"`
EditDuration int `json:"edit_duration"`
AdminEdit bool `json:"admin_edit"`
MaxCommentSize int `json:"max_comment_size"`
Admins []string `json:"admins"`
AdminEmail string `json:"admin_email"`
Auth []string `json:"auth_providers"`
AnonVote bool `json:"anon_vote"`
LowScore int `json:"low_score"`
CriticalScore int `json:"critical_score"`
PositiveScore bool `json:"positive_score"`
ReadOnlyAge int `json:"readonly_age"`
MaxImageSize int `json:"max_image_size"`
EmailNotifications bool `json:"email_notifications"`
TelegramBotUsername string `json:"telegram_bot_username"`
EmojiEnabled bool `json:"emoji_enabled"`
SimpleView bool `json:"simple_view"`
SendJWTHeader bool `json:"send_jwt_header"`
SubscribersOnly bool `json:"subscribers_only"`
Version string `json:"version"`
EditDuration int `json:"edit_duration"`
AdminEdit bool `json:"admin_edit"`
MinCommentSize int `json:"min_comment_size"`
MaxCommentSize int `json:"max_comment_size"`
Admins []string `json:"admins"`
AdminEmail string `json:"admin_email"`
Auth []string `json:"auth_providers"`
AnonVote bool `json:"anon_vote"`
LowScore int `json:"low_score"`
CriticalScore int `json:"critical_score"`
PositiveScore bool `json:"positive_score"`
ReadOnlyAge int `json:"readonly_age"`
MaxImageSize int `json:"max_image_size"`
EmailNotifications bool `json:"email_notifications"`
TelegramNotifications bool `json:"telegram_notifications"`
EmojiEnabled bool `json:"emoji_enabled"`
SimpleView bool `json:"simple_view"`
SendJWTHeader bool `json:"send_jwt_header"`
SubscribersOnly bool `json:"subscribers_only"`
}{
Version: s.Version,
EditDuration: int(s.DataService.EditDuration.Seconds()),
AdminEdit: s.DataService.AdminEdits,
MaxCommentSize: s.DataService.MaxCommentSize,
Admins: admins,
AdminEmail: emails,
LowScore: s.ScoreThresholds.Low,
CriticalScore: s.ScoreThresholds.Critical,
PositiveScore: s.DataService.PositiveScore,
ReadOnlyAge: s.ReadOnlyAge,
MaxImageSize: s.ImageService.MaxSize,
EmailNotifications: s.EmailNotifications,
TelegramBotUsername: s.TelegramBotUsername,
EmojiEnabled: s.EmojiEnabled,
AnonVote: s.AnonVote,
SimpleView: s.SimpleView,
SendJWTHeader: s.SendJWTHeader,
SubscribersOnly: s.SubscribersOnly,
Version: s.Version,
EditDuration: int(s.DataService.EditDuration.Seconds()),
AdminEdit: s.DataService.AdminEdits,
MinCommentSize: s.DataService.MinCommentSize,
MaxCommentSize: s.DataService.MaxCommentSize,
Admins: admins,
AdminEmail: emails,
LowScore: s.ScoreThresholds.Low,
CriticalScore: s.ScoreThresholds.Critical,
PositiveScore: s.DataService.PositiveScore,
ReadOnlyAge: s.ReadOnlyAge,
MaxImageSize: s.ImageService.MaxSize,
EmailNotifications: s.EmailNotifications,
TelegramNotifications: s.TelegramNotifications,
EmojiEnabled: s.EmojiEnabled,
AnonVote: s.AnonVote,
SimpleView: s.SimpleView,
SendJWTHeader: s.SendJWTHeader,
SubscribersOnly: s.SubscribersOnly,
}
cnf.Auth = []string{}
@@ -462,8 +472,7 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
if cnf.Admins == nil { // prevent json serialization to nil
cnf.Admins = []string{}
}
render.Status(r, http.StatusOK)
render.JSON(w, r, cnf)
R.RenderJSON(w, cnf)
}
// serves static files from the webRoot directory or files embedded into the compiled binary if that directory is absent
@@ -482,7 +491,7 @@ func addFileServer(r chi.Router, embedFS embed.FS, webRoot, version string) {
webFS = http.StripPrefix("/web", webFS)
r.Get("/web", http.RedirectHandler("/web/", http.StatusMovedPermanently).ServeHTTP)
r.With(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(20, nil)),
r.With(rateLimiter(20),
middleware.Timeout(10*time.Second),
cacheControl(time.Hour, version),
).Get("/web/*", func(w http.ResponseWriter, r *http.Request) {
@@ -609,19 +618,22 @@ func cacheControl(expiration time.Duration, version string) func(http.Handler) h
}
}
// frameAncestors is a middleware setting Content-Security-Policy "frame-ancestors host1 host2 ..."
// prevents loading of comments widgets from any other origins. In case if the list of allowed empty, ignored.
func frameAncestors(hosts []string) func(http.Handler) http.Handler {
return func(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
if len(hosts) == 0 {
h.ServeHTTP(w, r)
return
// securityHeadersMiddleware sets security-related headers: Content-Security-Policy and Permissions-Policy
func securityHeadersMiddleware(imageProxyEnabled bool, allowedAncestors []string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
imgSrc := "*"
if imageProxyEnabled {
imgSrc = "'self'"
}
w.Header().Set("Content-Security-Policy", "frame-ancestors "+strings.Join(hosts, " ")+";")
h.ServeHTTP(w, r)
}
return http.HandlerFunc(fn)
frameAncestors := "*"
if len(allowedAncestors) > 0 {
frameAncestors = strings.Join(allowedAncestors, " ")
}
w.Header().Set("Content-Security-Policy", fmt.Sprintf("default-src 'none'; base-uri 'none'; form-action 'none'; connect-src 'self'; frame-src 'self' mailto:; img-src %s; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; font-src data:; object-src 'none'; frame-ancestors %s;", imgSrc, frameAncestors))
w.Header().Set("Permissions-Policy", "accelerometer=(), autoplay=(), camera=(), cross-origin-isolated=(), display-capture=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), unload=(), window-management=()")
next.ServeHTTP(w, r)
})
}
}
@@ -715,3 +727,16 @@ func parseError(err error, defaultCode int) (code int) {
return code
}
// rateLimiter creates a rate limiting middleware with proper IP lookup configuration.
// tollbooth v8 requires explicit IP lookup method to be set.
// uses RemoteAddr which is set by chi's middleware.RealIP to the real client IP
// from X-Forwarded-For, X-Real-IP, or True-Client-IP headers.
func rateLimiter(maxReq float64) func(http.Handler) http.Handler {
lmt := tollbooth.NewLimiter(maxReq, nil)
lmt.SetIPLookup(limiter.IPLookup{
Name: "RemoteAddr",
IndexFromRight: 0,
})
return tollbooth.HTTPMiddleware(lmt)
}
+56 -57
View File
@@ -6,6 +6,7 @@ import (
"crypto/rand"
"crypto/sha1" //nolint:gosec //not used for security
"encoding/json"
"errors"
"fmt"
"html/template"
"io"
@@ -14,13 +15,12 @@ import (
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/hashicorp/go-multierror"
"github.com/umputun/remark42/backend/app/notify"
@@ -33,16 +33,17 @@ import (
)
type private struct {
dataService privStore
cache LoadingCache
readOnlyAge int
commentFormatter *store.CommentFormatter
imageService *image.Service
notifyService *notify.Service
authenticator *auth.Service
telegramService telegramService
remarkURL string
anonVote bool
dataService privStore
cache LoadingCache
readOnlyAge int
commentFormatter *store.CommentFormatter
imageService *image.Service
notifyService *notify.Service
authenticator *auth.Service
telegramService telegramService
remarkURL string
anonVote bool
disableFancyTextFormatting bool // disables SmartyPants in the comment text rendering of the posted comments
}
// telegramService is a subset of Telegram service used for setting up user telegram notifications
@@ -75,7 +76,7 @@ func (s *private) previewCommentCtrl(w http.ResponseWriter, r *http.Request) {
user := rest.MustGetUserInfo(r)
comment := store.Comment{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &comment); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&comment); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't bind comment", rest.ErrDecode)
return
}
@@ -87,25 +88,24 @@ func (s *private) previewCommentCtrl(w http.ResponseWriter, r *http.Request) {
return
}
comment = s.commentFormatter.Format(comment)
comment = s.commentFormatter.Format(comment, s.disableFancyTextFormatting)
comment.Sanitize()
// check if images are valid
for _, id := range s.imageService.ExtractPictures(comment.Text) {
// check if images are valid, omit proxied images as they are lazy-loaded
for _, id := range s.imageService.ExtractNonProxiedPictures(comment.Text) {
err := s.imageService.ResetCleanupTimer(id)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't renew staged picture cleanup timer", rest.ErrImgNotFound)
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't load picture from the comment", rest.ErrImgNotFound)
return
}
}
render.HTML(w, r, comment.Text)
rest.HTMLResponse(w, http.StatusOK, comment.Text)
}
// POST /comment - adds comment, resets all immutable fields
func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
comment := store.Comment{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &comment); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&comment); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't bind comment", rest.ErrDecode)
return
}
@@ -127,10 +127,10 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid comment", rest.ErrCommentValidation)
return
}
comment = s.commentFormatter.Format(comment)
comment = s.commentFormatter.Format(comment, s.disableFancyTextFormatting)
// check if images are valid
for _, id := range s.imageService.ExtractPictures(comment.Text) {
// check if images are valid, omit proxied images as they are lazy-loaded
for _, id := range s.imageService.ExtractNonProxiedPictures(comment.Text) {
_, err := s.imageService.Load(id)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't load picture from the comment", rest.ErrImgNotFound)
@@ -150,7 +150,7 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
id, err := s.dataService.Create(comment)
if err == service.ErrRestrictedWordsFound {
if errors.Is(err, service.ErrRestrictedWordsFound) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid comment", rest.ErrCommentRestrictWords)
return
}
@@ -174,8 +174,7 @@ func (s *private) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[DEBUG] created comment %+v", finalComment)
render.Status(r, http.StatusCreated)
render.JSON(w, r, &finalComment)
_ = R.EncodeJSON(w, http.StatusCreated, &finalComment)
}
// PUT /comment/{id}?site=siteID&url=post-url - update comment
@@ -186,7 +185,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
Delete bool
}{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &edit); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&edit); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't read comment details from body", rest.ErrDecode)
return
}
@@ -211,7 +210,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
editReq := service.EditRequest{
Text: s.commentFormatter.FormatText(edit.Text),
Text: s.commentFormatter.FormatText(edit.Text, s.disableFancyTextFormatting),
Orig: edit.Text,
Summary: edit.Summary,
Delete: edit.Delete,
@@ -219,7 +218,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
res, err := s.dataService.EditComment(locator, id, editReq)
if err == service.ErrRestrictedWordsFound {
if errors.Is(err, service.ErrRestrictedWordsFound) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid comment", rest.ErrCommentValidation)
return
}
@@ -231,7 +230,7 @@ func (s *private) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
}
s.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.SiteID, locator.URL, lastCommentsScope, user.ID))
render.JSON(w, r, res)
R.RenderJSON(w, res)
}
// GET /user?site=siteID - returns user info
@@ -244,12 +243,12 @@ func (s *private) userInfoCtrl(w http.ResponseWriter, r *http.Request) {
if err != nil {
log.Printf("[WARN] can't read email for %s, %v", user.ID, err)
}
if len(email) > 0 {
if email != "" {
user.EmailSubscription = true
}
}
render.JSON(w, r, user)
R.RenderJSON(w, user)
}
// PUT /vote/{id}?site=siteID&url=post-url&vote=1 - vote for/against comment
@@ -290,7 +289,7 @@ func (s *private) voteCtrl(w http.ResponseWriter, r *http.Request) {
return
}
s.cache.Flush(cache.Flusher(locator.SiteID).Scopes(locator.URL, comment.User.ID))
render.JSON(w, r, R.JSON{"id": comment.ID, "score": comment.Score})
R.RenderJSON(w, R.JSON{"id": comment.ID, "score": comment.Score})
}
// getEmailCtrl gets email address for authenticated user.
@@ -303,7 +302,7 @@ func (s *private) getEmailCtrl(w http.ResponseWriter, r *http.Request) {
log.Printf("[WARN] can't read email for %s, %v", user.ID, err)
}
render.JSON(w, r, R.JSON{"user": user, "address": address})
R.RenderJSON(w, R.JSON{"user": user, "address": address})
}
// sendEmailConfirmationCtrl gets address and siteID from query, makes confirmation token and sends it to user.
@@ -320,7 +319,7 @@ func (s *private) sendEmailConfirmationCtrl(w http.ResponseWriter, r *http.Reque
Address string
autoConfirm bool
}{autoConfirm: true}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &subscribe); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&subscribe); err != nil {
if err != io.EOF {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't parse request body", rest.ErrDecode)
return
@@ -360,10 +359,10 @@ func (s *private) sendEmailConfirmationCtrl(w http.ResponseWriter, r *http.Reque
claims := token.Claims{
Handshake: &token.Handshake{ID: user.ID + "::" + subscribe.Address},
StandardClaims: jwt.StandardClaims{
Audience: r.URL.Query().Get("site"),
ExpiresAt: time.Now().Add(30 * time.Minute).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{r.URL.Query().Get("site")},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(30 * time.Minute)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
Issuer: "remark42",
},
}
@@ -383,7 +382,7 @@ func (s *private) sendEmailConfirmationCtrl(w http.ResponseWriter, r *http.Reque
},
)
render.JSON(w, r, R.JSON{"user": user, "address": subscribe.Address, "updated": false})
R.RenderJSON(w, R.JSON{"user": user, "address": subscribe.Address, "updated": false})
}
// telegramSubscribeCtrl generates and verifies telegram notification request
@@ -421,7 +420,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
s.telegramService.AddToken(tkn, user.ID, siteID, expires)
render.JSON(w, r, R.JSON{"token": tkn, "bot": s.telegramService.GetBotUsername()})
R.RenderJSON(w, R.JSON{"token": tkn, "bot": s.telegramService.GetBotUsername()})
return
}
@@ -430,7 +429,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
var address, siteID string
address, siteID, err := s.telegramService.CheckToken(queryToken, user.ID)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't set telegram for user", rest.ErrInternal)
rest.SendErrorJSON(w, r, http.StatusNotFound, err, "request is not verified yet", rest.ErrInternal)
return
}
@@ -443,7 +442,7 @@ func (s *private) telegramSubscribeCtrl(w http.ResponseWriter, r *http.Request)
return
}
render.JSON(w, r, R.JSON{"updated": true, "address": val})
R.RenderJSON(w, R.JSON{"updated": true, "address": val})
}
// setConfirmedEmailCtrl uses provided token parameter (generated by sendEmailConfirmationCtrl) to set email and add it to user token
@@ -455,7 +454,7 @@ func (s *private) setConfirmedEmailCtrl(w http.ResponseWriter, r *http.Request)
Site string
Token string
}{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, hardBodyLimit), &confirm); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, hardBodyLimit)).Decode(&confirm); err != nil {
if err != io.EOF {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't parse request body", rest.ErrDecode)
return
@@ -511,7 +510,7 @@ func (s *private) setEmail(w http.ResponseWriter, r *http.Request, userID, siteI
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "failed to set token", rest.ErrInternal)
return
}
render.JSON(w, r, R.JSON{"updated": true, "address": val})
R.RenderJSON(w, R.JSON{"updated": true, "address": val})
}
// POST/GET /email/unsubscribe.html?site=siteID&tkn=jwt - unsubscribe the user in token from email notifications
@@ -595,7 +594,7 @@ func (s *private) emailUnsubscribeCtrl(w http.ResponseWriter, r *http.Request) {
tmpl := template.Must(template.New("unsubscribe").Parse(tmplstr))
msg := bytes.Buffer{}
MustExecute(tmpl, &msg, nil)
render.HTML(w, r, msg.String())
rest.HTMLResponse(w, http.StatusOK, msg.String())
}
// DELETE /email?site=siteID - removes user's email
@@ -622,7 +621,7 @@ func (s *private) deleteEmailCtrl(w http.ResponseWriter, r *http.Request) {
return
}
}
render.JSON(w, r, R.JSON{"deleted": true})
R.RenderJSON(w, R.JSON{"deleted": true})
}
// DELETE /telegram?site=siteID - removes user's telegram
@@ -636,7 +635,7 @@ func (s *private) deleteTelegramCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't delete telegram for user", code)
return
}
render.JSON(w, r, R.JSON{"deleted": true})
R.RenderJSON(w, R.JSON{"deleted": true})
}
// GET /userdata?site=siteID - exports all data about the user as a json with user info and list of all comments
@@ -702,11 +701,11 @@ func (s *private) deleteMeCtrl(w http.ResponseWriter, r *http.Request) {
siteID := r.URL.Query().Get("site")
claims := token.Claims{
StandardClaims: jwt.StandardClaims{
Audience: siteID,
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{siteID},
Issuer: "remark42",
ExpiresAt: time.Now().AddDate(0, 3, 0).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
ExpiresAt: jwt.NewNumericDate(time.Now().AddDate(0, 3, 0)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
},
User: &token.User{
ID: user.ID,
@@ -724,7 +723,7 @@ func (s *private) deleteMeCtrl(w http.ResponseWriter, r *http.Request) {
}
link := fmt.Sprintf("%s/web/deleteme.html?token=%s", s.remarkURL, tokenStr)
render.JSON(w, r, R.JSON{"site": siteID, "user_id": user.ID, "token": tokenStr, "link": link})
R.RenderJSON(w, R.JSON{"site": siteID, "user_id": user.ID, "token": tokenStr, "link": link})
}
// POST /image - save image with form request
@@ -749,7 +748,7 @@ func (s *private) savePictureCtrl(w http.ResponseWriter, r *http.Request) {
return
}
render.JSON(w, r, R.JSON{"id": id})
R.RenderJSON(w, R.JSON{"id": id})
}
func (s *private) isReadOnly(locator store.Locator) bool {
+182 -33
View File
@@ -10,20 +10,21 @@ import (
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
"github.com/go-chi/render"
"github.com/go-pkgz/auth/token"
"github.com/go-pkgz/auth/v2/token"
"github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/golang-jwt/jwt"
"github.com/golang-jwt/jwt/v5"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/notify"
"github.com/umputun/remark42/backend/app/rest/proxy"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/image"
)
@@ -69,7 +70,7 @@ func TestRest_CreateFilteredCode(t *testing.T) {
c := R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
require.NoError(t, err, string(b))
loc := c["locator"].(map[string]interface{})
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
@@ -79,6 +80,86 @@ func TestRest_CreateFilteredCode(t *testing.T) {
assert.True(t, len(c["id"].(string)) > 8)
}
// based on issue https://github.com/umputun/remark42/issues/1631
func TestRest_CreateAndPreviewWithImage(t *testing.T) {
ts, srv, teardown := startupT(t)
ts.Close()
defer teardown()
srv.ImageService.ProxyAPI = srv.RemarkURL + "/api/v1/img"
srv.ImageProxy = &proxy.Image{
HTTP2HTTPS: true,
CacheExternal: true,
RoutePath: "/api/v1/img",
RemarkURL: srv.RemarkURL,
ImageService: srv.ImageService,
}
srv.CommentFormatter = store.NewCommentFormatter(srv.ImageProxy)
// need to recreate the server with new ImageProxy, otherwise old one will be used
ts = httptest.NewServer(srv.routes())
defer ts.Close()
var pngRead bool
// server with the test PNG image
pngServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, e := io.Copy(w, gopherPNG())
assert.NoError(t, e)
pngRead = true
}))
defer pngServer.Close()
t.Run("create", func(t *testing.T) {
resp, err := post(t, ts.URL+"/api/v1/comment",
`{"text": "![](`+pngServer.URL+`/gopher.png)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode, string(b))
assert.NoError(t, resp.Body.Close())
c := R.JSON{}
err = json.Unmarshal(b, &c)
require.NoError(t, err, string(b))
assert.NotContains(t, c["text"], pngServer.URL)
assert.Contains(t, c["text"], srv.RemarkURL)
loc := c["locator"].(map[string]interface{})
assert.Equal(t, "remark42", loc["site"])
assert.Equal(t, "https://radio-t.com/blah1", loc["url"])
assert.True(t, len(c["id"].(string)) > 8)
assert.Equal(t, false, pngRead, "original image is not yet accessed by server")
})
t.Run("preview", func(t *testing.T) {
resp, err := post(t, ts.URL+"/api/v1/preview",
`{"text": "![](`+pngServer.URL+`/gopher.png)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
assert.NoError(t, err)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode, string(b))
assert.NoError(t, resp.Body.Close())
assert.NotContains(t, string(b), pngServer.URL)
assert.Contains(t, string(b), srv.RemarkURL)
assert.Equal(t, false, pngRead, "original image is not yet accessed by server")
// retrieve the image from the cache
imgURL := strings.Split(strings.Split(string(b), "src=\"")[1], "\"")[0]
// replace srv.RemarkURL with ts.URL
imgURL = strings.ReplaceAll(imgURL, srv.RemarkURL, ts.URL)
resp, err = http.Get(imgURL)
assert.NoError(t, err)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode, string(b))
assert.NoError(t, resp.Body.Close())
// compare image to original gopher png after decoding from base64
assert.Equal(t, gopher, base64.StdEncoding.EncodeToString(b))
assert.Equal(t, true, pngRead, "original image accessed to be shown to user")
})
}
func TestRest_CreateOldPost(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
@@ -166,6 +247,25 @@ func TestRest_CreateWithRestrictedWord(t *testing.T) {
assert.Equal(t, "invalid comment", c["details"])
}
func TestRest_CreateRelativeURL(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
// check that it's not possible to click insert URL button and not alter the URL in it (which is `url` by default)
relativeURLText := `{"text": "here is a link with relative URL: [google.com](url)", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`
resp, err := post(t, ts.URL+"/api/v1/comment", relativeURLText)
assert.NoError(t, err)
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
c := R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
assert.Equal(t, "links should start with mailto:, http:// or https://", c["error"])
assert.Equal(t, "invalid comment", c["details"])
}
func TestRest_CreateRejected(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
@@ -267,6 +367,56 @@ func TestRest_CreateAndGet(t *testing.T) {
assert.Equal(t, store.User{Name: "admin", ID: "admin", Admin: true, Blocked: false, IP: ""}, comment.User, "no ip")
}
func TestRest_CreateWithQuotes(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
// create comment with quotes with smartypants
resp, err := post(t, ts.URL+"/api/v1/comment",
`{"text": "smartpants \"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
c := R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
id := c["id"].(string)
// get created comment by id as non-admin
res, code := getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah1", ts.URL, id))
assert.Equal(t, http.StatusOK, code)
comment := store.Comment{}
err = json.Unmarshal([]byte(res), &comment)
assert.NoError(t, err)
assert.Equal(t, "<p>smartpants «quoted» text</p>\n", comment.Text)
assert.Equal(t, "smartpants \"quoted\" text", comment.Orig)
// create comment with quotes without smartypants
srv.privRest.disableFancyTextFormatting = true
resp, err = post(t, ts.URL+"/api/v1/comment",
`{"text": "no_smartpants \"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "remark42"}}`)
require.NoError(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
c = R.JSON{}
err = json.Unmarshal(b, &c)
assert.NoError(t, err)
id = c["id"].(string)
// get created comment by id as non-admin
res, code = getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah1", ts.URL, id))
assert.Equal(t, http.StatusOK, code)
comment = store.Comment{}
err = json.Unmarshal([]byte(res), &comment)
assert.NoError(t, err)
assert.Equal(t, "<p>no_smartpants &#34;quoted&#34; text</p>\n", comment.Text)
assert.Equal(t, "no_smartpants \"quoted\" text", comment.Orig)
}
func TestRest_Update(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
@@ -574,7 +724,7 @@ func TestRest_Vote(t *testing.T) {
req, err := http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/vote/%s?site=remark42&url=https://radio-t.com/blah&vote=%d", ts.URL, id1, val), http.NoBody)
assert.NoError(t, err)
req.Header.Add("X-JWT", devToken)
req.Header.Add("X-JWT", dev2Token)
resp, err := client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
@@ -583,7 +733,7 @@ func TestRest_Vote(t *testing.T) {
assert.Equal(t, http.StatusOK, vote(1), "first vote allowed")
assert.Equal(t, http.StatusBadRequest, vote(1), "second vote rejected")
body, code := getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
body, code := getWithDev2Auth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
assert.Equal(t, http.StatusOK, code)
cr := store.Comment{}
err := json.Unmarshal([]byte(body), &cr)
@@ -594,7 +744,7 @@ func TestRest_Vote(t *testing.T) {
assert.Equal(t, map[string]store.VotedIPInfo(nil), cr.VotedIPs, "hidden")
assert.Equal(t, http.StatusOK, vote(-1), "opposite vote allowed")
body, code = getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
body, code = getWithDev2Auth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
assert.Equal(t, http.StatusOK, code)
cr = store.Comment{}
err = json.Unmarshal([]byte(body), &cr)
@@ -603,7 +753,7 @@ func TestRest_Vote(t *testing.T) {
assert.Equal(t, 0, cr.Vote)
assert.Equal(t, http.StatusOK, vote(-1), "opposite vote allowed one more time")
body, code = getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
body, code = getWithDev2Auth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
assert.Equal(t, http.StatusOK, code)
cr = store.Comment{}
err = json.Unmarshal([]byte(body), &cr)
@@ -612,7 +762,7 @@ func TestRest_Vote(t *testing.T) {
assert.Equal(t, -1, cr.Vote)
assert.Equal(t, http.StatusBadRequest, vote(-1), "dbl vote not allowed")
body, code = getWithDevAuth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
body, code = getWithDev2Auth(t, fmt.Sprintf("%s/api/v1/id/%s?site=remark42&url=https://radio-t.com/blah", ts.URL, id1))
assert.Equal(t, http.StatusOK, code)
cr = store.Comment{}
err = json.Unmarshal([]byte(body), &cr)
@@ -709,11 +859,11 @@ func TestRest_EmailAndTelegram(t *testing.T) {
// issue good token
claims := token.Claims{
Handshake: &token.Handshake{ID: "dev::good@example.com"},
StandardClaims: jwt.StandardClaims{
Audience: "remark42",
ExpiresAt: time.Now().Add(10 * time.Minute).Unix(),
NotBefore: time.Now().Add(-1 * time.Minute).Unix(),
Handshake: &token.Handshake{ID: "provider1_dev::good@example.com"},
RegisteredClaims: jwt.RegisteredClaims{
Audience: jwt.ClaimStrings{"remark42"},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(10 * time.Minute)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-1 * time.Minute)),
Issuer: "remark42",
},
}
@@ -765,7 +915,6 @@ func TestRest_EmailAndTelegram(t *testing.T) {
client := http.Client{}
defer client.CloseIdleConnections()
for _, x := range testData {
x := x
t.Run(x.description, func(t *testing.T) {
reqBody := io.NopCloser(strings.NewReader(x.body))
if x.body == "" {
@@ -808,7 +957,7 @@ func TestRest_EmailNotification(t *testing.T) {
// create new comment from dev user
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
`{"text": "test 123",
"user": {"name": "dev::good@example.com"},
"user": {"name": "provider1_dev::good@example.com"},
"locator":{"url": "https://radio-t.com/blah1",
"site": "remark42"}}`))
assert.NoError(t, err)
@@ -820,7 +969,7 @@ func TestRest_EmailNotification(t *testing.T) {
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
parentComment := store.Comment{}
require.NoError(t, render.DecodeJSON(strings.NewReader(string(body)), &parentComment))
require.NoError(t, json.Unmarshal(body, &parentComment))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 1, len(mockDestination.Get()))
@@ -882,7 +1031,7 @@ func TestRest_EmailNotification(t *testing.T) {
var clearUser store.User
err = json.Unmarshal(body, &clearUser)
assert.NoError(t, err)
assert.Equal(t, store.User{Name: "developer one", ID: "dev", EmailSubscription: false,
assert.Equal(t, store.User{Name: "developer one", ID: "provider1_dev", EmailSubscription: false,
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, clearUser)
// verify email
@@ -916,7 +1065,7 @@ func TestRest_EmailNotification(t *testing.T) {
var subscribedUser store.User
err = json.Unmarshal(body, &subscribedUser)
assert.NoError(t, err)
assert.Equal(t, store.User{Name: "developer one", ID: "dev", EmailSubscription: true,
assert.Equal(t, store.User{Name: "developer one", ID: "provider1_dev", EmailSubscription: true,
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, subscribedUser)
// create child comment from another user, email notification expected
@@ -1058,7 +1207,7 @@ func TestRest_TelegramNotification(t *testing.T) {
// create new comment from dev user
req, err := http.NewRequest("POST", ts.URL+"/api/v1/comment", strings.NewReader(
`{"text": "test 123",
"user": {"name": "dev::good@example.com"},
"user": {"name": "provider1_dev::good@example.com"},
"locator":{"url": "https://radio-t.com/blah1",
"site": "remark42"}}`))
assert.NoError(t, err)
@@ -1070,7 +1219,7 @@ func TestRest_TelegramNotification(t *testing.T) {
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusCreated, resp.StatusCode, string(body))
parentComment := store.Comment{}
require.NoError(t, render.DecodeJSON(strings.NewReader(string(body)), &parentComment))
require.NoError(t, json.Unmarshal(body, &parentComment))
// wait for mock notification Submit to kick off
time.Sleep(time.Millisecond * 30)
require.Equal(t, 1, len(mockDestination.Get()))
@@ -1137,8 +1286,8 @@ func TestRest_TelegramNotification(t *testing.T) {
body, err = io.ReadAll(resp.Body)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusInternalServerError, resp.StatusCode, string(body))
require.Equal(t, `{"code":0,"details":"can't set telegram for user","error":"not verified"}`+"\n", string(body))
require.Equal(t, http.StatusNotFound, resp.StatusCode, string(body))
require.Equal(t, `{"code":0,"details":"request is not verified yet","error":"not verified"}`+"\n", string(body))
mockTlgrm.notVerified = false
@@ -1186,7 +1335,7 @@ func TestRest_TelegramNotification(t *testing.T) {
var user store.User
err = json.Unmarshal(body, &user)
assert.NoError(t, err)
assert.Equal(t, store.User{Name: "developer one", ID: "dev",
assert.Equal(t, store.User{Name: "developer one", ID: "provider1_dev",
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, user)
// create child comment from another user, telegram notification expected
@@ -1245,7 +1394,7 @@ func TestRest_UserAllData(t *testing.T) {
defer teardown()
// write 3 comments
user := store.User{ID: "dev", Name: "user name 1"}
user := store.User{ID: "provider1_dev", Name: "user name 1"}
c1 := store.Comment{User: user, Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 5, 27, 1, 14, 10, 0, time.Local)}
c2 := store.Comment{User: user, Text: "test test #2", ParentID: "p1", Locator: store.Locator{SiteID: "remark42",
@@ -1270,13 +1419,13 @@ func TestRest_UserAllData(t *testing.T) {
require.Equal(t, "application/gzip", resp.Header.Get("Content-Type"))
ungzReader, err := gzip.NewReader(resp.Body)
assert.NoError(t, err)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
ungzBody, err := io.ReadAll(ungzReader)
assert.NoError(t, err)
require.NoError(t, err)
strUungzBody := string(ungzBody)
assert.True(t, strings.HasPrefix(strUungzBody,
`{"info": {"name":"developer one","id":"dev","picture":"http://example.com/pic.png","ip":"127.0.0.1","admin":false,"site_id":"remark42"}, "comments":[{`))
`{"info": {"name":"developer one","id":"provider1_dev","picture":"http://example.com/pic.png","ip":"127.0.0.1","admin":false,"site_id":"remark42"}, "comments":[{`))
assert.Equal(t, 3, strings.Count(strUungzBody, `"text":`), "3 comments inside")
parsed := struct {
@@ -1286,7 +1435,7 @@ func TestRest_UserAllData(t *testing.T) {
err = json.Unmarshal(ungzBody, &parsed)
assert.NoError(t, err)
assert.Equal(t, store.User{Name: "developer one", ID: "dev",
assert.Equal(t, store.User{Name: "developer one", ID: "provider1_dev",
Picture: "http://example.com/pic.png", IP: "127.0.0.1", SiteID: "remark42"}, parsed.Info)
assert.Equal(t, 3, len(parsed.Comments))
@@ -1302,7 +1451,7 @@ func TestRest_UserAllDataManyComments(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
user := store.User{ID: "dev", Name: "user name 1"}
user := store.User{ID: "provider1_dev", Name: "user name 1"}
c := store.Comment{User: user, Text: "test test #1", Locator: store.Locator{SiteID: "remark42",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 5, 27, 1, 14, 10, 0, time.Local)}
@@ -1329,7 +1478,7 @@ func TestRest_UserAllDataManyComments(t *testing.T) {
assert.NoError(t, err)
strUngzBody := string(ungzBody)
assert.True(t, strings.HasPrefix(strUngzBody,
`{"info": {"name":"developer one","id":"dev","picture":"http://example.com/pic.png","ip":"127.0.0.1","admin":false,"site_id":"remark42"}, "comments":[{`))
`{"info": {"name":"developer one","id":"provider1_dev","picture":"http://example.com/pic.png","ip":"127.0.0.1","admin":false,"site_id":"remark42"}, "comments":[{`))
assert.Equal(t, 51, strings.Count(strUngzBody, `"text":`), "51 comments inside")
}
@@ -1353,12 +1502,12 @@ func TestRest_DeleteMe(t *testing.T) {
err = json.Unmarshal(body, &m)
assert.NoError(t, err)
assert.Equal(t, "remark42", m["site"])
assert.Equal(t, "dev", m["user_id"])
assert.Equal(t, "provider1_dev", m["user_id"])
tkn := m["token"]
claims, err := srv.Authenticator.TokenService().Parse(tkn)
assert.NoError(t, err)
assert.Equal(t, "dev", claims.User.ID)
assert.Equal(t, "provider1_dev", claims.User.ID)
assert.Equal(t, "https://demo.remark42.com/web/deleteme.html?token="+tkn, m["link"])
req, err = http.NewRequest(http.MethodPost, fmt.Sprintf("%s/api/v1/deleteme?site=remark42", ts.URL), http.NoBody)
+92 -20
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"crypto/sha1" // nolint
"encoding/base64"
"encoding/json"
"fmt"
"io"
"net/http"
@@ -12,10 +13,10 @@ import (
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
R "github.com/go-pkgz/rest"
"github.com/google/uuid"
"github.com/skip2/go-qrcode"
"github.com/umputun/remark42/backend/app/rest"
@@ -48,8 +49,18 @@ type pubStore interface {
Counts(siteID string, postIDs []string) ([]store.PostInfo, error)
}
// GET /find?site=siteID&url=post-url&format=[tree|plain]&sort=[+/-time|+/-score|+/-controversy]&view=[user|all]&since=unix_ts_msec
// find comments for given post. Returns in tree or plain formats, sorted
// GET /find?site=siteID&url=post-url&format=[tree|plain]&sort=[+/-time|+/-score|+/-controversy]&view=[user|all]&since=unix_ts_msec&limit=100&offset_id={id}
// find comments for given post. Returns in tree or plain formats, sorted.
//
// When `url` parameter is not set (e.g. request is for site-wide comments), does not return deleted comments.
//
// When `limit` is set, first {limit} comments are returned. When `offset_id` is set, comments are returned starting
// after the comment with the given id.
// format="tree" limits comments by top-level comments and all their replies,
// and never returns parent comment with only part of replies.
//
// `count` in the response refers to total number of non-deleted comments,
// `count_left` to amount of comments left to be returned _including deleted_.
func (s *public) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
sort := r.URL.Query().Get("sort")
@@ -68,7 +79,24 @@ func (s *public) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
since = time.Time{} // since doesn't make sense for tree
}
log.Printf("[DEBUG] get comments for %+v, sort %s, format %s, since %v", locator, sort, format, since)
limitParam := r.URL.Query().Get("limit")
var limit int
if limitParam != "" {
if limit, err = strconv.Atoi(limitParam); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "bad limit value", rest.ErrCommentNotFound)
return
}
}
offsetID := r.URL.Query().Get("offset_id")
if offsetID != "" {
if _, err = uuid.Parse(offsetID); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "bad offset_id value", rest.ErrCommentNotFound)
return
}
}
log.Printf("[DEBUG] get comments for %+v, sort %s, format %s, since %v, limit %d, offset %s", locator, sort, format, since, limit, offsetID)
key := cache.NewKey(locator.SiteID).ID(URLKeyWithUser(r)).Scopes(locator.SiteID, locator.URL)
data, err := s.cache.Get(key, func() ([]byte, error) {
@@ -77,22 +105,44 @@ func (s *public) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
comments = []store.Comment{} // error should clear comments and continue for post info
}
comments = s.applyView(comments, view)
var commentsInfo store.PostInfo
if info, ee := s.dataService.Info(locator, s.readOnlyAge); ee == nil {
commentsInfo = info
}
if !since.IsZero() { // if since is set, number of comments can be different from total in the DB
commentsInfo.Count = 0
for _, c := range comments {
if !c.Deleted {
commentsInfo.Count++
}
}
}
// post might be readonly without any comments, Info call will fail then and ReadOnly flag should be checked separately
if !commentsInfo.ReadOnly && locator.URL != "" && s.dataService.IsReadOnly(locator) {
commentsInfo.ReadOnly = true
}
var b []byte
switch format {
case "tree":
tree := service.MakeTree(comments, sort, s.readOnlyAge)
if tree.Nodes == nil { // eliminate json nil serialization
tree.Nodes = []*service.Node{}
withInfo := treeWithInfo{Tree: service.MakeTree(comments, sort, limit, offsetID), Info: commentsInfo}
withInfo.Info.CountLeft = withInfo.CountLeft()
withInfo.Info.LastComment = withInfo.LastComment()
if withInfo.Nodes == nil { // eliminate json nil serialization
withInfo.Nodes = []*service.Node{}
}
if s.dataService.IsReadOnly(locator) {
tree.Info.ReadOnly = true
}
b, e = encodeJSONWithHTML(tree)
b, e = encodeJSONWithHTML(withInfo)
default:
withInfo := commentsWithInfo{Comments: comments}
if info, ee := s.dataService.Info(locator, s.readOnlyAge); ee == nil {
withInfo.Info = info
if limit > 0 || offsetID != "" {
comments, commentsInfo.CountLeft = limitComments(comments, limit, offsetID)
}
if limit > 0 && len(comments) > 0 {
commentsInfo.LastComment = comments[len(comments)-1].ID
}
withInfo := commentsWithInfo{Comments: comments, Info: commentsInfo}
b, e = encodeJSONWithHTML(withInfo)
}
return b, e
@@ -182,7 +232,6 @@ func (s *public) commentByIDCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get comment by id", rest.ErrCommentNotFound)
return
}
render.Status(r, http.StatusOK)
if err = R.RenderJSONWithHTML(w, r, comment); err != nil {
log.Printf("[WARN] can't render last comments for url=%s, id=%s", url, id)
@@ -247,7 +296,7 @@ func (s *public) countCtrl(w http.ResponseWriter, r *http.Request) {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get count", rest.ErrPostNotFound)
return
}
render.JSON(w, r, R.JSON{"count": count, "locator": locator})
R.RenderJSON(w, R.JSON{"count": count, "locator": locator})
}
// POST /counts?site=siteID - get number of comments for posts from post body
@@ -255,7 +304,7 @@ func (s *public) countMultiCtrl(w http.ResponseWriter, r *http.Request) {
const countBodyLimit int64 = 1024 * 128 // count request can be big for some site because it lists all urls
siteID := r.URL.Query().Get("site")
posts := []string{}
if err := render.DecodeJSON(http.MaxBytesReader(w, r.Body, countBodyLimit), &posts); err != nil {
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, countBodyLimit)).Decode(&posts); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get list of posts from request", rest.ErrSiteNotFound)
return
}
@@ -343,13 +392,14 @@ func (s *public) loadPictureCtrl(w http.ResponseWriter, r *http.Request) {
}
// GET /robots.txt
func (s *public) robotsCtrl(w http.ResponseWriter, r *http.Request) {
func (s *public) robotsCtrl(w http.ResponseWriter, _ *http.Request) {
allowed := []string{"/find", "/last", "/id", "/count", "/counts", "/list", "/config", "/user",
"/img", "/avatar", "/picture"}
for i := range allowed {
allowed[i] = "Allow: /api/v1" + allowed[i]
}
render.PlainText(w, r, "User-agent: *\nDisallow: /auth/\nDisallow: /api/\n"+strings.Join(allowed, "\n")+"\n")
responseText := fmt.Sprintf("User-agent: *\nDisallow: /auth/\nDisallow: /api/\n%s\n", strings.Join(allowed, "\n"))
rest.PlainTextResponse(w, http.StatusOK, responseText)
}
// GET /qr/telegram - generates QR for provided URL, used for Telegram auth and notifications subscription. The first
@@ -416,3 +466,25 @@ func (s *public) parseSince(r *http.Request) (time.Time, error) {
}
return sinceTS, nil
}
// limitComments returns limited list of comments and count of comments left after limit.
// If offsetID is provided, the list will be sliced starting from the comment with this ID.
// If offsetID is not found, the full list will be returned.
// It's used for only "
func limitComments(c []store.Comment, limit int, offsetID string) (comments []store.Comment, countLeft int) {
if offsetID != "" {
for i, comment := range c {
if comment.ID == offsetID {
c = c[i+1:]
break
}
}
}
if limit > 0 && len(c) > limit {
countLeft = len(c) - limit
c = c[:limit]
}
return c, countLeft
}
+327 -24
View File
@@ -5,13 +5,16 @@ import (
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"strconv"
"strings"
"testing"
"time"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -76,13 +79,31 @@ func TestRest_Preview(t *testing.T) {
assert.NoError(t, resp.Body.Close())
assert.Contains(t,
string(b),
"{\"code\":20,\"details\":\"can't renew staged picture cleanup timer\","+
"\"error\":\"can't get image stats for dev_user/bad_picture: stat",
`{"code":20,"details":"can't load picture from the comment",`+
`"error":"can't get image stats for dev_user/bad_picture: stat`,
)
assert.Contains(t,
string(b),
"/pics-remark42/staging/dev_user/62/bad_picture: no such file or directory\"}\n",
)
// test quotes with and without smartypants
resp, err = post(t, ts.URL+"/api/v1/preview", `{"text": "\"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, "<p>«quoted» text</p>\n", string(b))
srv.privRest.disableFancyTextFormatting = true
resp, err = post(t, ts.URL+"/api/v1/preview", `{"text": "\"quoted\" text", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err = io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, "<p>&#34;quoted&#34; text</p>\n", string(b))
}
func TestRest_PreviewWithWrongImage(t *testing.T) {
@@ -97,8 +118,8 @@ func TestRest_PreviewWithWrongImage(t *testing.T) {
assert.NoError(t, resp.Body.Close())
assert.Contains(t,
string(b),
"{\"code\":20,\"details\":\"can't renew staged picture cleanup timer\","+
"\"error\":\"can't get image stats for dev_user/bad_picture: stat ",
`{"code":20,"details":"can't load picture from the comment",`+
`"error":"can't get image stats for dev_user/bad_picture: stat `,
)
assert.Contains(t,
string(b),
@@ -120,9 +141,9 @@ srv, ts := prep(t)
}
BKT
`
text = strings.Replace(text, "BKT", "```", -1)
text = strings.ReplaceAll(text, "BKT", "```")
j := fmt.Sprintf(`{"text": %q, "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, text)
j = strings.Replace(j, "\n", "\\n", -1)
j = strings.ReplaceAll(j, "\n", "\\n")
resp, err := post(t, ts.URL+"/api/v1/preview", j)
assert.NoError(t, err)
@@ -131,10 +152,10 @@ BKT
assert.NoError(t, err)
assert.Equal(t,
`<h1>h1</h1>
<pre class="chroma"><code><span class="line"><span class="cl">func TestRest_Preview(t *testing.T) {
</span></span><span class="line"><span class="cl">srv, ts := prep(t)
</span></span><span class="line"><span class="cl"> require.NotNil(t, srv)
</span></span><span class="line"><span class="cl">}
<pre class="chroma"><code><span class="line"><span class="cl"><span class="k">func</span> <span class="n">TestRest_Preview</span><span class="p">(</span><span class="n">t</span> <span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"><span class="n">srv</span><span class="p">,</span> <span class="n">ts</span> <span class="p">:</span><span class="o">=</span> <span class="n">prep</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"> <span class="n">require</span><span class="o">.</span><span class="n">NotNil</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="n">srv</span><span class="p">)</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre>`,
string(b))
assert.NoError(t, resp.Body.Close())
@@ -148,17 +169,17 @@ func TestRest_PreviewCode(t *testing.T) {
func main(aa string) int {return 0}
BKT
`
text = strings.Replace(text, "BKT", "```", -1)
text = strings.ReplaceAll(text, "BKT", "```")
j := fmt.Sprintf(`{"text": %q, "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`, text)
j = strings.Replace(j, "\n", "\\n", -1)
j = strings.ReplaceAll(j, "\n", "\\n")
resp, err := post(t, ts.URL+"/api/v1/preview", j)
assert.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, `<pre class="chroma"><code><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">(</span><span class="nx">aa</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span><span class="k">return</span> <span class="mi">0</span><span class="p">}</span>
</span></span></code></pre>`, string(b))
assert.Equal(t, `<pre class="chroma"><code><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="nx">aa</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="p">{</span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">}</span><span class="w">
</span></span></span></code></pre>`, string(b))
assert.NoError(t, resp.Body.Close())
}
@@ -209,7 +230,7 @@ func TestRest_Find(t *testing.T) {
assert.Equal(t, id2, comments.Comments[0].ID)
// get in tree mode
tree := service.Tree{}
tree := treeWithInfo{}
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, http.StatusOK, code)
err = json.Unmarshal([]byte(res), &tree)
@@ -235,7 +256,7 @@ func TestRest_FindAge(t *testing.T) {
_, err = srv.DataService.Create(c2)
require.NoError(t, err)
tree := service.Tree{}
tree := treeWithInfo{}
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, http.StatusOK, code)
@@ -278,7 +299,7 @@ func TestRest_FindReadOnly(t *testing.T) {
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
tree := service.Tree{}
tree := treeWithInfo{}
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, http.StatusOK, code)
err = json.Unmarshal([]byte(res), &tree)
@@ -286,7 +307,7 @@ func TestRest_FindReadOnly(t *testing.T) {
assert.Equal(t, "https://radio-t.com/blah1", tree.Info.URL)
assert.True(t, tree.Info.ReadOnly, "post is ro")
tree = service.Tree{}
tree = treeWithInfo{}
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah2&format=tree")
assert.Equal(t, http.StatusOK, code)
err = json.Unmarshal([]byte(res), &tree)
@@ -325,8 +346,8 @@ func TestRest_FindUserView(t *testing.T) {
require.Equal(t, 2, len(comments.Comments), "should have 2 comments")
assert.Equal(t, id1, comments.Comments[0].ID)
assert.Equal(t, id2, comments.Comments[1].ID)
assert.Equal(t, "dev", comments.Comments[0].User.ID)
assert.Equal(t, "dev", comments.Comments[1].User.ID)
assert.Equal(t, "provider1_dev", comments.Comments[0].User.ID)
assert.Equal(t, "provider1_dev", comments.Comments[1].User.ID)
assert.Equal(t, "", comments.Comments[0].Text)
assert.Equal(t, "", comments.Comments[1].Text)
@@ -440,7 +461,7 @@ func TestRest_FindUserComments(t *testing.T) {
assert.Equal(t, http.StatusOK, code, "noting for user blah")
assert.Equal(t, `{"comments":[],"count":0}`+"\n", comments)
{
res, code := get(t, ts.URL+"/api/v1/comments?site=remark42&user=dev")
res, code := get(t, ts.URL+"/api/v1/comments?site=remark42&user=provider1_dev")
assert.Equal(t, http.StatusOK, code)
resp := struct {
@@ -459,7 +480,7 @@ func TestRest_FindUserComments(t *testing.T) {
}
{
res, code := get(t, ts.URL+"/api/v1/comments?site=remark42&user=dev&skip=1&limit=2")
res, code := get(t, ts.URL+"/api/v1/comments?site=remark42&user=provider1_dev&skip=1&limit=2")
assert.Equal(t, http.StatusOK, code)
resp := struct {
@@ -477,6 +498,288 @@ func TestRest_FindUserComments(t *testing.T) {
}
}
func TestRest_FindUserComments_CWE_918(t *testing.T) {
ts, srv, teardown := startupT(t)
srv.DataService.TitleExtractor = service.NewTitleExtractor(http.Client{Timeout: time.Second}, []string{"radio-t.com"}) // required for extracting the title, bad URL test
defer srv.DataService.TitleExtractor.Close()
defer teardown()
backendRequestedArbitraryServer := false
arbitraryServer := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
t.Logf("request received: %+v", r)
backendRequestedArbitraryServer = true
}))
defer arbitraryServer.Close()
arbitraryURLComment := store.Comment{Text: "arbitrary URL request test",
Locator: store.Locator{SiteID: "remark42", URL: arbitraryServer.URL}}
assert.False(t, backendRequestedArbitraryServer)
addComment(t, arbitraryURLComment, ts)
assert.False(t, backendRequestedArbitraryServer,
"no request is expected to the test server as it's not in the list of the allowed domains for the title extractor")
res, code := get(t, ts.URL+"/api/v1/comments?site=remark42&user=provider1_dev")
assert.Equal(t, http.StatusOK, code)
resp := struct {
Comments []store.Comment
Count int
}{}
err := json.Unmarshal([]byte(res), &resp)
assert.NoError(t, err)
require.Equal(t, 1, len(resp.Comments), "should have 2 comments")
assert.Equal(t, "", resp.Comments[0].PostTitle, "empty from the first post")
assert.Equal(t, arbitraryServer.URL, resp.Comments[0].Locator.URL, "arbitrary URL provided by the request")
}
func TestPublic_FindCommentsCtrl_ConsistentCount(t *testing.T) {
// test that comment counting is consistent between tree and plain formats
ts, srv, teardown := startupT(t)
defer teardown()
commentLocator := store.Locator{URL: "test-url", SiteID: "remark42"}
// vote for comment multiple times
setScore := func(locator store.Locator, id string, val int) {
abs := func(x int) int {
if x < 0 {
return -x
}
return x
}
for i := 0; i < abs(val); i++ {
_, err := srv.DataService.Vote(service.VoteReq{
Locator: locator,
CommentID: id,
// unique user ID is needed for correct counting of controversial votes
UserID: "user" + strconv.Itoa(val) + strconv.Itoa(i),
Val: val > 0,
})
require.NoError(t, err)
}
}
// Adding initial comments (8 to test-url and 1 to another-url) and voting, and delete two of comments to the first post.
// With sleep so that at least few millisecond pass between each comment
// and later we would be able to use that in "since" filter with millisecond precision
ids := make([]string, 9)
timestamps := make([]time.Time, 9)
c1 := store.Comment{Text: "top-level comment 1", Locator: commentLocator}
ids[0], timestamps[0] = addCommentGetCreatedTime(t, c1, ts)
// #3 by score
setScore(commentLocator, ids[0], 1)
time.Sleep(time.Millisecond * 5)
c2 := store.Comment{Text: "top-level comment 2", Locator: commentLocator}
ids[1], timestamps[1] = addCommentGetCreatedTime(t, c2, ts)
// #2 by score
setScore(commentLocator, ids[1], 2)
time.Sleep(time.Millisecond * 5)
c3 := store.Comment{Text: "second-level comment 1", ParentID: ids[0], Locator: commentLocator}
ids[2], timestamps[2] = addCommentGetCreatedTime(t, c3, ts)
// #1 by score
setScore(commentLocator, ids[2], 10)
time.Sleep(time.Millisecond * 5)
c4 := store.Comment{Text: "third-level comment 1", ParentID: ids[2], Locator: commentLocator}
ids[3], timestamps[3] = addCommentGetCreatedTime(t, c4, ts)
// #5 by score, #1 by controversy
setScore(commentLocator, ids[3], 4)
setScore(commentLocator, ids[3], -4)
time.Sleep(time.Millisecond * 5)
c5 := store.Comment{Text: "second-level comment 2", ParentID: ids[1], Locator: commentLocator}
ids[4], timestamps[4] = addCommentGetCreatedTime(t, c5, ts)
// #5 by score, #2 by controversy
setScore(commentLocator, ids[4], 2)
setScore(commentLocator, ids[4], -3)
time.Sleep(time.Millisecond * 5)
c6 := store.Comment{Text: "deleted third-level comment 2", ParentID: ids[4], Locator: commentLocator}
ids[5], timestamps[5] = addCommentGetCreatedTime(t, c6, ts)
// deleted later so not visible in site-wide requests
setScore(commentLocator, ids[5], 10)
setScore(commentLocator, ids[5], -10)
time.Sleep(time.Millisecond * 5)
c7 := store.Comment{Text: "top-level comment 3", Locator: commentLocator}
ids[6], timestamps[6] = addCommentGetCreatedTime(t, c7, ts)
// #6 by score, #4 by controversy
setScore(commentLocator, ids[6], -3)
setScore(commentLocator, ids[6], 1)
time.Sleep(time.Millisecond * 5)
c8 := store.Comment{Text: "deleted second-level comment 3", ParentID: ids[6], Locator: commentLocator}
ids[7], timestamps[7] = addCommentGetCreatedTime(t, c8, ts)
// deleted later so not visible in site-wide requests
setScore(commentLocator, ids[7], -20)
c9 := store.Comment{Text: "comment to post 2", Locator: store.Locator{URL: "another-url", SiteID: "remark42"}}
ids[8], timestamps[8] = addCommentGetCreatedTime(t, c9, ts)
// #7 by score
setScore(store.Locator{URL: "another-url", SiteID: "remark42"}, ids[8], -25)
// delete two comments bringing the total from 9 to 6
err := srv.DataService.Delete(commentLocator, ids[7], store.SoftDelete)
assert.NoError(t, err)
err = srv.DataService.Delete(commentLocator, ids[5], store.HardDelete)
assert.NoError(t, err)
srv.Cache.Flush(cache.FlusherRequest{})
commentLocator.URL = "readonly-test"
// set post without comments to read-only
assert.NoError(t, srv.DataService.SetReadOnly(commentLocator, true))
sinceTenSecondsAgo := strconv.FormatInt(time.Now().Add(-time.Second*10).UnixNano()/1000000, 10)
sinceTS := make([]string, 9)
formattedTS := make([]string, 9)
for i, created := range timestamps {
sinceTS[i] = strconv.FormatInt(created.UnixNano()/1000000, 10)
formattedTS[i] = created.Format(time.RFC3339Nano)
}
t.Logf("last timestamp: %v", timestamps[7])
testCases := []struct {
params string
expectedBody string
}{
// test parameters url, format, since, sort
{"", fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url", fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"format=plain", fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"format=plain&url=test-url", fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTenSecondsAgo, fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTenSecondsAgo, fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTS[0], fmt.Sprintf(`"info":{"count":7,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTS[0], fmt.Sprintf(`"info":{"url":"test-url","count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTS[1], fmt.Sprintf(`"info":{"count":6,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTS[1], fmt.Sprintf(`"info":{"url":"test-url","count":5,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"since=" + sinceTS[4], fmt.Sprintf(`"info":{"count":3,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[8])},
{"url=test-url&since=" + sinceTS[4], fmt.Sprintf(`"info":{"url":"test-url","count":2,"count_left":0,"first_time":%q,"last_time":%q}`, formattedTS[0], formattedTS[7])},
{"format=tree", `"info":{"count":7`},
{"format=tree&url=test-url", `"info":{"url":"test-url","count":6`},
{"format=tree&sort=+time", `"info":{"count":7`},
{"format=tree&url=test-url&sort=+time", `"info":{"url":"test-url","count":6`},
{"format=tree&sort=-score", `"info":{"count":7`},
{"format=tree&url=test-url&sort=-score", `"info":{"url":"test-url","count":6`},
{"sort=+time", fmt.Sprintf(`"score":-25,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[8])},
{"sort=-time", fmt.Sprintf(`"score":1,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[0])},
{"sort=+score", fmt.Sprintf(`"score":10,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[2])},
{"sort=+score&url=test-url", fmt.Sprintf(`"score":10,"vote":0,"time":%q}],"info":{"url":"test-url","count":6`, formattedTS[2])},
{"sort=-score", fmt.Sprintf(`"score":-25,"vote":0,"time":%q}],"info":{"count":7`, formattedTS[8])},
{"sort=-score&url=test-url", fmt.Sprintf(`"score":-2,"vote":0,"controversy":1.5874010519681994,"time":%q}],"info":{"url":"test-url","count":6`, formattedTS[6])},
{"sort=-time&since=" + sinceTS[4], fmt.Sprintf(`"score":-1,"vote":0,"controversy":2.924017738212866,"time":%q}],"info":{"count":3`, formattedTS[4])},
{"sort=-score&since=" + sinceTS[3], fmt.Sprintf(`"score":-25,"vote":0,"time":%q}],"info":{"count":4`, formattedTS[8])},
{"sort=-score&url=test-url&since=" + sinceTS[3], fmt.Sprintf(`"score":-2,"vote":0,"controversy":1.5874010519681994,"time":%q}],"info":{"url":"test-url","count":3`, formattedTS[6])},
{"sort=+controversy&url=test-url&since=" + sinceTS[5], fmt.Sprintf(`"score":-2,"vote":0,"controversy":1.5874010519681994,"time":%q}],"info":{"url":"test-url","count":1`, formattedTS[6])},
// three comments of which last one deleted and doesn't have controversy so returned last
{"sort=-controversy&url=test-url&since=" + sinceTS[5], fmt.Sprintf(`"score":0,"vote":0,"time":%q,"delete":true}],"info":{"url":"test-url","count":1`, formattedTS[7])},
// test readonly status for the post without comments
{"url=readonly-test", `"info":{"count":0,"count_left":0,"read_only":true`},
{"format=tree&url=readonly-test", `"info":{"count":0,"count_left":0,"read_only":true`},
// test parameters limit, offset_id for format=plain
{"limit=bad", `{"code":1,"details":"bad limit value","error":"strconv.Atoi: parsing \"bad\": invalid syntax"}`},
{"offset_id=bad", `{"code":1,"details":"bad offset_id value","error":"invalid UUID length: 3"}`},
{"limit=2", `"info":{"count":7,"count_left":5,"last_comment":"` + ids[1]},
{"limit=6", `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{"limit=7", `"info":{"count":7,"count_left":0,"last_comment":"` + ids[8]},
{"limit=2&url=test-url", `"info":{"url":"test-url","count":6,"count_left":6,"last_comment":"` + ids[1]},
{"limit=6&url=test-url", `"info":{"url":"test-url","count":6,"count_left":2,"last_comment":"` + ids[5]},
{"limit=7&url=test-url", `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("limit=2&offset_id=%s", ids[2]), `"info":{"count":7,"count_left":2,"last_comment":"` + ids[4]},
{fmt.Sprintf("limit=2&offset_id=%s", ids[3]), `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("limit=2&offset_id=%s", ids[4]), `"info":{"count":7,"count_left":0`},
{fmt.Sprintf("limit=1&offset_id=%s", ids[6]), `"info":{"count":7,"count_left":0`},
{fmt.Sprintf("limit=2&offset_id=%s", ids[8]), `"info":{"count":7,"count_left":0`},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[2]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[4]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[3]), `"info":{"url":"test-url","count":6,"count_left":2,"last_comment":"` + ids[5]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[4]), `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("limit=1&url=test-url&offset_id=%s", ids[6]), `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[7]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[8]), `"info":{"url":"test-url","count":6,"count_left":6,`},
// deleted comment, offset is ignored in site-wide request but not for particular URL
{fmt.Sprintf("limit=2&offset_id=%s", ids[5]), `"info":{"count":7,"count_left":5,"last_comment":"` + ids[1]},
{fmt.Sprintf("limit=2&url=test-url&offset_id=%s", ids[5]), `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[7]},
// non-existing comment, offset is ignored, deleted comments included into request with "url"
{fmt.Sprintf("limit=1&offset_id=%s", uuid.New().String()), `"info":{"count":7,"count_left":6,"last_comment":"` + ids[0]},
{fmt.Sprintf("limit=1&url=test-url&offset_id=%s", uuid.New().String()), `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[0]},
// since is ignored for tree format, so we test it only for plain
{"limit=6&since=" + sinceTenSecondsAgo, `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{"limit=1&since=" + sinceTS[4], `"info":{"count":3,"count_left":2,"last_comment":"` + ids[4]},
{"limit=6&url=test-url&since=" + sinceTenSecondsAgo, `"info":{"url":"test-url","count":6,"count_left":2,"last_comment":"` + ids[5]},
{"limit=1&url=test-url&since=" + sinceTS[4], `"info":{"url":"test-url","count":2,"count_left":3,"last_comment":"` + ids[4]},
// start with deleted comment timestamp
{"limit=1&since=" + sinceTS[5], `"info":{"count":2,"count_left":1,"last_comment":"` + ids[6]},
{"limit=1&since=" + sinceTS[6], `"info":{"count":2,"count_left":1,"last_comment":"` + ids[6]},
{"limit=1&url=test-url&since=" + sinceTS[5], `"info":{"url":"test-url","count":1,"count_left":2,"last_comment":"` + ids[5]},
{"limit=1&url=test-url&since=" + sinceTS[6], `"info":{"url":"test-url","count":1,"count_left":1,"last_comment":"` + ids[6]},
// test sort
{"limit=1&sort=+time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[0]},
{"limit=1&sort=-time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[7]},
{"limit=1&sort=+score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[6]},
{"limit=1&sort=-score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[2]},
{"limit=1&sort=+controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[0]},
{"limit=1&sort=-controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":7,"last_comment":"` + ids[3]},
// test parameters limit, offset_id for format=tree
{"format=tree&limit=bad", `{"code":1,"details":"bad limit value","error":"strconv.Atoi: parsing \"bad\": invalid syntax"}`},
{"format=tree&offset_id=bad", `{"code":1,"details":"bad offset_id value","error":"invalid UUID length: 3"}`},
{"format=tree&limit=2", `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{"format=tree&limit=6", `"info":{"count":7,"count_left":2,"last_comment":"` + ids[1]},
{"format=tree&limit=7", `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{"format=tree&url=test-url&limit=2", `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{"format=tree&url=test-url&limit=6", `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[1]},
{"format=tree&url=test-url&limit=7", `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[6]},
// start after first top-level comment
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[0]), `"info":{"count":7,"count_left":2,"last_comment":"` + ids[1]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[0]), `"info":{"url":"test-url","count":6,"count_left":1,"last_comment":"` + ids[1]},
// start after second top-level comment
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[1]), `"info":{"count":7,"count_left":1,"last_comment":"` + ids[6]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[1]), `"info":{"url":"test-url","count":6,"count_left":0,"last_comment":"` + ids[6]},
// start after third top-level comment, so expect comment to post 2, or no comments on post 1 if "url" is set
{fmt.Sprintf("format=tree&limit=1&offset_id=%s", ids[6]), `"info":{"count":7,"count_left":0,"last_comment":"` + ids[8]},
{fmt.Sprintf("format=tree&url=test-url&limit=1&offset_id=%s", ids[6]), `"info":{"url":"test-url","count":6,"count_left":0`},
// non-root comment IDs or non-existing IDs are ignored
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[2]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[3]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[4]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=2&offset_id=%s", ids[7]), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&limit=1&offset_id=%s", uuid.New().String()), `"info":{"count":7,"count_left":4,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[2]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[3]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[4]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=2&offset_id=%s", ids[7]), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{fmt.Sprintf("format=tree&url=test-url&limit=1&offset_id=%s", uuid.New().String()), `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
// test sort
{"format=tree&limit=1&sort=+time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{"format=tree&limit=1&sort=-time&url=test-url", `"info":{"url":"test-url","count":6,"count_left":5,"last_comment":"` + ids[6]},
{"format=tree&limit=1&sort=+score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":5,"last_comment":"` + ids[6]},
{"format=tree&limit=1&sort=-score&url=test-url", `"info":{"url":"test-url","count":6,"count_left":4,"last_comment":"` + ids[1]},
{"format=tree&limit=1&sort=+controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":3,"last_comment":"` + ids[0]},
{"format=tree&limit=1&sort=-controversy&url=test-url", `"info":{"url":"test-url","count":6,"count_left":5,"last_comment":"` + ids[6]},
}
for _, tc := range testCases {
t.Run(tc.params, func(t *testing.T) {
url := fmt.Sprintf(ts.URL+"/api/v1/find?site=remark42&%s", tc.params)
body, code := get(t, url)
expectedStatus := http.StatusOK
if strings.Contains(tc.params, "=bad") {
expectedStatus = http.StatusBadRequest
}
assert.Equal(t, expectedStatus, code)
assert.Contains(t, body, tc.expectedBody)
t.Log(body)
// prevent hit limiter from engaging
time.Sleep(80 * time.Millisecond)
})
}
}
func TestRest_UserInfo(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
@@ -486,7 +789,7 @@ func TestRest_UserInfo(t *testing.T) {
user := store.User{}
err := json.Unmarshal([]byte(body), &user)
assert.NoError(t, err)
assert.Equal(t, store.User{Name: "developer one", ID: "dev", Picture: "http://example.com/pic.png",
assert.Equal(t, store.User{Name: "developer one", ID: "provider1_dev", Picture: "http://example.com/pic.png",
IP: "127.0.0.1", SiteID: "remark42"}, user)
}
+106 -49
View File
@@ -16,10 +16,11 @@ import (
"testing"
"time"
"github.com/go-pkgz/auth"
"github.com/go-pkgz/auth/avatar"
"github.com/go-pkgz/auth/token"
cache "github.com/go-pkgz/lcw"
"github.com/go-pkgz/auth/v2"
"github.com/go-pkgz/auth/v2/avatar"
"github.com/go-pkgz/auth/v2/provider"
"github.com/go-pkgz/auth/v2/token"
cache "github.com/go-pkgz/lcw/v2"
R "github.com/go-pkgz/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -40,13 +41,15 @@ import (
// To generate a token, enter one of the tokens here into https://jwt.io, change the secret to one you're using in your test
// ("secret" in case of startupT), and alter the fields you want to be changed.
var devToken = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0MiIsImV4cCI6Mzc4OTE5MTgyMiwianRpIjoicmFuZG9tIGlkIiwiaXNzIjoicmVtYXJrNDIiLCJuYmYiOjE1MjE4ODQyMjIsInVzZXIiOnsibmFtZSI6ImRldmVsb3BlciBvbmUiLCJpZCI6ImRldiIsInBpY3R1cmUiOiJodHRwOi8vZXhhbXBsZS5jb20vcGljLnBuZyIsImlwIjoiMTI3LjAuMC4xIiwiZW1haWwiOiJtZUBleGFtcGxlLmNvbSJ9fQ.aKUAXiZxXypgV7m1wEOgUcyPOvUDXHDi3A06YWKbcLg`
var devToken = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0MiIsImV4cCI6Mzc4OTE5MTgyMiwianRpIjoicmFuZG9tIGlkIiwiaXNzIjoicmVtYXJrNDIiLCJuYmYiOjE1MjE4ODQyMjIsInVzZXIiOnsibmFtZSI6ImRldmVsb3BlciBvbmUiLCJpZCI6InByb3ZpZGVyMV9kZXYiLCJwaWN0dXJlIjoiaHR0cDovL2V4YW1wbGUuY29tL3BpYy5wbmciLCJpcCI6IjEyNy4wLjAuMSIsImVtYWlsIjoibWVAZXhhbXBsZS5jb20ifX0.dirTS_ahSF6375sdO2iodm2K2UmRTzQNQMFiHuTQCVs`
var dev2Token = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0MiIsImV4cCI6Mzc4OTE5MTgyMiwianRpIjoicmFuZG9tIGlkIiwiaXNzIjoicmVtYXJrNDIiLCJuYmYiOjE1MjE4ODQyMjIsInVzZXIiOnsibmFtZSI6ImRldmVsb3BlciBvbmUiLCJpZCI6InByb3ZpZGVyMV9kZXYyIiwicGljdHVyZSI6Imh0dHA6Ly9leGFtcGxlLmNvbS9waWMucG5nIiwiaXAiOiIxMjcuMC4wLjEiLCJlbWFpbCI6Im1lQGV4YW1wbGUuY29tIn19.qsR_PupfjIq7uw0eAuyGV8nsUoMx9v541c9olnRInRQ`
var anonToken = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0MiIsImV4cCI6Mzc4OTE5MTgyMiwianRpIjoicmFuZG9tIGlkIiwiaXNzIjoicmVtYXJrNDIiLCJuYmYiOjE1MjE4ODQyMjIsInVzZXIiOnsibmFtZSI6ImFub255bW91cyB0ZXN0IHVzZXIiLCJpZCI6ImFub255bW91c190ZXN0X3VzZXIiLCJwaWN0dXJlIjoiaHR0cDovL2V4YW1wbGUuY29tL3BpYy5wbmciLCJpcCI6IjEyNy4wLjAuMSIsImVtYWlsIjoiYW5vbkBleGFtcGxlLmNvbSJ9fQ.gAae2WMxZNZE5ebVboptPEyQ7Nk6EQxciNnGJ_mPOuU`
var emailUserToken = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0MiIsImV4cCI6Mzc4OTE5MTgyMiwianRpIjoicmFuZG9tIGlkIiwiaXNzIjoicmVtYXJrNDIiLCJuYmYiOjE1MjE4ODQyMjIsInVzZXIiOnsibmFtZSI6Imdvb2RAZXhhbXBsZS5jb20gdGVzdCB1c2VyIiwiaWQiOiJlbWFpbF9mNWRmZTlkMmU2YmQ3NWZjNzRlYTVmYWJmMjczYjQ1YjViYWViMTk1IiwicGljdHVyZSI6Imh0dHA6Ly9leGFtcGxlLmNvbS9waWMucG5nIiwiaXAiOiIxMjcuMC4wLjEiLCJlbWFpbCI6Imdvb2RAZXhhbXBsZS5jb20ifX0.vH2HN1JpuXL8okTJq1A-zGHQ-l2ILcwxvDDEmu2zwks`
var devTokenBadAud = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0Ml9iYWQiLCJleHAiOjM3ODkxOTE4MjIsImp0aSI6InJhbmRvbSBpZCIsImlzcyI6InJlbWFyazQyIiwibmJmIjoxNTIxODg0MjIyLCJ1c2VyIjp7Im5hbWUiOiJkZXZlbG9wZXIgb25lIiwiaWQiOiJkZXYiLCJwaWN0dXJlIjoiaHR0cDovL2V4YW1wbGUuY29tL3BpYy5wbmciLCJpcCI6IjEyNy4wLjAuMSIsImVtYWlsIjoibWVAZXhhbXBsZS5jb20ifX0.FuTTocVtcxr4VjpfIICvU2yOb3su28VkDzj94H9Q3xY`
var devTokenBadAud = `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZW1hcms0Ml9iYWQiLCJleHAiOjM3ODkxOTE4MjIsImp0aSI6InJhbmRvbSBpZCIsImlzcyI6InJlbWFyazQyIiwibmJmIjoxNTIxODg0MjIyLCJ1c2VyIjp7Im5hbWUiOiJkZXZlbG9wZXIgb25lIiwiaWQiOiJwcm92aWRlcjFfZGV2IiwicGljdHVyZSI6Imh0dHA6Ly9leGFtcGxlLmNvbS9waWMucG5nIiwiaXAiOiIxMjcuMC4wLjEiLCJlbWFpbCI6Im1lQGV4YW1wbGUuY29tIn19.X-lvnHvBz6VfEbVV4f-bjcZuLY5pYtvEansk_TQMrX8`
var adminUmputunToken = `eyJhbGciOiJIUzI1NiJ9.eyJhdWQiOiJyZW1hcms0MiIsImV4cCI6MTk1NDU5Nzk4MCwianRpIjoiOTdhMmUwYWM0ZGM3ZDVmNjkyNmQ1ZTg2MjBhY2VmOWE0MGMwIiwiaWF0IjoxNDU0NTk3NjgwLCJpc3MiOiJyZW1hcms0MiIsInVzZXIiOnsibmFtZSI6IlVtcHV0dW4iLCJpZCI6ImdpdGh1Yl9lZjBmNzA2YTciLCJwaWN0dXJlIjoiaHR0cHM6Ly9yZW1hcms0Mi5yYWRpby10LmNvbS9hcGkvdjEvYXZhdGFyL2NiNDJmZjQ5M2FkZTY5NmQ4OGEzYTU5MGYxMzZhZTllMzRkZTdjMWIuaW1hZ2UiLCJhdHRycyI6eyJhZG1pbiI6dHJ1ZSwiYmxvY2tlZCI6ZmFsc2V9fX0.dZiOjWHguo9f42XCMooMcv4EmYFzifl_-LEvPZHCtks`
@@ -125,7 +128,7 @@ func TestRest_RunStaticSSLMode(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
@@ -175,7 +178,7 @@ func TestRest_RunAutocertModeHTTPOnly(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
}
@@ -191,7 +194,7 @@ func TestRest_RunAutocertModeHTTPOnly(t *testing.T) {
}
func TestRest_rejectAnonUser(t *testing.T) {
ts := httptest.NewServer(fakeAuth(rejectAnonUser(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ts := httptest.NewServer(fakeAuth(rejectAnonUser(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprintln(w, "Hello")
}))))
defer ts.Close()
@@ -224,7 +227,6 @@ func Test_URLKey(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
r, err := http.NewRequest("GET", tt.url, http.NoBody)
require.NoError(t, err)
@@ -249,7 +251,6 @@ func Test_URLKeyWithUser(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
r, err := http.NewRequest("GET", tt.url, http.NoBody)
require.NoError(t, err)
@@ -276,7 +277,6 @@ func TestRest_parseError(t *testing.T) {
}
for n, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(n), func(t *testing.T) {
res := parseError(tt.err, rest.ErrInternal)
assert.Equal(t, tt.res, res)
@@ -299,12 +299,11 @@ func TestRest_cacheControl(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", tt.url, nil)
req := httptest.NewRequest("GET", tt.url, http.NoBody)
w := httptest.NewRecorder()
h := cacheControl(tt.exp, tt.version)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h := cacheControl(tt.exp, tt.version)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
@@ -317,30 +316,54 @@ func TestRest_cacheControl(t *testing.T) {
}
func TestRest_frameAncestors(t *testing.T) {
tbl := []struct {
hosts []string
header string
}{
{[]string{"http://example.com"}, "frame-ancestors http://example.com;"},
{[]string{}, ""},
{[]string{"http://example.com", "http://example2.com"}, "frame-ancestors http://example.com http://example2.com;"},
}
ts, _, teardown := startupT(t, func(o *Rest) {
o.AllowedAncestors = []string{"'self'", "https://example.com"}
})
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", nil)
w := httptest.NewRecorder()
// Test case with frame-ancestors
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "frame-ancestors 'self' https://example.com;")
teardown()
h := frameAncestors(tt.hosts)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
t.Logf("%+v", resp.Header)
assert.Equal(t, tt.header, resp.Header.Get("Content-Security-Policy"))
})
}
// Test case without frame-ancestors
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.AllowedAncestors = []string{}
})
defer teardown()
resp, err = client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "frame-ancestors *;")
}
// check CSP, img-src should be 'self' with proxy enabled and * without it
func TestRest_securityHeaders(t *testing.T) {
ts, _, teardown := startupT(t)
// with proxy disabled
client := http.Client{}
resp, err := client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src *;")
teardown()
// check CSP with proxy enabled
ts, _, teardown = startupT(t, func(srv *Rest) {
srv.ExternalImageProxy = true
})
defer teardown()
resp, err = client.Get(ts.URL + "/web/index.html")
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Contains(t, resp.Header.Get("Content-Security-Policy"), "img-src 'self';")
}
func TestRest_subscribersOnly(t *testing.T) {
@@ -361,14 +384,13 @@ func TestRest_subscribersOnly(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com", nil)
req := httptest.NewRequest("GET", "http://example.com", http.NoBody)
if tt.setUser {
req = token.SetUserInfo(req, tt.user)
}
w := httptest.NewRecorder()
h := subscribersOnly(tt.subsOnly)(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h := subscribersOnly(tt.subsOnly)(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
@@ -400,7 +422,7 @@ func Test_validEmailAuth(t *testing.T) {
t.Run(strconv.Itoa(i), func(t *testing.T) {
req := httptest.NewRequest("GET", "http://example.com"+tt.req, http.NoBody)
w := httptest.NewRecorder()
h := validEmailAuth()(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
h := validEmailAuth()(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
h.ServeHTTP(w, req)
resp := w.Result()
assert.Equal(t, tt.status, resp.StatusCode)
@@ -435,7 +457,7 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
b, err := engine.NewBoltDB(bolt.Options{}, engine.BoltSite{FileName: testDB, SiteID: "remark42"})
require.NoError(t, err)
memCache := cache.NewScache(cache.NewNopCache())
memCache := cache.NewScache[[]byte](cache.NewNopCache[[]byte]())
astore := adminstore.NewStaticStore("123456", []string{"remark42"}, []string{"a1", "a2"}, "admin@remark-42.com")
restrictedWordsMatcher := service.NewRestrictedWordsMatcher(service.StaticRestrictedWordsLister{Words: []string{"duck"}})
@@ -455,7 +477,7 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
DataService: dataStore,
Authenticator: auth.NewService(auth.Opts{
AdminPasswd: "password",
SecretReader: token.SecretFunc(func(aud string) (string, error) { return "secret", nil }),
SecretReader: token.SecretFunc(func(string) (string, error) { return "secret", nil }),
AvatarStore: avatar.NewLocalFS(tmp + "/ava-remark42"),
}),
Cache: memCache,
@@ -484,16 +506,26 @@ func startupT(t *testing.T, srvHook ...func(srv *Rest)) (ts *httptest.Server, sr
Cache: memCache,
KeyStore: astore,
},
NotifyService: notify.NopService,
EmojiEnabled: true,
NotifyService: notify.NopService,
EmojiEnabled: true,
openRouteLimiter: 100,
}
srv.ScoreThresholds.Low, srv.ScoreThresholds.Critical = -5, -10
// add some providers. Needed because we don't allow users with unlisted providers to authenticate
providers := []string{"provider1", "anonymous", "github", "email"}
for _, p := range providers {
srv.Authenticator.AddDirectProvider(p, provider.CredCheckerFunc(func(_, _ string) (ok bool, err error) {
return true, nil
}))
}
for _, h := range srvHook {
h(srv)
}
ts = httptest.NewServer(srv.routes())
routes := srv.routes()
ts = httptest.NewServer(routes)
teardown = func() {
ts.Close()
@@ -552,6 +584,20 @@ func getWithDevAuth(t *testing.T, url string) (body string, code int) {
return string(b), r.StatusCode
}
func getWithDev2Auth(t *testing.T, url string) (body string, code int) {
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("GET", url, http.NoBody)
require.NoError(t, err)
req.Header.Add("X-JWT", dev2Token)
r, err := client.Do(req)
require.NoError(t, err)
b, err := io.ReadAll(r.Body)
assert.NoError(t, err)
require.NoError(t, r.Body.Close())
return string(b), r.StatusCode
}
func getWithAdminAuth(t *testing.T, url string) (response string, statusCode int) {
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
@@ -574,7 +620,7 @@ func post(t *testing.T, url, body string) (*http.Response, error) {
return client.Do(req)
}
func addComment(t *testing.T, c store.Comment, ts *httptest.Server) string {
func addCommentGetCreatedTime(t *testing.T, c store.Comment, ts *httptest.Server) (id string, created time.Time) {
b, err := json.Marshal(c)
require.NoError(t, err, "can't marshal comment %+v", c)
@@ -594,7 +640,14 @@ func addComment(t *testing.T, c store.Comment, ts *httptest.Server) string {
err = json.Unmarshal(b, &crResp)
require.NoError(t, err)
time.Sleep(time.Nanosecond * 10)
return crResp["id"].(string)
created, err = time.Parse(time.RFC3339, crResp["time"].(string))
require.NoError(t, err)
return crResp["id"].(string), created
}
func addComment(t *testing.T, c store.Comment, ts *httptest.Server) string {
id, _ := addCommentGetCreatedTime(t, c, ts)
return id
}
func requireAdminOnly(t *testing.T, req *http.Request) {
@@ -633,5 +686,9 @@ func waitForHTTPSServerStart(port int) {
}
func TestMain(m *testing.M) {
goleak.VerifyTestMain(m)
goleak.VerifyTestMain(
m,
// this will be fixed in https://github.com/hashicorp/golang-lru/issues/159
goleak.IgnoreTopFunction("github.com/hashicorp/golang-lru/v2/expirable.NewLRU[...].func1"),
)
}
+1 -2
View File
@@ -5,7 +5,7 @@ import (
"net/http"
"time"
cache "github.com/go-pkgz/lcw"
cache "github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
"github.com/gorilla/feeds"
@@ -56,7 +56,6 @@ func (s *rss) postCommentsCtrl(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml; charset=utf-8")
w.WriteHeader(http.StatusOK)
if _, err = w.Write(data); err != nil {
log.Printf("[WARN] failed to send response to %s, %s", r.RemoteAddr, err)
}
+4 -7
View File
@@ -271,10 +271,7 @@ func TestServer_RssReplies(t *testing.T) {
}
func waitOnSecChange() {
for {
if time.Now().Nanosecond() < 100000000 {
break
}
for time.Now().Nanosecond() >= 100000000 {
time.Sleep(10 * time.Nanosecond)
}
}
@@ -283,11 +280,11 @@ func waitOnSecChange() {
func cleanRssFormatting(expected, actual string) (cleanExp, cleanAct string) {
reSpaces := regexp.MustCompile(`[\s\p{Zs}]{2,}`)
expected = strings.Replace(expected, "\n", " ", -1)
expected = strings.Replace(expected, "\t", " ", -1)
expected = strings.ReplaceAll(expected, "\n", " ")
expected = strings.ReplaceAll(expected, "\t", " ")
expected = reSpaces.ReplaceAllString(expected, " ")
actual = strings.Replace(actual, "\n", " ", -1)
actual = strings.ReplaceAll(actual, "\n", " ")
actual = reSpaces.ReplaceAllString(actual, " ")
return expected, actual
}
+2 -2
View File
@@ -21,7 +21,7 @@ func TestSSL_Redirect(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
@@ -56,7 +56,7 @@ func TestSSL_ACME_HTTPChallengeRouter(t *testing.T) {
client := http.Client{
// prevent http redirect
CheckRedirect: func(req *http.Request, via []*http.Request) error {
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
}
+116
View File
@@ -0,0 +1,116 @@
{
"version": 1,
"comments": [
{
"commentHex": "e7a2ef4b4aa1414a7ee65a989889aaecd9d5e7e3bca598ea7a967b4dbcaa8e11",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2022-10-25T07:25:46.807555Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "a29e741145daceb4ca5b3e5e279e05b56f73c04703d93b944718ef757e15317f",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-07-26T12:24:55.058552Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "46baf36433830a4e8bda1de56290cf5fd74c08bfa844fee4ec1744985dc77010",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-10-31T11:03:25.403282Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "6d3bb64ff73b5f9d6a959212ffde472a51abf8bdefaa5ed843659796bceef9de",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "46baf36433830a4e8bda1de56290cf5fd74c08bfa844fee4ec1744985dc77010",
"score": 0,
"state": "approved",
"creationDate": "2023-11-01T22:23:47.112062Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "23fcfcd03745ed71a9d23a9b59387a313df57e5c0faad8ba5dc96112766312c5",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-10-23T12:33:03.370182Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "d0ad6f11cf0c5f8e17457a378a6bb789f412c6b7ef7ada4ae06ec8451f7a18aa",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "root",
"score": 0,
"state": "approved",
"creationDate": "2023-10-18T01:18:38.193625Z",
"direction": 0,
"deleted": false
},
{
"commentHex": "098960fd01c1fc7c0d3ea428f52fab97ea5c18aa52f3565bba679224daddc687",
"domain": "example.com",
"url": "/example",
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"markdown": "",
"html": "",
"parentHex": "23fcfcd03745ed71a9d23a9b59387a313df57e5c0faad8ba5dc96112766312c5",
"score": 0,
"state": "approved",
"creationDate": "2023-11-01T22:24:04.639965Z",
"direction": 0,
"deleted": false
}
],
"commenters": [
{
"commenterHex": "018407e4b12b35f43b1d804d82607b341bef80c4325dd047d93f2cbb439cff85",
"email": "undefined",
"name": "blank",
"link": "undefined",
"photo": "undefined",
"provider": "anon",
"joinDate": "2022-06-09T15:54:29.865919Z",
"isModerator": false,
"deleted": false
}
]
}
+20 -5
View File
@@ -10,7 +10,6 @@ import (
"runtime"
"strings"
"github.com/go-chi/render"
log "github.com/go-pkgz/lgr"
"github.com/go-pkgz/rest"
@@ -67,20 +66,36 @@ func SendErrorHTML(w http.ResponseWriter, r *http.Request, httpStatusCode int, e
tmplstr := MustRead("error_response.html.tmpl")
tmpl := template.Must(template.New("error").Parse(tmplstr))
log.Printf("[WARN] %s", errDetailsMsg(r, httpStatusCode, err, details, errCode))
render.Status(r, httpStatusCode)
msg := bytes.Buffer{}
MustExecute(tmpl, &msg, errTmplData{
Error: err.Error(),
Details: details,
})
render.HTML(w, r, msg.String())
HTMLResponse(w, httpStatusCode, msg.String())
}
// SendErrorJSON makes {error: blah, details: blah, code: 42} json body and responds with error code
func SendErrorJSON(w http.ResponseWriter, r *http.Request, httpStatusCode int, err error, details string, errCode int) {
log.Printf("[WARN] %s", errDetailsMsg(r, httpStatusCode, err, details, errCode))
render.Status(r, httpStatusCode)
render.JSON(w, r, rest.JSON{"error": err.Error(), "details": details, "code": errCode})
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(httpStatusCode)
rest.RenderJSON(w, rest.JSON{"error": err.Error(), "details": details, "code": errCode})
}
// HTMLResponse writes HTML content with the given status code
func HTMLResponse(w http.ResponseWriter, status int, html string) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(status)
_, _ = w.Write([]byte(html))
}
// PlainTextResponse writes plain text content with the given status code
func PlainTextResponse(w http.ResponseWriter, status int, text string) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(status)
_, _ = w.Write([]byte(text))
}
func errDetailsMsg(r *http.Request, httpStatusCode int, err error, details string, errCode int) string {
+14 -5
View File
@@ -12,7 +12,7 @@ import (
"github.com/PuerkitoBio/goquery"
log "github.com/go-pkgz/lgr"
"github.com/go-pkgz/repeater"
"github.com/go-pkgz/repeater/v2"
"github.com/umputun/remark42/backend/app/rest"
"github.com/umputun/remark42/backend/app/store/image"
@@ -57,7 +57,7 @@ func (p Image) extract(commentHTML string, imgSrcPred func(string) bool) ([]stri
return nil, fmt.Errorf("can't create document: %w", err)
}
result := []string{}
doc.Find("img").Each(func(i int, s *goquery.Selection) {
doc.Find("img").Each(func(_ int, s *goquery.Selection) {
if im, ok := s.Attr("src"); ok {
if imgSrcPred(im) {
result = append(result, im)
@@ -72,7 +72,7 @@ func (p Image) replace(commentHTML string, imgs []string) string {
for _, img := range imgs {
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(img))
resImgURL := p.RemarkURL + p.RoutePath + "?src=" + encodedImgURL
commentHTML = strings.Replace(commentHTML, img, resImgURL, -1)
commentHTML = strings.ReplaceAll(commentHTML, img, resImgURL)
}
return commentHTML
@@ -98,6 +98,10 @@ func (p Image) Handler(w http.ResponseWriter, r *http.Request) {
if img == nil {
img, err = p.downloadImage(context.Background(), imgURL)
if err != nil {
if strings.Contains(err.Error(), "invalid content type") {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "invalid content type", rest.ErrImgNotFound)
return
}
rest.SendErrorJSON(w, r, http.StatusNotFound, err, "can't get image "+imgURL, rest.ErrAssetNotFound)
return
}
@@ -147,7 +151,7 @@ func (p Image) downloadImage(ctx context.Context, imgURL string) ([]byte, error)
client := http.Client{Timeout: 30 * time.Second}
defer client.CloseIdleConnections()
var resp *http.Response
err := repeater.NewDefault(5, time.Second).Do(ctx, func() error {
err := repeater.NewFixed(5, time.Second).Do(ctx, func() error {
var e error
req, e := http.NewRequest("GET", imgURL, http.NoBody)
if e != nil {
@@ -159,12 +163,17 @@ func (p Image) downloadImage(ctx context.Context, imgURL string) ([]byte, error)
if err != nil {
return nil, fmt.Errorf("can't download image %s: %w", imgURL, err)
}
defer resp.Body.Close()
defer resp.Body.Close() //nolint gosec // we don't care about response body
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("got unsuccessful response status %d while fetching %s", resp.StatusCode, imgURL)
}
contentType := resp.Header.Get("Content-Type")
if !strings.HasPrefix(contentType, "image/") {
return nil, fmt.Errorf("invalid content type %s", contentType)
}
imgData, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("unable to read image body")
+39 -26
View File
@@ -77,7 +77,6 @@ func TestImage_Extract(t *testing.T) {
img := Image{HTTP2HTTPS: true}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
res, err := img.extract(tt.inp, func(src string) bool { return strings.HasPrefix(src, "http://") })
assert.NoError(t, err)
@@ -95,7 +94,7 @@ func TestImage_Replace(t *testing.T) {
func TestImage_Routes(t *testing.T) {
// no image supposed to be cached
imageStore := image.StoreMock{LoadFunc: func(id string) ([]byte, error) { return nil, nil }}
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
HTTP2HTTPS: true,
RemarkURL: "https://demo.remark42.com",
@@ -108,31 +107,45 @@ func TestImage_Routes(t *testing.T) {
httpSrv := imgHTTPTestsServer(t)
defer httpSrv.Close()
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/img1.png"))
t.Run("valid image", func(t *testing.T) {
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/img1.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "1462", resp.Header["Content-Length"][0])
assert.Equal(t, "image/png", resp.Header["Content-Type"][0])
})
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "1462", resp.Header["Content-Length"][0])
assert.Equal(t, "image/png", resp.Header["Content-Type"][0])
t.Run("no image", func(t *testing.T) {
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/no-such-image.png"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
})
encodedImgURL = base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "/image/no-such-image.png"))
resp, err = http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusNotFound, resp.StatusCode)
t.Run("bad encoding", func(t *testing.T) {
encodedImgURL := base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "bad encoding"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
assert.Equal(t, 2, len(imageStore.LoadCalls()))
})
encodedImgURL = base64.URLEncoding.EncodeToString([]byte(httpSrv.URL + "bad encoding"))
resp, err = http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
assert.Equal(t, 2, len(imageStore.LoadCalls()))
t.Run("non-image reference", func(t *testing.T) {
encodedImgURL := base64.URLEncoding.EncodeToString([]byte("https://google.com"))
resp, err := http.Get(ts.URL + "/?src=" + encodedImgURL)
require.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
assert.Equal(t, 3, len(imageStore.LoadCalls()))
})
}
func TestImage_DisabledCachingAndHTTP2HTTPS(t *testing.T) {
imageStore := image.StoreMock{LoadFunc: func(id string) ([]byte, error) { return nil, nil }}
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
RemarkURL: "https://demo.remark42.com",
RoutePath: "/api/v1/proxy",
@@ -158,10 +171,10 @@ func TestImage_DisabledCachingAndHTTP2HTTPS(t *testing.T) {
func TestImage_RoutesCachingImage(t *testing.T) {
imageStore := image.StoreMock{
LoadFunc: func(id string) ([]byte, error) {
LoadFunc: func(string) ([]byte, error) {
return nil, nil
},
SaveFunc: func(id string, img []byte) error {
SaveFunc: func(string, []byte) error {
return nil
},
}
@@ -196,7 +209,7 @@ func TestImage_RoutesCachingImage(t *testing.T) {
func TestImage_RoutesUsingCachedImage(t *testing.T) {
// In order to validate that cached data used cache "will return" some other data from what http server would
testImage := []byte(fmt.Sprintf("%256s", "X"))
imageStore := image.StoreMock{LoadFunc: func(id string) ([]byte, error) {
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) {
return testImage, nil
}}
img := Image{
@@ -226,7 +239,7 @@ func TestImage_RoutesUsingCachedImage(t *testing.T) {
func TestImage_RoutesTimedOut(t *testing.T) {
// no image supposed to be cached
imageStore := image.StoreMock{LoadFunc: func(id string) ([]byte, error) { return nil, nil }}
imageStore := image.StoreMock{LoadFunc: func(string) ([]byte, error) { return nil, nil }}
img := Image{
HTTP2HTTPS: true,
RemarkURL: "https://demo.remark42.com",
@@ -249,7 +262,7 @@ func TestImage_RoutesTimedOut(t *testing.T) {
assert.NoError(t, resp.Body.Close())
require.NoError(t, err)
t.Log(string(b))
assert.True(t, strings.Contains(string(b), "deadline exceeded"))
assert.Contains(t, string(b), "deadline exceeded")
assert.Equal(t, 1, len(imageStore.LoadCalls()))
}
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"fmt"
"net/http"
"github.com/go-pkgz/auth/token"
"github.com/go-pkgz/auth/v2/token"
"github.com/umputun/remark42/backend/app/store"
)
+3
View File
@@ -8,6 +8,9 @@ import (
log "github.com/go-pkgz/lgr"
)
// NOTE: matryer/moq should be installed globally and works with `go generate ./...`
//go:generate moq --out admin_mock.go . Store
// Store defines interface returning admins info for given site
type Store interface {
Key(siteID string) (key string, err error)
+256
View File
@@ -0,0 +1,256 @@
// Code generated by moq; DO NOT EDIT.
// github.com/matryer/moq
package admin
import (
"sync"
)
// Ensure, that StoreMock does implement Store.
// If this is not the case, regenerate this file with moq.
var _ Store = &StoreMock{}
// StoreMock is a mock implementation of Store.
//
// func TestSomethingThatUsesStore(t *testing.T) {
//
// // make and configure a mocked Store
// mockedStore := &StoreMock{
// AdminsFunc: func(siteID string) ([]string, error) {
// panic("mock out the Admins method")
// },
// EmailFunc: func(siteID string) (string, error) {
// panic("mock out the Email method")
// },
// EnabledFunc: func(siteID string) (bool, error) {
// panic("mock out the Enabled method")
// },
// KeyFunc: func(siteID string) (string, error) {
// panic("mock out the Key method")
// },
// OnEventFunc: func(siteID string, et EventType) error {
// panic("mock out the OnEvent method")
// },
// }
//
// // use mockedStore in code that requires Store
// // and then make assertions.
//
// }
type StoreMock struct {
// AdminsFunc mocks the Admins method.
AdminsFunc func(siteID string) ([]string, error)
// EmailFunc mocks the Email method.
EmailFunc func(siteID string) (string, error)
// EnabledFunc mocks the Enabled method.
EnabledFunc func(siteID string) (bool, error)
// KeyFunc mocks the Key method.
KeyFunc func(siteID string) (string, error)
// OnEventFunc mocks the OnEvent method.
OnEventFunc func(siteID string, et EventType) error
// calls tracks calls to the methods.
calls struct {
// Admins holds details about calls to the Admins method.
Admins []struct {
// SiteID is the siteID argument value.
SiteID string
}
// Email holds details about calls to the Email method.
Email []struct {
// SiteID is the siteID argument value.
SiteID string
}
// Enabled holds details about calls to the Enabled method.
Enabled []struct {
// SiteID is the siteID argument value.
SiteID string
}
// Key holds details about calls to the Key method.
Key []struct {
// SiteID is the siteID argument value.
SiteID string
}
// OnEvent holds details about calls to the OnEvent method.
OnEvent []struct {
// SiteID is the siteID argument value.
SiteID string
// Et is the et argument value.
Et EventType
}
}
lockAdmins sync.RWMutex
lockEmail sync.RWMutex
lockEnabled sync.RWMutex
lockKey sync.RWMutex
lockOnEvent sync.RWMutex
}
// Admins calls AdminsFunc.
func (mock *StoreMock) Admins(siteID string) ([]string, error) {
if mock.AdminsFunc == nil {
panic("StoreMock.AdminsFunc: method is nil but Store.Admins was just called")
}
callInfo := struct {
SiteID string
}{
SiteID: siteID,
}
mock.lockAdmins.Lock()
mock.calls.Admins = append(mock.calls.Admins, callInfo)
mock.lockAdmins.Unlock()
return mock.AdminsFunc(siteID)
}
// AdminsCalls gets all the calls that were made to Admins.
// Check the length with:
//
// len(mockedStore.AdminsCalls())
func (mock *StoreMock) AdminsCalls() []struct {
SiteID string
} {
var calls []struct {
SiteID string
}
mock.lockAdmins.RLock()
calls = mock.calls.Admins
mock.lockAdmins.RUnlock()
return calls
}
// Email calls EmailFunc.
func (mock *StoreMock) Email(siteID string) (string, error) {
if mock.EmailFunc == nil {
panic("StoreMock.EmailFunc: method is nil but Store.Email was just called")
}
callInfo := struct {
SiteID string
}{
SiteID: siteID,
}
mock.lockEmail.Lock()
mock.calls.Email = append(mock.calls.Email, callInfo)
mock.lockEmail.Unlock()
return mock.EmailFunc(siteID)
}
// EmailCalls gets all the calls that were made to Email.
// Check the length with:
//
// len(mockedStore.EmailCalls())
func (mock *StoreMock) EmailCalls() []struct {
SiteID string
} {
var calls []struct {
SiteID string
}
mock.lockEmail.RLock()
calls = mock.calls.Email
mock.lockEmail.RUnlock()
return calls
}
// Enabled calls EnabledFunc.
func (mock *StoreMock) Enabled(siteID string) (bool, error) {
if mock.EnabledFunc == nil {
panic("StoreMock.EnabledFunc: method is nil but Store.Enabled was just called")
}
callInfo := struct {
SiteID string
}{
SiteID: siteID,
}
mock.lockEnabled.Lock()
mock.calls.Enabled = append(mock.calls.Enabled, callInfo)
mock.lockEnabled.Unlock()
return mock.EnabledFunc(siteID)
}
// EnabledCalls gets all the calls that were made to Enabled.
// Check the length with:
//
// len(mockedStore.EnabledCalls())
func (mock *StoreMock) EnabledCalls() []struct {
SiteID string
} {
var calls []struct {
SiteID string
}
mock.lockEnabled.RLock()
calls = mock.calls.Enabled
mock.lockEnabled.RUnlock()
return calls
}
// Key calls KeyFunc.
func (mock *StoreMock) Key(siteID string) (string, error) {
if mock.KeyFunc == nil {
panic("StoreMock.KeyFunc: method is nil but Store.Key was just called")
}
callInfo := struct {
SiteID string
}{
SiteID: siteID,
}
mock.lockKey.Lock()
mock.calls.Key = append(mock.calls.Key, callInfo)
mock.lockKey.Unlock()
return mock.KeyFunc(siteID)
}
// KeyCalls gets all the calls that were made to Key.
// Check the length with:
//
// len(mockedStore.KeyCalls())
func (mock *StoreMock) KeyCalls() []struct {
SiteID string
} {
var calls []struct {
SiteID string
}
mock.lockKey.RLock()
calls = mock.calls.Key
mock.lockKey.RUnlock()
return calls
}
// OnEvent calls OnEventFunc.
func (mock *StoreMock) OnEvent(siteID string, et EventType) error {
if mock.OnEventFunc == nil {
panic("StoreMock.OnEventFunc: method is nil but Store.OnEvent was just called")
}
callInfo := struct {
SiteID string
Et EventType
}{
SiteID: siteID,
Et: et,
}
mock.lockOnEvent.Lock()
mock.calls.OnEvent = append(mock.calls.OnEvent, callInfo)
mock.lockOnEvent.Unlock()
return mock.OnEventFunc(siteID, et)
}
// OnEventCalls gets all the calls that were made to OnEvent.
// Check the length with:
//
// len(mockedStore.OnEventCalls())
func (mock *StoreMock) OnEventCalls() []struct {
SiteID string
Et EventType
} {
var calls []struct {
SiteID string
Et EventType
}
mock.lockOnEvent.RLock()
calls = mock.calls.OnEvent
mock.lockOnEvent.RUnlock()
return calls
}
+14
View File
@@ -6,6 +6,20 @@ import (
"github.com/stretchr/testify/assert"
)
func TestStaticStore_StoreWithoutSites(t *testing.T) {
var ks Store = NewStaticKeyStore("key123")
enabled, err := ks.Enabled("any")
assert.NoError(t, err)
assert.True(t, enabled, "on empty store all sites are enabled")
assert.NoError(t, ks.OnEvent("test", EvCreate), "static store does nothing OnEvent")
// empty key
ks = NewStaticKeyStore("")
key, err := ks.Key("any")
assert.Error(t, err, "empty key")
assert.Empty(t, key)
}
func TestStaticStore_Get(t *testing.T) {
var ks Store = NewStaticStore("key123", []string{"s1", "s2", "s3"},
[]string{"123", "xyz"}, "aa@example.com")
+22 -15
View File
@@ -45,11 +45,13 @@ type Edit struct {
// PostInfo holds summary for given post url
type PostInfo struct {
URL string `json:"url"`
Count int `json:"count"`
ReadOnly bool `json:"read_only,omitempty" bson:"read_only,omitempty"`
FirstTS time.Time `json:"first_time,omitempty" bson:"first_time,omitempty"`
LastTS time.Time `json:"last_time,omitempty" bson:"last_time,omitempty"`
URL string `json:"url,omitempty"` // can be attached to site-wide comments but won't be set then
Count int `json:"count"`
CountLeft int `json:"count_left"` // used only with returning search results limited by number, otherwise zero
LastComment string `json:"last_comment,omitempty"` // used only with returning search results limited by number
ReadOnly bool `json:"read_only,omitempty" bson:"read_only,omitempty"` // can be attached to site-wide comments but won't be set then
FirstTS time.Time `json:"first_time,omitempty" bson:"first_time,omitempty"`
LastTS time.Time `json:"last_time,omitempty" bson:"last_time,omitempty"`
}
// BlockedUser holds id and ts for blocked user
@@ -98,6 +100,7 @@ func (c *Comment) SetDeleted(mode DeleteMode) {
c.Text = ""
c.Orig = ""
c.Score = 0
c.Controversy = 0
c.Votes = map[string]bool{}
c.VotedIPs = make(map[string]VotedIPInfo)
c.Edit = nil
@@ -121,8 +124,8 @@ func (c *Comment) Sanitize() {
// special case for embedding the quotes from Twitter
p.AllowAttrs("class").Matching(regexp.MustCompile("^twitter-tweet$")).OnElements("blockquote")
// this is list of <span> tag classes which could be produced by chroma code renderer
// source: https://github.com/alecthomas/chroma/blob/cc2dd5b/types.go#L211-L307
const codeSpanClassRegex = "^(bg|chroma|line|ln|lnt|hl|lntable|lntd|cl|w|err|x|k|kc" +
// source: https://github.com/alecthomas/chroma/blob/c263f6f/types.go#L209-L306
const codeSpanClassRegex = "^(bg|chroma|line|ln|lnt|hl|lntable|lntd|lnlinks|cl|w|err|x|k|kc" +
"|kd|kn|kp|kr|kt|n|na|nb|bp|nc|no|nd|ni|ne|nf|fm|py|nl|nn|nx|nt|nv|vc|vg" +
"|vi|vm|l|ld|s|sa|sb|sc|dl|sd|s2|se|sh|si|sx|sr|s1|ss|m|mb|mf|mh|mi|il" +
"|mo|o|ow|p|c|ch|cm|cp|cpf|c1|cs|g|gd|ge|gr|gh|gi|go|gp|gs|gu|gt|gl)$"
@@ -141,12 +144,12 @@ func (c *Comment) Snippet(limit int) string {
if limit <= 0 {
limit = snippetLen
}
cleanText := strings.Replace(c.Text, "\n", " ", -1)
cleanText := strings.ReplaceAll(c.Text, "\n", " ")
size := len([]rune(cleanText))
if size < limit {
return cleanText
}
snippet := []rune(cleanText)[:size]
snippet := []rune(cleanText)[:limit]
// go back in snippet and found the first space
for i := len(snippet) - 1; i >= 0; i-- {
if snippet[i] == ' ' {
@@ -154,6 +157,10 @@ func (c *Comment) Snippet(limit int) string {
break
}
}
// Don't add a space if comment is just a one single word which has been truncated.
if len(snippet) == limit {
return string(snippet) + "..."
}
return string(snippet) + " ..."
}
@@ -172,14 +179,14 @@ func (c *Comment) SanitizeAsURL(inp string) string {
func (c *Comment) escapeHTMLWithSome(inp string) string {
res := template.HTMLEscapeString(inp)
res = strings.Replace(res, "&amp;", "&", -1)
res = strings.Replace(res, "&#34;", "\"", -1)
res = strings.Replace(res, "&#39;", "'", -1)
res = strings.ReplaceAll(res, "&amp;", "&")
res = strings.ReplaceAll(res, "&#34;", "\"")
res = strings.ReplaceAll(res, "&#39;", "'")
return res
}
// SanitizeText used to sanitize any input string
// SanitizeText used to sanitize any input string, and removes any HTML tags
func (c *Comment) SanitizeText(inp string) string {
clean := bluemonday.UGCPolicy().Sanitize(inp)
return c.escapeHTMLWithSome(clean)
clean := bluemonday.StrictPolicy().Sanitize(inp)
return strings.TrimSpace(c.escapeHTMLWithSome(clean))
}
+12 -7
View File
@@ -22,7 +22,7 @@ func TestComment_Sanitize(t *testing.T) {
},
out: Comment{
Text: "blah XSS\n\t",
User: User{ID: `&lt;a href=&#34;http://blah.com&#34;&gt;username&lt;/a&gt;`, Name: "name &lt;b/&gt;"},
User: User{ID: `&lt;a href=&#34;http://blah.com&#34;&gt;username&lt;/a&gt;`, Name: "name"},
},
},
{
@@ -88,6 +88,14 @@ func TestComment_Sanitize(t *testing.T) {
inp: Comment{Text: "blah blah", PostTitle: "<script>alert()</script>something"},
out: Comment{Text: "blah blah", PostTitle: "something"},
},
{
inp: Comment{Text: "blah blah", PostTitle: "<a href=\"https://example.com\">test</a>"},
out: Comment{Text: "blah blah", PostTitle: "test"},
},
{
inp: Comment{Text: "blah blah", PostTitle: "https://example.com/blah"}, // link is left as-is, but not rendered as <a>
out: Comment{Text: "blah blah", PostTitle: "https://example.com/blah"},
},
{
inp: Comment{Text: `<blockquote class="twitter-tweet"><p lang="es" dir="ltr">Silicon iMac Concept<a href="https://t.co/7ga95QxVXn">https://t.co/7ga95QxVXn</a> by <a href="https://twitter.com/marcsheep?ref_src=twsrc%5Etfw">@marcsheep</a> <a href="https://t.co/ULnVpG8w55">pic.twitter.com/ULnVpG8w55</a></p>&mdash; Andreas Storm (@avstorm) <a href="https://twitter.com/avstorm/status/1325693387798933504?ref_src=twsrc%5Etfw">November 9, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>`, PostTitle: "Twitter quote"},
out: Comment{Text: `<blockquote class="twitter-tweet"><p lang="es" dir="ltr">Silicon iMac Concept<a href="https://t.co/7ga95QxVXn" rel="nofollow">https://t.co/7ga95QxVXn</a> by <a href="https://twitter.com/marcsheep?ref_src=twsrc%5Etfw" rel="nofollow">@marcsheep</a> <a href="https://t.co/ULnVpG8w55" rel="nofollow">pic.twitter.com/ULnVpG8w55</a></p>— Andreas Storm (@avstorm) <a href="https://twitter.com/avstorm/status/1325693387798933504?ref_src=twsrc%5Etfw" rel="nofollow">November 9, 2020</a></blockquote> `, PostTitle: "Twitter quote"},
@@ -196,12 +204,11 @@ func TestComment_Snippet(t *testing.T) {
{0, "", ""},
{-1, "test\nblah", "test blah"},
{5, "test\nblah", "test ..."},
{5, "xyz12345 xxx", "xyz12345 ..."},
{10, "xyz12345 xxx\ntest 123456", "xyz12345 xxx test ..."},
{5, "xyz12345 xxx", "xyz12..."},
{10, "xyz12345 xxx\ntest 123456", "xyz12345 ..."},
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
c := Comment{Text: tt.inp}
out := c.Snippet(tt.limit)
@@ -237,7 +244,6 @@ func TestComment_sanitizeAsURL(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
c := Comment{}
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.out, c.SanitizeAsURL(tt.inp))
@@ -263,12 +269,11 @@ func TestComment_sanitizeText(t *testing.T) {
},
{
"<a href=javascript:alert(document.domain)//>xxx</a>",
"&lt;a/&gt;xxx&lt;/a&gt;",
"xxx",
},
}
for i, tt := range tbl {
tt := tt
c := Comment{}
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.out, c.SanitizeText(tt.inp))
+9 -11
View File
@@ -170,7 +170,7 @@ func (b *BoltDB) Find(req FindRequest) (comments []store.Comment, err error) {
return e
}
return bucket.ForEach(func(k, v []byte) error {
return bucket.ForEach(func(_, v []byte) error {
comment := store.Comment{}
if e = json.Unmarshal(v, &comment); e != nil {
return fmt.Errorf("failed to unmarshal: %w", e)
@@ -306,9 +306,8 @@ func (b *BoltDB) Info(req InfoRequest) ([]store.PostInfo, error) {
})
// set read-only from age and manual bucket
readOnlyAge := req.ReadOnlyAge
info.ReadOnly = readOnlyAge > 0 && !info.FirstTS.IsZero() && info.FirstTS.AddDate(0, 0, readOnlyAge).Before(time.Now())
if b.checkFlag(FlagRequest{Locator: req.Locator, Flag: ReadOnly}) {
info.ReadOnly = req.ReadOnlyAge > 0 && !info.FirstTS.IsZero() && info.FirstTS.AddDate(0, 0, req.ReadOnlyAge).Before(time.Now())
if !info.ReadOnly && b.checkFlag(FlagRequest{Locator: req.Locator, Flag: ReadOnly}) {
info.ReadOnly = true
}
return []store.PostInfo{info}, err
@@ -425,11 +424,11 @@ func (b *BoltDB) Close() error {
}
// Last returns up to max last comments for given siteID
func (b *BoltDB) lastComments(siteID string, max int, since time.Time) (comments []store.Comment, err error) {
func (b *BoltDB) lastComments(siteID string, maximum int, since time.Time) (comments []store.Comment, err error) {
comments = []store.Comment{}
if max > lastLimit || max == 0 {
max = lastLimit
if maximum > lastLimit || maximum == 0 {
maximum = lastLimit
}
bdb, err := b.db(siteID)
@@ -467,7 +466,7 @@ func (b *BoltDB) lastComments(siteID string, max int, since time.Time) (comments
continue
}
comments = append(comments, comment)
if len(comments) >= max {
if len(comments) >= maximum {
break
}
}
@@ -725,7 +724,7 @@ func (b *BoltDB) listDetails(loc store.Locator) (result []UserDetailEntry, err e
err = bdb.View(func(tx *bolt.Tx) error {
var entry UserDetailEntry
bucket := tx.Bucket([]byte(userDetailsBucketName))
return bucket.ForEach(func(userID, value []byte) error {
return bucket.ForEach(func(_, value []byte) error {
if err = json.Unmarshal(value, &entry); err != nil {
return fmt.Errorf("failed to unmarshal entry: %w", e)
}
@@ -866,11 +865,10 @@ func (b *BoltDB) deleteUser(bdb *bolt.DB, siteID, userID string, mode store.Dele
// get list of commentID for all user's comment
comments := []commentInfo{}
for _, postInfo := range posts {
postInfo := postInfo
err = bdb.View(func(tx *bolt.Tx) error {
postsBkt := tx.Bucket([]byte(postsBucketName))
postBkt := postsBkt.Bucket([]byte(postInfo.URL))
err = postBkt.ForEach(func(postURL []byte, commentVal []byte) error {
err = postBkt.ForEach(func(_ []byte, commentVal []byte) error {
comment := store.Comment{}
if err = json.Unmarshal(commentVal, &comment); err != nil {
return fmt.Errorf("failed to unmarshal: %w", err)
+1 -1
View File
@@ -324,7 +324,7 @@ func TestBoltDB_InfoPost(t *testing.T) {
b, teardown := prep(t) // two comments for https://radio-t.com
defer teardown()
ts := func(min int) time.Time { return time.Date(2017, 12, 20, 15, 18, min, 0, time.Local) }
ts := func(minute int) time.Time { return time.Date(2017, 12, 20, 15, 18, minute, 0, time.Local) }
// add one more for https://radio-t.com/2
comment := store.Comment{
+33 -20
View File
@@ -35,24 +35,17 @@ func NewCommentFormatter(converters ...CommentConverter) *CommentFormatter {
}
// Format comment fields
func (f *CommentFormatter) Format(c Comment) Comment {
c.Text = f.FormatText(c.Text)
func (f *CommentFormatter) Format(c Comment, raw bool) Comment {
c.Text = f.FormatText(c.Text, raw)
return c
}
// FormatText converts text with markdown processor, applies external converters and shortens links
func (f *CommentFormatter) FormatText(txt string) (res string) {
mdExt := bf.NoIntraEmphasis | bf.Tables | bf.FencedCode |
bf.Strikethrough | bf.SpaceHeadings | bf.HardLineBreak |
bf.BackslashLineBreak | bf.Autolink
rend := bf.NewHTMLRenderer(bf.HTMLRendererParameters{
Flags: bf.Smartypants | bf.SmartypantsFractions | bf.SmartypantsDashes | bf.SmartypantsAngledQuotes,
})
extRend := bfchroma.NewRenderer(bfchroma.Extend(rend), bfchroma.ChromaOptions(html.WithClasses(true)))
res = string(bf.Run([]byte(txt), bf.WithExtensions(mdExt), bf.WithRenderer(extRend)))
//
// raw=true disables SmartyPants for HTML rendering (replacement of quotes, dashes, fractions, etc).
func (f *CommentFormatter) FormatText(txt string, raw bool) (res string) {
mdExt, rend := GetMdExtensionsAndRenderer(raw)
res = string(bf.Run([]byte(txt), bf.WithExtensions(mdExt), bf.WithRenderer(rend)))
res = f.unEscape(res)
for _, conv := range f.converters {
@@ -64,14 +57,14 @@ func (f *CommentFormatter) FormatText(txt string) (res string) {
}
// Shortens all the automatic links in HTML: auto link has equal "href" and "text" attributes.
func (f *CommentFormatter) shortenAutoLinks(commentHTML string, max int) (resHTML string) {
func (f *CommentFormatter) shortenAutoLinks(commentHTML string, maximum int) (resHTML string) {
doc, err := goquery.NewDocumentFromReader(strings.NewReader(commentHTML))
if err != nil {
return commentHTML
}
doc.Find("a").Each(func(i int, s *goquery.Selection) {
doc.Find("a").Each(func(_ int, s *goquery.Selection) {
if href, ok := s.Attr("href"); ok {
if href != s.Text() || len(href) < max+3 || max < 3 {
if href != s.Text() || len(href) < maximum+3 || maximum < 3 {
return
}
commentURL, e := url.Parse(href)
@@ -84,7 +77,7 @@ func (f *CommentFormatter) shortenAutoLinks(commentHTML string, max int) (resHTM
return
}
short := string([]rune(href)[:max-3])
short := string([]rune(href)[:maximum-3])
if len(short) < len(host) {
short = host
}
@@ -106,7 +99,7 @@ func (f *CommentFormatter) unEscape(txt string) (res string) {
}
res = txt
for _, e := range elems {
res = strings.Replace(res, e.from, e.to, -1)
res = strings.ReplaceAll(res, e.from, e.to)
}
return res
}
@@ -117,7 +110,7 @@ func (f *CommentFormatter) lazyImage(commentHTML string) (resHTML string) {
if err != nil {
return commentHTML
}
doc.Find("img").Each(func(i int, s *goquery.Selection) {
doc.Find("img").Each(func(_ int, s *goquery.Selection) {
s.SetAttr("loading", "lazy")
})
resHTML, err = doc.Find("body").Html()
@@ -126,3 +119,23 @@ func (f *CommentFormatter) lazyImage(commentHTML string) (resHTML string) {
}
return resHTML
}
// GetMdExtensionsAndRenderer returns blackfriday extensions and renderer used for rendering markdown
// within store module.
//
// raw=true disables SmartyPants for HTML rendering (replacement of quotes, dashes, fractions, etc).
func GetMdExtensionsAndRenderer(raw bool) (bf.Extensions, *bfchroma.Renderer) {
mdExt := bf.NoIntraEmphasis | bf.Tables | bf.FencedCode |
bf.Strikethrough | bf.SpaceHeadings | bf.HardLineBreak |
bf.BackslashLineBreak | bf.Autolink
flags := bf.HTMLFlags(0)
if !raw {
flags = bf.Smartypants | bf.SmartypantsFractions | bf.SmartypantsDashes | bf.SmartypantsAngledQuotes
}
rend := bf.NewHTMLRenderer(bf.HTMLRendererParameters{Flags: flags})
extRend := bfchroma.NewRenderer(bfchroma.Extend(rend), bfchroma.ChromaOptions(html.WithClasses(true)))
return mdExt, extRend
}
+16 -9
View File
@@ -2,6 +2,7 @@ package store
import (
"strconv"
"strings"
"testing"
"time"
@@ -35,30 +36,37 @@ func TestFormatter_FormatText(t *testing.T) {
"lazy image",
},
{"&mdash; not translated #354", "<p>— not translated #354</p>\n!converted", "mdash"},
{`no_smartpants "quoted" text`, "<p>no_smartpants &#34;quoted&#34; text</p>\n!converted", "normal quotes without smartpants"},
{`"quoted" text`, "<p>«quoted» text</p>\n!converted", "normal quotes with smartpants"},
{`no_smartpants “quoted” text`, "<p>no_smartpants “quoted” text</p>\n!converted", "curly quotes without smartpants"},
{`“quoted” text`, "<p>“quoted” text</p>\n!converted", "curly quotes with smartpants"},
{`no_smartpants «quoted» text`, "<p>no_smartpants «quoted» text</p>\n!converted", "French guillemets without smartpants"},
{`«quoted» text`, "<p>«quoted» text</p>\n!converted", "French guillemets with smartpants"},
{"smth\n```go\nfunc main(aa string) int {return 0}\n```", `<p>smth</p>
<pre tabindex="0" class="chroma"><code><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">(</span><span class="nx">aa</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span><span class="k">return</span> <span class="mi">0</span><span class="p">}</span>
</span></span></code></pre>!converted`, "code with language"},
{"```\ntest_code\n```", `<pre tabindex="0" class="chroma"><code><span class="line"><span class="cl">test_code
<pre class="chroma"><code><span class="line"><span class="cl"><span class="kd">func</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="nx">aa</span><span class="w"> </span><span class="kt">string</span><span class="p">)</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="p">{</span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">}</span><span class="w">
</span></span></span></code></pre>!converted`, "code with language"},
{"```\ntest_code\n```", `<pre class="chroma"><code><span class="line"><span class="cl">test_code
</span></span></code></pre>!converted`, "code without language"},
}
f := NewCommentFormatter(mockConverter{})
for _, tt := range tbl {
tt := tt
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.out, f.FormatText(tt.in))
raw := strings.HasPrefix(tt.in, `no_smartpants`)
t.Logf("raw: %v", raw)
assert.Equal(t, tt.out, f.FormatText(tt.in, raw))
})
}
}
func TestFormatter_FormatTextNoConverter(t *testing.T) {
f := NewCommentFormatter()
assert.Equal(t, "<p>12345</p>\n", f.FormatText("12345"))
assert.Equal(t, "<p>12345</p>\n", f.FormatText("12345", false))
}
func TestFormatter_FormatTextConverterFunc(t *testing.T) {
fn := CommentConverterFunc(func(text string) string { return "zz!" + text })
f := NewCommentFormatter(fn)
assert.Equal(t, "zz!<p>12345</p>\n", f.FormatText("12345"))
assert.Equal(t, "zz!<p>12345</p>\n", f.FormatText("12345", false))
}
func TestFormatter_FormatComment(t *testing.T) {
@@ -78,7 +86,7 @@ func TestFormatter_FormatComment(t *testing.T) {
f := NewCommentFormatter(mockConverter{})
exp := comment
exp.Text = "<p>blah</p>\n\n<p>xyz</p>\n!converted"
assert.Equal(t, exp, f.Format(comment))
assert.Equal(t, exp, f.Format(comment, false))
}
func TestFormatter_ShortenAutoLinks(t *testing.T) {
@@ -137,7 +145,6 @@ func TestCommentFormatter_lazyImage(t *testing.T) {
f := NewCommentFormatter(nil)
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
assert.Equal(t, tt.out, f.lazyImage(tt.inp))
})
+13
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/binary"
"errors"
"fmt"
"time"
@@ -118,6 +119,18 @@ func (b *Bolt) Load(id string) ([]byte, error) {
return data, nil
}
// Delete image from storage
func (b *Bolt) Delete(id string) error {
return b.db.Update(func(tx *bolt.Tx) error {
// deleting a non-existing key doesn't return an error, so joining errors from deleting an image
// from both buckets is safe and will return nil if there are no errors on the real delete
// or image is absent in both buckets
err := tx.Bucket([]byte(imagesBktName)).Delete([]byte(id))
err = errors.Join(err, tx.Bucket([]byte(imagesStagedBktName)).Delete([]byte(id)))
return err
})
}
// Cleanup runs scan of staging and removes old data based on ttl
func (b *Bolt) Cleanup(_ context.Context, ttl time.Duration) error {
return b.db.Update(func(tx *bolt.Tx) error {
@@ -58,6 +58,36 @@ func TestBoltStore_LoadAfterSave(t *testing.T) {
assert.Error(t, err)
}
func TestBoltStore_LoadAfterDelete(t *testing.T) {
svc, teardown := prepareBoltImageStorageTest(t)
defer teardown()
// delete image from permanent storage
id := "test_img"
err := svc.Save(id, gopherPNGBytes())
assert.NoError(t, err)
err = svc.Commit(id)
require.NoError(t, err)
err = svc.Delete(id)
assert.NoError(t, err)
_, err = svc.Load(id)
assert.Error(t, err)
// delete staging image
id = "staging_img"
err = svc.Save(id, gopherPNGBytes())
assert.NoError(t, err)
err = svc.Delete(id)
assert.NoError(t, err)
_, err = svc.Load(id)
assert.Error(t, err)
}
func TestBoltStore_Cleanup(t *testing.T) {
svc, teardown := prepareBoltImageStorageTest(t)
defer teardown()
+25 -2
View File
@@ -23,7 +23,8 @@ type FileSystem struct {
Staging string
Partitions int
crc struct {
moveLock sync.Mutex // needed only for deleting images or moving them from staging to permanent storage
crc struct {
*crc64.Table
sync.Once
mask string
@@ -50,6 +51,8 @@ func (f *FileSystem) Save(id string, img []byte) error {
// Commit file stored in staging location by moving it to permanent location
func (f *FileSystem) Commit(id string) error {
f.moveLock.Lock()
defer f.moveLock.Unlock()
log.Printf("[DEBUG] Commit image %s", id)
stagingImage, permImage := f.location(f.Staging, id), f.location(f.Location, id)
@@ -107,11 +110,31 @@ func (f *FileSystem) Load(id string) ([]byte, error) {
return io.ReadAll(fh)
}
// Delete image from storage
func (f *FileSystem) Delete(id string) error {
f.moveLock.Lock()
defer f.moveLock.Unlock()
staging := f.location(f.Staging, id)
// file doesn't exist on staging, delete from permanent location
if _, err := os.Stat(staging); os.IsNotExist(err) {
file := f.location(f.Location, id)
e := os.Remove(file)
_ = os.Remove(path.Dir(file)) // try to remove directory
return e
}
// delete file from staging
err := os.Remove(staging)
_ = os.Remove(path.Dir(staging)) // try to remove directory
return err
}
// Cleanup runs scan of staging and removes old files based on ttl
func (f *FileSystem) Cleanup(_ context.Context, ttl time.Duration) error {
if _, err := os.Stat(f.Staging); os.IsNotExist(err) {
return nil
}
f.moveLock.Lock()
defer f.moveLock.Unlock()
// we can ignore context as on local FS remove is relatively fast operation
err := filepath.Walk(f.Staging, func(fpath string, info os.FileInfo, err error) error {
@@ -143,7 +166,7 @@ func (f *FileSystem) Info() (StoreInfo, error) {
}
var ts time.Time
err := filepath.Walk(f.Staging, func(fpath string, info os.FileInfo, err error) error {
err := filepath.Walk(f.Staging, func(_ string, info os.FileInfo, err error) error {
if err != nil {
return err
}
+25 -1
View File
@@ -131,6 +131,31 @@ func TestFsStore_LoadAfterCommit(t *testing.T) {
assert.Error(t, err)
}
func TestFsStore_LoadAfterDelete(t *testing.T) {
svc, teardown := prepareImageTest(t)
defer teardown()
id := "test_img"
err := svc.Save(id, gopherPNGBytes())
assert.NoError(t, err)
err = svc.Commit(id)
require.NoError(t, err)
err = svc.Delete(id)
require.NoError(t, err)
_, err = svc.Load(id)
assert.Error(t, err)
// create file on staging
err = svc.Save(id, gopherPNGBytes())
assert.NoError(t, err)
err = svc.Delete(id)
require.NoError(t, err)
_, err = svc.Load(id)
assert.Error(t, err)
}
func TestFsStore_location(t *testing.T) {
tbl := []struct {
partitions int
@@ -148,7 +173,6 @@ func TestFsStore_location(t *testing.T) {
{0, "user/12345", "/tmp/user/12345"},
}
for n, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(n), func(t *testing.T) {
svc := FileSystem{Location: "/tmp", Partitions: tt.partitions}
assert.Equal(t, tt.res, svc.location("/tmp", tt.id))
+53 -35
View File
@@ -70,6 +70,7 @@ type Store interface {
Info() (StoreInfo, error) // get meta information about storage
Save(id string, img []byte) error // store image with passed id to staging
Load(id string) ([]byte, error) // load image by ID
Delete(id string) error // delete image by ID
ResetCleanupTimer(id string) error // resets cleanup timer for the image, called on comment preview
Commit(id string) error // move image from staging to permanent
@@ -88,8 +89,8 @@ func NewService(s Store, p ServiceParams) *Service {
return &Service{ServiceParams: p, store: s}
}
// SubmitAndCommit multiple ids immediately
func (s *Service) SubmitAndCommit(idsFn func() []string) error {
// Commit multiple ids immediately
func (s *Service) Commit(idsFn func() []string) error {
errs := new(multierror.Error)
for _, id := range idsFn() {
err := s.store.Commit(id)
@@ -117,7 +118,7 @@ func (s *Service) Submit(idsFn func() []string) {
for atomic.LoadInt32(&s.term) == 0 && time.Since(req.TS) <= s.EditDuration {
time.Sleep(time.Millisecond * 10) // small sleep to relive busy wait but keep reactive for term (close)
}
err := s.SubmitAndCommit(req.idsFn)
err := s.Commit(req.idsFn)
if err != nil {
log.Printf("[WARN] image commit error %v", err)
}
@@ -141,39 +142,14 @@ func (s *Service) Submit(idsFn func() []string) {
// ExtractPictures gets list of images from the doc html and convert from urls to ids, i.e. user/pic.png
func (s *Service) ExtractPictures(commentHTML string) (ids []string) {
doc, err := goquery.NewDocumentFromReader(strings.NewReader(commentHTML))
if err != nil {
log.Printf("[ERROR] can't parse commentHTML to parse images: %q, error: %v", commentHTML, err)
return nil
}
doc.Find("img").Each(func(i int, sl *goquery.Selection) {
if im, ok := sl.Attr("src"); ok {
if strings.Contains(im, s.ImageAPI) {
elems := strings.Split(im, "/")
if len(elems) >= 2 {
id := elems[len(elems)-2] + "/" + elems[len(elems)-1]
ids = append(ids, id)
}
}
if strings.Contains(im, s.ProxyAPI) {
proxiedURL, err := url.Parse(im)
if err != nil {
return
}
imgURL, err := base64.URLEncoding.DecodeString(proxiedURL.Query().Get("src"))
if err != nil {
return
}
imgID, err := CachedImgID(string(imgURL))
if err != nil {
return
}
ids = append(ids, imgID)
}
}
})
return s.extractImageIDs(commentHTML, true)
}
return ids
// ExtractNonProxiedPictures gets list of non-proxied images from the doc html and convert from urls to ids, i.e. user/pic.png
// This method is used in image check on post preview and load, as proxied images have lazy loading
// and wouldn't be present on disk but still valid as they will be loaded the first time someone requests them.
func (s *Service) ExtractNonProxiedPictures(commentHTML string) (ids []string) {
return s.extractImageIDs(commentHTML, false)
}
// Cleanup runs periodic cleanup with 1.5*ServiceParams.EditDuration. Blocking loop, should be called inside of goroutine by consumer
@@ -237,6 +213,11 @@ func (s *Service) Load(id string) ([]byte, error) {
return s.store.Load(id)
}
// Delete wraps storage Delete function.
func (s *Service) Delete(id string) error {
return s.store.Delete(id)
}
// Save wraps storage Save function, validating and resizing the image before calling it.
func (s *Service) Save(userID string, r io.Reader) (id string, err error) {
id = path.Join(userID, guid())
@@ -262,6 +243,43 @@ func (s *Service) ImgContentType(img []byte) string {
return contentType
}
// returns list of image IDs from the comment html, including proxied images if includeProxied is true
func (s *Service) extractImageIDs(commentHTML string, includeProxied bool) (ids []string) {
doc, err := goquery.NewDocumentFromReader(strings.NewReader(commentHTML))
if err != nil {
log.Printf("[ERROR] can't parse commentHTML to parse images: %q, error: %v", commentHTML, err)
return nil
}
doc.Find("img").Each(func(_ int, sl *goquery.Selection) {
if im, ok := sl.Attr("src"); ok {
if strings.Contains(im, s.ImageAPI) {
elems := strings.Split(im, "/")
if len(elems) >= 2 {
id := elems[len(elems)-2] + "/" + elems[len(elems)-1]
ids = append(ids, id)
}
}
if includeProxied && strings.Contains(im, s.ProxyAPI) {
proxiedURL, err := url.Parse(im)
if err != nil {
return
}
imgURL, err := base64.URLEncoding.DecodeString(proxiedURL.Query().Get("src"))
if err != nil {
return
}
imgID, err := CachedImgID(string(imgURL))
if err != nil {
return
}
ids = append(ids, imgID)
}
}
})
return ids
}
// prepareImage calls readAndValidateImage and resize on provided image.
func (s *Service) prepareImage(r io.Reader) ([]byte, error) {
data, err := readAndValidateImage(r, s.MaxSize)
+46 -2
View File
@@ -4,9 +4,9 @@
package image
import (
context "context"
"context"
"sync"
time "time"
"time"
)
// Ensure, that StoreMock does implement Store.
@@ -25,6 +25,9 @@ var _ Store = &StoreMock{}
// CommitFunc: func(id string) error {
// panic("mock out the Commit method")
// },
// DeleteFunc: func(id string) error {
// panic("mock out the Delete method")
// },
// InfoFunc: func() (StoreInfo, error) {
// panic("mock out the Info method")
// },
@@ -50,6 +53,9 @@ type StoreMock struct {
// CommitFunc mocks the Commit method.
CommitFunc func(id string) error
// DeleteFunc mocks the Delete method.
DeleteFunc func(id string) error
// InfoFunc mocks the Info method.
InfoFunc func() (StoreInfo, error)
@@ -76,6 +82,11 @@ type StoreMock struct {
// ID is the id argument value.
ID string
}
// Delete holds details about calls to the Delete method.
Delete []struct {
// ID is the id argument value.
ID string
}
// Info holds details about calls to the Info method.
Info []struct {
}
@@ -99,6 +110,7 @@ type StoreMock struct {
}
lockCleanup sync.RWMutex
lockCommit sync.RWMutex
lockDelete sync.RWMutex
lockInfo sync.RWMutex
lockLoad sync.RWMutex
lockResetCleanupTimer sync.RWMutex
@@ -173,6 +185,38 @@ func (mock *StoreMock) CommitCalls() []struct {
return calls
}
// Delete calls DeleteFunc.
func (mock *StoreMock) Delete(id string) error {
if mock.DeleteFunc == nil {
panic("StoreMock.DeleteFunc: method is nil but Store.Delete was just called")
}
callInfo := struct {
ID string
}{
ID: id,
}
mock.lockDelete.Lock()
mock.calls.Delete = append(mock.calls.Delete, callInfo)
mock.lockDelete.Unlock()
return mock.DeleteFunc(id)
}
// DeleteCalls gets all the calls that were made to Delete.
// Check the length with:
//
// len(mockedStore.DeleteCalls())
func (mock *StoreMock) DeleteCalls() []struct {
ID string
} {
var calls []struct {
ID string
}
mock.lockDelete.RLock()
calls = mock.calls.Delete
mock.lockDelete.RUnlock()
return calls
}
// Info calls InfoFunc.
func (mock *StoreMock) Info() (StoreInfo, error) {
if mock.InfoFunc == nil {
+12 -22
View File
@@ -19,10 +19,10 @@ import (
func TestService_SaveAndLoad(t *testing.T) {
store := StoreMock{
SaveFunc: func(id string, img []byte) error {
SaveFunc: func(string, []byte) error {
return nil
},
LoadFunc: func(id string) ([]byte, error) {
LoadFunc: func(string) ([]byte, error) {
return nil, nil
},
}
@@ -105,6 +105,7 @@ func TestService_ExtractPictures(t *testing.T) {
ids = svc.ExtractPictures(html)
require.Equal(t, 1, len(ids), "one image in")
assert.Equal(t, "cached_images/12318fbd4c55e9d177b8b5ae197bc89c5afd8e07-a41fcb00643f28d700504256ec81cbf2e1aac53e", ids[0])
require.Empty(t, svc.ExtractNonProxiedPictures(html), "no non-proxied images expected to be found")
// bad url
html = `<img src=" https://remark42.radio-t.com/api/v1/img">`
@@ -125,7 +126,7 @@ func TestService_ExtractPictures(t *testing.T) {
func TestService_Cleanup(t *testing.T) {
store := StoreMock{
CleanupFunc: func(ctx context.Context, ttl time.Duration) error {
CleanupFunc: func(context.Context, time.Duration) error {
return nil
},
}
@@ -140,17 +141,13 @@ func TestService_Cleanup(t *testing.T) {
func TestService_Submit(t *testing.T) {
store := StoreMock{
CommitFunc: func(id string) error {
return nil
},
ResetCleanupTimerFunc: func(id string) error {
return nil
},
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
svc := NewService(&store, ServiceParams{ImageAPI: "/blah/", EditDuration: time.Millisecond * 100})
svc.Submit(func() []string { return []string{"id1", "id2", "id3"} })
assert.Equal(t, 3, len(store.ResetCleanupTimerCalls()))
err := svc.SubmitAndCommit(func() []string { return []string{"id4", "id5"} })
err := svc.Commit(func() []string { return []string{"id4", "id5"} })
assert.NoError(t, err)
svc.Submit(func() []string { return []string{"id6", "id7"} })
assert.Equal(t, 5, len(store.ResetCleanupTimerCalls()))
@@ -163,12 +160,8 @@ func TestService_Submit(t *testing.T) {
func TestService_Close(t *testing.T) {
store := StoreMock{
CommitFunc: func(id string) error {
return nil
},
ResetCleanupTimerFunc: func(id string) error {
return nil
},
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
svc := Service{store: &store, ServiceParams: ServiceParams{ImageAPI: "/blah/", EditDuration: time.Hour * 24}}
svc.Submit(func() []string { return []string{"id1", "id2", "id3"} })
@@ -181,10 +174,8 @@ func TestService_Close(t *testing.T) {
func TestService_SubmitDelay(t *testing.T) {
store := StoreMock{
CommitFunc: func(id string) error {
return nil
},
ResetCleanupTimerFunc: func(id string) error {
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error {
return nil
},
}
@@ -268,7 +259,6 @@ func TestGetProportionalSizes(t *testing.T) {
}
for i, tt := range tbl {
tt := tt
t.Run(strconv.Itoa(i), func(t *testing.T) {
resW, resH := getProportionalSizes(tt.inpW, tt.inpH, tt.limitW, tt.limitH)
assert.Equal(t, tt.resW, resW, "width")
@@ -287,7 +277,7 @@ func TestCachedImgID(t *testing.T) {
assert.Equal(t, "cached_images/"+Sha1Str("example.org")+"-"+Sha1Str(imgURL), img)
}
func TestService_DoubleClose(t *testing.T) {
func TestService_DoubleClose(*testing.T) {
store := StoreMock{}
svc := NewService(&store, ServiceParams{EditDuration: 20 * time.Millisecond})
svc.Close(context.TODO())
+6
View File
@@ -41,6 +41,12 @@ func (r *RPC) Load(id string) ([]byte, error) {
return io.ReadAll(base64.NewDecoder(base64.StdEncoding, strings.NewReader(rawImg)))
}
// Delete image from storage
func (r *RPC) Delete(id string) error {
_, err := r.Call("image.delete", id)
return err
}
// Commit file stored in staging location by moving it to permanent location
func (r *RPC) Commit(id string) error {
_, err := r.Call("image.commit", id)
@@ -41,6 +41,18 @@ func TestRemote_Load(t *testing.T) {
assert.Equal(t, gopherPNGBytes(), res)
}
func TestRemote_Delete(t *testing.T) {
ts := testServer(t, `{"method":"image.delete","params":"54321","id":1}`, `{}`)
defer ts.Close()
c := RPC{Client: jrpc.Client{API: ts.URL, Client: http.Client{}}}
var a Store = &c
_ = a
err := c.Delete("54321")
assert.NoError(t, err)
}
func TestRemote_Commit(t *testing.T) {
ts := testServer(t, `{"method":"image.commit","params":"gopher_id","id":1}`, `{"id":1}`)
defer ts.Close()
+80 -12
View File
@@ -5,15 +5,17 @@ package service
import (
"fmt"
"math"
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/go-pkgz/lcw"
"github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
"github.com/google/uuid"
"github.com/hashicorp/go-multierror"
bf "github.com/russross/blackfriday/v2"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/admin"
@@ -26,6 +28,7 @@ type DataStore struct {
Engine engine.Interface
EditDuration time.Duration
AdminStore admin.Store
MinCommentSize int
MaxCommentSize int
MaxVotes int
RestrictSameIPVotes struct {
@@ -46,7 +49,7 @@ type DataStore struct {
}
repliesCache struct {
lcw.LoadingCache
lcw.LoadingCache[struct{}]
once sync.Once
}
}
@@ -68,7 +71,7 @@ type PostMetaData struct {
ReadOnly bool `json:"read_only"`
}
const defaultCommentMaxSize = 2000
const defaultCommentMaxSize = 2048
const maxLastCommentsReply = 5000
// UnlimitedVotes doesn't restrict MaxVotes
@@ -278,7 +281,7 @@ func (s *DataStore) submitImages(comment store.Comment) {
var err error
if comment.Imported {
err = s.ImageService.SubmitAndCommit(idsFn)
err = s.ImageService.Commit(idsFn)
} else {
s.ImageService.Submit(idsFn)
}
@@ -531,7 +534,8 @@ func (s *DataStore) EditComment(locator store.Locator, commentID string, req Edi
func (s *DataStore) HasReplies(comment store.Comment) bool {
s.repliesCache.once.Do(func() {
// default expiration time of 5 minutes and cleanup time of 2.5 minutes
s.repliesCache.LoadingCache, _ = lcw.NewExpirableCache(lcw.TTL(5 * time.Minute))
o := lcw.NewOpts[struct{}]()
s.repliesCache.LoadingCache, _ = lcw.NewExpirableCache[struct{}](o.TTL(5 * time.Minute))
})
if _, found := s.repliesCache.Peek(comment.ID); found {
@@ -551,7 +555,7 @@ func (s *DataStore) HasReplies(comment store.Comment) bool {
// When this code is reached, key "comment.ID" is not in cache.
// Calling cache.Get on it will put it in cache with 5 minutes TTL.
// We call it with empty struct as value as we care about keys and not values.
_, _ = s.repliesCache.Get(comment.ID, func() (interface{}, error) { return struct{}{}, nil })
_, _ = s.repliesCache.Get(comment.ID, func() (struct{}, error) { return struct{}{}, nil })
return true
}
}
@@ -627,7 +631,9 @@ func (s *DataStore) Counts(siteID string, postIDs []string) ([]store.PostInfo, e
return res, nil
}
// ValidateComment checks if comment size below max and user fields set
// ValidateComment checks if comment size below max and user fields set.
// It also validates the absence of relative links as they are almost never the intention of the commenter,
// usually added by mistakes and only create confusion.
func (s *DataStore) ValidateComment(c *store.Comment) error {
maxSize := s.MaxCommentSize
if s.MaxCommentSize <= 0 {
@@ -636,13 +642,31 @@ func (s *DataStore) ValidateComment(c *store.Comment) error {
if c.Orig == "" {
return fmt.Errorf("empty comment text")
}
if len([]rune(c.Orig)) > maxSize {
return fmt.Errorf("comment text exceeded max allowed size %d (%d)", maxSize, len([]rune(c.Orig)))
commentLength := len([]rune(c.Orig))
if commentLength > maxSize {
return fmt.Errorf("comment text exceeded max allowed size %d (%d)", maxSize, commentLength)
}
if s.MinCommentSize > 0 && commentLength < s.MinCommentSize {
return fmt.Errorf("comment text is smaller than min allowed size %d (%d)", s.MinCommentSize, commentLength)
}
if c.User.ID == "" || c.User.Name == "" {
return fmt.Errorf("empty user info")
}
return nil
// for validation purposes it's not important if SmartyPants formatting is disabled or enabled,
// while for storing the comment that flag is set based on user preference
mdExt, rend := store.GetMdExtensionsAndRenderer(false)
parser := bf.New(bf.WithRenderer(rend), bf.WithExtensions(bf.CommonExtensions), bf.WithExtensions(mdExt))
var wrongLinkError error
parser.Parse([]byte(c.Orig)).Walk(func(node *bf.Node, _ bool) bf.WalkStatus {
if len(node.Destination) != 0 &&
(!strings.HasPrefix(string(node.Destination), "http://") && !strings.HasPrefix(string(node.Destination), "https://") && !strings.HasPrefix(string(node.Destination), "mailto:")) {
wrongLinkError = fmt.Errorf("links should start with mailto:, http:// or https://")
return bf.Terminate
}
return bf.GoToNext
})
return wrongLinkError
}
// IsAdmin checks if usesID in the list of admins
@@ -737,7 +761,23 @@ func (s *DataStore) Info(locator store.Locator, readonlyAge int) (store.PostInfo
if len(res) == 0 {
return store.PostInfo{}, fmt.Errorf("post %+v not found", locator)
}
return res[0], nil
// URL request
if locator.URL != "" {
return res[0], nil
}
// site-wide request which returned multiple store.PostInfo, so that URL and ReadOnly flags don't make sense
var info store.PostInfo
for _, i := range res {
info.Count += i.Count
if info.FirstTS.IsZero() || i.FirstTS.Before(info.FirstTS) {
info.FirstTS = i.FirstTS
}
if info.LastTS.IsZero() || i.LastTS.After(info.LastTS) {
info.LastTS = i.LastTS
}
}
return info, nil
}
// Delete comment by id
@@ -756,6 +796,34 @@ func (s *DataStore) Delete(locator store.Locator, commentID string, mode store.D
s.repliesCache.Delete(commentID)
s.repliesCache.Delete(comment.ParentID)
}
// delete images from the comment if they are not reused elsewhere in comments to the same page
idsFn := func() []string { // get IDs of all images from the same URL to verify if image from deleted comment was reused
comments, e := s.Engine.Find(engine.FindRequest{Locator: locator})
if e != nil {
log.Printf("[WARN] can't get comments %s text for deleted comment image check, %v", comment.ID, err)
return nil
}
var imgIDs = []string{}
for _, cc := range comments {
// exclude the comment we are deleting
if cc.ID != commentID {
imgIDs = append(imgIDs, s.ImageService.ExtractPictures(cc.Text)...)
}
}
return imgIDs
}
commentImgIDs := s.ImageService.ExtractPictures(comment.Text)
pageImgIDs := idsFn()
for _, id := range commentImgIDs {
if !slices.Contains(pageImgIDs, id) {
if err := s.ImageService.Delete(id); err != nil {
log.Printf("[WARN] failed to delete image %s on comment %s deletion, %v", id, commentID, err)
}
}
}
log.Printf("[ERROR] commentImgIDs: %v, pageImgIDs: %v", commentImgIDs, pageImgIDs)
req := engine.DeleteRequest{Locator: locator, CommentID: commentID, DeleteMode: mode}
return s.Engine.Delete(req)
}
@@ -911,7 +979,7 @@ func (s *DataStore) Last(siteID string, limit int, since time.Time, user store.U
func (s *DataStore) Close() error {
errs := new(multierror.Error)
if s.repliesCache.LoadingCache != nil {
errs = multierror.Append(errs, s.repliesCache.LoadingCache.Close())
errs = multierror.Append(errs, s.repliesCache.Close())
}
if s.TitleExtractor != nil {
errs = multierror.Append(errs, s.TitleExtractor.Close())
+288 -25
View File
@@ -2,6 +2,7 @@ package service
import (
"context"
"errors"
"fmt"
"math/rand"
"net/http"
@@ -131,7 +132,7 @@ func TestService_CreateFromPartialWithTitle(t *testing.T) {
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: ks,
TitleExtractor: NewTitleExtractor(http.Client{Timeout: 5 * time.Second})}
TitleExtractor: NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{"127.0.0.1"})}
defer b.Close()
postPath := "/post/42"
@@ -171,6 +172,50 @@ func TestService_CreateFromPartialWithTitle(t *testing.T) {
assert.Equal(t, "post blah", res.PostTitle, "keep comment title")
}
func TestService_Put(t *testing.T) {
ks := admin.NewStaticKeyStore("secret 123")
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: ks}
comment := store.Comment{
ID: "c-1",
ParentID: "id-1",
Text: "test text",
User: store.User{ID: "user2", Name: "user name 2"},
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
}
_, err := b.Create(comment)
require.NoError(t, err)
// create new comment with everything different to replace the first one with fields below
updatedComment := store.Comment{
ID: "c-1",
ParentID: "id-new",
Text: "new text",
User: store.User{ID: "user3", Name: "user name 3"},
Locator: store.Locator{URL: "https://example.com", SiteID: "example"},
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
}
err = b.Put(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, updatedComment)
require.NoError(t, err)
// request with wrong user, should not affect the comment user
got, err := b.Get(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "c-1", store.User{ID: "user1", Name: "user name 1"})
require.NoError(t, err)
assert.Equal(t, "c-1", got.ID)
assert.Equal(t, "new text", got.Text)
assert.Equal(t, "id-1", got.ParentID, "should be unaltered")
assert.Equal(t, "user2", got.User.ID, "should be unaltered")
assert.Equal(t, "user name 2", got.User.Name, "should be unaltered")
assert.Equal(t, "https://radio-t.com", got.Locator.URL, "should be unaltered")
assert.Equal(t, "radio-t", got.Locator.SiteID, "should be unaltered")
assert.Equal(t, time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local), got.Timestamp, "should be unaltered")
}
func TestService_SetTitle(t *testing.T) {
var titleEnable int32
tss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -195,7 +240,7 @@ func TestService_SetTitle(t *testing.T) {
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: ks,
TitleExtractor: NewTitleExtractor(http.Client{Timeout: 5 * time.Second})}
TitleExtractor: NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{"127.0.0.1"})}
defer b.Close()
comment := store.Comment{
Text: "text",
@@ -413,7 +458,6 @@ func TestService_VoteAggressive(t *testing.T) {
assert.Equal(t, 0, len(res[0].VotedIPs), "vote ips hidden")
// random +1/-1 result should be [0..2]
rand.Seed(time.Now().UnixNano())
for i := 0; i < 100; i++ {
wg.Add(1)
go func() {
@@ -546,6 +590,111 @@ func TestService_VoteControversy(t *testing.T) {
assert.InDelta(t, 1.73, res[0].Controversy, 0.01)
}
func TestService_RestrictedWords(t *testing.T) {
ks := admin.NewStaticKeyStore("secret 123")
eng, teardown := prepStoreEngine(t)
defer teardown()
restictedWordLister := StaticRestrictedWordsLister{Words: []string{"restricted"}}
b := DataStore{Engine: eng, AdminStore: ks, RestrictedWordsMatcher: NewRestrictedWordsMatcher(restictedWordLister)}
// test creating a comment with restricted words which should fail with appropriate error
reply := store.Comment{
ID: "c-1",
ParentID: "id-1",
Text: "restricted word",
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name 2"},
}
id, err := b.Create(reply)
assert.EqualError(t, err, ErrRestrictedWordsFound.Error(), "should fail with RestrictedWordError")
assert.Empty(t, id)
}
func TestDataStore_AdminStoreErrors(t *testing.T) {
badKey := true
badEnabled := true
as := admin.StoreMock{
OnEventFunc: func(string, admin.EventType) error { return errors.New("err") },
KeyFunc: func(string) (string, error) {
if badKey {
return "", errors.New("mock key err")
}
return "secret", nil
},
EnabledFunc: func(string) (bool, error) {
if badEnabled {
return false, errors.New("mock enabled err")
}
return true, nil
},
AdminsFunc: func(string) ([]string, error) { return nil, errors.New("mock admins err") },
}
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: &as, MaxVotes: -1}
comment := store.Comment{
ID: "c-1",
ParentID: "id-1",
Text: "restricted word",
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name 2"},
}
// Key call error
id, err := b.Create(comment)
assert.ErrorContainsf(t, err, "mock key err", "should fail with mock error")
assert.Empty(t, id)
assert.Equal(t, len(as.KeyCalls()), 1)
assert.Equal(t, len(as.EnabledCalls()), 0)
assert.Equal(t, len(as.OnEventCalls()), 0)
// Enabled call error
badKey = false
id, err = b.Create(comment)
assert.ErrorContains(t, err, "mock enabled err", "should fail with mock error")
assert.Empty(t, id)
assert.Equal(t, len(as.KeyCalls()), 2)
assert.Equal(t, len(as.EnabledCalls()), 1)
assert.Equal(t, len(as.OnEventCalls()), 0)
// only OnEvent error
badEnabled = false
id, err = b.Create(comment)
assert.NoError(t, err, "OnEvent error should be just logged")
assert.Equal(t, id, "c-1")
assert.Equal(t, len(as.KeyCalls()), 3)
assert.Equal(t, len(as.EnabledCalls()), 2)
assert.Equal(t, len(as.OnEventCalls()), 1)
// OnEvent error on Vote call
_, err = b.Vote(VoteReq{Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, CommentID: "c-1",
UserID: "user4", Val: true})
assert.NoError(t, err, "OnEvent error should be just logged")
assert.Equal(t, len(as.KeyCalls()), 4)
assert.Equal(t, len(as.EnabledCalls()), 3)
assert.Equal(t, len(as.OnEventCalls()), 2)
// Admins error
isAdmin := b.IsAdmin("radio-t", "user2")
assert.False(t, isAdmin)
assert.Equal(t, len(as.AdminsCalls()), 1)
assert.Equal(t, len(as.OnEventCalls()), 2)
// OnEvent error on EditComment call
_, err = b.EditComment(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "c-1", EditRequest{Text: "new text"})
assert.NoError(t, err, "OnEvent error should be just logged")
// OnEvent error on Delete call
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "c-1", store.SoftDelete)
assert.NoError(t, err, "OnEvent error should be just logged")
// OnEvent error on EditComment Delete call
_, err = b.EditComment(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "c-1", EditRequest{Delete: true})
assert.NoError(t, err, "OnEvent error should be just logged")
}
func TestService_VoteSameIP(t *testing.T) {
eng, teardown := prepStoreEngine(t)
defer teardown()
@@ -622,7 +771,6 @@ func TestService_Controversy(t *testing.T) {
b := DataStore{}
for i, tt := range tbl {
tt := tt
t.Run(fmt.Sprintf("check-%d-%d:%d", i, tt.ups, tt.downs), func(t *testing.T) {
assert.InDelta(t, tt.res, b.controversy(tt.ups, tt.downs), 0.01)
})
@@ -775,27 +923,31 @@ func TestService_EditCommentAdmin(t *testing.T) {
}
func TestService_ValidateComment(t *testing.T) {
b := DataStore{MaxCommentSize: 2000, AdminStore: admin.NewStaticKeyStore("secret 123")}
b := DataStore{MinCommentSize: 6, MaxCommentSize: 2000, AdminStore: admin.NewStaticKeyStore("secret 123")}
longText := fmt.Sprintf("%4000s", "X")
tbl := []struct {
inp store.Comment
err error
err string
}{
{inp: store.Comment{}, err: fmt.Errorf("empty comment text")},
{inp: store.Comment{Orig: "something blah", User: store.User{ID: "myid", Name: "name"}}, err: nil},
{inp: store.Comment{Orig: "something blah", User: store.User{ID: "myid"}}, err: fmt.Errorf("empty user info")},
{inp: store.Comment{Orig: longText, User: store.User{ID: "myid", Name: "name"}}, err: fmt.Errorf("comment text exceeded max allowed size 2000 (4000)")},
{inp: store.Comment{}, err: "empty comment text"},
{inp: store.Comment{Orig: "something blah", User: store.User{ID: "myid", Name: "name"}}, err: ""},
{inp: store.Comment{Orig: "something blah", User: store.User{ID: "myid"}}, err: "empty user info"},
{inp: store.Comment{Orig: "short", User: store.User{ID: "myid", Name: "name"}}, err: "comment text is smaller than min allowed size 6 (5)"},
{inp: store.Comment{Orig: longText, User: store.User{ID: "myid", Name: "name"}}, err: "comment text exceeded max allowed size 2000 (4000)"},
{inp: store.Comment{Orig: "here is a link with relative URL: [google.com](url)", User: store.User{ID: "myid", Name: "name"}}, err: "links should start with mailto:, http:// or https://"},
{inp: store.Comment{Orig: "here is a link with relative URL: [google.com](url)", User: store.User{ID: "myid", Name: "name"}}, err: "links should start with mailto:, http:// or https://"},
{inp: store.Comment{Orig: "multiple links, one is bad: [test](http://test) [test2](bad_url) [test3](https://test3)", User: store.User{ID: "myid", Name: "name"}}, err: "links should start with mailto:, http:// or https://"},
}
for n, tt := range tbl {
err := b.ValidateComment(&tt.inp)
if tt.err == nil {
if tt.err == "" {
assert.NoError(t, err, "check #%d", n)
continue
}
require.Error(t, err)
assert.EqualError(t, tt.err, err.Error(), "check #%d", n)
assert.EqualError(t, err, tt.err, "check #%d", n)
}
}
@@ -1154,7 +1306,7 @@ func TestService_Find(t *testing.T) {
assert.InDelta(t, 0, res[1].Controversy, 0.01)
// make sure title sanitized
assert.Equal(t, "some title, &lt;a href=\"http://radio-t.com\" rel=\"nofollow\"&gt;link&lt;/a&gt;", res[0].PostTitle)
assert.Equal(t, "some title, link", res[0].PostTitle)
}
func TestService_FindSince(t *testing.T) {
@@ -1183,18 +1335,55 @@ func TestService_Info(t *testing.T) {
b := DataStore{Engine: eng, EditDuration: 100 * time.Millisecond,
AdminStore: admin.NewStaticStore("secret 123", nil, []string{"user2"}, "user@email.com")}
info, err := b.Info(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, 0)
// add one more comment for another URL to test site-wide Info request
comment := store.Comment{
ID: "123456xyz",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2018, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com/another", SiteID: "radio-t"},
User: store.User{ID: "user2", Name: "user name"},
}
_, err := b.Create(comment)
require.NoError(t, err)
// get non-existing URL info
info, err := b.Info(store.Locator{URL: "bad", SiteID: "radio-t"}, 0)
assert.Error(t, err)
assert.Empty(t, info)
// get non-existing site info
info, err = b.Info(store.Locator{SiteID: "bad"}, 0)
assert.Error(t, err)
assert.Empty(t, info)
// test two initially created comments and store first comment FirstTS
info, err = b.Info(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, 0)
require.NoError(t, err)
assert.Equal(t, "https://radio-t.com", info.URL)
assert.Equal(t, 2, info.Count)
assert.False(t, info.ReadOnly)
assert.True(t, info.LastTS.After(info.FirstTS))
firstTS := info.FirstTS
time.Sleep(1 * time.Second) // make post RO in 1sec
info, err = b.Info(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, 1)
require.NoError(t, err)
assert.Equal(t, "https://radio-t.com", info.URL)
assert.True(t, info.ReadOnly)
// get last created comment LastTS
info, err = b.Info(store.Locator{URL: "https://radio-t.com/another", SiteID: "radio-t"}, 0)
require.NoError(t, err)
lastTS := info.LastTS
// site-level request
info, err = b.Info(store.Locator{SiteID: "radio-t"}, 1)
require.NoError(t, err)
assert.Equal(t, 3, info.Count)
assert.Empty(t, info.URL, "site-level request should not set URL")
assert.False(t, info.ReadOnly, "site-level request should not set ReadOnly")
assert.Equal(t, firstTS, info.FirstTS, "site-level request should have FirstTS from the first post")
assert.Equal(t, lastTS, info.LastTS, "site-level request should have LastTS from the last post")
}
func TestService_Delete(t *testing.T) {
@@ -1216,6 +1405,80 @@ func TestService_Delete(t *testing.T) {
assert.NoError(t, err)
}
func TestService_deleteImagesOnCommentDelete(t *testing.T) {
lgr.Setup(lgr.Debug, lgr.CallerFile, lgr.CallerFunc)
mockStore := image.StoreMock{
DeleteFunc: func(string) error { return nil },
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
EditDuration: 50 * time.Millisecond,
ImageAPI: "/images/dev/",
ProxyAPI: "/non_existent",
})
defer imgSvc.Close(context.TODO())
// two comments for https://radio-t.com
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, EditDuration: 50 * time.Millisecond,
AdminStore: admin.NewStaticKeyStore("secret 123"), ImageService: imgSvc}
c := store.Comment{
ID: "id-22",
Text: `some text <img src="/images/dev/pic1.png"/> xx <img src="/images/dev/pic2.png"/>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
// reply to the first comment with one new image and one existing one
c = store.Comment{
ID: "id-23",
ParentID: "id-22",
Text: `some text <img src="/images/dev/pic2.png"/> xx <img src="/images/dev/pic3.png"/>`,
Locator: store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err = b.Engine.Create(c) // create directly with engine, doesn't call submitImages
assert.NoError(t, err)
b.submitImages(c)
// verify that images are in staging store
assert.Equal(t, 4, len(mockStore.ResetCleanupTimerCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.ResetCleanupTimerCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[1].ID)
assert.Equal(t, "dev/pic2.png", mockStore.ResetCleanupTimerCalls()[2].ID)
assert.Equal(t, "dev/pic3.png", mockStore.ResetCleanupTimerCalls()[3].ID)
time.Sleep(b.EditDuration + 100*time.Millisecond)
// verify that they got into the main store
assert.Equal(t, 4, len(mockStore.CommitCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.CommitCalls()[0].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[1].ID)
assert.Equal(t, "dev/pic2.png", mockStore.CommitCalls()[2].ID)
assert.Equal(t, "dev/pic3.png", mockStore.CommitCalls()[3].ID)
// delete the first comment
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "id-22", store.SoftDelete)
assert.NoError(t, err)
// verify that images are deleted from the main store
assert.Equal(t, 1, len(mockStore.DeleteCalls()))
assert.Equal(t, "dev/pic1.png", mockStore.DeleteCalls()[0].ID)
// delete the second comment
err = b.Delete(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, "id-23", store.SoftDelete)
assert.NoError(t, err)
// verify that images are deleted from the main store
assert.Equal(t, 3, len(mockStore.DeleteCalls()))
assert.Equal(t, "dev/pic2.png", mockStore.DeleteCalls()[1].ID)
assert.Equal(t, "dev/pic3.png", mockStore.DeleteCalls()[2].ID)
}
// DeleteUser removes all comments from user
func TestService_DeleteUser(t *testing.T) {
// two comments for https://radio-t.com, no reply
@@ -1396,8 +1659,8 @@ func TestService_submitImages(t *testing.T) {
lgr.Setup(lgr.Debug, lgr.CallerFile, lgr.CallerFunc)
mockStore := image.StoreMock{
CommitFunc: func(id string) error { return nil },
ResetCleanupTimerFunc: func(id string) error { return nil },
CommitFunc: func(string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
@@ -1438,10 +1701,10 @@ func TestService_ResubmitStagingImages(t *testing.T) {
InfoFunc: func() (image.StoreInfo, error) {
return image.StoreInfo{FirstStagingImageTS: time.Time{}.Add(time.Second)}, nil
},
CommitFunc: func(id string) error {
CommitFunc: func(string) error {
return nil
},
ResetCleanupTimerFunc: func(id string) error { return nil },
ResetCleanupTimerFunc: func(string) error { return nil },
}
imgSvc := image.NewService(&mockStore,
image.ServiceParams{
@@ -1533,7 +1796,7 @@ func TestService_ResubmitStagingImages_EngineError(t *testing.T) {
first := true
engineMock := engine.InterfaceMock{
FindFunc: func(req engine.FindRequest) ([]store.Comment, error) {
FindFunc: func(engine.FindRequest) ([]store.Comment, error) {
if first {
first = false
return nil, nil
@@ -1558,7 +1821,7 @@ func TestService_ResubmitStagingImages_EngineError(t *testing.T) {
func TestService_alterComment(t *testing.T) {
engineMock := engine.InterfaceMock{
FlagFunc: func(req engine.FlagRequest) (bool, error) {
FlagFunc: func(engine.FlagRequest) (bool, error) {
return false, nil
},
}
@@ -1580,7 +1843,7 @@ func TestService_alterComment(t *testing.T) {
first := true
engineMock = engine.InterfaceMock{
FlagFunc: func(req engine.FlagRequest) (bool, error) {
FlagFunc: func(engine.FlagRequest) (bool, error) {
if first {
first = false
return false, nil
@@ -1598,7 +1861,7 @@ func TestService_alterComment(t *testing.T) {
first = true
engineMock = engine.InterfaceMock{
FlagFunc: func(req engine.FlagRequest) (bool, error) {
FlagFunc: func(engine.FlagRequest) (bool, error) {
if first {
first = false
return true, nil
@@ -1656,10 +1919,10 @@ func TestService_DoubleClose_Static(t *testing.T) {
eng, teardown := prepStoreEngine(t)
defer teardown()
b := DataStore{Engine: eng, AdminStore: ks,
TitleExtractor: NewTitleExtractor(http.Client{Timeout: 5 * time.Second})}
b.Close()
TitleExtractor: NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{})}
assert.NoError(t, b.Close())
// second call should not result in panic or errors
b.Close()
assert.NoError(t, b.Close())
}
// makes new boltdb, put two records
+2 -8
View File
@@ -242,11 +242,5 @@
"time": "2017-12-25T19:47:22Z"
}
}
],
"info": {
"url": "url",
"count": 12,
"first_time": "2017-12-25T19:46:01Z",
"last_time": "2017-12-25T19:47:22Z"
}
}
]
}
+2 -8
View File
@@ -176,11 +176,5 @@
}
}]
}]
}],
"info": {
"url": "url",
"count": 8,
"first_time": "2017-12-25T19:46:01Z",
"last_time": "2017-12-25T19:47:05Z"
}
}
}]
}
+39 -17
View File
@@ -4,10 +4,11 @@ import (
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
"github.com/go-pkgz/lcw"
"github.com/go-pkgz/lcw/v2"
log "github.com/go-pkgz/lgr"
"golang.org/x/net/html"
)
@@ -19,32 +20,53 @@ const (
// TitleExtractor gets html title from remote page, cached
type TitleExtractor struct {
client http.Client
cache lcw.LoadingCache
client http.Client
cache lcw.LoadingCache[string]
allowedDomains []string
}
// NewTitleExtractor makes extractor with cache. If memory cache failed, switching to no-cache
func NewTitleExtractor(client http.Client) *TitleExtractor {
func NewTitleExtractor(client http.Client, allowedDomains []string) *TitleExtractor {
log.Printf("[DEBUG] creating extractor, allowed domains %+v", allowedDomains)
res := TitleExtractor{
client: client,
client: client,
allowedDomains: allowedDomains,
}
var err error
res.cache, err = lcw.NewExpirableCache(lcw.TTL(teCacheTTL), lcw.MaxKeySize(teCacheMaxRecs))
o := lcw.NewOpts[string]()
res.cache, err = lcw.NewExpirableCache(o.TTL(teCacheTTL), o.MaxKeySize(teCacheMaxRecs))
if err != nil {
log.Printf("[WARN] failed to make cache, caching disabled for titles, %v", err)
res.cache = &lcw.Nop{}
res.cache = &lcw.Nop[string]{}
}
return &res
}
// Get page for url and return title
func (t *TitleExtractor) Get(url string) (string, error) {
func (t *TitleExtractor) Get(pageURL string) (string, error) {
// parse domain of the URL and check if it's in the allowed list
u, err := url.Parse(pageURL)
if err != nil {
return "", fmt.Errorf("failed to parse url %s: %w", pageURL, err)
}
allowed := false
for _, domain := range t.allowedDomains {
if u.Hostname() == domain ||
(strings.HasSuffix(u.Hostname(), domain) && // suffix match, e.g. "example.com" matches "www.example.com"
u.Hostname()[len(u.Hostname())-len(domain)-1] == '.') { // but we should not match "notexample.com"
allowed = true
break
}
}
if !allowed {
return "", fmt.Errorf("domain %s is not allowed", u.Host)
}
client := http.Client{Timeout: t.client.Timeout, Transport: t.client.Transport}
defer client.CloseIdleConnections()
b, err := t.cache.Get(url, func() (interface{}, error) {
resp, err := client.Get(url)
if err != nil {
return nil, fmt.Errorf("failed to load page %s: %w", url, err)
b, err := t.cache.Get(pageURL, func() (string, error) {
resp, e := client.Get(pageURL)
if e != nil {
return "", fmt.Errorf("failed to load page %s: %w", pageURL, e)
}
defer func() {
if err = resp.Body.Close(); err != nil {
@@ -52,23 +74,23 @@ func (t *TitleExtractor) Get(url string) (string, error) {
}
}()
if resp.StatusCode != 200 {
return nil, fmt.Errorf("can't load page %s, code %d", url, resp.StatusCode)
return "", fmt.Errorf("can't load page %s, code %d", pageURL, resp.StatusCode)
}
title, ok := t.getTitle(resp.Body)
if !ok {
return nil, fmt.Errorf("can't get title for %s", url)
return "", fmt.Errorf("can't get title for %s", pageURL)
}
return title, nil
})
// on error save result (empty string) to cache too and return "" title
if err != nil {
_, _ = t.cache.Get(url, func() (interface{}, error) { return "", nil })
_, _ = t.cache.Get(pageURL, func() (string, error) { return "", nil })
return "", err
}
return b.(string), nil
return b, nil
}
// Close title extractor
@@ -94,7 +116,7 @@ func (t *TitleExtractor) isTitleElement(n *html.Node) bool {
func (t *TitleExtractor) traverse(n *html.Node) (string, bool) {
if t.isTitleElement(n) {
title := n.FirstChild.Data
title = strings.Replace(title, "\n", "", -1)
title = strings.ReplaceAll(title, "\n", "")
title = strings.TrimSpace(title)
return title, true
}
+8 -9
View File
@@ -28,10 +28,9 @@ func TestTitle_GetTitle(t *testing.T) {
{`<html><body> 2222</body></html>`, false, ""},
}
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second})
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{})
defer ex.Close()
for i, tt := range tbl {
tt := tt
t.Run(fmt.Sprintf("check-%d", i), func(t *testing.T) {
title, ok := ex.getTitle(strings.NewReader(tt.page))
assert.Equal(t, tt.ok, ok)
@@ -41,7 +40,7 @@ func TestTitle_GetTitle(t *testing.T) {
}
func TestTitle_Get(t *testing.T) {
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second})
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{"127.0.0.1"})
defer ex.Close()
var hits int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -75,7 +74,7 @@ func TestTitle_GetConcurrent(t *testing.T) {
for n := 0; n < 1000; n++ {
body += "something something blah blah\n"
}
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second})
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{"127.0.0.1"})
defer ex.Close()
var hits int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -104,10 +103,10 @@ func TestTitle_GetConcurrent(t *testing.T) {
}
func TestTitle_GetFailed(t *testing.T) {
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second})
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{"127.0.0.1"})
defer ex.Close()
var hits int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(404)
}))
@@ -125,8 +124,8 @@ func TestTitle_GetFailed(t *testing.T) {
}
func TestTitle_DoubleClosed(t *testing.T) {
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second})
ex.Close()
ex := NewTitleExtractor(http.Client{Timeout: 5 * time.Second}, []string{})
assert.NoError(t, ex.Close())
// second call should not result in panic
ex.Close()
assert.NoError(t, ex.Close())
}

Some files were not shown because too many files have changed in this diff Show More