Dmitry VerkhoturovandGitHub
4793c1cd2c
* Assert what the image endpoints promise rather than the compressor's output Three tests pinned the exact bytes or the exact length of an encoded image, so they fail on any toolchain whose deflate or png encoder emits something different. CI pins go 1.25 and passes; go 1.27 fails all three, while the images themselves are perfectly valid. TestRest_QR now decodes both the golden file and the response and compares the pixels, which is the same assertion about the qr code and none about the encoder. The two resize cases assert the decoded image fits the box resize was given and touches one of its sides, which is what fitting to a box means and what the function actually promises. Resolves #2200. * Fill the instance URL into the embedded frontend at serve time The widget falls back to a compiled-in URL whenever a page omits `remark_config.host`. The bundler cannot know that URL, so it emits `{% REMARK_URL %}` and each distribution substitutes it: the docker image rewrites the files under its web root at container start, and the release binary, which serves the build embedded in itself, had nothing doing it. `prepare-release-assets.sh` filled the marker with `http://127.0.0.1:8080` before the embed instead, so every copy of the binary shipped pointing at the visitor's own loopback address, and on an https site the request is blocked as mixed content besides. It has been that way since v1.11.0, the first release to embed the frontend, and the earlier binaries embedded none, so the tarball has never served a correctly addressed widget. The placeholder now survives into the embedded copy and the file server fills it with the configured `REMARK_URL` as it serves, which is what the docker image already does to its own copy. The image no longer bakes the loopback address into its embedded copy either, so the fallback it keeps for a missing web root is correct rather than misleading. Substituted in html, js and mjs, the same set `docker-init.sh` rewrites, and the served size is the substituted one so a response is neither truncated nor left hanging. Nothing exercised the marker the frontend build emits wherever the instance url belongs. Every page in the suite sets `remark_config.host` from its own origin, so the compiled-in fallback is never read, and a distribution that stopped substituting would keep the suite green. Two tests. The first reads the served bundles and pages back and asserts the marker is gone from each and that what replaced it is this instance. The second covers what the substitution is for: the widget document carries no host of its own, since `iframe.html` builds its config from a query string the parent never puts one in, so everything it requests is addressed with the compiled-in url. It asserts the widget renders and that the config request went to this instance. The demo pages cannot show the second. Their loader builds the bundle's own script url from `remark_config.host`, so a page without one never gets as far as loading the widget. Verified by disabling both substitution paths, the serve-time one and the docker image's, and rebuilding: both tests fail. Editing the files on disk is not enough, since the file server substitutes as it serves. The served body now depends on remarkURL, but cacheControl builds its etag from version and path only. An operator who notices the widget is addressed to the wrong host, corrects REMARK_URL and restarts the same binary gets 304 on revalidation, so the client keeps a bundle pointing at the old host. Cache-Control is no-cache, so it revalidates every time and never ages out of that state either. That is the exact situation this substitution exists to fix, so the validator has to carry the url.
title, aliases
| title | aliases | |
|---|---|---|
| Backend Development Guidelines |
|
You can use a fully functional local version to develop and test both frontend and backend. It requires at least 2GB RAM or swap enabled.
To bring it up, run:
cp compose-dev-backend.yml compose-private.yml
# now, edit / debug `compose-private.yml` to your heart's content
# build and run
docker compose -f compose-private.yml up --build
It starts Remark42 on 127.0.0.1:8080 and adds local OAuth2 provider "Dev". To access the UI demo page go to http://127.0.0.1:8080/web/. By default, you would be logged in as dev_user, defined as admin. You can tweak any of the supported parameters in corresponded yml file.
{{< note "🚨" >}}
Please use 127.0.0.1 and not localhost to access the server; otherwise, CORS will prevent your browser from authentication to work correctly. You could alter the address for dev auth with the REMARK_URL environment variable.
{{< /note >}}
Backend Docker Compose config (compose-dev-backend.yml) by default skips running frontend related tests. Frontend Docker Compose config (compose-dev-frontend.yml) by default skips running backend related tests and sets NODE_ENV=development for frontend build.
Backend development
With Docker
Run tests in your IDE, and re-run make rundev each time you want to see how your code changes behave to test them at http://127.0.0.1:8080/web/.
Without Docker
You have to install the latest stable go toolchain to run the backend locally.
In order to have working Remark42 installation you need once to copy frontend static files to ./backend/app/cmd/web directory from master docker image, as it is expected to be where application compiles:
# frontend files
docker pull ghcr.io/umputun/remark42:master
docker create -ti --name remark42files ghcr.io/umputun/remark42:master sh
docker cp remark42files:/srv/web/ ./backend/app/cmd/
docker rm -f remark42files
# fix frontend files to point to the right URL
## Mac version
find -E ./backend/app/cmd/web -regex '.*\.(html|js|mjs)$' -print -exec sed -i '' "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
## Linux version
find ./backend/app/cmd/web -regex '.*\.\(html\|js\|mjs\)$' -print -exec sed -i "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
The assets under /web the frontend does not build (privacy.html, markdown-help.html,
400x400.jpeg) come from backend/app/webassets/assets and are embedded in the binary, so the copy
above neither covers them nor needs to. At runtime a file of the same name under web-root /
REMARK_WEB_ROOT is served in preference to the embedded one.
To run backend - cd backend; go run app/main.go server --dbg --secret=12345 --url=http://127.0.0.1:8080 --admin-passwd=password --site=remark. It stars backend service with embedded bolt store on port 8080 with basic auth, allowing to authenticate and run requests directly, like this:
HTTP http://admin:password@127.0.0.1:8080/api/v1/find?site=remark&sort=-active&format=tree&url=http://127.0.0.1:8080
Technical Details
Data stored in boltdb (embedded key/value database) files under STORE_BOLT_PATH. Each site is stored in a separate boltdb file.
To migrate/move Remark42 to another host, these boltdb files must be transferred. Additionally, the avatars directory AVATAR_FS_PATH (default ./var/avatars) and images directory IMAGE_FS_PATH (default ./var/pictures) must be transferred. As an alternative to storing avatars and images in the file system, boltdb can be used by setting AVATAR_TYPE or IMAGE_TYPE to bolt. Files in the IMAGE_FS_STAGING directory can be safely ignored because they are moved to the image store between app restarts.
The automatic backup process runs every 24h and exports all content in JSON-like format to backup-remark-YYYYMMDD.gz.
Authentication implemented with go-pkgz/auth stored in a cookie. It uses HttpOnly, secure cookies.
All heavy REST calls cached internally in LRU cache limited by CACHE_MAX_ITEMS and CACHE_MAX_SIZE with go-pkgz/rest.
User's activity throttled globally (up to 1000 simultaneous requests) and limited locally (per user, usually up to 10 req/sec).
Request timeout set to 60sec.
Admin authentication (--admin-password set) allows to hit Remark42 API without social login and admin privileges. Adds basic-auth for username: admin, password: ${ADMIN_PASSWD}. Enable it only for the initial comment import or for manual backups. Do not leave the server running with admin password set if you don't have an intention to keep creating backups manually!
User can vote for the comment multiple times but only to change the vote. Double voting is not allowed.
User can edit comments in 5 mins (configurable) window after creation.
User ID hashed and prefixed by OAuth provider name to avoid collisions and potential abuse.
All avatars resized and cached locally to prevent rate limiters from OAuth providers, part of go-pkgz/auth functionality.
Images served over HTTP can be proxied to HTTPS (IMAGE_PROXY_HTTP2HTTPS=true) to prevent mixed HTTP/HTTPS.
All images can be proxied and saved locally (IMAGE_PROXY_CACHE_EXTERNAL=true) instead of serving from the original location. Beware, images that are posted with this parameter enabled will be served from proxy even after it is disabled.
Docker build uses publicly available base images.