Dmitry VerkhoturovandGitHub
a5b2fe3cfc
Four upgrades that were finished but never merged, the compiler collapse
they enable, and the dependency sweep that follows. Direct
devDependencies go from 78 to 60 and dependencies from 10 to 9.
Three were doing the same job: `ts-loader` stripped types in webpack,
`babel-loader` did everything else, and `@swc/jest` repeated both for the
tests with its own copy of the JSX settings. Babel is the one that
survives, because the `data-testid` stripper has no equivalent elsewhere.
`ts-loader` ran `transpileOnly: true`, so it only stripped types, which
`@babel/preset-typescript` does; `fork-ts-checker-webpack-plugin` was
already what type-checks. Jest runs `babel-jest` against the same
`.babelrc.js` the bundle uses, passed as `configFile` because a
file-relative babel config does not reach the `node_modules` packages in
`transformIgnorePatterns`, and `jest.config.mjs` is plain ESM because a
`.ts` config is compiled against `tsconfig.json`, whose
`verbatimModuleSyntax` rejects ESM syntax in a file the package has not
declared as a module.
That removes `ts-loader`, `@swc/jest` and `@swc/core`. The last was
pinned to 1.2.205 from 2022 with no way forward, because newer builds
emit non-configurable exports and break `jest.spyOn` across 13 suites.
Babel compiles a file at a time with no type information, so it cannot
tell a type-only import from a real one and keeps the module. One line,
`import { boundActions } from './connected-comment'`, pulled the whole
redux store into `last-comments.mjs` and doubled it. `verbatimModuleSyntax`
and `@typescript-eslint/consistent-type-imports` mark them properly; the
statement has to be a separate `import type`, since verbatim semantics
keep an inline `import { type X }` and load the module anyway.
The legacy and modern compilations produced the same bytes. Both read the
same browserslist query, `defaults, not IE 11, not samsung 12` resolves to
chrome 109 and up, and nothing in the source needs transforming for that
set, so 28 of the 29 output pairs were byte-identical.
That made the module/nomodule switch worse than redundant: it served the
`.js` file to browsers with no ES module support, and those files carried
`??`, `?.` and class fields, so the fallback handed its own audience a
syntax error. There is now one bundle, always loaded as a module, in the
five templates and in the seven `site/` documents integrators copy from.
A production build emits 29 files rather than 58, in about 3 seconds
rather than 17. Two of those documents did not work at all beforehand:
the SPA snippet could not parse, and the subdomain example had an
unterminated string.
`@babel/core` 8 declares `^22.18 || >=24.11` and `size-limit` 13 declares
`^22.18 || ^24 || >=26`, so 20 was below the floor of two things installed
here; pnpm only warns, which is why every build passed. All seven places
the frontend pins it move together. `site/` is untouched: it builds with
yarn and eleventy and installs neither.
`eslint --print-config` before and after gives 173 active rules on an
application file against 172, and 172 on a spec file and a plain JS file
against 171. What is gone is three `flowtype` rules with no Flow here,
`no-new-object` and `no-new-symbol` whose upstream replacements are on,
`react/forbid-foreign-prop-types` with no propTypes anywhere, and, on TS
only, `no-useless-constructor`, whose typescript-eslint version is on at
error. `@babel/core` is pinned to 8 across the workspace because
`@jest/transform` and `istanbul-lib-instrument` depend on 7 outright; a
second scoped override holds `eslint-config-preact` on 7, since its
`@babel/eslint-parser` loads babel 7 syntax plugins.
`fast-async` rewrote every async function into nodent promise chains,
calls babel's `transform` synchronously, which babel 8 removed, and every
browser in the target list runs async natively. `prefresh` blew its stack
on `createContext` under babel 8 with no newer release to move to, which
compiled `intl.tsx` and `store/context.tsx` into throwing stubs, so
`pnpm dev:app` could not run the widget at all. `core-js` is not injected
now that `useBuiltIns` is gone, `postcss-custom-properties` was reached
directly although nothing declared it and resolved only through pnpm's
private hoist directory, and `cssnano` ran in both postcss chains although
`CssMinimizerPlugin` already uses it.
`pnpm lint`, `pnpm test` and `pnpm build` now work from `frontend/` as
`CLAUDE.md` and the contributing guide have always said they do; the
workspace root defined none of them.
title, aliases
| title | aliases | |
|---|---|---|
| Backend Development Guidelines |
|
You can use a fully functional local version to develop and test both frontend and backend. It requires at least 2GB RAM or swap enabled.
To bring it up, run:
cp compose-dev-backend.yml compose-private.yml
# now, edit / debug `compose-private.yml` to your heart's content
# build and run
docker compose -f compose-private.yml up --build
It starts Remark42 on 127.0.0.1:8080 and adds local OAuth2 provider "Dev". To access the UI demo page go to http://127.0.0.1:8080/web/. By default, you would be logged in as dev_user, defined as admin. You can tweak any of the supported parameters in corresponded yml file.
{{< note "🚨" >}}
Please use 127.0.0.1 and not localhost to access the server; otherwise, CORS will prevent your browser from authentication to work correctly. You could alter the address for dev auth with the REMARK_URL environment variable.
{{< /note >}}
Backend Docker Compose config (compose-dev-backend.yml) by default skips running frontend related tests. Frontend Docker Compose config (compose-dev-frontend.yml) by default skips running backend related tests and sets NODE_ENV=development for frontend build.
Backend development
With Docker
Run tests in your IDE, and re-run make rundev each time you want to see how your code changes behave to test them at http://127.0.0.1:8080/web/.
Without Docker
You have to install the latest stable go toolchain to run the backend locally.
In order to have working Remark42 installation you need once to copy frontend static files to ./backend/app/cmd/web directory from master docker image, as it is expected to be where application compiles:
# frontend files
docker pull ghcr.io/umputun/remark42:master
docker create -ti --name remark42files ghcr.io/umputun/remark42:master sh
docker cp remark42files:/srv/web/ ./backend/app/cmd/
docker rm -f remark42files
# fix frontend files to point to the right URL
## Mac version
find -E ./backend/app/cmd/web -regex '.*\.(html|js|mjs)$' -print -exec sed -i '' "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
## Linux version
find ./backend/app/cmd/web -regex '.*\.\(html\|js\|mjs\)$' -print -exec sed -i "s|{% REMARK_URL %}|http://127.0.0.1:8080|g" {} \;
The assets under /web the frontend does not build (privacy.html, markdown-help.html,
400x400.jpeg) come from backend/app/webassets/assets and are embedded in the binary, so the copy
above neither covers them nor needs to. At runtime a file of the same name under web-root /
REMARK_WEB_ROOT is served in preference to the embedded one.
To run backend - cd backend; go run app/main.go server --dbg --secret=12345 --url=http://127.0.0.1:8080 --admin-passwd=password --site=remark. It stars backend service with embedded bolt store on port 8080 with basic auth, allowing to authenticate and run requests directly, like this:
HTTP http://admin:password@127.0.0.1:8080/api/v1/find?site=remark&sort=-active&format=tree&url=http://127.0.0.1:8080
Technical Details
Data stored in boltdb (embedded key/value database) files under STORE_BOLT_PATH. Each site is stored in a separate boltdb file.
To migrate/move Remark42 to another host, these boltdb files must be transferred. Additionally, the avatars directory AVATAR_FS_PATH (default ./var/avatars) and images directory IMAGE_FS_PATH (default ./var/pictures) must be transferred. As an alternative to storing avatars and images in the file system, boltdb can be used by setting AVATAR_TYPE or IMAGE_TYPE to bolt. Files in the IMAGE_FS_STAGING directory can be safely ignored because they are moved to the image store between app restarts.
The automatic backup process runs every 24h and exports all content in JSON-like format to backup-remark-YYYYMMDD.gz.
Authentication implemented with go-pkgz/auth stored in a cookie. It uses HttpOnly, secure cookies.
All heavy REST calls cached internally in LRU cache limited by CACHE_MAX_ITEMS and CACHE_MAX_SIZE with go-pkgz/rest.
User's activity throttled globally (up to 1000 simultaneous requests) and limited locally (per user, usually up to 10 req/sec).
Request timeout set to 60sec.
Admin authentication (--admin-password set) allows to hit Remark42 API without social login and admin privileges. Adds basic-auth for username: admin, password: ${ADMIN_PASSWD}. Enable it only for the initial comment import or for manual backups. Do not leave the server running with admin password set if you don't have an intention to keep creating backups manually!
User can vote for the comment multiple times but only to change the vote. Double voting is not allowed.
User can edit comments in 5 mins (configurable) window after creation.
User ID hashed and prefixed by OAuth provider name to avoid collisions and potential abuse.
All avatars resized and cached locally to prevent rate limiters from OAuth providers, part of go-pkgz/auth functionality.
Images served over HTTP can be proxied to HTTPS (IMAGE_PROXY_HTTP2HTTPS=true) to prevent mixed HTTP/HTTPS.
All images can be proxied and saved locally (IMAGE_PROXY_CACHE_EXTERNAL=true) instead of serving from the original location. Beware, images that are posted with this parameter enabled will be served from proxy even after it is disabled.
Docker build uses publicly available base images.