mirrors/remark42
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

move technical part of readme to docs site

Browse Source
This commit is contained in:
Dmitry Verkhoturov
2021-10-21 13:06:40 -05:00
committed by Umputun
parent 3d06a04acd
commit 032f88c61e
3 changed files with 26 additions and 259 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -245
View File
@@ -35,6 +35,8 @@ Comments example:
For admin screenshots see [Admin UI wiki](https://github.com/umputun/remark42/wiki/Admin-UI)
</details>
All remark42 documentation is available [by the link](https://remark42.com/docs/).
#
- [Install](#install)
@@ -657,232 +659,6 @@ npm run dev
Developer build running by `webpack-dev-server` supports devtools for [React](https://github.com/facebook/react-devtools) and
[Redux](https://github.com/zalmoxisus/redux-devtools-extension).
## API
### Authorization
* `GET /auth/{provider}/login?from=http://url&site=site_id&session=1` - perform "social" login with one of [supported providers](#register-oauth2-providers) and redirect to `url`. Presence of `session` (any non-zero value) change the default cookie expiration and makes them session-only
* `GET /auth/logout` - logout
```go
type User struct {
Name string `json:"name"`
ID string `json:"id"`
Picture string `json:"picture"`
Admin bool `json:"admin"`
Blocked bool `json:"block"`
Verified bool `json:"verified"`
PaidSub bool `json:"paid_sub"` // is paid Patreon subscriber
}
```
### Commenting
* `POST /api/v1/comment` - add a comment, _auth required_
```go
type Comment struct {
ID string `json:"id"` // comment ID, read only
ParentID string `json:"pid"` // parent ID
Text string `json:"text"` // comment text, after md processing
Orig string `json:"orig"` // original comment text
User User `json:"user"` // user info, read only
Locator Locator `json:"locator"` // post locator
Score int `json:"score"` // comment score, read only
Vote int `json:"vote"` // vote for the current user, -1/1/0
Controversy float64 `json:"controversy,omitempty"` // comment controversy, read only
Timestamp time.Time `json:"time"` // time stamp, read only
Edit *Edit `json:"edit,omitempty" bson:"edit,omitempty"` // pointer to have empty default in JSON response
Pin bool `json:"pin"` // pinned status, read only
Delete bool `json:"delete"` // delete status, read only
PostTitle string `json:"title"` // post title
}
type Locator struct {
SiteID string `json:"site"` // site ID
URL string `json:"url"` // post URL
}
type Edit struct {
Timestamp time.Time `json:"time" bson:"time"`
Summary string `json:"summary"`
}
```
* `POST /api/v1/preview` - preview comment in HTML. Body is `Comment` to render
* `GET /api/v1/find?site=site-id&url=post-url&sort=fld&format=tree|plain` - find all comments for given post
This is the primary call used by UI to show comments for the given post. It can return comments in two formats - `plain` and `tree`. In plain format result will be sorted list of `Comment`. In tree format this is going to be tree-like object with this structure:
```go
type Tree struct {
Nodes []Node `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
}
type Node struct {
Comment store.Comment `json:"comment"`
Replies []Node `json:"replies,omitempty"`
}
```
Sort can be `time`, `active` or `score`. Supported sort order with prefix -/+, i.e. `-time`. For `tree` mode sort will be applied to top-level comments only and all replies are always sorted by time.
* `PUT /api/v1/comment/{id}?site=site-id&url=post-url` - edit comment, allowed once in `EDIT_TIME` minutes since creation. Body is `EditRequest` JSON
```go
type EditRequest struct {
Text string `json:"text"` // updated text
Summary string `json:"summary"` // optional, summary of the edit
Delete bool `json:"delete"` // delete flag
}{}
```
* `GET /api/v1/last/{max}?site=site-id&since=ts-msec` - get up to `{max}` last comments, `since` (epoch time, milliseconds) is optional
* `GET /api/v1/id/{id}?site=site-id` - get comment by `comment id`
* `GET /api/v1/comments?site=site-id&user=id&limit=N` - get comment by `user id`, returns `response` object
```go
type response struct {
Comments []store.Comment `json:"comments"`
Count int `json:"count"`
}{}
```
* `GET /api/v1/count?site=site-id&url=post-url` - get comment's count for `{url}`
* `POST /api/v1/count?site=siteID` - get number of comments for posts from post body (list of post IDs)
* `GET /api/v1/list?site=site-id&limit=5&skip=2` - list commented posts, returns array or `PostInfo`, limit=0 will return all posts
```go
type PostInfo struct {
URL string `json:"url"`
Count int `json:"count"`
ReadOnly bool `json:"read_only,omitempty"`
FirstTS time.Time `json:"first_time,omitempty"`
LastTS time.Time `json:"last_time,omitempty"`
}
```
* `GET /api/v1/user` - get user info, _auth required_
* `PUT /api/v1/vote/{id}?site=site-id&url=post-url&vote=1` - vote for comment. `vote`=1 will increase score, -1 decrease, _auth required_
* `GET /api/v1/userdata?site=site-id` - export all user data to gz stream, _auth required_
* `POST /api/v1/deleteme?site=site-id` - request deletion of user data, _auth required_
* `GET /api/v1/config?site=site-id` - returns configuration (parameters) for given site
```go
type Config struct {
Version string `json:"version"`
EditDuration int `json:"edit_duration"`
MaxCommentSize int `json:"max_comment_size"`
Admins []string `json:"admins"`
AdminEmail string `json:"admin_email"`
Auth []string `json:"auth_providers"`
LowScore int `json:"low_score"`
CriticalScore int `json:"critical_score"`
PositiveScore bool `json:"positive_score"`
ReadOnlyAge int `json:"readonly_age"`
MaxImageSize int `json:"max_image_size"`
EmojiEnabled bool `json:"emoji_enabled"`
SubscribersOnly bool `json:"subscribers_only"` // enable commenting only for Patreon subscribers
}
```
* `GET /api/v1/info?site=site-idd&url=post-url` - returns `PostInfo` for site and URL
### Streaming API
Streaming API provides server-sent events for post updates as well as a site update:
* `GET /api/v1/stream/info?site=site-idd&url=post-url&since=unix_ts_msec` - returns stream (`event: info`) with `PostInfo` records for the site and URL. `since` is optional
* `GET /api/v1/stream/last?site=site-id&since=unix_ts_msec` - returns updates stream (`event: last`) with comments for the site, `since` is optional
<details><summary>Response example</summary>
```
data: {"url":"https://radio-t.com/blah1","count":2,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.142872-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":3,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.157709-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":4,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.172991-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":5,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.188429-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":6,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.204742-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":7,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.220692-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":8,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.23817-05:00"}
event: info
data: {"url":"https://radio-t.com/blah1","count":9,"first_time":"2019-06-18T12:53:48.125686-05:00","last_time":"2019-06-18T12:53:48.254669-05:00"}
```
</details>
### RSS feeds
* `GET /api/v1/rss/post?site=site-id&url=post-url` - RSS feed for a post
* `GET /api/v1/rss/site?site=site-id` - RSS feed for given site
* `GET /api/v1/rss/reply?site=site-id&user=user-id` - RSS feed for replies to user's comments
### Images management
* `GET /api/v1/picture/{user}/{id}` - load stored image
* `POST /api/v1/picture` - upload and store image, uses post form with `FormFile("file")`. Returns `{"id": user/imgid}`, _auth required_
_returned ID should be appended to load image URL on caller side_
### Email subscription
* `GET /api/v1/email?site=site-id` - get user's email, _auth required_
* `POST /api/v1/email/subscribe?site=site-id&address=user@example.org` - makes confirmation token and sends it to user over email, _auth required_
Trying to subscribe to the same email a second time will return response code `409 Conflict` and explaining error message
* `POST /api/v1/email/confirm?site=site-id&tkn=token` - uses provided token parameter to set email for the user, _auth required_
Setting email subscribe user for all first-level replies to his messages
* `DELETE /api/v1/email?site=siteID` - removes user's email, _auth required_
### Admin
* `DELETE /api/v1/admin/comment/{id}?site=site-id&url=post-url` - delete comment by `id`
* `PUT /api/v1/admin/user/{userid}?site=site-id&block=1&ttl=7d` - block or unblock user with optional TTL (default=permanent)
* `GET api/v1/admin/blocked&site=site-id` - list of blocked user IDs
```go
type BlockedUser struct {
ID string `json:"id"`
Name string `json:"name"`
Until time.Time `json:"time"`
}
```
* `GET /api/v1/admin/export?site=site-id&mode=[stream|file]` - export all comments to JSON stream or gz file
* `POST /api/v1/admin/import?site=site-id` - import comments from the backup, uses post body
* `POST /api/v1/admin/import/form?site=site-id` - import comments from the backup, user post form
* `POST /api/v1/admin/remap?site=site-id` - remap comments to different URLs. Expect list of "from-url new-url" pairs separated by \n. From-url and new-url parts are separated by space. If URLs end with an asterisk (*) it means matching by the prefix. Remap procedure based on export/import chain so make the backup first
```
http://oldsite.com* https://newsite.com*
http://oldsite.com/from-old-page/1 https://newsite.com/to-new-page/1
```
* `GET /api/v1/admin/wait?site=site-id` - wait for completion for any async migration ops (import or remap)
* `PUT /api/v1/admin/pin/{id}?site=site-id&url=post-url&pin=1` - pin or unpin comment
* `GET /api/v1/admin/user/{userid}?site=site-id` - get user's info
* `DELETE /api/v1/admin/user/{userid}?site=site-id` - delete all user's comments
* `PUT /api/v1/admin/readonly?site=site-id&url=post-url&ro=1` - set read-only status
* `PUT /api/v1/admin/verify/{userid}?site=site-id&verified=1` - set verified status
* `GET /api/v1/admin/deleteme?token=token` - process deleteme user's request
_all admin calls require auth and admin privilege_
## Privacy
* Remark42 is trying to be very sensitive to any private or semi-private information.
@@ -897,25 +673,6 @@ _all admin calls require auth and admin privilege_
* Cookie lifespan can be restricted to session-only.
* All potentially sensitive data stored by Remark42 hashed and encrypted.
## Technical details
* Data stored in [boltdb](https://github.com/etcd-io/bbolt) (embedded key/value database) files under `STORE_BOLT_PATH`.
* Each site is stored in a separate boltdb file.
* To migrate/move Remark42 to another host boltdb files as well as avatars directory `AVATAR_FS_PATH` should be transferred. Optionally, boltdb can be used to store avatars as well.
* Automatic backup process runs every 24h and exports all content in JSON-like format to `backup-remark-YYYYMMDD.gz`.
* Authentication implemented with [go-pkgz/auth](https://github.com/go-pkgz/auth) stored in a cookie. It uses HttpOnly, secure cookies.
* All heavy REST calls cached internally in LRU cache limited by `CACHE_MAX_ITEMS` and `CACHE_MAX_SIZE` with [go-pkgz/rest](https://github.com/go-pkgz/rest).
* User's activity throttled globally (up to 1000 simultaneous requests) and limited locally (per user, usually up to 10 req/sec).
* Request timeout set to 60sec.
* Admin authentication (`--admin-password` set) allows to hit Remark42 API without social login and with admin privileges. Adds basic-auth for username: `admin`, password: `${ADMIN_PASSWD}`.
* User can vote for the comment multiple times but only to change the vote. Double voting is not allowed.
* User can edit comments in 5 mins (configurable) window after creation.
* User ID hashed and prefixed by OAuth provider name to avoid collisions and potential abuse.
* All avatars resized and cached locally to prevent rate limiters from OAuth providers, part of [go-pkgz/auth](https://github.com/go-pkgz/auth) functionality.
* Images served over HTTP can be proxied to HTTPS (`IMAGE_PROXY_HTTP2HTTPS=true`) to prevent mixed HTTP/HTTPS.
* All images can be proxied and saved locally (`IMAGE_PROXY_CACHE_EXTERNAL=true`) instead of serving from the original location. Beware, images that are posted with this parameter enabled will be served from proxy even after it will be disabled.
* Docker build uses [publicly available](https://github.com/umputun/baseimage) base images.
## Related projects
* [A Helm chart for Remark42 on Kubernetes](https://github.com/groundhog2k/helm-charts/tree/master/charts/remark42)
site/src/docs/contributing/api/index.md
+5 -1
View File
@@ -134,11 +134,15 @@ type Config struct {
## Streaming API
Streaming API provides server-sent events for post updates as well as a site update:
<details><summary>Not available</summary>
Streaming API supposed to provide server-sent events for post updates as well as a site update:
- `GET /api/v1/stream/info?site=site-idd&url=post-url&since=unix_ts_msec` - returns stream (`event: info`) with `PostInfo` records for the site and URL. `since` is optional
- `GET /api/v1/stream/last?site=site-id&since=unix_ts_msec` - returns updates stream (`event: last`) with comments for the site, `since` is optional
It was removed in https://github.com/umputun/remark42/pull/826 due to not being used and affecting tests flakiness and could be returned if there will be a developer who would be willing to write frontend support for it.
</details>
<details><summary>Response example</summary>
```
site/src/docs/contributing/technical-details/index.md
+19 -13
View File
@@ -2,26 +2,32 @@
title: Technical Details
---
Remark42 uses boltdb (embedded key/value database) files under `STORE_BOLT_PATH` for storing data. Each site stored in a separate boltbd file.
Data stored in [boltdb](https://github.com/etcd-io/bbolt) (embedded key/value database) files under `STORE_BOLT_PATH`. Each site is stored in a separate boltdb file.
In order to migrate/move Remark42 to another host boltbd files as well as avatars directory `AVATAR_FS_PATH` should be transferred. Optionally, boltdb can be used to store avatars as well.
To migrate/move Remark42 to another host boltdb files as well as avatars directory `AVATAR_FS_PATH` should be transferred. Optionally, boltdb can be used to store avatars as well.
Automatic backup process runs every 24 hours and exports all content in json-like format to `backup-remark-YYYYMMDD.gz`.
Automatic backup process runs every 24h and exports all content in JSON-like format to `backup-remark-YYYYMMDD.gz`.
Authentication implemented with `go-pkgz/auth` stored in a cookie. It uses HttpOnly, secure cookies.
Authentication implemented with [go-pkgz/auth](https://github.com/go-pkgz/auth) stored in a cookie. It uses HttpOnly, secure cookies.
All heavy REST calls cached internally in LRU cache limited by `CACHE_MAX_ITEMS` and `CACHE_MAX_SIZE` with `go-pkgz/rest`
All heavy REST calls cached internally in LRU cache limited by `CACHE_MAX_ITEMS` and `CACHE_MAX_SIZE` with [go-pkgz/rest](https://github.com/go-pkgz/rest).
User's activity throttled globally (up to 1000 simultaneous requests) and limited locally (per user, usually up to 10 req/sec). Request timeout set to 60 seconds.
User's activity throttled globally (up to 1000 simultaneous requests) and limited locally (per user, usually up to 10 req/sec).
Admin authentication (`--admin-password` set) allows hitting Remark42 API without social login and with admin privileges. Adds basic-auth for username: admin, password: `${ADMIN_PASSWD}`.
Request timeout set to 60sec.
User can vote for the comment multiple times but only to change the vote. Double-voting not allowed.
Admin authentication (`--admin-password` set) allows to hit Remark42 API without social login and with admin privileges. Adds basic-auth for username: `admin`, password: `${ADMIN_PASSWD}`.
User can edit comments in 5 minutes (configurable) window after creation.
User ID hashed and prefixed by oauth provider name to avoid collisions and potential abuse.
User can vote for the comment multiple times but only to change the vote. Double voting is not allowed.
All avatars resized and cached locally to prevent rate limiters from oauth providers, part of `go-pkgz/auth` functionality.
User can edit comments in 5 mins (configurable) window after creation.
Images can be proxied (`IMAGE_PROXY_HTTP2HTTPS=true`) to prevent mixed `http/https`. All images can be proxied and saved (`IMAGE_PROXY_CACHE_EXTERNAL=true`) instead of serving from original location. Beware, images which are posted with this parameter enabled will be served from proxy even after it will be disabled.
Docker build uses publicly available base images.
User ID hashed and prefixed by OAuth provider name to avoid collisions and potential abuse.
All avatars resized and cached locally to prevent rate limiters from OAuth providers, part of [go-pkgz/auth](https://github.com/go-pkgz/auth) functionality.
Images served over HTTP can be proxied to HTTPS (`IMAGE_PROXY_HTTP2HTTPS=true`) to prevent mixed HTTP/HTTPS.
All images can be proxied and saved locally (`IMAGE_PROXY_CACHE_EXTERNAL=true`) instead of serving from the original location. Beware, images that are posted with this parameter enabled will be served from proxy even after it will be disabled.
Docker build uses [publicly available](https://github.com/umputun/baseimage) base images.