mirrors/remark42
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feature/ts - read-only support (#57)

Browse Source 
* extend post info with ts and add info implemenation to engine

* info controller

* remove unused

* docker froendly new test

* tree find to include post info struct

* remove leftovers from codeconv

* test for empty tree request

* add func name to error rest report

* test with disabled proxy

* add read only age and reject updates on old posts

* lint: shaddow err

* make info calls fast. needs migration!

* fix backup restore script

* increase limiter timeout

* add local exporter

* migrate list to new info

* fix rest list tests

* add manual backup script

* default backup path

* fix eait on sec change failing rss tests

* add RO status bucket and rest

* add admin RO controller test
This commit is contained in:
Umputun
2018-05-28 14:57:36 -05:00
committed by GitHub
parent 90569b9c62
commit 6aee34bb1e
27 changed files with 948 additions and 301 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.travis.yml
-1
View File
@@ -5,7 +5,6 @@ install:
script:
- docker build
--build-arg COVERALLS_TOKEN=$COVERALLS_TOKEN
--build-arg CODECOV_TOKEN=$CODECOV_TOKEN
--build-arg CI=$CI
--build-arg TRAVIS=$TRAVIS
--build-arg TRAVIS_BRANCH=$TRAVIS_BRANCH
Dockerfile
+3 -1
View File
@@ -50,10 +50,12 @@ WORKDIR /srv
ADD scripts/import-disqus.sh /srv/import-disqus.sh
ADD scripts/restore-backup.sh /srv/restore-backup.sh
ADD scripts/migrate-data.sh /srv/migrate-data.sh
ADD scripts/create-backup.sh /srv/create-backup.sh
ADD start.sh /srv/start.sh
RUN chmod +x /srv/start.sh /srv/import-disqus.sh /srv/restore-backup.sh
RUN chmod +x /srv/start.sh /srv/import-disqus.sh /srv/restore-backup.sh /srv/migrate-data.sh /srv/create-backup.sh
COPY --from=build-backend /go/src/github.com/umputun/remark/remark /srv/
COPY --from=build-frontend /srv/web/public/ /srv/web
README.md
+31 -4
View File
@@ -33,13 +33,14 @@ Remark42 is a self-hosted, lightweight, and simple (yet functional) comment engi
| Command line | Environment | Default | Multi | Description |
| ----------------- | -------------------- | ---------------------- | ----- | --------------------------------------- |
| --url | REMARK_URL | `https://remark42.com` | no | url to remark server |
| --bolt | BOLTDB_PATH | `/tmp` | no | path to data directory |
| --bolt | BOLTDB_PATH | `./var` | no | path to data directory |
| --site | SITE | `remark` | yes | site name(s) |
| --admin | ADMIN | | yes | admin names (list of user ids) |
| --backup | BACKUP_PATH | `/tmp` | no | backups location |
| --backup | BACKUP_PATH | `./var/backup` | no | backups location |
| --max-back | MAX_BACKUP_FILES | `10` | no | max backup files to keep |
| --max-cache-items | MAX_CACHE_ITEMS | `1000` | no | max number of cached items, 0-unlimited |
| --max-cache-value | MAX_CACHE_VALUE | `65536` | no | max size of cached value, o-unlimited |
| --avatars | AVATAR_STORE | `./var/avatars` | no | avatars location |
| --secret | SECRET | | no | secret key, required |
| --max-comment | MAX_COMMENT_SIZE | 2048 | no | comment's size limit |
| --google-cid | REMARK_GOOGLE_CID | | no | Google OAuth client ID |
@@ -104,6 +105,26 @@ _instructions for google oauth2 setup borrowed from [oauth2_proxy](https://githu
2. Move this file to your remark42 host within `.var` and unzip, i.e. `gunzip <disqus-export-name>.xml.gz`.
3. Run import command - `docker-compose exec remark /srv/import-disqus.sh <disqus-export-name>.xml <your site id>`
#### Backup and restore
##### Automatic backups
Remark42 by default makes daily backup files under `${BACKUP_PATH}` (default `./var/backup`). Backups kept up to `${MAX_BACKUP_FILES}` (default 10). Each backup file contains exported and gzipped content, i.e., all comments. At any point, the user can restore such backup and revert all comments to the desirable state. Note: restore procedure cleans the current data store and replaces all comments with comments from the backup file.
For safety and security reasons restore functionality not exposed outside of your server by default. The recommended way to restore from the backup is to use provided `scripts/restore-backup.sh`. It can run inside the container:
`docker-compose exec remark /srv/restore-backup.sh <backup-filename.gz> <your site id>`
##### Schema migration
One special case for backup/restore is schema migration. Some versions or remark42 may extend or change the schema
and for such upgrades migration required. Provided migration script `scripts/migrate-data.sh` makes a fresh backup and then loads it back to your remark42 instance.
`docker-compose exec remark /srv/migrate-data.sh <your site id>`
##### Manual backup
#### Admin users
@@ -284,6 +305,7 @@ In plain format result will be sorted list of `Comment`. In tree format this is
```go
type Tree struct {
Nodes []Node `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
}
type Node struct {
@@ -318,8 +340,11 @@ Sort can be `time`, `active` or `score`. Supported sort order with prefix -/+, i
* `GET /api/v1/list?site=site-id&limit=5&skip=2` - list commented posts, returns array or `PostInfo`, limit=0 will return all posts
```go
type PostInfo struct {
URL string `json:"url"`
Count int `json:"count"`
URL string `json:"url"`
Count int `json:"count"`
ReadOnly bool `json:"read_only,omitempty"`
FirstTS time.Time `json:"first_time,omitempty"`
LastTS time.Time `json:"last_time,omitempty"`
}
```
* `GET /api/v1/user` - get user info, _auth required_
@@ -336,6 +361,7 @@ Sort can be `time`, `active` or `score`. Supported sort order with prefix -/+, i
CriticalScore int `json:"critical_score"`
}
```
* `GET /api/v1/info?site=site-idd&url=post-ur` - returns `PostInfo` for site and url
### RSS feeds
@@ -358,6 +384,7 @@ Sort can be `time`, `active` or `score`. Supported sort order with prefix -/+, i
* `POST /api/v1/admin/import?site=side-id` - import comments from the backup.
* `PUT /api/v1/admin/pin/{id}?site=site-id&url=post-url&pin=1` - pin or unpin comment.
* `DELETE /api/v1/admin/user/{userid}?site=site-id&block=1` - delete all user's comments.
* `PUT /api/v1/admin/readonly?site=site-id&url=post-url&ro=1` - set read-only status
_all admin calls require auth and admin privilege_
app/main.go
+16 -14
View File
@@ -46,15 +46,15 @@ type Opts struct {
SecretKey string `long:"secret" env:"SECRET" required:"true" description:"secret key"`
LowScore int `long:"low-score" env:"LOW_SCORE" default:"-5" description:"low score threshold"`
CriticalScore int `long:"critical-score" env:"CRITICAL_SCORE" default:"-10" description:"critical score threshold"`
GoogleCID string `long:"google-cid" env:"REMARK_GOOGLE_CID" description:"Google OAuth client ID"`
GoogleCSEC string `long:"google-csec" env:"REMARK_GOOGLE_CSEC" description:"Google OAuth client secret"`
GithubCID string `long:"github-cid" env:"REMARK_GITHUB_CID" description:"Github OAuth client ID"`
GithubCSEC string `long:"github-csec" env:"REMARK_GITHUB_CSEC" description:"Github OAuth client secret"`
FacebookCID string `long:"facebook-cid" env:"REMARK_FACEBOOK_CID" description:"Facebook OAuth client ID"`
FacebookCSEC string `long:"facebook-csec" env:"REMARK_FACEBOOK_CSEC" description:"Facebook OAuth client secret"`
DisqusCID string `long:"disqus-cid" env:"REMARK_DISQUS_CID" description:"Disqus OAuth client ID"`
DisqusCSEC string `long:"disqus-csec" env:"REMARK_DISQUS_CSEC" description:"Disqus OAuth client secret"`
ReadOnlyAge int `long:"read-age" env:"READONLY_AGE" default:"0" description:"read-only age of comments"`
GoogleCID string `long:"google-cid" env:"REMARK_GOOGLE_CID" description:"Google OAuth client ID"`
GoogleCSEC string `long:"google-csec" env:"REMARK_GOOGLE_CSEC" description:"Google OAuth client secret"`
GithubCID string `long:"github-cid" env:"REMARK_GITHUB_CID" description:"Github OAuth client ID"`
GithubCSEC string `long:"github-csec" env:"REMARK_GITHUB_CSEC" description:"Github OAuth client secret"`
FacebookCID string `long:"facebook-cid" env:"REMARK_FACEBOOK_CID" description:"Facebook OAuth client ID"`
FacebookCSEC string `long:"facebook-csec" env:"REMARK_FACEBOOK_CSEC" description:"Facebook OAuth client secret"`
DisqusCID string `long:"disqus-cid" env:"REMARK_DISQUS_CID" description:"Disqus OAuth client ID"`
DisqusCSEC string `long:"disqus-csec" env:"REMARK_DISQUS_CSEC" description:"Disqus OAuth client secret"`
Port int `long:"port" env:"REMARK_PORT" default:"8080" description:"port"`
WebRoot string `long:"web-root" env:"REMARK_WEB_ROOT" default:"./web" description:"web root directory"`
@@ -66,7 +66,7 @@ var revision = "unknown"
type Application struct {
Opts
srv *api.Rest
importer *api.Import
migrator *api.Migrator
exporter migrator.Exporter
terminated chan struct{}
}
@@ -127,11 +127,12 @@ func New(opts Opts) (*Application, error) {
exporter := &migrator.Remark{DataStore: &dataService}
importer := &api.Import{
migrator := &api.Migrator{
Version: revision,
Cache: cache,
NativeImporter: &migrator.Remark{DataStore: &dataService},
DisqusImporter: &migrator.Disqus{DataStore: &dataService},
NativeExported: &migrator.Remark{DataStore: &dataService},
SecretKey: opts.SecretKey,
}
@@ -142,6 +143,7 @@ func New(opts Opts) (*Application, error) {
WebRoot: opts.WebRoot,
ImageProxy: &proxy.Image{Enabled: opts.ImageProxy, RoutePath: "/api/v1/img", RemarkURL: opts.RemarkURL},
AvatarProxy: avatarProxy,
ReadOnlyAge: opts.ReadOnlyAge,
Authenticator: auth.Authenticator{
JWTService: jwtService,
Admins: opts.Admins,
@@ -152,7 +154,7 @@ func New(opts Opts) (*Application, error) {
}
srv.ScoreThresholds.Low, srv.ScoreThresholds.Critical = opts.LowScore, opts.CriticalScore
tch := make(chan struct{})
return &Application{srv: srv, importer: importer, exporter: exporter, Opts: opts, terminated: tch}, nil
return &Application{srv: srv, migrator: migrator, exporter: exporter, Opts: opts, terminated: tch}, nil
}
// Run all application objects
@@ -165,10 +167,10 @@ func (a *Application) Run(ctx context.Context) error {
// shutdown on context cancellation
<-ctx.Done()
a.srv.Shutdown()
a.importer.Shutdown()
a.migrator.Shutdown()
}()
a.activateBackup(ctx) // runs in goroutine for each site
go a.importer.Run(a.Port + 1)
go a.migrator.Run(a.Port + 1)
a.srv.Run(a.Port)
close(a.terminated)
return nil
app/rest/api/admin.go
+14
View File
@@ -35,6 +35,7 @@ func (a *admin) routes(middlewares ...func(http.Handler) http.Handler) chi.Route
router.Get("/export", a.exportCtrl)
router.Put("/pin/{id}", a.setPinCtrl)
router.Get("/blocked", a.blockedUsersCtrl)
router.Put("/readonly", a.setReadOnlyCtrl)
return router
}
@@ -96,6 +97,19 @@ func (a *admin) blockedUsersCtrl(w http.ResponseWriter, r *http.Request) {
render.JSON(w, r, users)
}
// PUT /readonly?site=siteID&url=post-url&ro=1 - set or reset read-only status for the post
func (a *admin) setReadOnlyCtrl(w http.ResponseWriter, r *http.Request) {
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
roStatus := r.URL.Query().Get("ro") == "1"
if err := a.dataService.SetReadOnly(locator, roStatus); err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't set readonly status")
return
}
a.cache.Flush(locator.SiteID)
render.JSON(w, r, JSON{"locator": locator, "read-only": roStatus})
}
// PUT /pin/{id}?site=siteID&url=post-url&pin=1
// mark/unmark comment as a special
func (a *admin) setPinCtrl(w http.ResponseWriter, r *http.Request) {
app/rest/api/admin_test.go
+44
View File
@@ -231,6 +231,50 @@ func TestAdmin_BlockedList(t *testing.T) {
assert.Equal(t, "user2", users[1].ID)
}
func TestAdmin_ReadOnly(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
c1 := store.Comment{Text: "test test #1", Locator: store.Locator{SiteID: "radio-t",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user1 name", ID: "user1"}}
c2 := store.Comment{Text: "test test #2", ParentID: "p1", Locator: store.Locator{SiteID: "radio-t",
URL: "https://radio-t.com/blah"}, User: store.User{Name: "user2", ID: "user2"}}
_, err := srv.DataService.Create(c1)
assert.Nil(t, err)
_, err = srv.DataService.Create(c2)
assert.Nil(t, err)
info, err := srv.DataService.Info(store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah"}, 0)
assert.Nil(t, err)
assert.False(t, info.ReadOnly)
client := http.Client{}
// set post to read-only
req, err := http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/readonly?site=radio-t&url=https://radio-t.com/blah&ro=1", ts.URL), nil)
assert.Nil(t, err)
withBasicAuth(req, "dev", "password")
_, err = client.Do(req)
require.Nil(t, err)
info, err = srv.DataService.Info(store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah"}, 0)
assert.Nil(t, err)
assert.True(t, info.ReadOnly)
// resset post's read-only
req, err = http.NewRequest(http.MethodPut,
fmt.Sprintf("%s/api/v1/admin/readonly?site=radio-t&url=https://radio-t.com/blah&ro=0", ts.URL), nil)
assert.Nil(t, err)
withBasicAuth(req, "dev", "password")
_, err = client.Do(req)
require.Nil(t, err)
info, err = srv.DataService.Info(store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah"}, 0)
assert.Nil(t, err)
assert.False(t, info.ReadOnly)
}
func TestAdmin_ExportStream(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
app/rest/api/import.go
-100
View File
@@ -1,100 +0,0 @@
package api
import (
"context"
"fmt"
"log"
"net/http"
"strings"
"sync"
"time"
"github.com/didip/tollbooth"
"github.com/didip/tollbooth_chi"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/go-chi/render"
"github.com/umputun/remark/app/migrator"
"github.com/umputun/remark/app/rest"
"github.com/umputun/remark/app/rest/cache"
)
// Import rest runs on unexposed port and available for local requests only
type Import struct {
Version string
Cache cache.LoadingCache
NativeImporter migrator.Importer
DisqusImporter migrator.Importer
SecretKey string
httpServer *http.Server
lock sync.Mutex
}
// Run the listener and request's router, activate rest server
// this server doesn't have any authentication and SHOULDN'T BE EXPOSED in any way
func (s *Import) Run(port int) {
log.Printf("[INFO] activate import server on port %d", port)
router := s.routes()
s.lock.Lock()
s.httpServer = &http.Server{Addr: fmt.Sprintf("127.0.0.1:%d", port), Handler: router}
s.lock.Unlock()
err := s.httpServer.ListenAndServe()
log.Printf("[WARN] http server terminated, %s", err)
}
// Shutdown import http server
func (s *Import) Shutdown() {
log.Print("[WARN] shutdown import server")
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
s.lock.Lock()
if err := s.httpServer.Shutdown(ctx); err != nil {
log.Printf("[DEBUG] importer shutdown error, %s", err)
}
s.lock.Unlock()
log.Print("[DEBUG] shutdown import server completed")
}
func (s *Import) routes() chi.Router {
router := chi.NewRouter()
router.Use(middleware.RealIP, Recoverer)
router.Use(middleware.Throttle(1000), middleware.Timeout(60*time.Second))
router.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
router.Use(AppInfo("remark42-importer", s.Version), Ping, Logger(LogAll))
router.Post("/api/v1/admin/import", s.importCtrl)
return router
}
// POST /import?secret=key&site=site-id&provider=disqus|remark
// imports comments from post body.
func (s *Import) importCtrl(w http.ResponseWriter, r *http.Request) {
secret := r.URL.Query().Get("secret")
if strings.TrimSpace(secret) == "" || secret != s.SecretKey {
render.Status(r, http.StatusForbidden)
render.JSON(w, r, JSON{"status": "error", "details": "secret key"})
return
}
siteID := r.URL.Query().Get("site")
importer := s.NativeImporter
if r.URL.Query().Get("provider") == "disqus" {
importer = s.DisqusImporter
}
size, err := importer.Import(r.Body, siteID)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "import failed")
return
}
s.Cache.Flush(siteID)
render.Status(r, http.StatusCreated)
render.JSON(w, r, JSON{"status": "ok", "size": size})
}
app/rest/api/migrator.go
+133
View File
@@ -0,0 +1,133 @@
package api
import (
"compress/gzip"
"context"
"fmt"
"log"
"net/http"
"strings"
"sync"
"time"
"github.com/didip/tollbooth"
"github.com/didip/tollbooth_chi"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/go-chi/render"
"github.com/umputun/remark/app/migrator"
"github.com/umputun/remark/app/rest"
"github.com/umputun/remark/app/rest/cache"
)
// Migrator rest runs on unexposed port and available for local requests only
type Migrator struct {
Version string
Cache cache.LoadingCache
NativeImporter migrator.Importer
DisqusImporter migrator.Importer
NativeExported migrator.Exporter
SecretKey string
httpServer *http.Server
lock sync.Mutex
}
// Run the listener and request's router, activate rest server
// this server doesn't have any authentication and SHOULDN'T BE EXPOSED in any way
func (m *Migrator) Run(port int) {
log.Printf("[INFO] activate import server on port %d", port)
router := m.routes()
m.lock.Lock()
m.httpServer = &http.Server{Addr: fmt.Sprintf("127.0.0.1:%d", port), Handler: router}
m.lock.Unlock()
err := m.httpServer.ListenAndServe()
log.Printf("[WARN] http server terminated, %s", err)
}
// Shutdown import http server
func (m *Migrator) Shutdown() {
log.Print("[WARN] shutdown import server")
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
m.lock.Lock()
if err := m.httpServer.Shutdown(ctx); err != nil {
log.Printf("[DEBUG] importer shutdown error, %s", err)
}
m.lock.Unlock()
log.Print("[DEBUG] shutdown import server completed")
}
func (m *Migrator) routes() chi.Router {
router := chi.NewRouter()
router.Use(middleware.RealIP, Recoverer)
router.Use(middleware.Throttle(1000), middleware.Timeout(15*time.Minute))
router.Use(tollbooth_chi.LimitHandler(tollbooth.NewLimiter(10, nil)))
router.Use(AppInfo("remark42-importer", m.Version), Ping, Logger(LogAll))
router.Post("/api/v1/admin/import", m.importCtrl)
router.Get("/api/v1/admin/export", m.exportCtrl)
return router
}
// POST /import?secret=key&site=site-id&provider=disqus|remark
// imports comments from post body.
func (m *Migrator) importCtrl(w http.ResponseWriter, r *http.Request) {
secret := r.URL.Query().Get("secret")
if strings.TrimSpace(secret) == "" || secret != m.SecretKey {
render.Status(r, http.StatusForbidden)
render.JSON(w, r, JSON{"status": "error", "details": "secret key"})
return
}
siteID := r.URL.Query().Get("site")
importer := m.NativeImporter
if r.URL.Query().Get("provider") == "disqus" {
importer = m.DisqusImporter
}
size, err := importer.Import(r.Body, siteID)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "import failed")
return
}
m.Cache.Flush(siteID)
render.Status(r, http.StatusCreated)
render.JSON(w, r, JSON{"status": "ok", "size": size})
}
// GET /export?site=site-id&secret=12345
// exports all comments for siteID as json stream or gz file
func (m *Migrator) exportCtrl(w http.ResponseWriter, r *http.Request) {
secret := r.URL.Query().Get("secret")
if strings.TrimSpace(secret) == "" || secret != m.SecretKey {
render.Status(r, http.StatusForbidden)
render.JSON(w, r, JSON{"status": "error", "details": "secret key"})
return
}
siteID := r.URL.Query().Get("site")
exportFile := fmt.Sprintf("%s-%s.json.gz", siteID, time.Now().Format("20060102"))
w.Header().Set("Content-Type", "application/gzip")
w.Header().Set("Content-Disposition", "attachment;filename="+exportFile)
w.WriteHeader(http.StatusOK)
gzWriter := gzip.NewWriter(w)
defer func() {
if e := gzWriter.Close(); e != nil {
log.Printf("[WARN] can't close gzip writer, %s", e)
}
}()
if _, err := m.NativeExported.Export(gzWriter, siteID); err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "export failed")
return
}
}
app/rest/api/import_test.go → app/rest/api/migrator_test.go
+46 -7
View File
@@ -1,6 +1,7 @@
package api
import (
"compress/gzip"
"io/ioutil"
"net/http"
"net/http/httptest"
@@ -18,7 +19,7 @@ import (
"github.com/umputun/remark/app/store/service"
)
func TestImport(t *testing.T) {
func TestMigrator_Import(t *testing.T) {
srv, ts := prepImportSrv(t)
assert.NotNil(t, srv)
defer cleanupImportSrv(srv, ts)
@@ -38,7 +39,7 @@ func TestImport(t *testing.T) {
assert.Equal(t, `{"size":2,"status":"ok"}`+"\n", string(b))
}
func TestImportRejected(t *testing.T) {
func TestMigrator_ImportRejected(t *testing.T) {
srv, ts := prepImportSrv(t)
assert.NotNil(t, srv)
defer cleanupImportSrv(srv, ts)
@@ -54,8 +55,45 @@ func TestImportRejected(t *testing.T) {
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
}
func TestImportShutdown(t *testing.T) {
srv := Import{}
func TestMigrator_Export(t *testing.T) {
srv, ts := prepImportSrv(t)
assert.NotNil(t, srv)
defer cleanupImportSrv(srv, ts)
r := strings.NewReader(`{"id":"2aa0478c-df1b-46b1-b561-03d507cf482c","pid":"","text":"<p>test test #1</p>","user":{"name":"developer one","id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"radio-t","url":"https://radio-t.com/blah1"},"score":0,"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}
{"id":"83fd97fd-ff64-48d1-9fb7-ca7769c77037","pid":"p1","text":"<p>test test #2</p>","user":{"name":"developer one","id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"radio-t","url":"https://radio-t.com/blah2"},"score":0,"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
client := &http.Client{Timeout: 1 * time.Second}
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=radio-t&provider=native&secret=123456", r)
require.Nil(t, err)
resp, err := client.Do(req)
require.Nil(t, err)
require.Equal(t, http.StatusCreated, resp.StatusCode)
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?site=radio-t&secret=123456", nil)
require.Nil(t, err)
resp, err = client.Do(req)
require.Nil(t, err)
require.Equal(t, 200, resp.StatusCode)
require.Equal(t, "application/gzip", resp.Header.Get("Content-Type"))
ungzReader, err := gzip.NewReader(resp.Body)
assert.NoError(t, err)
ungzBody, err := ioutil.ReadAll(ungzReader)
assert.NoError(t, err)
assert.Equal(t, 2, strings.Count(string(ungzBody), "\n"))
assert.Equal(t, 2, strings.Count(string(ungzBody), "\"text\""))
t.Logf("%s", string(ungzBody))
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?site=radio-t&secret=bad", nil)
require.Nil(t, err)
resp, err = client.Do(req)
require.Nil(t, err)
require.Equal(t, 403, resp.StatusCode)
}
func TestMigrator_Shutdown(t *testing.T) {
srv := Migrator{}
go func() {
time.Sleep(100 * time.Millisecond)
srv.Shutdown()
@@ -65,13 +103,14 @@ func TestImportShutdown(t *testing.T) {
assert.True(t, time.Since(st).Seconds() < 1, "should take about 100ms")
}
func prepImportSrv(t *testing.T) (svc *Import, ts *httptest.Server) {
func prepImportSrv(t *testing.T) (svc *Migrator, ts *httptest.Server) {
b, err := engine.NewBoltDB(bolt.Options{}, engine.BoltSite{FileName: testDb, SiteID: "radio-t"})
require.Nil(t, err)
dataStore := &service.DataStore{Interface: b}
svc = &Import{
svc = &Migrator{
DisqusImporter: &migrator.Disqus{DataStore: dataStore},
NativeImporter: &migrator.Remark{DataStore: dataStore},
NativeExported: &migrator.Remark{DataStore: dataStore},
Cache: &mockCache{},
SecretKey: "123456",
}
@@ -81,7 +120,7 @@ func prepImportSrv(t *testing.T) (svc *Import, ts *httptest.Server) {
return svc, ts
}
func cleanupImportSrv(srv *Import, ts *httptest.Server) {
func cleanupImportSrv(srv *Migrator, ts *httptest.Server) {
ts.Close()
os.Remove(testDb)
}
app/rest/api/rest.go
+41 -10
View File
@@ -33,15 +33,15 @@ import (
// Rest is a rest access server
type Rest struct {
Version string
DataService service.DataStore
Authenticator auth.Authenticator
Exporter migrator.Exporter
Cache cache.LoadingCache
AvatarProxy *proxy.Avatar
ImageProxy *proxy.Image
WebRoot string
Version string
DataService service.DataStore
Authenticator auth.Authenticator
Exporter migrator.Exporter
Cache cache.LoadingCache
AvatarProxy *proxy.Avatar
ImageProxy *proxy.Image
WebRoot string
ReadOnlyAge int
ScoreThresholds struct {
Low int
Critical int
@@ -142,6 +142,8 @@ func (s *Rest) routes() chi.Router {
ropen.Get("/list", s.listCtrl)
ropen.Get("/config", s.configCtrl)
ropen.Post("/preview", s.previewCommentCtrl)
ropen.Get("/info", s.infoCtrl)
ropen.Mount("/rss", s.rssRoutes())
ropen.Mount("/img", s.ImageProxy.Routes())
})
@@ -201,6 +203,13 @@ func (s *Rest) createCommentCtrl(w http.ResponseWriter, r *http.Request) {
return
}
if s.ReadOnlyAge > 0 {
if info, e := s.DataService.Info(comment.Locator, s.ReadOnlyAge); e == nil && info.ReadOnly {
rest.SendErrorJSON(w, r, http.StatusForbidden, errors.New("rejected"), "old post, read-only")
return
}
}
id, err := s.DataService.Create(comment)
if err != nil {
rest.SendErrorJSON(w, r, http.StatusInternalServerError, err, "can't save comment")
@@ -247,6 +256,26 @@ func (s *Rest) previewCommentCtrl(w http.ResponseWriter, r *http.Request) {
render.HTML(w, r, comment.Text)
}
// GET /info?site=siteID&url=post-url - get info about the post
func (s *Rest) infoCtrl(w http.ResponseWriter, r *http.Request) {
locator := store.Locator{SiteID: r.URL.Query().Get("site"), URL: r.URL.Query().Get("url")}
data, err := s.Cache.Get(cache.Key(cache.URLKey(r), locator.SiteID, locator.URL), 4*time.Hour, func() ([]byte, error) {
info, e := s.DataService.Info(locator, s.ReadOnlyAge)
if e != nil {
return nil, e
}
return encodeJSONWithHTML(info)
})
if err != nil {
rest.SendErrorJSON(w, r, http.StatusBadRequest, err, "can't get post info")
return
}
renderJSONFromBytes(w, r, data)
}
// PUT /comment/{id}?site=siteID&url=post-url - update comment
func (s *Rest) updateCommentCtrl(w http.ResponseWriter, r *http.Request) {
@@ -318,7 +347,7 @@ func (s *Rest) findCommentsCtrl(w http.ResponseWriter, r *http.Request) {
var b []byte
switch r.URL.Query().Get("format") {
case "tree":
b, e = encodeJSONWithHTML(rest.MakeTree(maskedComments, sort))
b, e = encodeJSONWithHTML(rest.MakeTree(maskedComments, sort, s.ReadOnlyAge))
default:
b, e = encodeJSONWithHTML(maskedComments)
}
@@ -432,6 +461,7 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
Auth []string `json:"auth_providers"`
LowScore int `json:"low_score"`
CriticalScore int `json:"critical_score"`
ReadOnlyAge int `json:"readonly_age"`
}
cnf := config{
@@ -441,6 +471,7 @@ func (s *Rest) configCtrl(w http.ResponseWriter, r *http.Request) {
Admins: s.Authenticator.Admins,
LowScore: s.ScoreThresholds.Low,
CriticalScore: s.ScoreThresholds.Critical,
ReadOnlyAge: s.ReadOnlyAge,
}
cnf.Auth = []string{}
app/rest/api/rest_test.go
+146 -22
View File
@@ -29,7 +29,7 @@ import (
var testDb = "/tmp/test-remark.db"
var testHTML = "/tmp/test-remark.html"
func TestServer_Ping(t *testing.T) {
func TestRest_Ping(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
defer cleanup(ts)
@@ -39,7 +39,7 @@ func TestServer_Ping(t *testing.T) {
assert.Equal(t, 200, code)
}
func TestServer_Create(t *testing.T) {
func TestRest_Create(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
defer cleanup(ts)
@@ -47,7 +47,7 @@ func TestServer_Create(t *testing.T) {
resp, err := post(t, ts.URL+"/api/v1/comment",
`{"text": "test 123", "locator":{"url": "https://radio-t.com/blah1", "site": "radio-t"}}`)
assert.Nil(t, err)
assert.Equal(t, http.StatusCreated, resp.StatusCode)
require.Equal(t, http.StatusCreated, resp.StatusCode)
b, err := ioutil.ReadAll(resp.Body)
assert.Nil(t, err)
@@ -60,7 +60,41 @@ func TestServer_Create(t *testing.T) {
assert.True(t, len(c["id"].(string)) > 8)
}
func TestServer_CreateTooBig(t *testing.T) {
func TestRest_CreateOldPost(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
defer cleanup(ts)
// make old, but not too old comment
old := store.Comment{Text: "test test old", ParentID: "", Timestamp: time.Now().AddDate(0, 0, -5),
Locator: store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah1"}, User: store.User{ID: "u1"}}
_, err := srv.DataService.Create(old)
assert.Nil(t, err)
comments, err := srv.DataService.Find(store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah1"}, "time")
assert.Nil(t, err)
assert.Equal(t, 1, len(comments))
// try to add new comment to the same old post
resp, err := post(t, ts.URL+"/api/v1/comment",
`{"text": "test 123", "locator":{"site": "radio-t","url": "https://radio-t.com/blah1"}}`)
assert.Nil(t, err)
assert.Equal(t, http.StatusCreated, resp.StatusCode)
assert.Nil(t, srv.DataService.DeleteAll("radio-t"))
// make too old comment
old = store.Comment{Text: "test test old", ParentID: "", Timestamp: time.Now().AddDate(0, 0, -15),
Locator: store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah1"}, User: store.User{ID: "u1"}}
_, err = srv.DataService.Create(old)
assert.Nil(t, err)
resp, err = post(t, ts.URL+"/api/v1/comment",
`{"text": "test 123", "locator":{"site": "radio-t","url": "https://radio-t.com/blah1"}}`)
assert.Nil(t, err)
assert.Equal(t, http.StatusForbidden, resp.StatusCode)
}
func TestRest_CreateTooBig(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
defer cleanup(ts)
@@ -80,7 +114,7 @@ func TestServer_CreateTooBig(t *testing.T) {
assert.Equal(t, "invalid comment", c["details"])
}
func TestServer_Preview(t *testing.T) {
func TestRest_Preview(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
defer cleanup(ts)
@@ -93,7 +127,7 @@ func TestServer_Preview(t *testing.T) {
assert.Equal(t, "<p>test 123</p>\n", string(b))
}
func TestServer_PreviewWithMD(t *testing.T) {
func TestRest_PreviewWithMD(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
defer cleanup(ts)
@@ -102,7 +136,7 @@ func TestServer_PreviewWithMD(t *testing.T) {
# h1
BKT
func TestServer_Preview(t *testing.T) {
func TestRest_Preview(t *testing.T) {
srv, ts := prep(t)
require.NotNil(t, srv)
}
@@ -118,10 +152,10 @@ BKT
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err := ioutil.ReadAll(resp.Body)
assert.Nil(t, err)
assert.Equal(t, "<h1>h1</h1>\n\n<pre><code>func TestServer_Preview(t *testing.T) {\nsrv, ts := prep(t)\n require.NotNil(t, srv)\n}\n</code></pre>\n", string(b))
assert.Equal(t, "<h1>h1</h1>\n\n<pre><code>func TestRest_Preview(t *testing.T) {\nsrv, ts := prep(t)\n require.NotNil(t, srv)\n}\n</code></pre>\n", string(b))
}
func TestServer_CreateAndGet(t *testing.T) {
func TestRest_CreateAndGet(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -153,7 +187,7 @@ func TestServer_CreateAndGet(t *testing.T) {
t.Logf("%+v", comment)
}
func TestServer_Find(t *testing.T) {
func TestRest_Find(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -198,9 +232,44 @@ func TestServer_Find(t *testing.T) {
assert.Nil(t, err)
assert.Equal(t, 1, len(tree.Nodes))
assert.Equal(t, 1, len(tree.Nodes[0].Replies))
assert.Equal(t, 2, tree.Info.Count)
assert.Equal(t, "https://radio-t.com/blah1", tree.Info.URL)
assert.False(t, tree.Info.ReadOnly, "post is fresh")
}
func TestServer_Update(t *testing.T) {
func TestRest_FindAge(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
c1 := store.Comment{Text: "test test #1", ParentID: "", Timestamp: time.Now().AddDate(0, 0, -5),
Locator: store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah1"}, User: store.User{ID: "u1"}}
_, err := srv.DataService.Create(c1)
require.Nil(t, err)
c2 := store.Comment{Text: "test test #2", ParentID: "", Timestamp: time.Now().AddDate(0, 0, -15),
Locator: store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah2"}, User: store.User{ID: "u1"}}
_, err = srv.DataService.Create(c2)
require.Nil(t, err)
tree := rest.Tree{}
res, code := get(t, ts.URL+"/api/v1/find?site=radio-t&url=https://radio-t.com/blah1&format=tree")
assert.Equal(t, 200, code)
err = json.Unmarshal([]byte(res), &tree)
assert.Nil(t, err)
assert.Equal(t, "https://radio-t.com/blah1", tree.Info.URL)
assert.False(t, tree.Info.ReadOnly, "post is fresh")
res, code = get(t, ts.URL+"/api/v1/find?site=radio-t&url=https://radio-t.com/blah2&format=tree")
assert.Equal(t, 200, code)
err = json.Unmarshal([]byte(res), &tree)
assert.Nil(t, err)
assert.Equal(t, "https://radio-t.com/blah2", tree.Info.URL)
assert.True(t, tree.Info.ReadOnly, "post is old")
}
func TestRest_Update(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -239,7 +308,7 @@ func TestServer_Update(t *testing.T) {
assert.Equal(t, c2, c3, "same as response from update")
}
func TestServer_Last(t *testing.T) {
func TestRest_Last(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -268,9 +337,23 @@ func TestServer_Last(t *testing.T) {
err = json.Unmarshal([]byte(res), &comments)
assert.Nil(t, err)
assert.Equal(t, 3, len(comments), "should have 3 comments")
res, code = get(t, ts.URL+"/api/v1/last/X?site=radio-t")
assert.Equal(t, 200, code)
err = json.Unmarshal([]byte(res), &comments)
assert.Nil(t, err)
assert.Equal(t, 3, len(comments), "should have 3 comments")
err = srv.DataService.Delete(store.Locator{SiteID: "radio-t", URL: "https://radio-t.com/blah1"}, id1, store.SoftDelete)
assert.Nil(t, err)
res, code = get(t, ts.URL+"/api/v1/last/5?site=radio-t")
assert.Equal(t, 200, code)
err = json.Unmarshal([]byte(res), &comments)
assert.Nil(t, err)
assert.Equal(t, 2, len(comments), "should have 2 comments")
}
func TestServer_FindUserComments(t *testing.T) {
func TestRest_FindUserComments(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -302,7 +385,7 @@ func TestServer_FindUserComments(t *testing.T) {
assert.Equal(t, 3, resp.Count, "should have 3 count")
}
func TestServer_UserInfo(t *testing.T) {
func TestRest_UserInfo(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -316,7 +399,7 @@ func TestServer_UserInfo(t *testing.T) {
Picture: "/api/v1/avatar/remark.image", Admin: true, Blocked: false, IP: ""}, user)
}
func TestServer_Vote(t *testing.T) {
func TestRest_Vote(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -360,7 +443,7 @@ func TestServer_Vote(t *testing.T) {
assert.Equal(t, map[string]bool{}, cr.Votes)
}
func TestServer_Count(t *testing.T) {
func TestRest_Count(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -390,7 +473,7 @@ func TestServer_Count(t *testing.T) {
assert.Equal(t, 2.0, j["count"])
}
func TestServer_Counts(t *testing.T) {
func TestRest_Counts(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -420,7 +503,7 @@ func TestServer_Counts(t *testing.T) {
{URL: "https://radio-t.com/blah2", Count: 2}}), j)
}
func TestServer_List(t *testing.T) {
func TestRest_List(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -441,10 +524,13 @@ func TestServer_List(t *testing.T) {
pi := []store.PostInfo{}
err := json.Unmarshal([]byte(body), &pi)
assert.Nil(t, err)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/blah2", Count: 2}, {URL: "https://radio-t.com/blah1", Count: 3}}, pi)
assert.Equal(t, "https://radio-t.com/blah2", pi[0].URL)
assert.Equal(t, 2, pi[0].Count)
assert.Equal(t, "https://radio-t.com/blah1", pi[1].URL)
assert.Equal(t, 3, pi[1].Count)
}
func TestServer_Config(t *testing.T) {
func TestRest_Config(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -459,10 +545,47 @@ func TestServer_Config(t *testing.T) {
assert.Equal(t, 4000., j["max_comment_size"])
assert.Equal(t, -5., j["low_score"])
assert.Equal(t, -10., j["critical_score"])
assert.Equal(t, 10., j["readonly_age"])
t.Logf("%+v", j)
}
func TestServer_FileServer(t *testing.T) {
func TestRest_Info(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
user := store.User{ID: "user1", Name: "user name 1"}
c1 := store.Comment{User: user, Text: "test test #1", Locator: store.Locator{SiteID: "radio-t",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 05, 27, 1, 14, 10, 0, time.Local)}
c2 := store.Comment{User: user, Text: "test test #2", ParentID: "p1", Locator: store.Locator{SiteID: "radio-t",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 05, 27, 1, 14, 20, 0, time.Local)}
c3 := store.Comment{User: user, Text: "test test #3", ParentID: "p1", Locator: store.Locator{SiteID: "radio-t",
URL: "https://radio-t.com/blah1"}, Timestamp: time.Date(2018, 05, 27, 1, 14, 25, 0, time.Local)}
_, err := srv.DataService.Create(c1)
require.Nil(t, err, "%+v", err)
_, err = srv.DataService.Create(c2)
require.Nil(t, err)
_, err = srv.DataService.Create(c3)
require.Nil(t, err)
body, code := get(t, ts.URL+"/api/v1/info?site=radio-t&url=https://radio-t.com/blah1")
assert.Equal(t, 200, code)
info := store.PostInfo{}
err = json.Unmarshal([]byte(body), &info)
assert.Nil(t, err)
exp := store.PostInfo{URL: "https://radio-t.com/blah1", Count: 3,
FirstTS: time.Date(2018, 05, 27, 1, 14, 10, 0, time.Local), LastTS: time.Date(2018, 05, 27, 1, 14, 25, 0, time.Local)}
assert.Equal(t, exp, info)
_, code = get(t, ts.URL+"/api/v1/info?site=radio-t&url=https://radio-t.com/blah-no")
assert.Equal(t, 400, code)
_, code = get(t, ts.URL+"/api/v1/info?site=radio-t-no&url=https://radio-t.com/blah-no")
assert.Equal(t, 400, code)
}
func TestRest_FileServer(t *testing.T) {
srv, ts := prep(t)
assert.NotNil(t, srv)
defer cleanup(ts)
@@ -472,7 +595,7 @@ func TestServer_FileServer(t *testing.T) {
assert.Equal(t, "some html", body)
}
func TestServer_Shutdown(t *testing.T) {
func TestRest_Shutdown(t *testing.T) {
srv := Rest{Authenticator: auth.Authenticator{},
AvatarProxy: &proxy.Avatar{StorePath: "/tmp", RoutePath: "/api/v1/avatar"}, ImageProxy: &proxy.Image{}}
go func() {
@@ -500,6 +623,7 @@ func prep(t *testing.T) (srv *Rest, ts *httptest.Server) {
WebRoot: "/tmp",
AvatarProxy: &proxy.Avatar{StorePath: "/tmp", RoutePath: "/api/v1/avatar"},
ImageProxy: &proxy.Image{},
ReadOnlyAge: 10,
}
srv.ScoreThresholds.Low, srv.ScoreThresholds.Critical = -5, -10
app/rest/api/rss_test.go
+5 -5
View File
@@ -16,7 +16,7 @@ func TestServer_RssPost(t *testing.T) {
assert.NotNil(t, srv)
defer cleanup(ts)
waitOnMinChange()
waitOnSecChange()
c1 := store.Comment{
Text: "test 123",
@@ -57,7 +57,7 @@ func TestServer_RssSite(t *testing.T) {
assert.NotNil(t, srv)
defer cleanup(ts)
waitOnMinChange()
waitOnSecChange()
pubDate := time.Now().Format(time.RFC1123Z)
@@ -111,7 +111,7 @@ func TestServer_RssWithReply(t *testing.T) {
assert.NotNil(t, srv)
defer cleanup(ts)
waitOnMinChange()
waitOnSecChange()
pubDate := time.Now().Format(time.RFC1123Z)
@@ -158,9 +158,9 @@ func TestServer_RssWithReply(t *testing.T) {
assert.Equal(t, expected, res)
}
func waitOnMinChange() {
func waitOnSecChange() {
for {
if time.Now().Nanosecond() > 500000000 {
if time.Now().Nanosecond() < 100000000 {
break
}
time.Sleep(10 * time.Nanosecond)
app/rest/httperrors.go
+9 -5
View File
@@ -23,18 +23,22 @@ func errDetailsMsg(r *http.Request, code int, err error, details string) string
if user, e := GetUserInfo(r); e == nil {
uinfoStr = user.Name + "/" + user.ID + " - "
}
q := r.URL.String()
if qun, e := url.QueryUnescape(q); e == nil {
q = qun
}
srcFileInfo := ""
if _, file, line, ok := runtime.Caller(2); ok {
if pc, file, line, ok := runtime.Caller(2); ok {
fnameElems := strings.Split(file, "/")
srcFileInfo = fmt.Sprintf(" [caused by %s:%d]", strings.Join(fnameElems[len(fnameElems)-3:], "/"), line)
funcNameElems := strings.Split(runtime.FuncForPC(pc).Name(), "/")
srcFileInfo = fmt.Sprintf(" [caused by %s:%d %s]", strings.Join(fnameElems[len(fnameElems)-3:], "/"),
line, funcNameElems[len(funcNameElems)-1])
}
return fmt.Sprintf("%s - %v - %d - %s%s - %s%s",
details, err, code, uinfoStr, strings.Split(r.RemoteAddr, ":")[0], q, srcFileInfo)
remoteIP := r.RemoteAddr
if pos := strings.Index(remoteIP, ":"); pos >= 0 {
remoteIP = remoteIP[:pos]
}
return fmt.Sprintf("%s - %v - %d - %s%s - %s%s", details, err, code, uinfoStr, remoteIP, q, srcFileInfo)
}
app/rest/httperrors_test.go
+15 -1
View File
@@ -9,6 +9,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark/app/store"
)
func TestSendErrorJSON(t *testing.T) {
@@ -39,8 +40,21 @@ func TestErrorDetailsMsg(t *testing.T) {
callerFn := func() {
req, err := http.NewRequest("GET", "https://example.com/test?k1=v1&k2=v2", nil)
require.Nil(t, err)
req.RemoteAddr = "1.2.3.4"
msg := errDetailsMsg(req, 500, errors.New("error 500"), "error details 123456")
assert.Equal(t, "error details 123456 - error 500 - 500 - - https://example.com/test?k1=v1&k2=v2 [caused by app/rest/httperrors_test.go:45]", msg)
assert.Equal(t, "error details 123456 - error 500 - 500 - 1.2.3.4 - https://example.com/test?k1=v1&k2=v2 [caused by app/rest/httperrors_test.go:47 rest.TestErrorDetailsMsg]", msg)
}
callerFn()
}
func TestErrorDetailsMsgWithUser(t *testing.T) {
callerFn := func() {
req, err := http.NewRequest("GET", "https://example.com/test?k1=v1&k2=v2", nil)
req.RemoteAddr = "127.0.0.1:1234"
req = SetUserInfo(req, store.User{Name: "test", ID: "id"})
require.Nil(t, err)
msg := errDetailsMsg(req, 500, errors.New("error 500"), "error details 123456")
assert.Equal(t, "error details 123456 - error 500 - 500 - test/id - 127.0.0.1 - https://example.com/test?k1=v1&k2=v2 [caused by app/rest/httperrors_test.go:59 rest.TestErrorDetailsMsgWithUser]", msg)
}
callerFn()
}
app/rest/proxy/image_test.go
+8
View File
@@ -94,6 +94,14 @@ func TestPicture_Convert(t *testing.T) {
r = img.Convert(`<img src="https://radio-t.com/img3.png"/> xyz <img src="http://images.pexels.com/67636/img4.jpeg">`)
assert.Equal(t, `<img src="https://radio-t.com/img3.png"/> xyz <img src="/img?src=aHR0cDovL2ltYWdlcy5wZXhlbHMuY29tLzY3NjM2L2ltZzQuanBlZw==">`, r)
img = Image{Enabled: true, RoutePath: "/img", RemarkURL: "http://example.com"}
r = img.Convert(`<img src="http://radio-t.com/img3.png"/> xyz`)
assert.Equal(t, `<img src="http://radio-t.com/img3.png"/> xyz`, r, "http:// remark url, no proxy")
img = Image{Enabled: false, RoutePath: "/img"}
r = img.Convert(`<img src="http://radio-t.com/img3.png"/> xyz`)
assert.Equal(t, `<img src="http://radio-t.com/img3.png"/> xyz`, r, "disabled, no proxy")
}
func imgHTTPServer(t *testing.T) *httptest.Server {
app/rest/tree.go
+47 -19
View File
@@ -10,25 +10,39 @@ import (
// Tree is formatter making tree from list of comments
type Tree struct {
Nodes []*Node `json:"comments"`
Nodes []*Node `json:"comments"`
Info store.PostInfo `json:"info,omitempty"`
}
// Node is a comment with optional replies
type Node struct {
Comment store.Comment `json:"comment"`
Replies []*Node `json:"replies,omitempty"`
ts time.Time
Comment store.Comment `json:"comment"`
Replies []*Node `json:"replies,omitempty"`
tsModified time.Time
tsCreated time.Time
}
// recurData wraps all fileds used in recursive processing as intermediate results
type recurData struct {
ts time.Time
visible bool
tsModified time.Time
tsCreated time.Time
visible bool
}
// MakeTree gets unsorted list of comments and produces Tree
func MakeTree(comments []store.Comment, sortType string) *Tree {
res := Tree{}
func MakeTree(comments []store.Comment, sortType string, readOnlyAge int) *Tree {
if len(comments) == 0 {
return &Tree{}
}
res := Tree{
Info: store.PostInfo{
URL: comments[0].Locator.URL,
Count: len(comments), // TODO: includes deleted?
FirstTS: comments[0].Timestamp,
LastTS: comments[0].Timestamp,
},
}
topComments := res.filter(comments, "")
res.Nodes = []*Node{}
@@ -36,12 +50,23 @@ func MakeTree(comments []store.Comment, sortType string) *Tree {
node := Node{Comment: rootComment}
rd := recurData{}
commentsTree, t := res.proc(comments, &node, &rd, rootComment.ID)
commentsTree, tsModified, tsCreated := res.proc(comments, &node, &rd, rootComment.ID)
// skip deleted with no sub-comments ar all sub-comments deleted
if rootComment.Deleted && (len(commentsTree.Replies) == 0 || !rd.visible) {
continue
}
commentsTree.ts = t
commentsTree.tsModified, commentsTree.tsCreated = tsModified, tsCreated
if commentsTree.tsCreated.Before(res.Info.FirstTS) {
res.Info.FirstTS = commentsTree.tsCreated
}
if commentsTree.tsModified.After(res.Info.LastTS) {
res.Info.LastTS = commentsTree.tsModified
}
res.Info.ReadOnly = readOnlyAge > 0 && !res.Info.FirstTS.IsZero() &&
res.Info.FirstTS.AddDate(0, 0, readOnlyAge).Before(time.Now())
res.Nodes = append(res.Nodes, commentsTree)
}
@@ -50,16 +75,19 @@ func MakeTree(comments []store.Comment, sortType string) *Tree {
}
// proc makes tree for one top-level comment recursively
func (t *Tree) proc(comments []store.Comment, node *Node, rd *recurData, parentID string) (*Node, time.Time) {
func (t *Tree) proc(comments []store.Comment, node *Node, rd *recurData, parentID string) (*Node, time.Time, time.Time) {
if rd.ts.IsZero() {
rd.ts = node.Comment.Timestamp
if rd.tsModified.IsZero() || rd.tsCreated.IsZero() {
rd.tsModified, rd.tsCreated = node.Comment.Timestamp, node.Comment.Timestamp
}
repComments := t.filter(comments, parentID)
for _, rc := range repComments {
if rc.Timestamp.After(rd.ts) {
rd.ts = rc.Timestamp
if rc.Timestamp.After(rd.tsModified) {
rd.tsModified = rc.Timestamp
}
if rc.Timestamp.Before(rd.tsCreated) {
rd.tsCreated = rc.Timestamp
}
if !rc.Deleted {
rd.visible = true
@@ -72,7 +100,7 @@ func (t *Tree) proc(comments []store.Comment, node *Node, rd *recurData, parentI
sort.Slice(node.Replies, func(i, j int) bool {
return node.Replies[i].Comment.Timestamp.Before(node.Replies[j].Comment.Timestamp)
})
return node, rd.ts
return node, rd.tsModified, rd.tsCreated
}
// filter returns comments for parentID
@@ -87,7 +115,7 @@ func (t *Tree) filter(comments []store.Comment, parentID string) (f []store.Comm
}
// sort list of nodes, i.e. top-level comments
// time sort uses ts from latest reply
// time sort uses tsModified from latest reply
func (t *Tree) sortNodes(sortType string) {
sort.Slice(t.Nodes, func(i, j int) bool {
@@ -100,9 +128,9 @@ func (t *Tree) sortNodes(sortType string) {
case "+active", "-active", "active":
if strings.HasPrefix(sortType, "-") {
return t.Nodes[i].ts.After(t.Nodes[j].ts)
return t.Nodes[i].tsModified.After(t.Nodes[j].tsModified)
}
return t.Nodes[i].ts.Before(t.Nodes[j].ts)
return t.Nodes[i].tsModified.Before(t.Nodes[j].tsModified)
case "+score", "-score", "score":
if strings.HasPrefix(sortType, "-") {
app/rest/tree_test.go
+93 -45
View File
@@ -4,6 +4,8 @@ import (
"bytes"
"encoding/json"
"io/ioutil"
"regexp"
"strings"
"testing"
"time"
@@ -14,36 +16,46 @@ import (
func TestMakeTree(t *testing.T) {
loc := store.Locator{URL: "url", SiteID: "site"}
ts := func(min int, sec int) time.Time { return time.Date(2017, 12, 25, 19, min, sec, 0, time.UTC) }
// unsorted by purpose
comments := []store.Comment{
{ID: "14", ParentID: "1", Timestamp: time.Date(2017, 12, 25, 19, 46, 14, 0, time.UTC)},
{ID: "1", Timestamp: time.Date(2017, 12, 25, 19, 46, 1, 0, time.UTC)},
{ID: "2", Timestamp: time.Date(2017, 12, 25, 19, 47, 2, 0, time.UTC)},
{ID: "11", ParentID: "1", Timestamp: time.Date(2017, 12, 25, 19, 46, 11, 0, time.UTC)},
{ID: "13", ParentID: "1", Timestamp: time.Date(2017, 12, 25, 19, 46, 13, 0, time.UTC)},
{ID: "12", ParentID: "1", Timestamp: time.Date(2017, 12, 25, 19, 46, 12, 0, time.UTC)},
{ID: "131", ParentID: "13", Timestamp: time.Date(2017, 12, 25, 19, 46, 31, 0, time.UTC)},
{ID: "132", ParentID: "13", Timestamp: time.Date(2017, 12, 25, 19, 46, 32, 0, time.UTC)},
{ID: "21", ParentID: "2", Timestamp: time.Date(2017, 12, 25, 19, 47, 21, 0, time.UTC)},
{ID: "22", ParentID: "2", Timestamp: time.Date(2017, 12, 25, 19, 47, 22, 0, time.UTC)},
{ID: "4", Timestamp: time.Date(2017, 12, 25, 19, 47, 22, 0, time.UTC)},
{ID: "3", Timestamp: time.Date(2017, 12, 25, 19, 47, 22, 0, time.UTC)},
{ID: "5", Deleted: true},
{ID: "6", Deleted: true},
{ID: "61", ParentID: "6", Deleted: true},
{ID: "62", ParentID: "6", Deleted: true},
{ID: "611", ParentID: "61", Deleted: true},
{Locator: loc, ID: "14", ParentID: "1", Timestamp: ts(46, 14)},
{Locator: loc, ID: "1", Timestamp: ts(46, 1)},
{Locator: loc, ID: "2", Timestamp: ts(47, 2)},
{Locator: loc, ID: "11", ParentID: "1", Timestamp: ts(46, 11)},
{Locator: loc, ID: "13", ParentID: "1", Timestamp: ts(46, 13)},
{Locator: loc, ID: "12", ParentID: "1", Timestamp: ts(46, 12)},
{Locator: loc, ID: "131", ParentID: "13", Timestamp: ts(46, 31)},
{Locator: loc, ID: "132", ParentID: "13", Timestamp: ts(46, 32)},
{Locator: loc, ID: "21", ParentID: "2", Timestamp: ts(47, 21)},
{Locator: loc, ID: "22", ParentID: "2", Timestamp: ts(47, 22)},
{Locator: loc, ID: "4", Timestamp: ts(47, 22)},
{Locator: loc, ID: "3", Timestamp: ts(47, 22)},
{Locator: loc, ID: "5", Deleted: true},
{Locator: loc, ID: "6", Deleted: true},
{Locator: loc, ID: "61", ParentID: "6", Deleted: true},
{Locator: loc, ID: "62", ParentID: "6", Deleted: true},
{Locator: loc, ID: "611", ParentID: "61", Deleted: true},
}
res := MakeTree(comments, "time")
res := MakeTree(comments, "time", 0)
buf := &bytes.Buffer{}
enc := json.NewEncoder(buf)
enc.SetIndent("", " ")
err := enc.Encode(res)
assert.Nil(t, err)
assert.Equal(t, expJSON, buf.String())
// t.Log(string(buf.Bytes()))
expected, actual := cleanFormatting(expJSON, buf.String())
assert.Equal(t, expected, actual)
assert.Equal(t, store.PostInfo{URL: "url", Count: 17, FirstTS: ts(46, 1), LastTS: ts(47, 22)}, res.Info)
res = MakeTree([]store.Comment{}, "time", 0)
assert.Equal(t, &Tree{}, res)
res = MakeTree(comments, "time", 10)
assert.Equal(t, store.PostInfo{URL: "url", Count: 17, FirstTS: ts(46, 1), LastTS: ts(47, 22), ReadOnly: true}, res.Info)
}
func TestTreeSortNodes(t *testing.T) {
@@ -65,33 +77,39 @@ func TestTreeSortNodes(t *testing.T) {
{ID: "5", Deleted: true},
}
res := MakeTree(comments, "+active")
res := MakeTree(comments, "+active", 0)
assert.Equal(t, "2", res.Nodes[0].Comment.ID)
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].ts)
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].tsModified)
res = MakeTree(comments, "-active")
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].ts)
res = MakeTree(comments, "-active", 0)
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].tsModified)
assert.Equal(t, "1", res.Nodes[0].Comment.ID)
res = MakeTree(comments, "+time")
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].ts)
res = MakeTree(comments, "-time")
res = MakeTree(comments, "+time", 0)
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].tsModified)
assert.Equal(t, "1", res.Nodes[0].Comment.ID)
res = MakeTree(comments, "-time", 0)
assert.Equal(t, "6", res.Nodes[0].Comment.ID)
res = MakeTree(comments, "score")
res = MakeTree(comments, "score", 0)
assert.Equal(t, "4", res.Nodes[0].Comment.ID)
assert.Equal(t, "3", res.Nodes[1].Comment.ID)
assert.Equal(t, "6", res.Nodes[2].Comment.ID)
assert.Equal(t, "1", res.Nodes[3].Comment.ID)
res = MakeTree(comments, "+score")
res = MakeTree(comments, "+score", 0)
assert.Equal(t, "4", res.Nodes[0].Comment.ID)
res = MakeTree(comments, "-score")
res = MakeTree(comments, "-score", 0)
assert.Equal(t, "2", res.Nodes[0].Comment.ID)
assert.Equal(t, "1", res.Nodes[1].Comment.ID)
assert.Equal(t, "3", res.Nodes[2].Comment.ID)
assert.Equal(t, "6", res.Nodes[3].Comment.ID)
res = MakeTree(comments, "undefined", 0)
t.Log(res.Nodes[0].Comment.ID, res.Nodes[0].tsModified)
assert.Equal(t, "1", res.Nodes[0].Comment.ID)
}
func BenchmarkTree(b *testing.B) {
@@ -102,7 +120,7 @@ func BenchmarkTree(b *testing.B) {
assert.Nil(b, err)
for i := 0; i < b.N; i++ {
res := MakeTree(comments, "time")
res := MakeTree(comments, "time", 0)
assert.NotNil(b, res)
}
}
@@ -120,8 +138,9 @@ const expJSON = `{
"picture": "",
"admin": false
},
"locator": {
"url": ""
"locator": {
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -140,7 +159,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -159,7 +179,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -178,7 +199,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -197,7 +219,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -216,7 +239,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -237,7 +261,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -258,7 +283,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -277,7 +303,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -296,7 +323,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -317,7 +345,8 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
@@ -336,13 +365,32 @@ const expJSON = `{
"admin": false
},
"locator": {
"url": ""
"site": "site",
"url": "url"
},
"score": 0,
"votes": null,
"time": "2017-12-25T19:47:22Z"
}
}
]
],
"info": {
"url": "url",
"count": 17,
"first_time": "2017-12-25T19:46:01Z",
"last_time": "2017-12-25T19:47:22Z"
}
}
`
func cleanFormatting(expected, actual string) (string, string) {
reSpaces := regexp.MustCompile(`[\s\p{Zs}]{2,}`)
expected = strings.Replace(expected, "\n", " ", -1)
expected = strings.Replace(expected, "\t", " ", -1)
expected = reSpaces.ReplaceAllString(expected, " ")
actual = strings.Replace(actual, "\n", " ", -1)
actual = reSpaces.ReplaceAllString(actual, " ")
return expected, actual
}
app/store/comment.go
+5 -2
View File
@@ -38,8 +38,11 @@ type Edit struct {
// PostInfo holds summary for given post url
type PostInfo struct {
URL string `json:"url"`
Count int `json:"count"`
URL string `json:"url"`
Count int `json:"count"`
ReadOnly bool `json:"read_only,omitempty"`
FirstTS time.Time `json:"first_time,omitempty"`
LastTS time.Time `json:"last_time,omitempty"`
}
// BlockedUser holds id and ts for blocked user
app/store/engine/bolt_accessor.go
+89 -46
View File
@@ -4,8 +4,8 @@ import (
"encoding/json"
"fmt"
"log"
"strconv"
"strings"
"time"
"github.com/coreos/bbolt"
"github.com/pkg/errors"
@@ -22,17 +22,19 @@ import (
// is a nested bucket named userID with kv as ts:reference
// - blocking info sits in "block" bucket. Key is userID, value - ts
// - counts per post to keep number of comments. Key is post url, value - count
// - readonly per post to keep status of manually set RO posts. Key is post url, value - ts
type BoltDB struct {
dbs map[string]*bolt.DB
}
const (
// top level buckets
postsBucketName = "posts"
lastBucketName = "last"
userBucketName = "users"
blocksBucketName = "block"
countsBucketName = "counts"
postsBucketName = "posts"
lastBucketName = "last"
userBucketName = "users"
blocksBucketName = "block"
infoBucketName = "info"
readonlyBucketName = "readonly"
// limits
lastLimit = 1000
@@ -59,7 +61,8 @@ func NewBoltDB(options bolt.Options, sites ...BoltSite) (*BoltDB, error) {
}
// make top-level buckets
topBuckets := []string{postsBucketName, lastBucketName, userBucketName, blocksBucketName, countsBucketName}
topBuckets := []string{postsBucketName, lastBucketName, userBucketName, blocksBucketName,
infoBucketName, readonlyBucketName}
err = db.Update(func(tx *bolt.Tx) error {
for _, bktName := range topBuckets {
if _, e := tx.CreateBucketIfNotExists([]byte(bktName)); e != nil {
@@ -85,6 +88,11 @@ func (b *BoltDB) Create(comment store.Comment) (commentID string, err error) {
if err != nil {
return "", err
}
if b.IsReadOnly(comment.Locator) {
return "", errors.Errorf("post %s is read-only", comment.Locator.URL)
}
err = bdb.Update(func(tx *bolt.Tx) error {
postBkt, e := b.makePostBucket(tx, comment.Locator.URL)
@@ -122,11 +130,10 @@ func (b *BoltDB) Create(comment store.Comment) (commentID string, err error) {
return errors.Wrapf(e, "failed to put user comment %s for %s", comment.ID, comment.User.ID)
}
// increment comments count for post url
if _, e = b.count(tx, comment.Locator.URL, 1); e != nil {
return errors.Wrapf(e, "failed to increment count for %s", comment.Locator)
// set info with countfor post url
if _, e = b.setInfo(tx, comment); e != nil {
return errors.Wrapf(e, "failed to set info for %s", comment.Locator)
}
return nil
})
@@ -188,8 +195,8 @@ func (b *BoltDB) Last(siteID string, max int) (comments []store.Comment, err err
return e
}
comment, e := b.load(postBkt, []byte(commentID))
if e != nil {
comment := store.Comment{}
if e := b.load(postBkt, []byte(commentID), &comment); e != nil {
log.Printf("[WARN] can't load comment for %s from store %s", commentID, url)
continue
}
@@ -244,11 +251,12 @@ func (b BoltDB) List(siteID string, limit, skip int) (list []store.PostInfo, err
continue
}
postURL := string(k)
count, e := b.count(tx, postURL, 0)
if e != nil {
return e
infoBkt := tx.Bucket([]byte(infoBucketName))
info := store.PostInfo{}
if e := b.load(infoBkt, []byte(postURL), &info); e != nil {
return errors.Wrapf(e, "can't load info for %s", postURL)
}
list = append(list, store.PostInfo{URL: postURL, Count: count})
list = append(list, info)
if limit > 0 && len(list) >= limit {
break
}
@@ -259,6 +267,30 @@ func (b BoltDB) List(siteID string, limit, skip int) (list []store.PostInfo, err
return list, err
}
// Info returns time range and count for locator
func (b *BoltDB) Info(locator store.Locator, readOnlyAge int) (store.PostInfo, error) {
bdb, err := b.db(locator.SiteID)
if err != nil {
return store.PostInfo{}, err
}
info := store.PostInfo{}
err = bdb.View(func(tx *bolt.Tx) error {
infoBkt := tx.Bucket([]byte(infoBucketName))
if e := b.load(infoBkt, []byte(locator.URL), &info); e != nil {
return errors.Wrapf(e, "can't load info for %s", locator.URL)
}
return nil
})
// set read-only from age and manual bucket
info.ReadOnly = readOnlyAge > 0 && !info.FirstTS.IsZero() && info.FirstTS.AddDate(0, 0, readOnlyAge).Before(time.Now())
if b.IsReadOnly(locator) {
info.ReadOnly = true
}
return info, err
}
// User extracts all comments for given site and given userID
// "users" bucket has sub-bucket for each userID, and keeps it as ts:ref
func (b *BoltDB) User(siteID string, userID string, limit int) (comments []store.Comment, totalComments int, err error) {
@@ -325,8 +357,7 @@ func (b *BoltDB) Get(locator store.Locator, commentID string) (comment store.Com
if e != nil {
return e
}
comment, e = b.load(bucket, []byte(commentID))
return e
return b.load(bucket, []byte(commentID), &comment)
})
return comment, err
}
@@ -391,54 +422,66 @@ func (b *BoltDB) getUserBucket(tx *bolt.Tx, userID string) (*bolt.Bucket, error)
return userIDBkt, nil
}
// save comment to key for bucket. Should run in update tx
func (b *BoltDB) save(bkt *bolt.Bucket, key []byte, comment store.Comment) (err error) {
jdata, jerr := json.Marshal(&comment)
// save marshaled value to key for bucket. Should run in update tx
func (b *BoltDB) save(bkt *bolt.Bucket, key []byte, value interface{}) (err error) {
if value == nil {
return errors.Errorf("can't save nil value for %s", key)
}
jdata, jerr := json.Marshal(value)
if jerr != nil {
return errors.Wrap(jerr, "can't marshal comment")
}
if err = bkt.Put([]byte(comment.ID), jdata); err != nil {
if err = bkt.Put(key, jdata); err != nil {
return errors.Wrapf(err, "failed to save key %s", key)
}
return nil
}
// load comment by key from bucket. Should run in view tx
func (b *BoltDB) load(bkt *bolt.Bucket, key []byte) (comment store.Comment, err error) {
commentVal := bkt.Get(key)
if commentVal == nil {
return comment, errors.Errorf("no comments for %s", key)
// load and unmarshal json value by key from bucket. Should run in view tx
func (b *BoltDB) load(bkt *bolt.Bucket, key []byte, res interface{}) error {
value := bkt.Get(key)
if value == nil {
return errors.Errorf("no value for %s", key)
}
if err = json.Unmarshal(commentVal, &comment); err != nil {
return comment, errors.Wrap(err, "failed to unmarshal")
if err := json.Unmarshal(value, &res); err != nil {
return errors.Wrap(err, "failed to unmarshal")
}
return comment, nil
return nil
}
// count adds val to counts key postURL. val can be negative to subtract. if val 0 can be used as accessor
// it uses separate counts bucket because boltdb Stat call is very slow
func (b *BoltDB) count(tx *bolt.Tx, postURL string, val int) (int, error) {
btoi := func(v []byte) int {
res, _ := strconv.Atoi(string(v))
return res
}
infoBkt := tx.Bucket([]byte(infoBucketName))
itob := func(v int) []byte {
return []byte(strconv.Itoa(v))
}
countBkt := tx.Bucket([]byte(countsBucketName))
countVal := countBkt.Get([]byte(postURL))
if countVal == nil {
countVal = itob(0)
info := store.PostInfo{}
if err := b.load(infoBkt, []byte(postURL), &info); err != nil {
info = store.PostInfo{}
}
if val == 0 { // get current count, don't update
return btoi(countVal), nil
return info.Count, nil
}
updatedCount := btoi(countVal) + val
return updatedCount, countBkt.Put([]byte(postURL), itob(updatedCount))
info.Count += val
return info.Count, b.save(infoBkt, []byte(postURL), &info)
}
func (b *BoltDB) setInfo(tx *bolt.Tx, comment store.Comment) (store.PostInfo, error) {
infoBkt := tx.Bucket([]byte(infoBucketName))
info := store.PostInfo{}
if err := b.load(infoBkt, []byte(comment.Locator.URL), &info); err != nil {
info = store.PostInfo{
Count: 0,
URL: comment.Locator.URL,
FirstTS: comment.Timestamp,
LastTS: comment.Timestamp,
}
}
info.Count++
info.LastTS = comment.Timestamp
return info, b.save(infoBkt, []byte(comment.Locator.URL), &info)
}
func (b *BoltDB) db(siteID string) (*bolt.DB, error) {
app/store/engine/bolt_accessor_test.go
+85 -4
View File
@@ -7,6 +7,7 @@ import (
"github.com/coreos/bbolt"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark/app/store"
)
@@ -32,6 +33,30 @@ func TestBoltDB_CreateAndFind(t *testing.T) {
assert.EqualError(t, err, `site "radio-t-bad" not found`)
}
func TestBoltDB_CreateReadOnly(t *testing.T) {
defer os.Remove(testDb)
var b = prep(t)
comment := store.Comment{
ID: "id-ro",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 22, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com/ro", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
err := b.SetReadOnly(comment.Locator, true)
require.Nil(t, err)
_, err = b.Create(comment)
assert.NotNil(t, err)
assert.Equal(t, "post https://radio-t.com/ro is read-only", err.Error())
err = b.SetReadOnly(comment.Locator, false)
require.Nil(t, err)
_, err = b.Create(comment)
assert.Nil(t, err)
}
func TestBoltDB_Get(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -73,6 +98,9 @@ func TestBoltDB_Put(t *testing.T) {
err = b.Put(store.Locator{URL: "https://radio-t.com", SiteID: "bad"}, comment)
assert.EqualError(t, err, `site "bad" not found`)
err = b.Put(store.Locator{URL: "https://radio-t.com-bad", SiteID: "radio-t"}, comment)
assert.EqualError(t, err, `no bucket https://radio-t.com-bad in store`)
}
func TestBoltDB_Last(t *testing.T) {
@@ -124,26 +152,74 @@ func TestBoltDB_List(t *testing.T) {
_, err := b.Create(comment)
assert.Nil(t, err)
ts := func(sec int) time.Time { return time.Date(2017, 12, 20, 15, 18, sec, 0, time.Local) }
res, err := b.List("radio-t", 0, 0)
assert.Nil(t, err)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/2", Count: 1}, {URL: "https://radio-t.com", Count: 2}}, res)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/2", Count: 1, FirstTS: ts(22), LastTS: ts(22)},
{URL: "https://radio-t.com", Count: 2, FirstTS: ts(22), LastTS: ts(23)}},
res)
res, err = b.List("radio-t", -1, -1)
assert.Nil(t, err)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/2", Count: 1}, {URL: "https://radio-t.com", Count: 2}}, res)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/2", Count: 1, FirstTS: ts(22), LastTS: ts(22)},
{URL: "https://radio-t.com", Count: 2, FirstTS: ts(22), LastTS: ts(23)}}, res)
res, err = b.List("radio-t", 1, 0)
assert.Nil(t, err)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/2", Count: 1}}, res)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com/2", Count: 1, FirstTS: ts(22), LastTS: ts(22)}}, res)
res, err = b.List("radio-t", 1, 1)
assert.Nil(t, err)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com", Count: 2}}, res)
assert.Equal(t, []store.PostInfo{{URL: "https://radio-t.com", Count: 2, FirstTS: ts(22), LastTS: ts(23)}}, res)
res, err = b.List("bad", 1, 1)
assert.EqualError(t, err, `site "bad" not found`)
}
func TestBoltDB_Info(t *testing.T) {
defer os.Remove(testDb)
b := prep(t) // two comments for https://radio-t.com
ts := func(min int) time.Time { return time.Date(2017, 12, 20, 15, 18, min, 0, time.Local) }
// add one more for https://radio-t.com/2
comment := store.Comment{
ID: "12345",
Text: `some text, <a href="http://radio-t.com">link</a>`,
Timestamp: time.Date(2017, 12, 20, 15, 18, 24, 0, time.Local),
Locator: store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"},
User: store.User{ID: "user1", Name: "user name"},
}
_, err := b.Create(comment)
assert.Nil(t, err)
r, err := b.Info(store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"}, 0)
require.Nil(t, err)
assert.Equal(t, store.PostInfo{URL: "https://radio-t.com/2", Count: 1, FirstTS: ts(24), LastTS: ts(24)}, r)
r, err = b.Info(store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"}, 10)
require.Nil(t, err)
assert.Equal(t, store.PostInfo{URL: "https://radio-t.com/2", Count: 1, FirstTS: ts(24), LastTS: ts(24), ReadOnly: true}, r)
r, err = b.Info(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t"}, 0)
require.Nil(t, err)
assert.Equal(t, store.PostInfo{URL: "https://radio-t.com", Count: 2, FirstTS: ts(22), LastTS: ts(23)}, r)
_, err = b.Info(store.Locator{URL: "https://radio-t.com/error", SiteID: "radio-t"}, 0)
require.NotNil(t, err)
_, err = b.Info(store.Locator{URL: "https://radio-t.com", SiteID: "radio-t-error"}, 0)
require.NotNil(t, err)
err = b.SetReadOnly(store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"}, true)
require.Nil(t, err)
r, err = b.Info(store.Locator{URL: "https://radio-t.com/2", SiteID: "radio-t"}, 0)
require.Nil(t, err)
assert.Equal(t, store.PostInfo{URL: "https://radio-t.com/2", Count: 1, FirstTS: ts(24), LastTS: ts(24), ReadOnly: true}, r)
}
func TestBoltDB_GetForUser(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -188,6 +264,11 @@ func TestBoltDB_Ref(t *testing.T) {
assert.NotNil(t, err)
}
func TestBoltDB_New(t *testing.T) {
_, err := NewBoltDB(bolt.Options{}, BoltSite{FileName: "/tmp/no-such-place/tmp.db", SiteID: "radio-t"})
assert.EqualError(t, err, "failed to make boltdb for /tmp/no-such-place/tmp.db: open /tmp/no-such-place/tmp.db: no such file or directory")
}
// makes new boltdb, put two records
func prep(t *testing.T) *BoltDB {
os.Remove(testDb)
app/store/engine/bolt_admin.go
+42 -3
View File
@@ -28,8 +28,8 @@ func (b *BoltDB) Delete(locator store.Locator, commentID string, mode store.Dele
return e
}
comment, err := b.load(postBkt, []byte(commentID))
if err != nil {
comment := store.Comment{}
if err := b.load(postBkt, []byte(commentID), &comment); err != nil {
return errors.Wrapf(err, "can't load key %s from bucket %s", commentID, locator.URL)
}
// set deleted status and clear fields
@@ -63,7 +63,7 @@ func (b *BoltDB) DeleteAll(siteID string) error {
}
// delete all buckets except blocked users
toDelete := []string{postsBucketName, lastBucketName, userBucketName, countsBucketName}
toDelete := []string{postsBucketName, lastBucketName, userBucketName, infoBucketName}
// delete top-level buckets
err = bdb.Update(func(tx *bolt.Tx) error {
@@ -217,3 +217,42 @@ func (b *BoltDB) Blocked(siteID string) (users []store.BlockedUser, err error) {
return users, err
}
// SetReadOnly makes post read-only or reset the ro flag
func (b *BoltDB) SetReadOnly(locator store.Locator, status bool) error {
bdb, err := b.db(locator.SiteID)
if err != nil {
return err
}
return bdb.Update(func(tx *bolt.Tx) error {
bucket := tx.Bucket([]byte(readonlyBucketName))
switch status {
case true:
if e := bucket.Put([]byte(locator.URL), []byte(time.Now().Format(tsNano))); e != nil {
return errors.Wrapf(e, "failed to set ro for %s to %s", locator.URL, status)
}
case false:
if e := bucket.Delete([]byte(locator.URL)); e != nil {
return errors.Wrapf(e, "failed to clean ro for %s", locator.URL)
}
}
return nil
})
}
// IsReadOnly checks if user blocked
func (b *BoltDB) IsReadOnly(locator store.Locator) (ro bool) {
bdb, err := b.db(locator.SiteID)
if err != nil {
return false
}
_ = bdb.View(func(tx *bolt.Tx) error {
bucket := tx.Bucket([]byte(readonlyBucketName))
ro = bucket.Get([]byte(locator.URL)) != nil
return nil
})
return ro
}
app/store/engine/bolt_admin_test.go
+36 -8
View File
@@ -4,14 +4,13 @@ import (
"os"
"testing"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark/app/store"
)
func TestBoltDB_Delete(t *testing.T) {
func TestBoltAdmin_Delete(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -43,9 +42,13 @@ func TestBoltDB_Delete(t *testing.T) {
loc.SiteID = "bad"
err = b.Delete(loc, res[0].ID, store.SoftDelete)
assert.EqualError(t, err, `site "bad" not found`)
loc = store.Locator{URL: "https://radio-t.com/bad", SiteID: "radio-t"}
err = b.Delete(loc, res[0].ID, store.SoftDelete)
assert.EqualError(t, err, `no bucket https://radio-t.com/bad in store`)
}
func TestBoltDB_DeleteHard(t *testing.T) {
func TestBoltAdmin_DeleteHard(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -65,7 +68,7 @@ func TestBoltDB_DeleteHard(t *testing.T) {
assert.Equal(t, store.User{Name: "deleted", ID: "deleted", Picture: "", Admin: false, Blocked: false, IP: ""}, res[0].User)
}
func TestBoltDB_DeleteAll(t *testing.T) {
func TestBoltAdmin_DeleteAll(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -89,7 +92,7 @@ func TestBoltDB_DeleteAll(t *testing.T) {
assert.EqualError(t, err, `site "bad" not found`)
}
func TestBoltDB_DeleteUser(t *testing.T) {
func TestBoltAdmin_DeleteUser(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
err := b.DeleteUser("radio-t", "user1")
@@ -112,9 +115,12 @@ func TestBoltDB_DeleteUser(t *testing.T) {
comments, err := b.Last("radio-t", 10)
assert.Nil(t, err)
assert.Equal(t, 0, len(comments), "nothing left")
err = b.DeleteUser("radio-t-bad", "user1")
assert.EqualError(t, err, `site "radio-t-bad" not found`)
}
func TestBoltDB_BlockUser(t *testing.T) {
func TestBoltAdmin_BlockUser(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -130,9 +136,11 @@ func TestBoltDB_BlockUser(t *testing.T) {
assert.EqualError(t, b.SetBlock("bad", "user1", true), `site "bad" not found`)
assert.NoError(t, b.SetBlock("radio-t", "userX", false))
assert.False(t, b.IsBlocked("radio-t-bad", "user1"), "nothing blocked on wrong site")
}
func TestBoltDB_BlockList(t *testing.T) {
func TestBoltAdmin_BlockList(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
@@ -151,3 +159,23 @@ func TestBoltDB_BlockList(t *testing.T) {
_, err = b.Blocked("bad")
assert.EqualError(t, err, `site "bad" not found`)
}
func TestBoltAdmin_ReadOnly(t *testing.T) {
defer os.Remove(testDb)
b := prep(t)
assert.False(t, b.IsReadOnly(store.Locator{SiteID: "radio-t", URL: "url-1"}), "nothing ro")
assert.NoError(t, b.SetReadOnly(store.Locator{SiteID: "radio-t", URL: "url-1"}, true))
assert.True(t, b.IsReadOnly(store.Locator{SiteID: "radio-t", URL: "url-1"}), "url-1 ro")
assert.False(t, b.IsReadOnly(store.Locator{SiteID: "radio-t", URL: "url-2"}), "url-2 still writable")
assert.NoError(t, b.SetReadOnly(store.Locator{SiteID: "radio-t", URL: "url-1"}, false))
assert.False(t, b.IsReadOnly(store.Locator{SiteID: "radio-t", URL: "url-1"}), "url-1 writable")
assert.EqualError(t, b.SetReadOnly(store.Locator{SiteID: "bad", URL: "url-1"}, true), `site "bad" not found`)
assert.NoError(t, b.SetReadOnly(store.Locator{SiteID: "radio-t", URL: "url-1xyz"}, false))
assert.False(t, b.IsReadOnly(store.Locator{SiteID: "radio-t-bad", URL: "url-1"}), "nothing blocked on wrong site")
}
app/store/engine/engine.go
+3
View File
@@ -27,6 +27,7 @@ type Accessor interface {
User(siteID string, userID string, limit int) ([]store.Comment, int, error) // comments by user, sorted by time
Count(locator store.Locator) (int, error) // number of comments for the post
List(siteID string, limit int, skip int) ([]store.PostInfo, error) // list of commented posts
Info(locator store.Locator, readonlyAge int) (store.PostInfo, error) // get post info
}
// Admin defines all store ops avail for admin only
@@ -37,6 +38,8 @@ type Admin interface {
SetBlock(siteID string, userID string, status bool) error // block or unblock user
IsBlocked(siteID string, userID string) bool // check if user blocked
Blocked(siteID string) ([]store.BlockedUser, error) // get list of blocked users
SetReadOnly(locator store.Locator, status bool) error // set/reset read-only flag
IsReadOnly(locator store.Locator) bool // check if post read-only
}
// sortComments is for engines can't sort data internally
remark.rest
+5 -3
View File
@@ -1,9 +1,9 @@
### find request with tree
GET {{host}}/api/v1/find?site=remark&sort=-active&format=tree&url=https://remark42.com/demo/
GET {{host}}/api/v1/find?site=radiot&sort=-active&format=tree&url=https://radio-t.com/p/2018/05/05/podcast-596/
### find request with plain
GET {{host}}/api/v1/find?site=remark&sort=-score&format=plain&url=https://remark42.com/demo/
GET {{host}}/api/v1/find?site=radiot&sort=-time&format=plain&url=https://radio-t.com/p/2018/05/08/prep-597/
### last 50 comments
GET {{host}}/api/v1/last/50?site=remark
@@ -84,7 +84,7 @@ Content-Type: application/json
]
### list commented posts
GET {{host}}/api/v1/list?site=remark&limit=10&skip=5
GET {{host}}/api/v1/list?site=radiot&limit=10&skip=5
### get config
GET {{host}}/api/v1/config
@@ -101,6 +101,8 @@ GET {{host}}/api/v1/admin/blocked?site=remark
### delete comment by id
DELETE {{host}}/api/v1/admin/comment/3665976683?site=remark&url=https://remark42.com/demo/
### get post info
GET {{host}}/api/v1/info?site=radiot&url=https://radio-t.com/p/2018/05/08/prep-597/
### post rss
GET {{host}}/api/v1/rss/post?site=remark&url=https://remark42.com/demo/
scripts/create-backup.sh
Executable
+8
View File
@@ -0,0 +1,8 @@
#!/bin/sh
# this scrips makes a backup file to /srv/var/userbackup-<site>-<timestamp>.gz
BACKUP_PATH=${BACKUP_PATH:-./var}
backup_file=${BACKUP_PATH}/userbackup-${1}-$(date +%s).gz
echo "make backup file for site $1 to $backup_file"
curl "http://127.0.0.1:8081/api/v1/admin/export?site=${1}&secret=${SECRET}" > ${backup_file}
scripts/migrate-data.sh
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# this scrips making a backup file to /tmp/export-remark.gz and loading it back
# useful to migrate data schema in case if new version of data store incomaptible with the stored comments.
echo "make backup file for site $1"
curl "http://127.0.0.1:8081/api/v1/admin/export?site=${1}&secret=${SECRET}" > /tmp/export-remark.gz
BOLTDB_PATH=${BOLTDB_PATH:-./var}
BACKUP_PATH=${BACKUP_PATH:-./var}
cp ${BOLTDB_PATH}/${1}.db ${BACKUP_PATH}/${1}-$(date +%s).db
echo "import backup to site $1"
echo "unpack /tmp/export-remark.gz"
gunzip -c /tmp/export-remark.gz >/tmp/backup.remark
ls -laH /tmp/backup.remark
echo "export to site $1"
curl -X POST -H "Content-Type: application/json" --data-binary @/tmp/backup.remark "http://127.0.0.1:8081/api/v1/admin/import?site=${1}&provider=native&secret=${SECRET}"
rm -f /tmp/backup.remark
rm -f /tmp/export-remark.gz
scripts/restore-backup.sh
+1 -1
View File
@@ -7,5 +7,5 @@ gunzip -c /srv/var/backup/$1 >/tmp/backup.remark
size=`stat -c "%s" /tmp/backup.remark`
echo "source file size ${size}"
curl -X POST -H "Content-Type: application/json" -d @/tmp/backup.remark "http://127.0.0.1:8081/api/v1/admin/import?site=${2}&provider=native&secret=${SECRET}"
curl -X POST -H "Content-Type: application/json" --data-binary @/tmp/backup.remark "http://127.0.0.1:8081/api/v1/admin/import?site=${2}&provider=native&secret=${SECRET}"
rm -fq /tmp/backup.remark