mirrors/remark42
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0b651dddd4f0ea2bec74cb2ea99a5dcc05d22fe8
remark42/backend/app/rest/api/migrator_test.go
T
Dmitry VerkhoturovandGitHub 0b651dddd4 Make backend tests wait on conditions instead of durations (#2190) 
* Make backend tests wait on conditions instead of durations

The backend workflow has a long tail of runs that fail once and pass on
a rerun. Every one of them comes down to a test assuming an operation
finishes within some duration rather than waiting for the state it
needs. Three were reproducible and each was reproduced against the old
code before being changed: TestServerAuthHooks minted a token that lived
one second and never tested expiry, so a slow runner turned the first
POST into a 401; TestServerApp_AnonMode saw "connection refused" because
waitForHTTPServerStart returned silently after three seconds and left a
later assertion to fail with something unrelated; TestFsStore_Cleanup
slept 200ms against a 300ms ttl that Cleanup widens to 400ms with its
commit grace, so roughly 100ms of stall collected an image meant to
survive.

Fixed sleeps before asserting on asynchronous work are replaced with
polls on the condition itself, using require.Eventually and
require.EventuallyWithT, and require.Never where the assertion is that
something did not happen. Polling closures assert on the CollectT they
are handed rather than on t, since testify runs them on another
goroutine, and polls that issue HTTP requests stay under the rate limit
on the routes they poll through.

Where a test needs time to have passed, the clock input is pinned
instead: staging ages are stamped with os.Chtimes on both sides of the
cleanup boundary right before each call, which also makes the 100ms
commit grace an exact case rather than something no assertion reaches,
and the RSS tests set store.Comment.Timestamp explicitly rather than
racing the wall clock into the first 100ms of a second so pubDate
matches.

chooseUnusedPort takes a port from the kernel's ephemeral range. Picking
at random out of a fixed 10000-port window let two package binaries,
which go test ./... runs concurrently, land on the same number between
the probe closing and the server binding. The start helpers fail naming
the port they waited on, and the SSL tests wait on the redirect port as
well as the TLS one.

Arbitrary budgets that nothing tests are gone: ten HTTP clients with a
one-second timeout against bolt-backed import and export, the "should
take about 100msec" assertions, and a one-second bound on noticing an
already cancelled context. Shutdown stays bounded at ten seconds so a
hang is still caught.

Two assertions get stronger. TestServerAuthHooks accepted 403 or 401
from a blocked user, an alternative that existed only because the short
token could expire mid-test; it is deterministically 403 now.
TestAdmin_BlockedList asserted two users blocked while one carried the
same 150ms ttl the next step waits to lapse, so the halves raced each
other.

goleak stops reporting the regexp2 clock goroutine, which chroma pulls
in for syntax highlighting and which lives for up to a second after the
last match with a timeout; it ends on its own but a binary finishing
inside that window was reported as leaking, and this suite now finishes
sooner. The ignore for net/http.(*Server).Shutdown goes the other way:
it no longer matches anything, with both packages run fifteen times each
under CPU oversubscription to confirm.

Two gaps the change would otherwise have opened are covered directly
rather than left to the side effects that used to cover them. The
one-second token was the only thing exercising the authenticator's
ClaimsUpd hook on refresh, so TestServerApp_ClaimsUpd now calls the hook
itself and checks admin, blocked, email and restricted-name
impersonation, including the two pass-through cases. Lifting the
open-route limit removed the last incidental exercise of the rate
limiter, so TestRateLimiter drives a burst past the allowance and checks
the refusals and that the limit is per client. Both run without a wall
clock, and both were confirmed to fail when the behaviour they cover is
removed.

Production code is untouched. The two sleeps outside test code, the 429
backoff in cmd/cleanup.go and the submit poll in store/image/image.go,
are left alone: no CI failure implicates them.

Test sleeps drop from 67 to 21, all of them either inside a
testing/synctest bubble or a poll interval. The suite runs in about 22
seconds instead of 46, mostly because
TestPublic_FindCommentsCtrl_ConsistentCount no longer paces a hundred
subtests with an 80ms sleep each to stay under the open route limit. The
300s per-package budget now matches across both workflows, the race_test
target and the documented command, and CLAUDE.md records the convention.

with '#' will be ignored, and an empty message aborts the commit. # #
Date: Sat Aug 22 01:12:31 2026 +0100 # # interactive rebase in progress;
onto 7c312da1 # Last command done (1 command done): # reword deb6cbf1 #
Make backend tests wait on conditions instead of durations # Next
command to do (1 remaining command): # reword 262e6dc2 # Apply go fix
under Go 1.27 # You are currently editing a commit while rebasing branch
'fix/backend-test-flakiness' on '7c312da1'. # # Changes to be committed:
.github/workflows/release.yml # modified: CLAUDE.md # modified: Makefile
modified: backend/_example/memory_store/server/rpc_test.go # modified:
backend/app/cmd/import_test.go # modified:
backend/app/cmd/server_test.go # modified: backend/app/main_test.go #
modified: backend/app/rest/api/admin_test.go # modified:
backend/app/rest/api/middleware_test.go # modified:
backend/app/rest/api/migrator_test.go # modified:
backend/app/rest/api/rest_private_test.go # modified:
backend/app/rest/api/rest_public_test.go # modified:
backend/app/rest/api/rest_test.go # modified:
backend/app/rest/api/rss_test.go # modified:
backend/app/rest/proxy/image_test.go # modified:
backend/app/store/image/fs_store_test.go # modified:
backend/app/store/service/service_test.go # modified:
docs/backlog/api-tests-deadlock-on-macos.md #

* Apply go fix under Go 1.27

Go 1.27 extends go fix with the modernizers, so `go fix ./...` now
rewrites patterns the language has since replaced. Running it across all
three modules produces this: legacy sync/atomic calls on plain integers
become the atomic types (notify.Service.closed, image.Service.term and
submitCount, and several test counters), reverse index loops become
slices.Backward, a Split-then-index becomes strings.Cut, counted loops
become range over an int, and interface{} becomes any in the e2e suite.

The example module needed no changes. The e2e module is behind a build
tag, so it only matches with `go fix -tags e2e ./...`.

One knock-on: prealloc can see the bound of a loop once it is written as
range over an int, so the slice it feeds is now preallocated.

with '#' will be ignored, and an empty message aborts the commit. # #
Date: Sat Aug 22 01:32:09 2026 +0100 # # interactive rebase in progress;
onto 7c312da1 # Last commands done (2 commands done): # reword deb6cbf1
262e6dc2 # Apply go fix under Go 1.27 # No commands remaining. # You are
currently editing a commit while rebasing branch
'fix/backend-test-flakiness' on '7c312da1'. # # Changes to be committed:
backend/app/migrator/native.go # modified: backend/app/notify/notify.go
backend/app/rest/api/rest_private_test.go # modified:
backend/app/store/comment.go # modified:
backend/app/store/image/image.go # modified:
backend/app/store/service/service_test.go # modified:
backend/app/store/service/title_test.go # modified: e2e/e2e_test.go #
modified: e2e/widgets_test.go #
2026-08-21 22:17:44 -05:00

764 lines
34 KiB
Go
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package api
import (
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/umputun/remark42/backend/app/store"
"github.com/umputun/remark42/backend/app/store/service"
)
func TestMigrator_Import(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(`{"version":1} {"id":"2aa0478c-df1b-46b1-b561-03d507cf482c","pid":"","text":"<p>test test #1</p>",
"user":{"name":"developer one","id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com",
"admin":true,"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},
"score":0,"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}
{"id":"83fd97fd-ff64-48d1-9fb7-ca7769c77037","pid":"p1","text":"<p>test test #2</p>","user":{"name":"developer one",
"id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah2"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
assert.NoError(t, err)
resp, err := client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportForm(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(`{"version":1} {"id":"2aa0478c-df1b-46b1-b561-03d507cf482c","pid":"","text":"<p>test test #1</p>",
"user":{"name":"developer one","id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com",
"admin":true,"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},
"score":0,"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}
{"id":"83fd97fd-ff64-48d1-9fb7-ca7769c77037","pid":"p1","text":"<p>test test #2</p>","user":{"name":"developer one",
"id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah2"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)
fileWriter, err := bodyWriter.CreateFormFile("file", "import.json")
require.NoError(t, err)
_, err = io.Copy(fileWriter, r)
require.NoError(t, err)
contentType := bodyWriter.FormDataContentType()
require.NoError(t, bodyWriter.Close())
authts := strings.Replace(ts.URL, "http://", "http://admin:password@", 1)
resp, err := http.Post(authts+"/api/v1/admin/import/form?site=remark42&provider=native", contentType, bodyBuf)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportFromWP(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(strings.ReplaceAll(xmlTestWP, "'", "`"))
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=wordpress", r)
assert.NoError(t, err)
req.Header.Add("Content-Type", "application/xml; charset=utf-8")
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://realmenweardress.es/2010/07/do-you-rp/")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 3, comments.Info.Count)
require.Equal(t, 3, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://realmenweardress.es/2010/07/do-you-rp/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 3, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments), "2 comments with 1 reply")
}
func TestMigrator_ImportFromCommento(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(`{"version":1,"comments":[{"commentHex":"7d77e39fcd813241d6281478cc8f21ab5f807d043c750bc1a936bc23b34fb854",
"domain":"example.com","url":"/blog/post/1","commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"markdown":"Example content","html":"","parentHex":"root","score":0,"state":"approved","creationDate":"2021-03-17T12:09:47.722181Z",
"direction":0,"deleted":false}],"commenters":[{"commenterHex":"a1ac58ed1146bd7fe3feff6a7276f73955c3bfd23cacee00e2e0a7a89b1a8c10",
"email":"somegreatmail@gmail.com","name":"User5276","link":"https://example.com/profile/257","photo":"https://secure.gravatar.com/avatar/8f279626d26175134b0d5c88648172f7",
"provider":"sso:example.com","joinDate":"2021-03-19T19:27:25.954285Z","isModerator":false}]}`)
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=commento", r)
assert.NoError(t, err)
req.Header.Add("Content-Type", "application/json; charset=utf-8")
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/blog/post/1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://example.com/blog/post/1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
}
func TestMigrator_ImportFromCommentoJSON(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r, err := os.Open("testdata/commento.json")
require.NoError(t, err)
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=commento", r)
assert.NoError(t, err)
req.Header.Add("Content-Type", "application/json; charset=utf-8")
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
assert.NoError(t, err)
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
assert.Equal(t, "{\"status\":\"import request accepted\"}\n", string(b))
assert.NoError(t, resp.Body.Close())
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 7, comments.Info.Count)
require.Equal(t, 7, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 7, comments.Info.Count)
require.Equal(t, 5, len(comments.Comments), "five comments with two replies")
}
func TestMigrator_ImportRejected(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(`{"version":1} {"id":"2aa0478c-df1b-46b1-b561-03d507cf482c","pid":"","text":"<p>test test #1</p>",
"user":{"name":"developer one","id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com",
"admin":true,"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},
"score":0,"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}
{"id":"83fd97fd-ff64-48d1-9fb7-ca7769c77037","pid":"p1","text":"<p>test test #2</p>","user":{"name":"developer one",
"id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah2"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native&secret=XYZ", r)
assert.NoError(t, err)
resp, err := client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
}
func TestMigrator_ImportDouble(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
tmpl := `{"id":"%d","pid":"","text":"<p>test test #1</p>","user":{"name":"developer one","id":"dev",
"picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}`
recs := make([]string, 0, 50)
for i := range 50 {
recs = append(recs, fmt.Sprintf(tmpl, i))
}
r := strings.NewReader(`{"version":1}` + strings.Join(recs, "\n")) // reader with 10k records
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
assert.NoError(t, err)
resp, err := client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
client = &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err = http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
assert.NoError(t, err)
resp, err = client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusConflict, resp.StatusCode)
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 50, comments.Info.Count)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://radio-t.com/blah1")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 50, comments.Info.Count)
}
func TestMigrator_ImportWaitExpired(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
tmpl := `{"id":"%d","pid":"","text":"<p>test test #1</p>","user":{"name":"developer one","id":"dev",
"picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}`
nRecs := 50
recs := make([]string, 0, nRecs)
for i := range nRecs {
recs = append(recs, fmt.Sprintf(tmpl, i))
}
r := strings.NewReader(`{"version":1}` + strings.Join(recs, "\n")) // reader with `nRecs` records
client := &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
require.NoError(t, err)
resp, err := client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusAccepted, resp.StatusCode)
client = &http.Client{Timeout: 5 * time.Second}
defer client.CloseIdleConnections()
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/wait?site=remark42&timeout=5ms", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
assert.NoError(t, err)
resp, err = client.Do(req)
assert.NoError(t, err)
assert.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusGatewayTimeout, resp.StatusCode)
waitForMigrationCompletion(t, ts)
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://example.com/example")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
}
func TestMigrator_Export(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
r := strings.NewReader(`{"version":1} {"id":"2aa0478c-df1b-46b1-b561-03d507cf482c","pid":"","text":"<p>test test #1</p>",
"user":{"name":"developer one","id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com",
"admin":true,"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah1"},
"score":0,"votes":{},"time":"2018-04-30T01:37:00.849053725-05:00"}
{"id":"83fd97fd-ff64-48d1-9fb7-ca7769c77037","pid":"p1","text":"<p>test test #2</p>","user":{"name":"developer one",
"id":"dev","picture":"/api/v1/avatar/remark.image","profile":"https://remark42.com","admin":true,
"ip":"ae12fe3b5f129b5cc4cdd2b136b7b7947c4d2741"},"locator":{"site":"remark42","url":"https://radio-t.com/blah2"},"score":0,
"votes":{},"time":"2018-04-30T01:37:00.861387771-05:00"}`)
// import comments first
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/import?site=remark42&provider=native", r)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusAccepted, resp.StatusCode)
waitForMigrationCompletion(t, ts)
// export unknown site is a client error, not internal
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=file&site=test", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
errBody, err := io.ReadAll(resp.Body)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusBadRequest, resp.StatusCode)
require.Equal(t, "application/json", resp.Header.Get("Content-Type"))
assert.Contains(t, string(errBody), `"code":6`) // rest.ErrSiteNotFound, not ErrInternal
assert.Contains(t, string(errBody), `not found`) // error detail names the missing site
// unknown site in stream mode is also a client error
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=stream&site=test", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
errBody, err = io.ReadAll(resp.Body)
require.NoError(t, err)
resp.Body.Close()
require.Equal(t, http.StatusBadRequest, resp.StatusCode)
assert.Contains(t, string(errBody), `"code":6`)
// check file mode
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=file&site=remark42", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode)
require.Equal(t, "application/gzip", resp.Header.Get("Content-Type"))
ungzReader, err := gzip.NewReader(resp.Body)
assert.NoError(t, err)
ungzBody, err := io.ReadAll(ungzReader)
assert.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, 3, strings.Count(string(ungzBody), "\n"))
assert.Equal(t, 2, strings.Count(string(ungzBody), "\"text\""))
t.Logf("%s", string(ungzBody))
// check stream mode
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?mode=stream&site=remark42", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err = client.Do(req)
require.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode)
require.Equal(t, "text/plain; charset=utf-8", resp.Header.Get("Content-Type"))
body, err := io.ReadAll(resp.Body)
assert.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, 3, strings.Count(string(body), "\n"))
assert.Equal(t, 2, strings.Count(string(body), "\"text\""))
t.Logf("%s", string(body))
req, err = http.NewRequest("GET", ts.URL+"/api/v1/admin/export?site=remark42", http.NoBody)
require.NoError(t, err)
resp, err = client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusUnauthorized, resp.StatusCode)
}
func TestMigrator_Remap(t *testing.T) {
ts, srv, teardown := startupT(t)
defer teardown()
// create 2 comments in https://remark42.com/demo/
c1 := store.Comment{Text: "first comment", Timestamp: time.Now(),
Locator: store.Locator{SiteID: "remark42", URL: "https://remark42.com/demo/"}, User: store.User{ID: "u1"}}
_, err := srv.DataService.Create(c1)
require.NoError(t, err)
c2 := store.Comment{Text: "second comment", Timestamp: time.Now(),
Locator: store.Locator{SiteID: "remark42", URL: "https://remark42.com/demo/"}, User: store.User{ID: "u2"}}
_, err = srv.DataService.Create(c2)
require.NoError(t, err)
// create 1 comment in https://remark42.com/demo-another/
c3 := store.Comment{Text: "third comment", Timestamp: time.Now(),
Locator: store.Locator{SiteID: "remark42", URL: "https://remark42.com/demo-another/"}, User: store.User{ID: "u3"}}
_, err = srv.DataService.Create(c3)
require.NoError(t, err)
// set url https://remark42.com/demo-another/ to be readonly
err = srv.DataService.SetMetas("remark42", []service.UserMetaData{}, []service.PostMetaData{{
URL: "https://remark42.com/demo-another/",
ReadOnly: true,
}})
require.NoError(t, err)
// check that comments created as expected
res, code := get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo/")
require.Equal(t, http.StatusOK, code)
comments := commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
// we want remap urls to another domain - www.remark42.com
rules := "https://remark42.com/* https://www.remark42.com/*"
resp, err := post(t, ts.URL+"/api/v1/admin/remap?site=remark42", rules) // auth as admin
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusAccepted, resp.StatusCode)
waitForMigrationCompletion(t, ts)
// after remap finished we should find comments from new urls
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://www.remark42.com/demo/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://www.remark42.com/demo/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 2, comments.Info.Count)
require.Equal(t, 2, len(comments.Comments))
require.False(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://www.remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&format=tree&url=https://www.remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 1, comments.Info.Count)
require.Equal(t, 1, len(comments.Comments))
require.True(t, comments.Info.ReadOnly)
// should find nothing from previous url
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
res, code = get(t, ts.URL+"/api/v1/find?site=remark42&url=https://remark42.com/demo-another/")
require.Equal(t, http.StatusOK, code)
comments = commentsWithInfo{}
err = json.Unmarshal([]byte(res), &comments)
require.NoError(t, err)
require.Equal(t, 0, comments.Info.Count)
require.Equal(t, 0, len(comments.Comments))
}
func TestMigrator_RemapReject(t *testing.T) {
ts, _, teardown := startupT(t)
defer teardown()
// without admin credentials
client := &http.Client{Timeout: waitTimeout}
defer client.CloseIdleConnections()
rules := strings.NewReader(`https://remark42.com/* https://www.remark42.com/*`)
req, err := http.NewRequest("POST", ts.URL+"/api/v1/admin/remap?site=remark42", rules)
require.NoError(t, err)
resp, err := client.Do(req)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusUnauthorized, resp.StatusCode)
}
func waitForMigrationCompletion(t *testing.T, ts *httptest.Server) {
client := &http.Client{Timeout: 10 * time.Second}
defer client.CloseIdleConnections()
req, err := http.NewRequest("GET", ts.URL+"/api/v1/admin/wait?site=remark42", http.NoBody)
require.NoError(t, err)
req.SetBasicAuth("admin", "password")
resp, err := client.Do(req)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
b, err := io.ReadAll(resp.Body)
require.NoError(t, err)
defer resp.Body.Close()
assert.Equal(t, "{\"site_id\":\"remark42\",\"status\":\"completed\"}\n", string(b))
}
var xmlTestWP = `
<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0"
xmlns:excerpt="http://wordpress.org/export/1.2/excerpt/"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:wp="http://wordpress.org/export/1.2/"
>
<channel>
<title>Real Men Wear Dress.es</title>
<link>https://realmenweardress.es</link>
<description>SuperAdmin&#039;s gaming and technological musings</description>
<pubDate>Mon, 23 Jul 2018 10:21:47 +0000</pubDate>
<language>en-US</language>
<wp:wxr_version>1.2</wp:wxr_version>
<wp:base_site_url>https://realmenweardress.es</wp:base_site_url>
<wp:base_blog_url>https://realmenweardress.es</wp:base_blog_url>
<wp:author><wp:author_id>2</wp:author_id><wp:author_login><![CDATA[SuperAdmin]]></wp:author_login><wp:author_email><![CDATA[superadmin@super.eu]]></wp:author_email><wp:author_display_name><![CDATA[SuperAdmin]]></wp:author_display_name><wp:author_first_name><![CDATA[SuperAdmin]]></wp:author_first_name><wp:author_last_name><![CDATA[superadmin]]></wp:author_last_name></wp:author>
<wp:author><wp:author_id>1</wp:author_id><wp:author_login><![CDATA[admin]]></wp:author_login><wp:author_email><![CDATA[superadmin@superadmin.co.uk]]></wp:author_email><wp:author_display_name><![CDATA[admin]]></wp:author_display_name><wp:author_first_name><![CDATA[]]></wp:author_first_name><wp:author_last_name><![CDATA[]]></wp:author_last_name></wp:author>
<wp:category>
<wp:term_id>25</wp:term_id>
<wp:category_nicename><![CDATA[cataclysm]]></wp:category_nicename>
<wp:category_parent><![CDATA[]]></wp:category_parent>
<wp:cat_name><![CDATA[Cataclysm]]></wp:cat_name>
</wp:category>
<wp:tag>
<wp:term_id>39</wp:term_id>
<wp:tag_slug><![CDATA[addons]]></wp:tag_slug>
<wp:tag_name><![CDATA[addons]]></wp:tag_name>
</wp:tag>
<generator>https://wordpress.org/?v=4.8.1</generator>
<item>
<title>Post without comments</title>
<link>https://realmenweardress.es/2010/06/hello-world/screenshot_013110_200413/</link>
<pubDate>Sat, 19 Jun 2010 08:34:13 +0000</pubDate>
<dc:creator><![CDATA[admin]]></dc:creator>
<guid isPermaLink="false">http://realmenweardress.es/wp-content/uploads/2010/06/ScreenShot_013110_200413.jpeg</guid>
<description></description>
<content:encoded><![CDATA[So you can actually fly into the well it appears and if your lucky you stay mounted. I imagine it terrifies the poor rats.]]></content:encoded>
<excerpt:encoded><![CDATA[]]></excerpt:encoded>
<wp:post_id>6</wp:post_id>
<wp:post_date><![CDATA[2010-06-19 08:34:13]]></wp:post_date>
<wp:post_date_gmt><![CDATA[2010-06-19 08:34:13]]></wp:post_date_gmt>
<wp:comment_status><![CDATA[open]]></wp:comment_status>
<wp:ping_status><![CDATA[open]]></wp:ping_status>
<wp:post_name><![CDATA[screenshot_013110_200413]]></wp:post_name>
<wp:status><![CDATA[inherit]]></wp:status>
<wp:post_parent>1</wp:post_parent>
<wp:menu_order>0</wp:menu_order>
<wp:post_type><![CDATA[attachment]]></wp:post_type>
<wp:post_password><![CDATA[]]></wp:post_password>
<wp:is_sticky>0</wp:is_sticky>
<wp:attachment_url><![CDATA[https://realmenweardress.es/wp-content/uploads/2010/06/ScreenShot_013110_200413-e1277214413194.jpeg]]></wp:attachment_url>
<wp:postmeta>
<wp:meta_key><![CDATA[_wp_attached_file]]></wp:meta_key>
<wp:meta_value><![CDATA[2010/06/ScreenShot_013110_200413-e1277214413194.jpeg]]></wp:meta_value>
</wp:postmeta>
</item>
<item>
<title>Post with comments. One is not approved</title>
<link>https://realmenweardress.es/2010/07/do-you-rp/</link>
<pubDate>Mon, 19 Jul 2010 14:24:22 +0000</pubDate>
<dc:creator><![CDATA[SuperAdmin]]></dc:creator>
<guid isPermaLink="false">http://realmenweardress.es/?p=100</guid>
<description></description>
<content:encoded><![CDATA[<a href="http://realmenweardress.es/wp-content/uploads/2010/07/ScreenShot_071410_230307-e1279546180886.jpeg"><img class="size-thumbnail wp-image-102 alignleft" title="I need to stand on things else I can't reach" src="http://realmenweardress.es/wp-content/uploads/2010/07/ScreenShot_071410_230307-e1279546270587-120x120.jpg" alt="I need to stand on things else I can't reach" width="120" height="120" /></a>Meet Grokknomel?]]></content:encoded>
<excerpt:encoded><![CDATA[]]></excerpt:encoded>
<wp:post_id>100</wp:post_id>
<wp:post_date><![CDATA[2010-07-19 14:24:22]]></wp:post_date>
<wp:post_date_gmt><![CDATA[2010-07-19 14:24:22]]></wp:post_date_gmt>
<wp:comment_status><![CDATA[open]]></wp:comment_status>
<wp:ping_status><![CDATA[open]]></wp:ping_status>
<wp:post_name><![CDATA[do-you-rp]]></wp:post_name>
<wp:status><![CDATA[publish]]></wp:status>
<wp:post_parent>0</wp:post_parent>
<wp:menu_order>0</wp:menu_order>
<wp:post_type><![CDATA[post]]></wp:post_type>
<wp:post_password><![CDATA[]]></wp:post_password>
<wp:is_sticky>0</wp:is_sticky>
<category domain="post_tag" nicename="alts"><![CDATA[alts]]></category>
<category domain="post_tag" nicename="role-playing"><![CDATA[role playing]]></category>
<category domain="category" nicename="stuff"><![CDATA[Stuff]]></category>
<category domain="post_tag" nicename="weird-in-a-cant-quite-help-myself-way"><![CDATA[weird in a can't quite help myself way]]></category>
<wp:postmeta>
<wp:meta_key><![CDATA[_edit_last]]></wp:meta_key>
<wp:meta_value><![CDATA[2]]></wp:meta_value>
</wp:postmeta>
<wp:comment>
<wp:comment_id>8</wp:comment_id>
<wp:comment_author><![CDATA[SuperUser1]]></wp:comment_author>
<wp:comment_author_email><![CDATA[superuser1@aol.com]]></wp:comment_author_email>
<wp:comment_author_url>http://superuser1.blogspot.com</wp:comment_author_url>
<wp:comment_author_IP><![CDATA[79.141.141.73]]></wp:comment_author_IP>
<wp:comment_date><![CDATA[2010-07-20 12:08:08]]></wp:comment_date>
<wp:comment_date_gmt><![CDATA[2010-07-20 12:08:08]]></wp:comment_date_gmt>
<wp:comment_content><![CDATA[I do catch myself]]></wp:comment_content>
<wp:comment_approved><![CDATA[1]]></wp:comment_approved>
<wp:comment_type><![CDATA[]]></wp:comment_type>
<wp:comment_parent>0</wp:comment_parent>
<wp:comment_user_id>0</wp:comment_user_id>
</wp:comment>
<wp:comment>
<wp:comment_id>9</wp:comment_id>
<wp:comment_author><![CDATA[SuperUser2]]></wp:comment_author>
<wp:comment_author_email><![CDATA[superuser2@gmail.com]]></wp:comment_author_email>
<wp:comment_author_url>http://thewowstorm.wordpress.com</wp:comment_author_url>
<wp:comment_author_IP><![CDATA[97.36.113.1]]></wp:comment_author_IP>
<wp:comment_date><![CDATA[2010-07-20 13:09:25]]></wp:comment_date>
<wp:comment_date_gmt><![CDATA[2010-07-20 13:09:25]]></wp:comment_date_gmt>
<wp:comment_content><![CDATA[I think it us inherent in the game to start seeing your character as a personality]]></wp:comment_content>
<wp:comment_approved><![CDATA[0]]></wp:comment_approved>
<wp:comment_type><![CDATA[]]></wp:comment_type>
<wp:comment_parent>0</wp:comment_parent>
<wp:comment_user_id>0</wp:comment_user_id>
</wp:comment>
<wp:comment>
<wp:comment_id>13</wp:comment_id>
<wp:comment_author><![CDATA[Wednesday Reading &laquo; Cynwise&#039;s Battlefield Manual]]></wp:comment_author>
<wp:comment_author_email><![CDATA[]]></wp:comment_author_email>
<wp:comment_author_url>http://cynwise.wordpress.com/2010/07/21/wednesday-reading-8/</wp:comment_author_url>
<wp:comment_author_IP><![CDATA[74.200.244.101]]></wp:comment_author_IP>
<wp:comment_date><![CDATA[2010-07-21 14:02:08]]></wp:comment_date>
<wp:comment_date_gmt><![CDATA[2010-07-21 14:02:08]]></wp:comment_date_gmt>
<wp:comment_content><![CDATA[[...] I know I&#8217;m a bit loony with my attachment to my bankers.  I&#8217;m glad I&#8217;m not the only one. [...]]]></wp:comment_content>
<wp:comment_approved><![CDATA[1]]></wp:comment_approved>
<wp:comment_type><![CDATA[pingback]]></wp:comment_type>
<wp:comment_parent>0</wp:comment_parent>
<wp:comment_user_id>0</wp:comment_user_id>
</wp:comment>
<wp:comment>
<wp:comment_id>14</wp:comment_id>
<wp:comment_author><![CDATA[SuperUser3]]></wp:comment_author>
<wp:comment_author_email><![CDATA[blablah@gmail.com]]></wp:comment_author_email>
<wp:comment_author_url>http://realmenweardress.es</wp:comment_author_url>
<wp:comment_author_IP><![CDATA[128.243.253.117]]></wp:comment_author_IP>
<wp:comment_date><![CDATA[2010-08-18 15:19:14]]></wp:comment_date>
<wp:comment_date_gmt><![CDATA[2010-08-18 15:19:14]]></wp:comment_date_gmt>
<wp:comment_content><![CDATA[Looks like http://releases.rancher.com/os/latest is no longer hosted - installs using this 'base-url' are failing.
I switched to Github with success:
'''
set base-url https://github.com/rancher/os/releases/download/v1.1.1-rc1
'''
Thanks for the article!]]></wp:comment_content>
<wp:comment_approved><![CDATA[1]]></wp:comment_approved>
<wp:comment_type><![CDATA[]]></wp:comment_type>
<wp:comment_parent>13</wp:comment_parent>
<wp:comment_user_id>2</wp:comment_user_id>
</wp:comment>
</item>
</channel>
</rss>
`
Reference in New Issue View Git Blame Copy Permalink