Greenfield Go multi-tenant IPFS Pinning Service wire-compatible with the
IPFS Pinning Services API spec. Paired 1:1 with Kubo over localhost RPC,
clustered via embedded NATS JetStream, Postgres source-of-truth with
RLS-enforced tenancy, Fiber + huma v2 for the HTTP surface, Authentik
OIDC for session login with kid-rotated HS256 JWT API tokens.
Feature-complete against the 22-milestone build plan, including the
ship-it v1.0 gap items:
* admin CLIs: drain/uncordon, maintenance, mint-token, rotate-key,
prune-denylist, rebalance --dry-run, cache-stats, cluster-presences
* TTL leader election via NATS KV, fence tokens, JetStream dedup
* rebalancer (plan/apply split), reconciler, requeue sweeper
* ristretto caches with NATS-backed cross-node invalidation
(placements live-nodes + token denylist)
* maintenance watchdog for stuck cluster-pause flag
* Prometheus /metrics with CIDR ACL, HTTP/pin/scheduler/cache gauges
* rate limiting: session (10/min) + anonymous global (120/min)
* integration tests: rebalance, refcount multi-org, RLS belt
* goreleaser (tar + deb/rpm/apk + Alpine Docker) targeting Gitea
Stack: Cobra/Viper, Fiber v2 + huma v2, embedded NATS JetStream,
pgx/sqlc/golang-migrate, ristretto, TypeID, prometheus/client_golang,
testcontainers-go.
66 lines
2.0 KiB
Go
66 lines
2.0 KiB
Go
package maintenance
|
|
|
|
import (
|
|
"context"
|
|
"log/slog"
|
|
"time"
|
|
)
|
|
|
|
// WatchdogInterval is how often the watchdog checks the maintenance
|
|
// flag's age. One minute strikes a balance: cluster-maintenance
|
|
// windows are usually measured in hours, so once-per-minute is plenty,
|
|
// and log collectors aggregating every 5 minutes will see at least
|
|
// one warn-line per collection window once the rail is tripped.
|
|
const WatchdogInterval = time.Minute
|
|
|
|
// Watchdog warns (via slog) whenever the cluster-wide maintenance flag
|
|
// has been set longer than maxDuration. It runs until ctx is cancelled.
|
|
//
|
|
// The whole point of the rail is that a "forgotten maintenance on"
|
|
// silently masks real outages (rebalancer and requeue sweeper stay
|
|
// no-op'd). A warning every WatchdogInterval ensures the operator's
|
|
// log collector will surface the situation before enough time passes
|
|
// for it to matter.
|
|
//
|
|
// maxDuration <= 0 disables the watchdog (returns immediately). The
|
|
// default from config is 1h; operators running very long planned
|
|
// upgrades should bump it rather than silencing the warning.
|
|
func Watchdog(ctx context.Context, mgr *Manager, maxDuration time.Duration) {
|
|
if mgr == nil || maxDuration <= 0 {
|
|
return
|
|
}
|
|
t := time.NewTicker(WatchdogInterval)
|
|
defer t.Stop()
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
case <-t.C:
|
|
check(ctx, mgr, maxDuration)
|
|
}
|
|
}
|
|
}
|
|
|
|
func check(ctx context.Context, mgr *Manager, maxDuration time.Duration) {
|
|
flag, on, err := mgr.IsOn(ctx)
|
|
if err != nil {
|
|
// Non-fatal — the KV bucket may be transiently unavailable.
|
|
// The next tick will retry.
|
|
return
|
|
}
|
|
if !on || flag == nil {
|
|
return
|
|
}
|
|
age := time.Since(flag.SetAt)
|
|
if age <= maxDuration {
|
|
return
|
|
}
|
|
slog.Warn("maintenance: cluster-wide flag has been on longer than maxDuration — rebalancer + sweeper are silently paused",
|
|
"age", age.Round(time.Second),
|
|
"max_duration", maxDuration,
|
|
"reason", flag.Reason,
|
|
"set_by", flag.SetBy,
|
|
"set_at", flag.SetAt,
|
|
"expires_at", flag.ExpiresAt)
|
|
}
|