3 Commits

Include renames

Author	SHA1	Message	Date
Chris LuandGitHub	82648cca53	test(s3/lifecycle/engine): pin delay-group dedup across buckets (#9418) ... Compile a 100-bucket × 5-rule snapshot where the five Days values include duplicates (1, 1, 7, 7, 30) and assert: - snap.actions has 500 entries — every (bucket, rule) compiles to its own ActionKey, no collapse. - snap.originalDelayGroups has exactly 3 entries — the routing index is keyed by Delay, so same-day rules across all buckets share a group. This is the property that lets the dispatcher index by delay group rather than per-rule. - Per-group key count = (rules with that day) × buckets, so every action is reachable from its group entry.	2026-05-10 10:36:54 -07:00
Chris LuandGitHub	05d31a04b6	fix(s3tests): wire lifecycle worker for expiration suite (#9374) ... * fix(s3tests): wire lifecycle worker for expiration suite The upstream s3-tests `test_lifecycle_expiration` / `test_lifecyclev2_expiration` exercise the "set rule, wait, verify deletion" path. Phase 4 (#9367) intentionally stripped the PUT-time back-stamp, so pre-existing objects no longer pick up TtlSec on a freshly-applied rule. The s3tests CI bare-bones `weed -s3` had nothing left driving expiration. Three changes that work together: - Engine scales `Days` by `util.LifeCycleInterval`. Production keeps the 24h day; the `s3tests` build tag shrinks it to 10s so a `Days: 1` rule completes inside the suite's 30s polling window. Exported `DaysToDuration` so sibling-package tests pin to the same scale. - Scheduler/dispatcher tick defaults split into `_default` / `_s3tests` files. Production stays 5s/30s/5m; the test build runs at 500ms/2s/2s so deletions land within a couple ticks of becoming due. - s3tests.yml spawns `weed shell s3.lifecycle.run-shard -shards 0-15 -events 0 -runtime 1800s` alongside the s3 server in both the basic and SQL blocks; the shell command runs the full pipeline (reader + scheduler + dispatcher) for the duration of the suite. `test_lifecycle_expiration_versioning_enabled` is left out for now — versioned-bucket expiration via the worker still needs its own pass. Drive-by: bump `TestWorkerDefaultJobTypes` to 7 to match the registered handler count (8b87ceb0d updated `mini_plugin_test.go` for the s3_lifecycle plugin but missed this twin test). Two retention-gate engine tests `t.Skip` under the s3tests build because they rely on absolute lookback-vs-retention math the day-rescale collapses; the prod build still covers them. * review: harden lifecycle worker spawn + assert handler identity - Workflow: aliveness check on the backgrounded `weed shell` (a bad command exits in <1s and the suite would otherwise just opaque-timeout); move worker/server teardown into a `trap cleanup EXIT` so failure paths still print the worker log and reap the data dir. - worker_test: check the actual job-type set by name, not just the count. * fix(shell): keep s3.lifecycle.run-shard alive when no rules exist yet The s3-tests CI runs the worker BEFORE any test creates a bucket, so LoadCompileInputs returns empty and the shell command was bailing out with "no buckets with enabled lifecycle rules found" within ~1s. The aliveness check then fired exit 1 before tox ever started. Two changes: - Don't early-exit on empty inputs. Compile against the empty set, log a one-liner, and let the pipeline run normally — the meta-log subscription is already up, so events for buckets created later DO arrive; they just need the engine to know about them when they do. - Add `-refresh <duration>` (default 5m, 2s in s3tests CI) that periodically re-runs LoadCompileInputs + engine.Compile so rules added after startup land in the snapshot the dispatcher reads on its next tick. Production deployments keep the 5m default; only the CI workflow drops to 2s. Workflow passes `-refresh 2s` in both basic and SQL blocks. * fix(shell): backfill pre-rule entries via bootstrap walker The reader-driven path only sees meta-log events created AFTER its engine snapshot knows the rule. The s3-tests CI scenario PUTs objects first, then PUTs the lifecycle config, so by the time the engine refresh picks up the new bucket the object events have already been seen-and-dropped (BucketActionKeys returned empty for the bucket). Wire bootstrap.Walk into the shell command: - bucketBootstrapper tracks buckets seen so far. kickOffNew spawns one loop goroutine per fresh bucket. - Each goroutine re-walks the bucket every walkInterval (defaults to the same value as -refresh, i.e. 2s in s3tests CI, 5m in prod) and feeds each entry through bootstrap.Walk; due actions dispatch via a direct LifecycleDelete RPC. Not-yet-due entries are silently skipped and picked up on a later iteration once they age past their (rescaled or real) threshold. - LifecycleDelete is called with no expected_identity; the server-side identityMatches treats nil as "skip CAS", which is the right call for bootstrap (the bootstrap entry doesn't carry chunk fid / extended hash anyway). The dispatcher's pkg-private toProtoActionKind is duplicated in the shell file rather than exported, since the shape is six lines and the reverse import would pull a proto dep into the s3lifecycle root. * refactor(s3/lifecycle): hoist bucket bootstrapper into scheduler pkg The shell command got the backfill in the previous commit but the worker plugin (weed/worker/tasks/s3_lifecycle/handler.go) drives Scheduler.Run directly and missed it — same root cause: the reader-driven path only sees events created after the rule lands, so a daily cron picking up a freshly-PUT rule wouldn't expire any pre-rule object. Move the looping bucket walker into scheduler.BucketBootstrapper: - Scheduler.Run now constructs one and calls KickOffNew on every engine refresh. Per-bucket goroutines re-walk every BootstrapWalkInterval (defaults to RefreshInterval — 5m in prod, 2s under s3tests). - The shell command consumes the same struct instead of its own copy so the two paths can't drift in semantics. * refactor(s3/lifecycle): walk-once + schedule via event injection Previous per-bucket walker re-listed every WalkInterval forever. For a bucket with N objects under a long rule, the worker did O(N * runtime / walkInterval) listings even when nothing was newly due — way too much for production-scale buckets. New approach: walk each bucket exactly once on first sight, synthesize one *reader.Event per existing entry, push it onto Pipeline.events. Router.Route builds a Match with DueTime=mtime+delay; future-due matches sit in the per-shard Schedule and fire when their DueTime arrives. Currently-due matches fire on the very next dispatch tick. Wiring: - dispatcher.Pipeline lifts its events channel into a struct field with sync.Once init, and exposes InjectEvent(ctx, ev). Reader no longer closes the channel — the dispatch goroutine exits on runCtx cancellation, which works the same as channel-close did. - scheduler.BucketBootstrapper drops the WalkInterval ticker. KickOffNew spawns one walker goroutine per fresh bucket; the goroutine lists, synthesizes events, then exits. - scheduler.Scheduler builds its pipelines up front and exposes a pipelineFanout (shard -> Pipeline) as the EventInjector, so a multi- worker scheduler routes each synthesized event to the pipeline that owns its shard. - Shell command's single-pipeline path passes pipeline.InjectEvent directly. Synthesized events carry TsNs=0; dispatcher.advance treats that as a no-op so the reader's persisted cursor isn't ratcheted past unprocessed meta-log events. Identity (HeadFid + ExtendedHash) is still computed from the real filer entry, so the server's identity-CAS catches an overwrite between bootstrap and dispatch. * debug(s3tests): make lifecycle worker progress visible in CI logs The previous CI failure dumped an empty $LC_LOG even though the worker was running. Two reasons: 1. weed shell suppresses glog by default (logtostderr / alsologtostderr set to false). Pass `-debug` so the bootstrapper's V(0) lines reach stderr instead of disappearing into /tmp/weed.*.log. 2. cleanup used `kill -9` which skips Go's stdout flush. SIGTERM first with a 1s grace, then SIGKILL the holdout, then read the log. While here: bump the bootstrap walker's two informational logs to V(0) so the diagnosis from CI doesn't require -v=1 on the worker. * fix(s3/lifecycle/dispatcher): refresh snap on every event Pipeline.Run captured snap at startup and only refreshed it on the dispatch tick. With bootstrap event injection, the walker pushes events seconds after engine.Compile sees the bucket — typically WITHIN the same dispatch interval. Routing against the cached (empty) snap then silently dropped every match because BucketActionKeys returned nil for the bucket-not-yet-in-snapshot case. Re-fetch on each event. Engine.Snapshot is an atomic.Pointer.Load, so the cost is negligible. The dispatch-tick branch keeps using a fresh local read for its own loop, so its semantics are unchanged.	2026-05-08 17:29:47 -07:00
Chris LuandGitHub	7f2b20d577	feat(s3/lifecycle): policy engine — XML conversion, Compile, decideMode, Match (#9348) ... * feat(s3/lifecycle): XML lifecycle config to canonical Rule LifecycleToCanonical takes a parsed *Lifecycle and returns []*s3lifecycle.Rule, the flat shape the engine compiles against. Filter resolution mirrors AWS: <And> sub-elements (Prefix + Tags + size filters) flatten into the canonical Rule's individual fields; single <Tag> filter populates FilterTags with one entry; <Prefix> filter takes precedence over the rule's top-level <Prefix>. Multi-action rules (Expiration + NoncurrentVersion + AbortMPU on the same XML <Rule>) populate every action field they declare. RuleActionKinds expands the canonical rule into its compiled actions downstream. * feat(s3/lifecycle): engine snapshot skeleton + ActionKey type Defines s3lifecycle.ActionKey{rule_hash, action_kind} as the engine's primary identity, and adds the engine package's Snapshot type. Snapshot is immutable after Compile (atomic-swapped on rebuild) and holds the ActionKey-keyed routing indexes: - originalDelayGroups: map[time.Duration][]ActionKey - predicateActions: []ActionKey - dateActions: map[ActionKey]time.Time - actions: map[ActionKey]*CompiledAction CompiledAction.engineState is an atomic.Uint32 so MarkActive (called after the durable bootstrap_complete + mode write commits) is visible to in-flight reader passes without a recompile. The reader filters on IsActive() before dispatching, so stale-snapshot dispatches are prevented. No callers yet; downstream commits add Compile, decideMode, and the Match functions. * feat(s3/lifecycle): decideMode + retention gate decideMode picks the scheduling mode for one (rule, kind) compiled action. Disabled rule -> DISABLED; EXPIRATION_DATE -> SCAN_AT_DATE; reader-driven kind whose eventLogHorizon + bootstrapLookbackMin exceeds metaLogRetention -> SCAN_ONLY; otherwise EVENT_DRIVEN. The gate runs per (rule, kind), so a 90d ExpirationDays sibling can degrade to scan_only while its 7d AbortMPU sibling stays active. MetaLogRetention=0 is treated as "unbounded" — matches the SeaweedFS default (Phase 0 verified that meta-log files are written without TtlSec by default), so the gate doesn't trip until an operator opts in to volume-TTL pruning of /topics/.system/log/. RuleMode is a Go-level enum here, separate from the wire-form LifecycleState.RuleMode in the proto package; the worker maps between them when reading/writing the durable state file. * feat(s3/lifecycle): Compile builds the engine snapshot per-action Compile produces a fresh Snapshot from per-bucket canonical rules. Each input rule expands into N CompiledActions via RuleActionKinds; mode comes from decideMode; activation requires both bootstrap_complete (from PriorStates) and mode==EVENT_DRIVEN. Routing indexes are populated by mode: - SCAN_AT_DATE: always indexed in dateActions (detector schedules at rule.date regardless of bootstrap status; the action runs once on the date and is then done). - EVENT_DRIVEN + active: indexed in originalDelayGroups (and in predicateActions when the rule has tag/size filters). - SCAN_ONLY / DISABLED / pending_bootstrap: not indexed; safety-scan tick or operator action handle these. snapshot_id is monotonic per process; pending writes stamp it. The new snapshot replaces the engine's atomic pointer; in-flight reader passes continue against their loaded snapshot. Tests cover: single-action rule, multi-action expansion (one rule -> three CompiledActions with three distinct delay groups), pending bootstrap exclusion from indexes, retention gate, sibling actions degrading independently under partial retention, ExpirationDate path, disabled rule, MarkActive flipping IsActive(), Compile producing monotonic snapshot ids. * feat(s3/lifecycle): MatchOriginalWrite / MatchPredicateChange / MatchPath The reader feeds events through the engine's match functions to find the active ActionKeys whose filter applies. The minimal Event shape the engine takes (bucket, path, tags, size, IsLatest, IsDeleteMarker, IsMPUInit) keeps engine free of filer_pb dependencies; the reader extracts these fields from the persisted *filer_pb.LogEntry payload in Phase 3. - MatchOriginalWrite: per-delay-group sweep entry. Filters on shape = EventShapeOriginalWrite, prefix, tag, size, then per-kind shape gating (ABORT_MPU only on IsMPUInit; EXPIRED_DELETE_MARKER only on IsLatest+IsDeleteMarker). - MatchPredicateChange: single near-now sweep. Returns only the predicate-sensitive subset of active ActionKeys. - MatchPath: bucket-level walker entry. Returns every active action whose filter matches; bootstrap iterates these per object and calls EvaluateAction per kind. All filter on a.IsActive() at routing time so MarkActive flips become visible without recompile. * fix(s3/lifecycle): scope ActionKey by bucket; defensive copies; tidy compile Three findings on the engine PR addressed: 1. Critical (cross-bucket collision): ActionKey was {RuleHash, ActionKind} only. Two buckets with rules whose XML is identical produce the same RuleHash; the second bucket's Compile would overwrite the first bucket's CompiledAction in snap.actions. Add Bucket to ActionKey so the engine's identity matches the on-disk path layout /etc/s3/lifecycle/<bucket>/<rule_hash>/<action_kind>/. Regression test pins it. 2. Major (immutability leak): OriginalDelayGroups, PredicateActions, DateActions returned the snapshot's internal maps/slices by reference, letting an external caller mutate routing state and break the documented immutability contract. Return defensive copies. 3. Minor (redundant condition): mode==EVENT_DRIVEN already implies kind != EXPIRATION_DATE because decideMode routes the date kind to SCAN_AT_DATE. Drop the redundant check. Tests updated to construct ActionKey with the new Bucket field. * fix(s3/lifecycle): drop size filters from rulePredicateSensitive An object's size is immutable once written: any content change is a fresh write that flows through the original-write stream, not the predicate-change one. Tagging rules really can flip post-PUT (operator adds/removes a tag without rewriting), so they belong; size filters do not. Including size filters here was adding rules to predicateActions for no purpose — every predicate-change sweep would waste cycles re-evaluating size predicates that physically can't have changed. * perf(s3/lifecycle): pre-sort AllActions at Compile time Snapshot is immutable after Compile (engineState bit-flips don't change membership), so the (bucket, rule_hash, action_kind) ordering is stable for the snapshot's lifetime. Build the sorted slice once and serve every AllActions() call from it; drop the per-call sort.Slice. The bootstrap walker is the primary caller and may iterate this on every task entry. * docs(s3/lifecycle): note the FilterSizeGreaterThan=0 ambiguity Per AWS S3 spec, <ObjectSizeGreaterThan>0</ObjectSizeGreaterThan> explicitly excludes 0-byte objects, but with the int64 zero value as the unset sentinel we can't distinguish that from omitted-and-default. Document the limitation inline so a future deployment that needs the distinction can switch to *int64 (or a paired set-bool) and update the matchers / RuleHash accordingly. Not fixing now: the explicit-zero configuration is unusual, the canonical Rule shape mirrors the same zero-as-unset convention as s3api.Filter, and a structural fix touches every filter-using site (evaluator, due_at, match, RuleHash). * fix(s3/lifecycle): make ObjectInfo.NoncurrentIndex *int The previous int field had a zero-value collision: 0 is both "newest non-current version" (a valid index) and "uninitialised by ObjectInfo{} literal." A caller who built &ObjectInfo{IsLatest: false} without explicitly setting NoncurrentIndex would have it implicitly read as "newest non-current," and the count-based NewerNoncurrent retention would use that bogus 0 to decide eligibility. Switch to *int so nil is explicitly "not a non-current version / index not yet computed." The evaluator's NoncurrentDays and NewerNoncurrent paths conservatively return ActionNone when the index is nil — the safety scan will revisit once the index is supplied. This removes a class of latent footguns in test setup and in any future code path that constructs ObjectInfo without a versioning-aware builder. idx() helper added in tests to keep the call sites a one-liner. * refactor(s3/lifecycle): trim narration from engine + helpers Drop "what" comments where well-named identifiers already say it (IsActive, MarkActive, AllActions, etc.); collapse multi-paragraph "why" docs to one-liners where the design rationale is already in the design doc. Keep WHY comments only at non-obvious load-bearing spots: the routing-index activation predicate, the *int rationale on NoncurrentIndex, the field-tag namespace in RuleHash, the SmallDelay horizon rule. Files: action_kind.go, rule.go, rule_hash.go, evaluate.go, due_at.go, min_trigger_age.go, event_log_horizon.go, engine/engine.go, engine/compile.go, engine/match.go, engine/mode.go. No behavior change; tests untouched and pass. * fix(s3/lifecycle): durable PriorState.Mode wins over decideMode PriorState.Mode was declared but never read; Compile recomputed mode via decideMode and stored that on every CompiledAction. Effect: an action durably persisted as SCAN_ONLY (lag fallback or operator pause) or DISABLED would silently re-promote to EVENT_DRIVEN on the next engine rebuild as soon as decideMode's XML+retention predicate said so. Defeats the durability of mode state. Use prior.Mode when set; fall through to decideMode only for new actions (no prior at all) and for legacy entries persisted before Mode existed (zero value). Regression test pins both branches. * fix(s3/lifecycle): MarkActive routability — index every EVENT_DRIVEN key MarkActive's documented contract was "flip visible without a recompile," but the routing indexes (originalDelayGroups, predicateActions) were only populated when active && mode == EVENT_DRIVEN at compile time. So a key compiled with BootstrapComplete=false would never enter the indexes; a later MarkActive flipped engineState but MatchOriginalWrite / MatchPredicateChange iterated the indexes and never saw the key. Only MatchPath (which walks bi.actionKeys) and DateActions worked. Index every EVENT_DRIVEN key regardless of `active`. The runtime IsActive() filter inside filterMatching already gates dispatch, so inactive entries are matched-but-not-fired; flipping MarkActive makes them routable without recompile, matching the documented contract. Tests updated: TestCompile_BootstrapPendingIndexedButInactive asserts the indexed-but-inactive shape; TestMatchOriginalWrite_MarkActiveBecomesRoutable asserts a MarkActive flip routes the next match. * test(s3/lifecycle): pin nil NoncurrentIndex no-op behavior Two regression tests for the *int pointer migration: nil index combined with NewerNoncurrent (either paired with NoncurrentDays or standalone) must short-circuit to ActionNone rather than guess at the version's position in the keep-N window. * refactor(s3/lifecycle): trim follow-up narration on engine + helpers Comments accumulated since the last sweep — the durable-Mode rationale, the MarkActive routability note, the routing-index doc, the NoncurrentIndex pointer rationale, and the EvaluateAction docblock. Trimmed each to one or two terse lines; the underlying contracts live in the design doc. * docs(s3/lifecycle): note CompileInput one-per-bucket invariant	2026-05-07 15:00:49 -07:00