mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-20 22:27:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(s3/lifecycle): cold-start walker covers pre-existing objects

Browse Source 
runShard only walked the bucket tree on the recovery branch (found
&& hash mismatch). For a fresh worker with no persisted cursor,
found=false, so the recovery walker never fired and the meta-log
replay only scanned runNow - maxTTL of events. Objects PUT before
that window — including pre-existing objects in a newly-rule-enabled
bucket — never matched the rule.

The streaming worker handled this with scheduler.BucketBootstrapper.
Daily-replay needed the equivalent: walk the live tree once on the
first run for each shard so pre-existing objects get evaluated even
when their PUT events are outside meta-log scan window.

Restructured the recovery branch to fire the walker on either
(found && mismatch) OR !found. On cold-start the cursor isn't
rewound — we keep TsNs=0 and let the drain below floor to
runNow - maxTTL like before; the walker just handles whatever the
sliding window can't reach.

TestLifecycleBootstrapWalkOnExistingObjects was the exact CI failure
this addresses (https://github.com/seaweedfs/seaweedfs/actions/runs/25777823522/job/75714014151).
This commit is contained in:
Chris Lu
2026-05-12 21:22:00 -07:00
parent 79a0c8541f
commit 7ed56b6728
1 changed files with 24 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
weed/s3api/s3lifecycle/dailyrun/run.go
+24 -10
View File
@@ -225,22 +225,36 @@ func runShard(ctx context.Context, cfg Config, snap *engine.Snapshot, runNow tim
})
}
// Recovery: rule-content edit (RuleSetHash mismatch) or partition
// flip (PromotedHash mismatch). Walk the rewritten rule set so
// already-due objects fire before the cursor rewinds; then rewind
// and let the sliding meta-log replay catch up steady state.
if found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted) {
// Recovery / cold-start walker:
// - found && hashes mismatch: rule edit or partition flip — walk
// the rewritten rule set so already-due objects fire before the
// cursor rewinds, then rewind for meta-log replay.
// - !found: first run for this shard. Pre-existing objects PUT
// before the rule was added live OUTSIDE the meta-log scan
// window (TsNs > runNow - maxTTL) and would never replay; the
// walker has to discover them. The streaming worker did this
// via BucketBootstrapper; daily-replay needs the same.
mustWalkRecovery := found && (persisted.RuleSetHash != rsh || persisted.PromotedHash != promoted)
mustWalkColdStart := !found
if mustWalkRecovery || mustWalkColdStart {
if cfg.Walker != nil {
if werr := cfg.Walker(ctx, engine.RecoveryView(snap), shardID); werr != nil {
return fmt.Errorf("shard=%d: recovery walk: %w", shardID, werr)
}
}
next := Cursor{
TsNs: runNow.Add(-maxTTL).UnixNano(),
RuleSetHash: rsh,
PromotedHash: promoted,
if mustWalkRecovery {
// Rule changed: rewind cursor so the sliding replay re-scans
// the new max-TTL window and the persisted hashes match the
// new rule set.
next := Cursor{
TsNs: runNow.Add(-maxTTL).UnixNano(),
RuleSetHash: rsh,
PromotedHash: promoted,
}
return cfg.Persister.Save(ctx, shardID, next)
}
return cfg.Persister.Save(ctx, shardID, next)
// Cold start: keep TsNs=0 so the drain below floors to
// runNow - maxTTL and the cursor is saved fresh after the run.
}
// Steady-state walker for walker-bound and scan_only-promoted rules.