From 9d11278d951cd55db10eb2e216c37acc4e7451da Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sat, 8 Aug 2026 09:24:58 -0700 Subject: [PATCH] filer: add filer.meta.scan to audit one directory's change history (#10645) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * filer: drain pending log chunk refs when the metadata stream ends In metadata chunks mode the server sends log file refs in responses of their own, and the client can only read them once it knows the run of refs is over. That was inferred solely from the arrival of a normal event, so refs still pending when the stream ended were dropped: the subscription returned no events and no error. A follower never noticed, because it runs forever and a live event always arrives to close the run. A bounded subscription — StopTsNs set, range already in the past — can receive nothing but refs and then EOF, and silently reports that nothing happened. For anything auditing a path that is the worst possible answer, since an empty result is indistinguishable from a quiet period. Drain on EOF as well as at the transition point. * filer: add filer.meta.scan to audit one directory's change history Reconstructing what happened to a path means replaying the metadata log, and filer.meta.tail is built for watching rather than auditing: it follows forever unless given a stop, prints multi-line JSON, and takes ranges only as durations before now, so an incident timestamp has to be converted by hand. Its -pattern also cannot find a versioned object. A versioned key is stored as .versions/v_, so the events carry the names ".versions" and "v_" and a pattern of the object's own name matches neither — the search comes back empty while the object is being written continuously. filer.meta.scan prints one line per change, stops at the end of the range, accepts absolute -since/-until with an explicit -tz, and reports versioned writes against the object key with the version id alongside, so -name matches the key a client would ask for. Delete markers are labelled as such rather than appearing as zero-length writes, and pointer flips on the .versions container are distinguished from writes of object data. * filer.meta.scan: read persisted log chunks from the volume servers Reading a range through the filer makes it decode every log entry in that range and filter each one, so the cost lands on the filer and does not shrink when the prefix is narrow — only the bytes on the wire do. On a cluster whose metadata log is dense that is the expensive part of a scan, and it is charged to the process least able to spare it. Enable metadata chunks mode: the filer hands out log chunk ids and the scan reads them from the volume servers itself. ReadLogFileRefs re-applies the same path filter client-side, so the output is unchanged — verified identical to the filer-read path over the same range, including after a restart drops the in-memory buffer and the data must come off disk. Direct read needs a route to the volume servers that the filer does not, so a failure before anything has been printed retries through the filer; retrying after partial output would duplicate lines. -directRead=false forces it. * filer.meta.scan: confirm an empty direct-read result through the filer An audit that returns nothing is read as "nothing happened here", so it is the one answer that must not be produced by a bug. Direct read has more ways to come back empty than the filer path does — it needs a route to the volume servers, and it depends on the ref-drain contract holding. When direct read yields no changes, re-run through the filer before reporting it, and warn if the two disagree. Re-running is safe only because nothing was printed; after partial output a replay would duplicate lines instead, so that case reports the error rather than retrying. --- weed/command/command.go | 1 + weed/command/filer_meta_scan.go | 352 +++++++++++++++++++++++++++ weed/command/filer_meta_scan_test.go | 177 ++++++++++++++ 3 files changed, 530 insertions(+) create mode 100644 weed/command/filer_meta_scan.go create mode 100644 weed/command/filer_meta_scan_test.go diff --git a/weed/command/command.go b/weed/command/command.go index 970b44ba2..daa3823a1 100644 --- a/weed/command/command.go +++ b/weed/command/command.go @@ -22,6 +22,7 @@ var Commands = []*Command{ cmdFilerCat, cmdFilerCopy, cmdFilerMetaBackup, + cmdFilerMetaScan, cmdFilerMetaTail, cmdFilerRemoteGateway, cmdFilerRemoteSynchronize, diff --git a/weed/command/filer_meta_scan.go b/weed/command/filer_meta_scan.go new file mode 100644 index 000000000..45305ee84 --- /dev/null +++ b/weed/command/filer_meta_scan.go @@ -0,0 +1,352 @@ +package command + +import ( + "context" + "fmt" + "io" + "os" + "strings" + "time" + + "google.golang.org/grpc" + + "github.com/seaweedfs/seaweedfs/weed/filer" + "github.com/seaweedfs/seaweedfs/weed/pb" + "github.com/seaweedfs/seaweedfs/weed/pb/filer_pb" + "github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants" + "github.com/seaweedfs/seaweedfs/weed/security" + "github.com/seaweedfs/seaweedfs/weed/util" +) + +func init() { + cmdFilerMetaScan.Run = runFilerMetaScan // break init cycle +} + +var cmdFilerMetaScan = &Command{ + UsageLine: "filer.meta.scan -pathPrefix=/some/dir [-since=...] [-until=...]", + Short: "audit what happened under a directory, one line per change", + Long: `Replay the filer's metadata log for one directory and print one line per change. + + Unlike filer.meta.tail this stops when it reaches the end of the requested + range instead of following, so it can be piped straight into grep or awk. + + Each line is: + +