Commit Graph
14497 Commits
Author SHA1 Message Date
Chris LuandGitHub e4eb5996d5 telemetry: tidy server module after prometheus bumps (#10380) 2026-07-20 23:13:27 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ce4b7f43bd build(deps): bump com.fasterxml.jackson.core:jackson-databind from 2.22.0 to 2.22.1 in /test/java/spark (#10378)
build(deps): bump com.fasterxml.jackson.core:jackson-databind

Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson) from 2.22.0 to 2.22.1.
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-version: 2.22.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:50:10 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
599585cafd build(deps): bump github.com/pkg/sftp from 1.13.10 to 1.13.11 (#10373)
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp) from 1.13.10 to 1.13.11.
- [Release notes](https://github.com/pkg/sftp/releases)
- [Commits](https://github.com/pkg/sftp/compare/v1.13.10...v1.13.11)

---
updated-dependencies:
- dependency-name: github.com/pkg/sftp
  dependency-version: 1.13.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:49:49 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>Chris Lu
71ae8097ac build(deps): bump github.com/prometheus/procfs from 0.20.1 to 0.21.1 (#10371)
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs) from 0.20.1 to 0.21.1.
- [Release notes](https://github.com/prometheus/procfs/releases)
- [Commits](https://github.com/prometheus/procfs/compare/v0.20.1...v0.21.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
  dependency-version: 0.21.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2026-07-20 17:49:24 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b8f919b75f build(deps): bump actions/setup-python from 6 to 7 (#10369)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:48:16 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2248f5f4f9 build(deps): bump actions/setup-go from 6 to 7 (#10370)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6 to 7.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:47:58 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
f6d746705e build(deps): bump github.com/prometheus/client_golang from 1.23.2 to 1.24.0 (#10372)
build(deps): bump github.com/prometheus/client_golang

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.23.2 to 1.24.0.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.24.0/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.23.2...v1.24.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:47:29 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
8c885ef5fe build(deps): bump google.golang.org/api from 0.287.0 to 0.289.0 (#10374)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.287.0 to 0.289.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.287.0...v0.289.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.289.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:47:18 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
34d5a5677c build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.143.0 to 3.144.5 (#10375)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.143.0 to 3.144.5.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.143.0...v3.144.5)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-version: 3.144.5
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-20 17:47:07 -07:00
ed1c54dae9 Bump Tarantool client library from 2.4.2 to 3.0.0 (#10377)
* Bump Tarantool client library from 2.4.2 to 3.0.0

* Fix review comments

---------

Co-authored-by: Marat Karimov <karimov_m@inbox.ru>
2026-07-20 17:35:20 -07:00
github-actions[bot] 875cd1f67e 4.40 4.40 2026-07-20 03:57:03 +00:00
Chris Lu eb6d8ebd5f remove unused files 2026-07-18 23:11:07 -07:00
Chris LuandGitHub 07c9e0db85 iceberg maintenance: write absolute s3:// locations into table metadata (#10363)
The maintenance operations built manifest, manifest-list, data-file and
metadata-log paths with a bare path.Join("metadata", ...), so a single
maintenance run wrote scheme-less relative paths into the new snapshot,
the metadata-log and the table xattr. SeaweedFS reads those back fine
because normalizeIcebergPath accepts both forms, but strict readers
resolve every location through S3FileIO and fail with "Invalid S3 URI,
cannot determine scheme", leaving the whole table unreadable after any
maintenance commit.

Add absoluteIcebergPath, the inverse of normalizeIcebergPath, and apply
it at every site that authors locations: rewrite_manifests, compact,
rewrite_position_delete_files, and the shared commit path. The base is
derived from the bucket and table path since that is where the worker
physically writes, matching the locations the REST catalog generates.
2026-07-18 19:21:35 -07:00
Chris LuandGitHub d8196428e7 s3: invalid tagging on CopyObject returns InvalidTag, not InvalidCopySource (#10362) 2026-07-18 13:58:10 -07:00
Chris LuandGitHub efdfeaba44 cassandra: do not mask KvGet backend errors as not-found (#10361)
* cassandra: do not mask KvGet backend errors as not-found

* use errors.Is for the gocql sentinel check
2026-07-18 13:57:24 -07:00
Chris LuandGitHub 02df1cf428 shell: cluster.ps lists s3 servers (#10359)
* shell: cluster.ps lists s3 servers

* explicit returns in listClusterNodes helper
2026-07-18 13:56:47 -07:00
Chris LuandGitHub fba01ae2dd worker: balance pre-move check compares addresses in http form (#10360)
The pre-move stale check compared the master's location urls (host:port)
literally against the proposal's node addresses, which carry the grpc
suffix (host:port.grpcPort). Every move failed as "stale move: volume no
longer on source" even though the volume was still there. Normalize both
sides through pb.ServerAddress before comparing.
2026-07-18 13:52:34 -07:00
Chris LuandGitHub 564803becd shell: show who holds the cluster lock (#10353)
* regenerate master_grpc.pb.go with protoc-gen-go-grpc v1.6.2

The other generated pb files are already on v1.6.2; this one was stale.

* shell: keep unlock from racing the lease renewal

A renewal RPC in flight while ReleaseLock runs re-creates the lock on the
master after the release deletes it, and can blank the client name if the
renewal reads it mid-release. The stale-token release is then ignored, so
the lock stays held (sometimes anonymously) until it expires. Serialize
the renew and release RPCs, and set the client name before flipping
isLocked so the renewal never sends a partial acquisition.

* shell: restart lease renewal after a failed renewal

The renewal goroutine exits on error but never cleared its running flag,
so later locks in the same process were never renewed and silently
expired after ten seconds.

* shell: show who holds the cluster lock

A blocked lock command gave no hint that another client holds the lock
(the refusals only surfaced at -v=2), and cluster.status reported the
shell's own lock state as if it were the cluster's. Add a
GetAdminLockStatus RPC to the master so lock prints the holder before
blocking and cluster.status shows the actual cluster-wide holder. Both
degrade silently against masters without the RPC.

* shell: bound admin lock RPC attempts with timeouts

The lease, renew, release, and holder-status calls all ran without a
deadline, so an unresponsive master could hang the renewal goroutine,
an unlock (which now waits on the renewal mutex), or the shell prompt.
Give each attempt its own short context; the retry loops still resolve
a fresh leader on the next try.

* master: reject admin token release on non-leaders

A follower holds no lock state, so it answered a release with success
while the leader kept the lock until expiry. Refuse like LeaseAdminToken
does so the client can try the leader instead.

* shell: leave the lock release call unbounded

A release cut short by a deadline leaves the lock held on the master
until it expires, so a slow master would turn every unlock into a
ten-second ghost lock. Restore the single fire-and-forget attempt;
the timeouts stay on the lease and renew paths, where a stalled call
forfeits the lease anyway.

* shell: release only the token unlock started with

A RequestLock racing a slow release (the admin presence lock does this
on shutdown) could have its freshly acquired token sent in the release
request or zeroed by the trailing stores. Capture the token once under
the mutex and compare on clear so a concurrent acquisition survives an
in-flight unlock.
2026-07-17 12:30:42 -07:00
Chris LuandGitHub 6f14be1138 stats: remote-mount bucket cache hit/miss metrics (#10352)
Reads of remote-backed entries now record hit or miss in
SeaweedFS_remote_cache_read_total{source,bucket,result} on the filer HTTP
path and the S3 gateway, so cache effectiveness of mounted buckets can be
graphed. Inline-content entries count as hits since they are served
locally without chunks. The filer purges the per-bucket series when the
bucket directory is deleted, so a standalone filer does not accumulate
series across bucket delete/recreate churn.
2026-07-16 16:58:45 -07:00
Chris LuandGitHub 1f8d0a9ccf volume server: fill in ModifiedAtSecond on the /status volume list (#10351)
Only the heartbeat path read the .dat mtime; collectStatForOneVolume left
the field at 0, so /status consumers could not tell how long a volume had
been idle.
2026-07-16 15:18:38 -07:00
Chris LuandGitHub 267f595660 batch delete: align the shard test and Rust server with continue-past-mismatch (#10349)
Commit 8bff3b32 changed BatchDelete to keep processing after a cookie
mismatch but left the integration test asserting the old early-break
behavior, breaking Volume Server Integration Tests (grpc - Shard 1) on
master. Align the test with the new semantics and port the same
break->continue to the Rust volume server, which runs the same suite
via VOLUME_SERVER_IMPL=rust.
2026-07-16 13:55:52 -07:00
Chris LuandGitHub 1fda7aa7f1 master: assign re-picks once after its growth concludes instead of shedding (#10348)
The initiator's shed check (initiatedGrow != HasGrowRequest) compares
against an err from a PickForWrite that may predate the growth
concluding: the grower registers its volumes before clearing the flag,
so when growth lands between the failed pick and the check, the assign
shed ResourceExhausted even though a writable volume was already
registered. Re-pick once after observing the conclusion and shed only
if the volume layout still has nothing writable. Applies to both the
gRPC Assign and the HTTP dirAssign paths, which share the shed logic.

Flaked in CI as TestAssignInitiatorWaitsForItsOwnGrowth; reproduced
deterministically by widening the enqueue-to-check window.
2026-07-16 13:55:38 -07:00
8a71327324 fix: report short S3 ReaderAt reads (#10345)
* fix: report short S3 ReaderAt reads

Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.

Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.

Fix: Clear io.EOF only when the requested buffer was completely filled.

Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>

* fix: validate S3 ReaderAt requests

Co-authored-by: Codex <noreply@openai.com>

* s3 ReadAt: simplify negative offset error

* s3 ReadAt: stop reading once the buffer is full

Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.

* s3 ReadAt: cover full reads that arrive with io.EOF

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-16 02:05:20 -07:00
8bff3b3213 fix(volume): reject overflowing needle ID deltas (#10342)
* fix: reject overflowing needle ID deltas

Problem: Parsing a file ID with a delta can wrap a valid maximum needle ID back to zero without returning an error.

Root cause: Needle.ParsePath added the parsed uint64 delta without checking whether the sum exceeded the needle ID range.

Fix: Compare the delta with the remaining uint64 capacity before addition and return a contextual overflow error when it does not fit.

Validation: go test ./weed/storage/needle -run ^TestNeedleParsePathRejectsDeltaOverflow$ -count=1; go test ./weed/storage/needle -count=1; git diff --check 10cdaf381875492a2c752d1038797e96ff18208f..HEAD
Co-authored-by: Codex <noreply@openai.com>

* fix: propagate needle ID delta parse errors

Co-authored-by: Codex <noreply@openai.com>

* print the needle id in hex in the delta overflow error

* batch delete: keep processing after a cookie mismatch

* rust volume: reject overflowing needle id deltas

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-15 23:24:04 -07:00
Chris LuandGitHub ae46e79e98 trigger dev container builds on seaweed-volume changes (#10344)
The dev container workflow prebuilds the Rust volume server from
seaweed-volume/ but did not watch that directory, so Rust build fixes
landed without this workflow validating them.
2026-07-15 23:22:13 -07:00
Chris LuandGitHub 7bfc44432d fix Rust build with protoc versions lacking native proto3 optional (#10343)
pass --experimental_allow_proto3_optional to protoc in the Rust build

filer.proto now carries a proto3 optional field, which the protoc 3.12
shipped in ubuntu-22.04 apt rejects unless this flag is set. Newer
protoc versions still accept the flag, so it is safe everywhere.
2026-07-15 22:52:17 -07:00
Chris LuandGitHub 5553e7b876 mount: surface ENOSPC instead of endless waiting when the cluster is full (#10341)
When every volume is full, a chunk upload fails only after the assign
retry budget, the failed chunk's data is dropped, and the mount kept
accepting writes anyway. cp would crawl for hours pushing the rest of
the file through a pipeline that could not persist it, and only close()
reported an error - a generic EIO.

Poison the file handle on the first failed chunk upload so subsequent
writes fail immediately, and map "no writable volumes" / "no free
volumes" upload errors to ENOSPC so the writing process aborts with
"No space left on device". Also guard lastErr with a mutex: it was
written concurrently by uploader goroutines, and keep the first error
so later failures do not mask the root cause.
2026-07-15 22:01:19 -07:00
Chris LuandGitHub 25ab4c3cac preserve Content-Encoding for remote-mounted objects (#10340)
* remote storage: carry Content-Encoding into mounted entries

A RemoteEntry now records the remote object's Content-Encoding, and every
path that materializes a local entry from remote metadata (lazy fetch, lazy
listing, remote.mount, remote.meta.sync, remote.cache) stamps it into the
entry extended attributes, so HTTP and S3 HeadObject/GetObject return the
header. GCS and Azure populate it on listing and stat; S3 only exposes it
via HeadObject, so listings leave it empty.

* remote storage: set Content-Encoding when uploading to the remote

An entry carrying Content-Encoding in its extended attributes (a native S3
upload, or a value pulled from the remote) now keeps it when
filer.remote.sync or remote.copy.local writes the object to GCS, S3, or
Azure, instead of silently dropping it.

* gcs: read remote objects without decompressive transcoding

GCS transparently decompresses gzip-encoded objects on download, which
ignores range requests and returns byte counts that disagree with the
tracked RemoteSize. Request the stored bytes instead; chunked reads of
gzip-encoded objects then behave like any other object.

* remote storage: track Content-Encoding presence so removals propagate

A listing that does not report encodings (S3) leaves the field unset and
the local header untouched, while an authoritative report of no encoding
(GCS, Azure, any stat) now clears a previously stamped header instead of
leaving it stale. remote.cache also schedules a metadata update when only
the reported encoding changes.

* remote storage: propagate Content-Encoding on metadata-only updates

filer.remote.sync routes same-content changes through UpdateFileMetadata,
which only touched custom metadata (GCS, Azure) or tags (S3), so a
Content-Encoding change in the extended attributes never reached the
remote object's real header. GCS now patches contentEncoding alongside
the metadata, and Azure reissues the blob's HTTP headers with the new
value, carrying the others over since the call replaces the full set.
S3 stays tags-only: changing the header there means rewriting the
object, which the sync already does whenever content changes.

* remote.meta.sync: optional per-file stat for listing-omitted metadata

S3 listings carry no Content-Encoding, so entries synced from them never
learn it and the lazy-stat path never runs once an entry exists. With
-statFiles, each new or changed file whose listing left the encoding
unreported is stat-ed before reconciling, and the stat-derived value is
persisted so the next run only stats files that changed. Off by default:
it costs one remote request per file, and GCS and Azure listings already
carry the encoding.

* s3: apply metadata-only Content-Encoding changes with an in-place copy

Content-Encoding is S3 system metadata, so the tags-only metadata update
silently left the object's real header untouched. When the encoding
differs, reissue the object as a self-copy with replaced metadata,
carrying the content type and configured storage class like a fresh
write does. CopyObject caps at 5 GiB; beyond that the change is logged
and applies on the next content write.

* azure: skip the metadata call when user metadata is unchanged

An encoding-only change reissues the blob's HTTP headers; sending the
unchanged user metadata alongside it wastes a round trip and bumps the
blob's ETag once more than needed.

* s3: carry existing object metadata through the encoding copy

The replace directive drops everything not resent, and a mounted entry
usually has no local mime or user metadata, so the in-place copy wiped
the object's Content-Type, Cache-Control, user metadata, encryption
settings, and storage class. Read them back with a HeadObject first and
carry them over, overriding only what SeaweedFS manages: the encoding,
a locally set mime, and the configured storage class. S3 reports
Expires as a string while the copy input wants a time, so it is parsed
and skipped when malformed.
2026-07-15 19:20:00 -07:00
Chris LuandGitHub c015cc3939 generate vtproto marshalers for filer_pb and use them on the metadata log path (#10337)
* generate vtproto marshalers for filer_pb and use them on the metadata log path

Reflection-based proto.Unmarshal allocates a fresh message tree through
reflect.New on every call. On the metadata subscription fan-out the same
event is decoded once per subscriber, so reflect.New tops the decode
churn under many mounts.

Generate MarshalVT/UnmarshalVT/SizeVT for filer.proto (a separate
filer_vtproto.pb.go, filer.pb.go untouched) and call them on the log
entry marshal and the subscribe/replay decode paths. UnmarshalVT
allocates message structs directly and copies byte and string fields, so
it stays wire-compatible with proto.Unmarshal and preserves the
non-aliasing the persisted-log cache depends on.

For SubscribeMetadataResponse this cuts decode allocations 69 -> 50 and
~4.5us -> ~2.1us per event; the win scales with subscriber overlap.

* marshal log entries directly into the buffer

SizeVT is allocation-free and MarshalToSizedBufferVT writes into a
pre-sized slice, so the log entry can be marshaled straight into
logBuffer.buf. This drops the per-entry MarshalVT allocation and the
follow-up copy on the write path.

* expand vtproto benchmarks: marshal, decode, and marshal-into-buffer by chunk count

Parametrize by nested-message count (chunks per event) and add encode +
zero-alloc marshal-into-buffer benchmarks alongside the decode one, so
the write-path win from MarshalToSizedBufferVT is measurable too.

* keep proto.Unmarshal for metadata events to preserve UTF-8 validation

UnmarshalVT skips proto3's UTF-8 validation of string fields, so a
SubscribeMetadataResponse with an invalid-UTF-8 string (e.g. Directory
"\xff") that proto.Unmarshal rejects would decode and reach path
filtering and subscribers. Decode events with proto.Unmarshal again;
UnmarshalVT stays on the log entry paths, whose only variable-length
fields are bytes and so carry no UTF-8 constraint.

Tests cover the codec difference and that a malformed event is skipped
before delivery.
2026-07-15 02:32:05 -07:00
Chris LuandGitHub 0ad83d5061 Fix object tagging writing back to the wrong object for nested keys (#10338)
Put/DeleteObjectTagging set the update directory to the bucket root for a
null version, so a tag change on allowed/protected.txt landed on
protected.txt at the bucket root instead. A principal scoped to one
nested key could overwrite a different object sharing the basename, the
same class of scope bypass fixed for PutObjectAcl. Share the object's
parent-directory resolver across both paths.
2026-07-15 02:31:51 -07:00
Chris LuandGitHub 8f29d0a91e route log buffer flush copies through the shared slab pool (#10336)
* route log buffer flush copies through the shared slab pool

Each flush copied the sealed window into a bytes.Buffer drawn from a
package-local sync.Pool. GC drains that pool, so once collections run
often (e.g. under GOMEMLIMIT) most flushes miss and Write grows a fresh
window-sized array, the dominant bytes.growSlice source under write load.

Copy into a size-classed slab from weed/util/mem instead. Slabs are
reused process-wide and returned to their exact size class after the
flush, so variable-sized flushes across many partitions stop churning
mismatched buffers.

* guard nil slab in flush releaseMemory

mem.Free(nil) resolves to the smallest slot pool and stores a zero-cap
slice, so a later mem.Allocate could return nil and panic. The current
call sites never pass nil, but the guard keeps a double release harmless.
2026-07-15 00:51:41 -07:00
Chris LuandGitHub 68e6fc5f7c fuse test: keep derived filer gRPC port below the ephemeral floor (#10334)
The FUSE harness picks the filer HTTP port and lets "weed mount" derive the
filer gRPC port as HTTP+10000. freePort kept the HTTP port under the Linux
ephemeral floor (32768) but not the derived gRPC port, which ranged up to
42000. When the gRPC port landed above the floor an outbound connection could
transiently hold it, so mini relocated its filer gRPC port while mount kept
dialing HTTP+10000, timing the mount out.

Cap the HTTP port at 22000 so the gRPC sibling stays at or below 32000, and
verify both ports are bindable before returning.
2026-07-14 12:46:54 -07:00
Chris LuandGitHub f3e4a73696 s3: return NoSuchKey/NoSuchBucket for a missing CopyObject source (#10332)
* s3: CopyObject returns NoSuchKey for a missing copy source

* s3: CopyObject returns NoSuchBucket for a missing source bucket
2026-07-14 11:53:53 -07:00
Chris LuandGitHub 311bc3a6df Fix PutObjectAcl writing back to the wrong object for nested keys (#10333)
* Fix PutObjectAcl writing back to the wrong object for nested keys

PutObjectAcl set the update directory to the bucket root, so an ACL
change on allowed/protected.txt landed on protected.txt at the bucket
root instead. A principal scoped to one nested key could overwrite a
different object sharing the basename. Target the object's own parent
directory.

* Test object ACL update directory resolution for nested keys
2026-07-14 11:46:37 -07:00
Chris LuandGitHub 10cdaf3818 Introduce weed shell command ec.check.replication. (#10328)
* Introduce weed shell command `ec.check.replication`.

This command performs a quick check of EC volume shard replication, reporting
volumes whose shards are over- or under-replicated. Each volume is checked
against its own data+parity ratio, obtained via
erasure_coding.EcShardsVolume{Data,Parity}Shards, so builds that derive the EC
ratio per volume report custom ratios correctly.

The name follows the shell's convention (cluster.check, volume.check.disk); the
closest normal-volume counterpart is volume.fix.replication.

* shell: ec.check.replication reports mixed under+over-replication in both lists
2026-07-13 17:18:42 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
cd7c8ae784 build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.141.0 to 3.143.0 (#10322)
build(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.141.0 to 3.143.0.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.141.0...v3.143.0)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-version: 3.143.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-13 11:45:03 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
a7073a9778 build(deps): bump golang.org/x/crypto from 0.53.0 to 0.54.0 (#10323)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.53.0 to 0.54.0.
- [Commits](https://github.com/golang/crypto/compare/v0.53.0...v0.54.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.54.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-13 11:44:56 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
0dc504733e build(deps): bump github.com/schollz/progressbar/v3 from 3.19.0 to 3.19.1 (#10324)
build(deps): bump github.com/schollz/progressbar/v3

Bumps [github.com/schollz/progressbar/v3](https://github.com/schollz/progressbar) from 3.19.0 to 3.19.1.
- [Release notes](https://github.com/schollz/progressbar/releases)
- [Commits](https://github.com/schollz/progressbar/compare/v3.19.0...v3.19.1)

---
updated-dependencies:
- dependency-name: github.com/schollz/progressbar/v3
  dependency-version: 3.19.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-13 11:44:46 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9964d5b82a build(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.19.26 to 1.19.28 (#10325)
build(deps): bump github.com/aws/aws-sdk-go-v2/credentials

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.19.26 to 1.19.28.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.26...credentials/v1.19.28)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.19.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-13 11:44:37 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7b2bee9890 build(deps): bump github.com/cognusion/imaging from 1.0.3 to 1.0.4 (#10326)
Bumps [github.com/cognusion/imaging](https://github.com/cognusion/imaging) from 1.0.3 to 1.0.4.
- [Commits](https://github.com/cognusion/imaging/compare/v1.0.3...v1.0.4)

---
updated-dependencies:
- dependency-name: github.com/cognusion/imaging
  dependency-version: 1.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-13 11:44:28 -07:00
Chris LuandGitHub bd9b5c25ff rust volume: verify the .dat ends at the last indexed needle (#10320)
* rust volume: verify the .dat ends at the last indexed needle

The Go loader quarantines a volume whose .dat extends past the last
indexed needle - the leftover of a torn shutdown - but the Rust check
only verified header fields of the trailing index entries and never
compared file sizes, so a torn tail loaded clean and writable. Appends
land at the raw file end, and past a misaligned tail the next needle
sits at an offset the 8-byte-unit .idx encoding rounds down, pointing
the index a few bytes before the needle.

Replace the last-10-entries walk with the current Go shape: find the
entry physically last in the .dat (append-ordered fast path, max-offset
scan for key-sorted rebuilds), verify that needle - tombstones with
their on-disk Size=0 - and require the file to end exactly at it,
marking the volume read-only otherwise.

Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb

* rust volume: buffer the .idx max-offset scan

The slow path read one 16-byte entry per syscall; a BufReader batches
the sequential scan like Go's WalkIndexFile does.

Claude-Session: https://claude.ai/code/session_01XgGXMLknzaNgQzHMyo2Vhb
2026-07-12 19:37:58 -07:00
Chris LuandGitHub 19dc085e33 master: statistics used size covers all collections and layouts (#10319)
StatFs on a mount reported cluster-wide total capacity but used size from
a single volume layout keyed by collection, replication, ttl, and disk
type. A mount without -collection therefore showed only the default
collection's usage, hiding data in named collections, and even a
collection-scoped mount missed volumes with a different replication,
ttl, or disk type.

Aggregate used size and file count across all layouts of the requested
collection, and across every collection when the collection is empty,
matching how Topology.Lookup treats an empty collection. Looking up
stats no longer creates a phantom collection as a side effect.
2026-07-12 12:56:10 -07:00
Chris LuandGitHub 013281b498 shell: volume.check.disk -resurrectMissingNeedles for never-vacuumed replicas (#10316)
An absent needle is normally indistinguishable from a vacuumed delete,
so check.disk skips it and replicas diverged by replication failures
cannot be reunited. But a replica with compaction revision 0 has never
been vacuumed: every delete it processed still holds its tombstone, so
a needle absent there is provably a missing write.

The new flag resurrects absent needles in exactly that case. The
receiving replica's live compaction revision is read after the index
snapshot and must be 0; vacuumed replicas keep the skip, now with the
revision in the message. The decision is passed per direction since
pass 2 runs pairs concurrently. Resurrected needles count toward
-nonRepairThreshold as before.
2026-07-12 00:14:16 -07:00
Chris LuandGitHub fa549e9c83 filer: harden TUS session authorization against cross-prefix access (#10315)
* filer: authorize TUS existing-session verbs against the validated stored target

Scope-check on TUS HEAD/PATCH/DELETE only populated a resource path for POST,
so a prefix-restricted token that learned another tenant's session id could act
on that session and land content at a TargetPath its own AllowedPrefixes forbid.

Split the filer JWT check into authenticateFilerJwt (signature and method) and
authorizeFilerJwtPaths (resource scope), and make the scope check fail closed:
a prefix-restricted token with no resolved resource path is denied instead of
authorized on signature alone. The TUS handler now authenticates first, reads
and validates the session once, authorizes the stored TargetPath, then operates
on that single pinned snapshot. readTusSessionInfo rejects a session whose id,
target or size is unusable, and getTusSession is split so the authorization
lookup no longer lists chunks.

* filer: reject non-canonical TUS upload ids

The uploads route took the first path component as the session id, so a trailing
path or other non-canonical spelling aliased one session under several URLs.
Require the id to be a canonical UUID, the only form the server mints, both at
routing and when reading a session's metadata, so one URL maps to one resource.

* filer: revalidate the pinned TUS session before completing an upload

Completion re-read chunks but not the session identity, so a PATCH finishing
after a concurrent DELETE or metadata replacement could still land at the id's
stored path. Before completing, confirm the session still exists and its target,
size and creation time are unchanged from the authorized snapshot; otherwise the
completion fails instead of writing to a path the request never authorized.

* filer: log TUS session lookup failures before returning not-found

readTusSessionInfo and loadTusSessionChunks failures answered "not found" with
no log line, so a transient filer or listing error was indistinguishable from a
genuinely missing session. Log the lookup at V(1) (a missing session is common
and benign) and the chunk-load error at Errorf (the session already resolved).
2026-07-11 22:45:28 -07:00
Chris LuandGitHub c1a1e3c1e3 shell: volume.tier.upload keeps volume replicas (#10314)
* volume: copying a remote-backed volume only needs space for the index

VolumeCopy sized its target-location check by the source .dat even when
that .dat lives in a cloud tier and only .idx/.vif land locally, so
re-replicating a tiered volume demanded the full remote size in free
disk. Require the index size instead.

* shell: volume.tier.upload keeps volume replicas

Tiering a replicated volume deleted every replica but the upload
source, leaving one server holding the only .idx and the only .vif
that knows the remote object key — losing that server orphaned the
volume even though its data sat intact in the cloud.

Replicate the uploaded .idx/.vif onto the other replica servers
instead (VolumeCopy skips the .dat for remote-backed volumes), so all
replicas serve reads from the same remote object and the volume keeps
its replica count. An already-tiered replica is preferred as the
upload source, so a rerun after a partial failure reuses the existing
remote object instead of uploading a second copy under a new key.

* shell: group tier upload locations instead of re-prepending

* rust volume: copying a remote-backed volume only needs space for the index

Mirror the Go VolumeCopy change: size the free-location check by the
source .idx when the .dat lives in a cloud tier, since only .idx/.vif
land locally.
2026-07-11 13:49:36 -07:00
Chris LuandGitHub 40ee4a08c5 admin: show bucket lifecycle rules in the Admin UI (#10313)
* admin: surface lifecycle rule counts in bucket listing

* admin: add bucket lifecycle JSON endpoint

* admin: show lifecycle rules on the buckets page

* admin: make the lifecycle count badge keyboard-accessible

* admin: match buckets empty-state colspan to the column count

* admin: drop stale lifecycle modal responses

* make
2026-07-11 11:31:30 -07:00
Chris LuandGitHub 29981f8d24 sftp: match permission paths on path-component boundaries (#10311)
* sftp: match permission paths on path-component boundaries

Permission checks compared paths with raw string prefixes, so a
permission entry for /tenants/alice also matched sibling paths such as
/tenants/alice-archive, letting a scoped user read and overwrite
another tenant's files. The home directory containment check had the
same flaw. Route both through pathWithin, which cleans the paths and
requires exact equality or a separator-delimited descendant.

* sftp: clean permission path into a local when ranking matches
2026-07-10 22:10:01 -07:00
Chris LuandGitHub e7be6bb2f8 filer: allow clearing a bucket read-only flag stuck after quota removal (#10310)
* filer: allow clearing a bucket read-only flag stuck after quota removal

Once s3.bucket.quota.enforce marks a bucket read-only, removing or
disabling the quota orphans the flag: enforcement skips buckets with a
non-positive quota, and fs.configure merges booleans with OR so
-readOnly=false could never turn it off. The only way out was deleting
the whole path rule.

- s3.bucket.quota -op=remove/disable and the admin server quota update
  now lift the read-only flag on the bucket's path rule
- fs.configure now honors explicitly passed false boolean flags
  (-readOnly, -fsync, -worm) instead of OR-merging them away

* filer: ClearBucketReadOnly reports unchanged when the save fails
2026-07-10 22:09:39 -07:00
c006dc563e ec: remove .ecsum sidecars on destroy / shard delete; align Go and Rust cleanup (#10307)
* fix(rust-volume): remove .ecsum sidecars on EC destroy / shard delete

Rust EcVolume::destroy removed shards and .ecx/.ecj/.vif but left bitrot
checksum sidecars (.ecsum / .ecsum.v*). On clusters that run weed-volume
(not Go weed volume), collection.delete therefore orphans every sidecar
while correctly wiping shards — observed live on 4.39 (14/14 .ecsum
survived after collection.delete on a freshly encoded EC volume).

Go Destroy already calls RemoveBitrotSidecars; this brings Rust to parity:
- hoist remove_bitrot_sidecars into ec_bitrot (shared helper)
- call it from EcVolume::destroy for dir / dir_idx / ecx_actual_dir
- call it from Store::delete_ec_shards when a disk has no remaining shards
- unit test: test_destroy_removes_bitrot_sidecar

* rust volume: gate the shard-delete sidecar sweep on a local shard removal

Only sweep a disk's .ecsum when this delete actually removed a shard file
there, matching Go's found gate: a delete that never touched a disk must not
strip a sidecar it does not own — a shared -dir.idx sibling with surviving
shards, or an ec.rebuild index-prep copy that lands .ecx/.ecsum before any
shard. The shard-presence probe now treats unexpected stat errors as
"exists" so a transient failure cannot orphan-classify live shards, and
check_all_ec_shards_deleted reuses it.

* rust volume: destroy() sidecar sweep needs only the data and idx bases

ecx_actual_dir is always one of the two, so the third branch could never
run; this is now exactly Go Destroy()'s two-base sweep.

* rust volume: call the shared sidecar removal helper directly

* rust volume: unit-test remove_bitrot_sidecars

Mirrors Go's TestRemoveBitrotSidecars: legacy and versioned sidecars are
removed, a shard file and a longer-vid sidecar survive, absent is success.

* rust volume: keep the shared idx-base sidecar while a sibling disk has shards

One -dir.idx serves every location, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards. Nothing
reads the idx-base sidecar today, but .ecx shows index-dir files are real;
this keeps the defensive sweep safe if a writer ever lands one there.

* ec shard delete: keep the shared idx-base sidecar while a sibling disk has shards

One -dir.idx serves every disk, so emptying one disk must not sweep
<idx>/<vol>.ecsum out from under a sibling that still holds shards of the
volume — the same gate the Rust volume server applies. A status error counts
as in-use so a transient failure never strips it early.

* rust volume: drop a shard-only disk's stale .vif with the node's last shard

Go's removeEcSharedIndexFiles also clears the data-base .vif in the
all-shards-gone pass, gated on .idx absence so a disk still hosting the
source volume keeps its live .vif; the Rust delete path left it behind.
Unexpected stat errors count as .idx-present so a transient failure never
strips a live volume's .vif.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-10 22:06:36 -07:00
Chris LuandGitHub ce82e3a057 filer: scope TUS HEAD/PATCH/DELETE against the session target path (#10309)
* filer: scope TUS HEAD/PATCH/DELETE against the session target path

checkTusJwtAuthorization only populated the scoped-path list for POST, so
a prefix-restricted token was scope-checked at session creation but not on
HEAD/PATCH/DELETE, which act on an existing session addressed by id. A
low-privilege tenant holding a valid write token who learns another
upload's session id could PATCH attacker bytes into that session, or
DELETE/HEAD it, landing content at a target path its own AllowedPrefixes
forbids. The session id is unguessable to an unauthenticated attacker but
is not an authorization boundary against a legitimate tenant.

Resolve the effective target for HEAD/PATCH/DELETE from the session's
stored TargetPath and scope the prefix check against it, the same way POST
scopes the create target. The scoped paths are resolved lazily so the
session read runs only for a prefix-restricted token; unrestricted tokens
and deployments without a signing key touch no extra state. An unknown
session stays unscoped so the handler still answers 404.

* filer: fail closed when a TUS session target cannot be resolved

The lazy scope resolver swallowed every readTusSessionInfo error and left
the request unscoped, so a filer read error or a corrupt session .info
would authorize a prefix-restricted token against a target the server
never resolved. Only a genuinely missing session should stay unscoped (so
the handler answers 404); any other failure now propagates and denies the
request. checkJwtAuthorizationScoped's resolver returns an error and a
failed resolution is treated as not-authorized.

* filer: fold the scoped-path resolver into checkJwtAuthorization

checkJwtAuthorization now takes the lazy resolver directly instead of a
separate checkJwtAuthorizationScoped wrapper, and the surrounding comments
are trimmed to the load-bearing why.
2026-07-10 20:33:01 -07:00