mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-22 14:32:35 +00:00
upload-parallel-submit
15630
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin
|
9d84ecca2f |
upload -dir: finish every file before reporting a failed upload
A failed batch stopped all remaining batches, and the file-list path returned before printing results, hiding ids of files that did upload. Now every file is still attempted, results print for each batch, and the command fails at the end if any upload failed. |
||
|
Devin
|
a1d2ebbb9b | operation: warm lazy log fds in the file-part leak test | ||
|
Devin
|
371d320854 |
operation: close open file parts when a batch fails to build
Batched -dir uploads now open up to 32 files at once; a mid-batch error left every earlier handle open until process exit. Close the parts already built, and close the handle when Stat fails after Open. |
||
  Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
84cc179b80 |
upload -dir: fetch file ids in batches instead of per file
Directory upload submitted every file through its own SubmitFiles call, so each file paid a separate serial assign RPC. Files are now collected during the walk and submitted in batches, sharing one batch assign and the parallel uploads. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
c8002d63bb |
upload: submit batched files concurrently
SubmitFiles fetched all file ids in one assign RPC but then uploaded the files one after another, so a multi-file upload spent its time in serial per-file round trips. Uploads now run through a small worker pool while results keep their input order. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
  Junker der ProvinzandGitHub
|
9b642d4636 | volume: use FALLOC_FL_KEEP_SIZE and log preallocation failures (#11920) | ||
 
|
a882565cd7 |
iceberg: keep a default-CRS geometry column without a CRS after compaction (#11899)
* iceberg: keep a default-CRS geometry column without a CRS after compaction Writers such as DuckDB write a GEOMETRY column whose Iceberg type has the default CRS (OGC:CRS84) with no CRS in the Parquet logical type. parquet-go spells the default out when it writes such a column (geometryType's LogicalType), even when the column it read declared none, so every compacted file declared GEOMETRY(crs=OGC:CRS84) where its inputs declared GEOMETRY. Both mean the same to the Parquet spec, but a reader may treat the file column as another type than the table column: DuckDB casts it to the table's GEOMETRY, and a cast column is no longer pruned by its row-group statistics, so a spatial filter read every row group of a compacted file. After a bin is merged, the footer's schema elements of every GEOMETRY column whose Iceberg type has the default CRS (top-level or nested) are rewritten to declare no CRS. The data, the page indexes and every offset stay byte for byte; only the footer is re-encoded. An explicit CRS is left as it is. Retyping the writer's column instead does not work: parquet-go refuses to convert between two GEOMETRY types that both mean the default, and its sorting writer converts. Tests: geometry_crs_test.go, a bin-packed and a sorted compaction of default-CRS inputs (one of them spelling the default out) write no CRS, with a control showing parquet-go spells it out on its own; an explicit CRS is kept; a nested geometry. On master the binpack and sort cases fail with CRS "OGC:CRS84". * iceberg: compress the row lineage columns of compacted v3 files A v3 compaction appends _row_id and _last_updated_sequence_number to the merged file. The two leaves were built without a codec, so they were written uncompressed, 16 bytes a row, whatever codec the data columns use; on a table of points the compacted files grew by about a third. The lineage leaves now take the codec of the reference file's first leaf that declares one, ZSTD when none does. Test: TestCompactCompressesLineageColumns; on master both columns are written UNCOMPRESSED next to ZSTD data columns. * iceberg: match list and map geometry by Iceberg name in files without field ids A Parquet footer path runs through the repeated group of a LIST or MAP (shapes.list.element, attrs.key_value.value), which Iceberg leaves out of its field names (shapes.element, attrs.value). The name lookup for a file without field ids therefore missed every geometry inside a list or a map and left the default CRS spelled out. Name the paths as Iceberg does, including the legacy two-level list forms. * iceberg: rewrite the geometry CRS footer inside the merged file's buffer Removing a CRS shortens the footer, so the new one fits where the old one was. Copying the whole merged file to replace a few hundred bytes of footer doubled the memory of a compaction near the target file size. * iceberg: take the lineage codec from the data columns, not an input's _row_id An input that already carries _row_id may list it before its data columns, and write it uncompressed; the new lineage columns then copied that codec instead of the data columns'. Choose from the columns the layout keeps. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
de2fc03c7d |
s3tables: accept the external-volume verification object at a table bucket's root (#11900)
* s3tables: accept the external-volume verification object at a table bucket's root Snowflake verifies an S3-compatible external volume before it creates or writes Iceberg tables in it (SYSTEM$VERIFY_EXTERNAL_VOLUME, also run implicitly): it writes, reads, lists and deletes one small object, <base>/verify_<epoch milliseconds>_<number>, a 21-byte body. A table bucket refuses every key outside the Iceberg layout, so the check failed and Snowflake could not use a table bucket as an external volume. PUT, GET, HEAD, DELETE and DeleteObjects now accept exactly that key in a table bucket, narrowly enough that it cannot be used to store data: - at the bucket root only, so nothing is encoded in a prefix, written into a hidden tree or placed on a namespace or table name; - one shape: verify_<13 digits of epoch ms within one hour of the server's clock>_<1-10 digits>; - a PUT must declare a Content-Length of at most 1 KiB (no chunked or undeclared body) and carry no user metadata, tagging, Content-Disposition, Cache-Control, Expires or website redirect. Multipart, copy, rename, ACL, tagging and POST policy writes keep the plain layout rule. Buckets that are not table buckets are unaffected. Tests: table_bucket_verify_object_test.go, the key accepted (with the Iceberg layout as the positive control and the plain rule still refusing it), 15 look-alikes refused (prefixed, hidden tree, wrong shape, outside the window, ordinary keys), and seven PUT refusals through PutObjectHandler. * s3tables: keep the verification key off namespaces and narrow its PUT A namespace may be named like a verification key (verify_<ms>_<n>), and the key then names the namespace's directory. A PUT replaced that directory with a prefix object, dropping the namespace's catalog metadata; a DELETE removed an empty namespace. The key now gets the plain layout rule when it names a directory or its entry cannot be read, and a PUT of it never falls back to a prefix object, so a namespace created after the check is not replaced either. The time window applies to PUT only: GET, HEAD and DELETE take a key of any date, so an object a check left behind for more than an hour can still be read and removed. A verification PUT also refuses an extraction request and the stored Content-Language, Response-Content-Disposition and Content-Encoding headers (aws-chunked framing alone is still accepted). The integration test runs the check's PUT, HEAD, GET, list and DELETE against a table bucket, and the PUT and DELETE over a namespace of the key's shape. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
21add33b6e |
s3 backend: optional HTTP client settings for the cloud tier (#11897)
* s3 backend: optional HTTP client settings for the cloud tier The S3 tier backend builds its client from a bare aws.Config, so it always uses Go's default transport: HTTP/2 wherever the endpoint offers it over TLS, no cap on connections per host, and a 90s idle timeout. Against some S3-compatible services that is a poor fit. Over HTTP/2 every upload and range read of a tiered volume shares one connection and its flow-control window; with a small window, tier uploads and concurrent reads of tiered volumes become throughput-bound on that one connection, while HTTP/1.1 opens a connection per concurrent request. Some services also limit parallel connections per client address, or drop idle connections before 90s. Add three optional settings to [storage.backend.s3.<id>], all off by default so existing deployments keep the SDK's default client: - disable_http2: keep the client on HTTP/1.1 (ForceAttemptHTTP2=false and an empty TLSNextProto, the documented opt-out); - max_connections: MaxConnsPerHost, 0 = no cap; - idle_connection_timeout_seconds: IdleConnTimeout, 0 = Go's default. The session cache key includes the options when any is set, so two backends on the same endpoint with different settings do not share a client. ToProperties reports the settings; the master.toml scaffold documents them. Tests: TestS3BackendHTTPOptionsReachTheClient fails without the change (the settings never reach the client); TestS3BackendDefaultHTTPClientUnchanged, TestS3BackendSessionsDifferByHTTPOptions, TestParseNonNegative. * s3 backend: reject an idle timeout too large for time.Duration An idle_connection_timeout_seconds above math.MaxInt64/1e9 overflowed when multiplied by time.Second. The negative result made the transport keep its default timeout and ToProperties export a negative value. Warn and use the default instead. * s3 backend: parse the idle timeout as 64-bit seconds parseIdleConnTimeout went through strconv.Atoi, so on 32-bit platforms any value above 2147483647 seconds was rejected while the same value was accepted on 64-bit, and the overflow test failed under GOARCH=386. Parse with ParseInt(…, 64) and export the effective value with FormatInt so the accepted range is the same on every platform. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
9a03c09396 |
s3api: cover the inline-content UploadPartCopy route at handler level (#11914)
s3api: exercise the inline-content route through CopyObjectPartHandler Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
f40267db44 |
s3: answer InvalidBucketName to a denied request on a malformed bucket name (#11893)
* s3: answer InvalidBucketName to a denied request on a malformed bucket name The store authorizes a request before it looks at the bucket, so a caller denied on a bucket whose name breaks the S3 naming rules (upper case, an underscore, too short, two adjacent periods, a non-ASCII letter) gets 403 AccessDenied, where AWS S3 answers 400 InvalidBucketName. Clients and compatibility suites that check the error code read the 403 as a permission problem. Auth and AuthPostPolicy now turn an AccessDenied for a resolved identity into InvalidBucketName when the bucket name fails VerifyS3BucketName or uses a character outside a-z, 0-9, '.' and '-' (VerifyS3BucketName checks with unicode.IsLower/IsNumber, which accept letters such as 'ü'). The answer depends on the name alone, so it reveals nothing about which buckets exist. An authentication failure, an allowed request and a denial on a well-formed name answer as before. Test: TestAuthDeniedInvalidBucketName drives iam.Auth with signed and anonymous requests; on master every malformed name answers 403, with the change 400, while the identity's own bucket stays allowed and a well-formed foreign name stays 403. TestRefineDenialForBucketName covers the rule table. * s3: refine POST-policy and deferred anonymous-read denials on a malformed bucket name Two denial paths bypass the refinement in Auth and AuthPostPolicy: - A browser POST upload carries its credentials in the form, so AuthPostPolicy delegates it to PostPolicyBucketHandler, which verifies the form and writes AccessDenied itself when the signer may not write the bucket or the per-key authorization denies the object. Both denials now go through refineDenialForBucketName with the verified identity. The coarse Write check used to run inside doesPolicySignatureV2Match and doesPolicySignatureV4Match before the signature was compared, so a form with a wrong signature was told whether the key may write the bucket. It now runs in doesPolicySignatureMatch after the signature verifies, as on the header-signed path; a form that fails verification answers SignatureDoesNotMatch (or MissingFields) and is never refined. - Unsigned GET/HEAD object reads are deferred by AuthWithPublicRead to the object handlers, where writeAnonymousAwareNotFound masks a miss as AccessDenied. That denial is refined with the anonymous identity, so a missing malformed bucket answers InvalidBucketName when an anonymous identity is configured, as Auth does. A denial on an entry that was found is left alone: the bucket exists. Test: TestPostPolicyDeniedInvalidBucketName drives AuthPostPolicy with signed forms (Write denied, per-key policy deny, bad signature, well-formed name); TestAnonymousReadDeniedInvalidBucketName drives AuthWithPublicRead with GET and HEAD, with and without an anonymous identity. Both fail before this change. * s3: refine a denied anonymous read of an existing key on a malformed bucket name A deferred anonymous GET/HEAD refined only its masked miss to InvalidBucketName, so in a bucket whose name breaks the S3 rules (one VerifyS3BucketName still accepts, such as a non-ASCII name) a missing key answered 400 and an existing private key 403, telling the caller which keys exist. The entry check now refines its denial the same way, so both answer alike. Without an anonymous identity no identity resolves and both stay 403, as in Auth. The deny-precedence case in TestAnonymousObjectACLConfiguredIdentity runs on the fixture bucket "b", shorter than S3's three-character minimum, so its denial now answers InvalidBucketName. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
6ee7c4c5c7 |
s3: no empty-directory marker in a "<dir>/" self-prefix listing (#11906)
An out-of-band empty directory listed under its own trailing-slash prefix answered the probe with a fake entry of itself: a Contents key on 4.48 and, since recently, a CommonPrefix. That listing enumerates children, where S3 returns nothing for an empty prefix, and clients that read every returned key (Pulumi lock listing) choked on the marker. The prefix object and directory key object paths are unchanged: a real "dir/" object still lists as its own key. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
 Javier de la TorreGitHubJavier de la TorrechrislusfDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
9c427b265f |
volume tier: accept deletes on a volume as soon as it is tiered (#11894)
* volume tier: accept deletes on a volume as soon as it is tiered
volume.tier.upload marks the volume read-only without deletes before the
upload (markVolumeReplicasWritable(..., false, false), i.e. noWriteOrDelete),
and nothing clears that until the volume server restarts, when load() gives
every tiered volume noWriteCanDelete. Every delete in between is refused with
"volume N is read only": the needle stays live in the index, and its bytes in
the remote object, until someone notices.
After the remote file is loaded, the upload handler now gives the volume the
flags a restart would (MarkTieredDeletable: writes refused, deletes accepted),
with two exceptions that keep the current behaviour:
- a replicated volume: the shell copies the uploaded replica's .idx and .vif
to the other replicas after the upload, and a tombstone written before that
copy would reach one replica only;
- a volume in I/O quarantine: that flag is not the upload's to lift.
Tests: TestTierUploadAcceptsDeletesOnASingleCopyVolume fails without the
change ("volume 1 is read only") and checks writes stay refused;
TestTierUploadKeepsRefusingDeletesOnAReplicatedVolume; TestMarkTieredDeletable
(quarantine kept).
* volume tier: check the I/O quarantine under noWriteLock
MarkTieredDeletable read the quarantine state before taking noWriteLock.
CollectHeartbeat could quarantine the volume and set noWriteOrDelete in
between, and the upload then cleared that refusal, so a quarantined volume
accepted deletes until the next heartbeat. Take noWriteLock first: the
heartbeat marks the quarantine before it takes noWriteLock, so either the
check sees it or the heartbeat's refusal lands afterwards and stays.
Also leave alone a volume past IoErrorTolerance that the next heartbeat
will quarantine, as the heartbeat itself does.
* volume: read noWriteOrDelete under noWriteLock in DeleteVolumeNeedle
DeleteVolumeNeedle checked v.noWriteOrDelete without the lock that
MarkVolumeReadonly, CollectHeartbeat and now MarkTieredDeletable hold when
they write it, so a delete racing a tier upload was a data race. Copy the
flag under the read lock; the delete path is otherwise unchanged.
* volume tier: mirror the delete acceptance rules in the Rust volume server
The Rust tier upload already promoted a freshly tiered volume to
no-write-can-delete via refresh_remote_write_mode, and did so for every
volume -- including the replicated and I/O-quarantined ones the Go
MarkTieredDeletable deliberately refuses. A live tier-up now goes
through mark_tiered_deletable (replicated volumes and quarantines keep
their upload-time mark until the next load), while load() keeps
promoting every tiered volume unconditionally, as Go does.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable
MarkVolumeWritable preserved noWriteCanDelete on a remote-tiered volume
only when it was already set. A tiered volume still holding the
upload-time mark (a replicated volume, or one whose tier upload ran
before MarkTieredDeletable existed) had it clear, so marking writable
published the volume as fully writable while its .dat lives remotely --
writes then failed against a backend that cannot append, after the
master was told the volume is writable again.
load() gives every remote-tiered volume noWriteCanDelete; deriving the
same flag here keeps mark-writable consistent with a restart: "writable"
on a remote volume resolves to can-delete.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume: keep a tiered volume delete-only when marked writable (rust)
set_writable preserved the remote delete-only mode only when the flag was
already set, so a tiered volume holding the upload-time read-only mark
(replicated, or quarantined under mark_tiered_deletable) was published
fully writable with its .dat remote. Derive no_write_can_delete from
has_remote_file() instead, matching load() and the Go MarkVolumeWritable.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* volume tier: serialize the quarantine check with the flag swap (rust)
A heartbeat refusal landing between should_quarantine() and the store
could be clobbered until the next heartbeat re-asserted it — the same
race the Go side closed by checking under noWriteLock. A
compare-exchange gives the same ordering: the swap fails, and the retry
sees the quarantine mark the heartbeat sets first.
* volume tier: MarkTieredDeletable refuses a non-tiered volume
A stray call on a local single-copy volume would pin it
read-only-can-delete. The only caller promotes right after
LoadRemoteFile, but the guard makes the method safe to reuse; the Rust
mark_tiered_deletable gets the same check, though its caller already
only reaches it under has_remote.
---------
Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com>
Co-authored-by: chrislusf <chris.lu@gmail.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
|
97f8ca13ff |
s3tables, iceberg: log client refusals at V(1), not ERROR (#11896)
* s3tables, iceberg: log client refusals at V(1), not ERROR Requests the S3 Tables and Iceberg REST APIs correctly refuse with a 4xx (access denied, not authorized, no such namespace, already exists, not empty, a commit conflict) were logged with glog.Errorf, so a client probing for what it may do, or retrying an optimistic commit, filled the ERROR log and hid real faults from anyone alerting on it. - s3tables: HandleRequest wraps the ResponseWriter in a statusRecorder (with Unwrap for http.ResponseController) and logs a handler error by the status it answered: 4xx at V(1) as "<op> refused (<status>)", 5xx or no status at ERROR as before. - s3tables: the "handleCreateNamespace called" trace moves from ERROR to V(3). - iceberg: writeManagerError's mapping is split into classifyManagerError, shared with a new logManagerError that logs 4xx-mapped errors at V(1) and everything else at ERROR; used by CreateNamespace, CommitTable and CommitTransaction. - storage: loading a remote-tier volume logged "readSuperBlock remote volume N: <nil>" at ERROR even when the read succeeded; it is now logged only when there is an error. Tests: TestRequestRefusedWith4xxIsNotLoggedAsError, TestLogRequestErrorByStatus, TestStatusRecorderKeepsTheFirstStatus, TestStatusRecorderUnwrapsForResponseController, TestNoTraceLoggedAtErrorLevel (fails on master on the CreateNamespace trace), TestLogManagerErrorRefusalsAreNotErrors, TestRemoteSuperBlockLogOnlyOnError. * s3tables: do not log a malformed CreateNamespace body at ERROR handleCreateNamespace logged a body it could not decode at ERROR before answering 400; the dispatcher already logs that refusal once at V(1). Also drop statusRecorder.Flush: it made the recorder an http.Flusher even over a writer that cannot flush, so http.NewResponseController reported success instead of ErrNotSupported. Unwrap already lets the controller reach the wrapped writer. * iceberg: require exactly one log line per classified manager error The test accepted a refusal that produced no log, or both logs. Require one V(1) refusal and no ERROR for a 4xx, and the reverse for a fault. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
780760fccc |
s3: report the copied bytes' MD5 in UploadPartCopy ETags (#11884)
* s3api: regression test for UploadPartCopy ETag on ranged copies Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: hash the copied bytes for a partial-chunk UploadPartCopy ETag A ranged UploadPartCopy that lands inside a source chunk stored the source chunk's ETag on the destination chunk, so the part reported the MD5 of the wrong bytes. streamCopyChunkRange now tees the forwarded plaintext into an optional digest writer; copySingleChunkForRange feeds it an md5 only when the range is not the whole source chunk, since a whole-chunk copy's stored bytes (and ETag) are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: stamp the copied range's MD5 on UploadPartCopy part entries The part entry got no Attributes.Md5, so copyEntryETag fell back to filer.ETagChunks: a single-chunk part inherited the source chunk's MD5 and a multi-chunk part reported a <md5>-N composite of source chunks. copyChunksForRange now returns the MD5 of the copied bytes — a single chunk's own digest, or for multi-chunk parts a second ordered stream of the source range hashed beside the concurrent chunk copies (md5 cannot combine per-chunk digests). The handler stores it on Attributes.Md5 and mirrors putToFiler's ExtETagKey, so CopyPartResult, ListParts and the completed object's composite ETag use each part's own MD5 (#11855).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: drop the source chunk's gzip flag on partial-range copies A ranged UploadPartCopy fetches raw (decompressed) bytes from the source volume and stores them uncompressed, but the destination chunk kept the source's IsCompressed flag, so reads of a partial copy of a gzip-stored chunk would try to decompress raw bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: hash the chunk range, not the entry, for the part MD5 getEncryptedStreamFromVolumes prefers inline Content over chunks, but copyChunksForRange copies the chunks — on an entry carrying both, the hash would describe different bytes than the copy.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cover sparse ranges in the part MD5 and fail on hash errors A single chunk's digest is the part MD5 only when it supplies the whole range; a sparse source zero-fills the rest, so gaps are hashed via the ordered range stream, and a fully chunkless range hashes zeros locally. A failed hash pass now fails the part copy instead of storing a chunk-composite ETag that describes the wrong bytes.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: rank copied-range destination chunks by source timestamps The concurrent chunk copies stamped each destination chunk with an arbitrary finish time, but reads resolve overlapping chunks by ModifiedTsNs — a copy of overlapping source chunks could expose different bytes than the source (and than the part MD5 just hashed). Destination chunks now get timestamps ranked by the source's, preserving its overlap precedence.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: cancel the part-MD5 stream when a chunk copy fails The hashing goroutine ran on context.Background(), so an early chunk-copy error return left it streaming and hashing the whole source range for a discarded result. A cancellable context now aborts it on every exit path.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: hash the copied range when the covering chunk has no usable ETag TUS and mount writes leave chunks without an ETag, so a whole-range UploadPartCopy would otherwise save the part with an empty Md5 and no ExtETagKey, and multipart completion rejects the part. * s3api: build the md5 source entry without copying the proto message * s3api: drop the issue reference from the part-copy etag test * s3api: route inline-content sources through the plaintext part copy The raw-byte path walks entry.GetChunks() only, so a ranged UploadPartCopy of an inline-Content source (filer HTTP or mount writes) produced a zero-filled part. Send those through copyObjectPartViaReencryption, which streams the real bytes and stamps their MD5 on the part. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3api: store the empty-part MD5 on writeEmptyCopyPart entries The slow path's empty part returned the right CopyPartResult ETag but saved no Attributes.Md5 or ExtETagKey, so ListParts and the completed object's composite ETag still fell back to a chunk-derived value. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
a3e97ec709 |
s3: apply Range to HeadObject (#11856)
* s3: apply Range to HeadObject * s3: apply Range to a HEAD of a stored directory key Directory-key entries answer through serveDirectoryContent before HeadObjectHandler's range branch, so a ranged HEAD returned 200 and the full length. * s3: keep the missing-directory check for plain directories only A stored directory-key marker is an object, so a range on its empty body answers 416 like on any other zero-byte object. * s3: let partNumber keep its priority over Range on a directory key HEAD * s3: serve ranged GETs of a stored directory key The same branch now applies the validated range to the inline content on GET, so a directory-key object answers 206 with the selected bytes like any other object. --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
  Lisandro PinGitHubClaude Opus 5.5Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Chris Lu |
23d8c62844 |
volume: route lookups to the leader master only (#11888)
* volume: route lookups to the leader master only (#11887) After a leader change, the volume server kept sending volume and EC shard lookups to masters that were not the leader: * currentMaster was set as soon as a heartbeat stream opened, before the peer said whether it was leader, and kept a follower through the redirect and its 3s sleep. A follower answers LookupVolume from its own vid cache, which right after an election can be empty, so reads went 404 rather than failing over. * EC lookups used Store.MasterAddress, a second, unlocked copy that the heartbeat loop set before each dial and cleared on errors, so EC and plain reads could disagree on the master or have none. * the receive goroutine called stream.Send itself when the volume options changed, racing the send loop, and exited silently on failure, after which leader redirects were no longer read. Publish currentMaster only once a master is known to be leader: the peer answered a new stream without naming a leader, which the leader always does, or a follower named the leader. Compare redirects against the connected peer. Replace Store.MasterAddress with MasterFn, backed by getCurrentMaster. Have the receiver ask the send loop for the extra heartbeat instead of sending it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: ask the masters for the leader after losing the heartbeat When the heartbeat stream to the leader failed, currentMaster kept the lost master until a new stream answered, so lookups on a cache miss kept dialing it through the retry delay even with a new leader already elected. On a regular heartbeat error, ask the configured masters for the raft leader and publish it, within the retry delay rather than on top of it. Without an answer the last known leader stays: clearing it would only trade a dead address for an empty one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: stop the heartbeat on a failed assertion The leader test closed stopChan only once every assertion had passed, so a failure left the stream open and the fake master's GracefulStop waited for the whole test run's timeout. Start the heartbeat through a helper whose cleanup stops it, safe to call twice, and waits for it to exit. It runs before the cleanups registered earlier, so the stream is closed before the store and the fake master go away. Both heartbeat tests use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: clear the leader when no master names one When the heartbeat lost its master and no configured master named a leader, refreshLeaderFromPeers kept the lost one. That master is dead or no longer leader, and a follower answers LookupVolume from its own cache instead of refusing, so lookups could return locations no leader vouched for. Clear currentMaster instead. Lookups then fail fast ("missing address") until a leader is known again, rather than reading a follower's cache or waiting on a dial to a dead master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * volume: keep volumeSizeLimit 64-bit aligned on 32-bit builds MasterFn (a func, 4 bytes on 386) replaced MasterAddress (a string, 8 bytes) ahead of volumeSizeLimit and moved it to a 4-byte-aligned offset, where atomic.LoadUint64 panics with an unaligned 64-bit operation. The atomic.Uint64 type carries the alignment itself. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish the answering leader before a duplicate-uuid retry Read the leader field first on each heartbeat reply. A follower reply carries nothing else, so it no longer runs through the option updates; and a leader that rejects the registration over duplicate uuids still answers lookups, so it stays the lookup target through the retry wait instead of leaving the previous master in place. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish the master only once it is known leader (rust) * volume: ask the masters for the leader after losing the heartbeat (rust) * volume: read the leader field first on each heartbeat reply (rust) * volume: drop the seed fallback from ec shard lookups (rust) cached_lookup_ec_shard_locations fell back to the static first seed when no leader was published, so a mid-election EC lookup could land on a follower and read its stale shard map. Error out like Go's masterAddress() returning "" instead, via a current_master() getter matching getCurrentMaster. * volume: send volume lookups to the confirmed leader, not the seed (rust) Every HTTP lookup path — proxy/redirect targets, replicated writes and deletes, batch chunk deletes, remote chunk reads — dialed the static first seed, so after a failover they kept asking a follower whose vid map can be stale. Go routes all of these through GetMaster; switch the lookup calls and their leader-known gates to current_master(). * volume: mark volumes readonly on the leader, retrying the named one (rust) notify_master_volume_readonly fell back to the seed when no leader was published, which the peer answers with "Not current leader". It now uses only the confirmed leader, skips the call in masterless mode, and on NotLeader asks the seed masters who leads and retries there — the same path Go's notifyMasterVolumeReadonly takes. The peer leader query is shared with refresh_leader_from_peers as lookup_leader_from_peers. * volume: ask the confirmed leader for copy prealloc options (rust) VolumeCopy's preallocate query went to the static seed; Go issues it through GetMaster. Route it through current_master() too — when no leader is known the lookup fails fast and the copy proceeds without preallocating, as before. * volume: keep the announced master port forms, not just http (rust) publish_current_master stripped to http form, so a leader configured with an explicit gRPC port (host:9333.19999) was redialed on the default offset. Store the announced host:port[.grpcPort] form like Go's currentMaster; callers pick to_http_address / to_grpc_address / parse_grpc_address per use, and Ping admission compares on the http form. * volume: fail replicated writes and deletes when no leader is known (rust) Gating replication on a known leader meant a write could be acknowledged on fewer copies than required, and a delete could leave the data readable on the other replicas. Go fails the request when the replica-location lookup cannot run; drop the gate so the empty-leader lookup errors the same way. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
4bdbfbaaa8 |
s3tables: judge an absent tag target by its canonical ARN (#11912)
* s3tables: judge an absent tag target by its canonical ARN authorizeAbsentTarget matched the bucket policy's Deny statements against the request's raw resource ARN. The ARN parsers accept any account and region but resolve the target by name, so a caller with an identity grant could reformat the ARN, miss the Deny match, and learn whether a denied name exists — the oracle the absent-target check was meant to close. Rebuild the ARN from the resolved names and the real bucket owner and evaluate the Deny against that, matching what the existing-target path checks. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: cover a foreign-account ARN on an absent denied table Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: keep the absent-target deny check behind a supplied ARN Callers that pass no resource ARN (ListTables, ListViews, CreateNamespace) must reach CheckTableOrBucketPermission's admin shortcut unchanged; running the bucket-policy Deny match first would deny the admin lists those callers take. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
e73c23a4ee |
fix(s3): set Access-Control-Allow-Credentials on CORS responses (#11886)
* fix(s3): set Access-Control-Allow-Credentials on CORS responses (fixes #11885) * fix(s3): expand wildcard exposed headers for credentialed CORS responses * fix(s3): keep wildcard in exposed headers for non-credentialed CORS requests * fix(s3): expose x-amz-tagging-count on credentialed CORS responses --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> |
||
|
|
8c3d8fcea0 |
s3tables: keep absent-target answers identical to denied ones (#11905)
* s3tables: authorize before revealing a list or tag target exists ListTables, ListViews and the tag operations read the target's catalog metadata before any permission check, so a denied caller could tell an existing name (403 or the resource's own error) from an absent one (an empty list, or 404). Evaluate the operation's permission with whatever bucket context could be read and answer a denied caller the same way for absent, damaged and denied targets. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: answer bucket reads uniformly when the entry is not a table bucket A plain S3 bucket directory answers GetTableBucket and ListNamespaces with 500 attribute-not-found while a missing name gets 404, and a denied CreateNamespace is 403 while a missing bucket is 404. Run the permission check before the absence answer and read not-a-table-bucket as NoSuchTableBucket. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3tables: cover absent-target answers matching denied ones Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
bb1ba8b247 |
iam: take the implicit user name from the verified identity (#11903)
* iam: take the implicit user name from the verified identity For self-service actions sent without UserName, the implicit user was resolved by re-parsing the raw Authorization header for a SigV4 credential and looking that access key up. The header is not what authentication verified: a request can be authenticated by one mechanism while carrying a header that names another user's access key, and the action then ran against that other user. The auth middleware already records the verified identity in the request context, so use it, on both the standalone iam server and the embedded IAM surface. When no verified identity is present the name is left unset rather than guessed. * iam: test implicit user name resolution Cover a request authenticated by presigned credentials that also carries a junk SigV4 header naming another user: the self-service actions must run as the verified signer. Update the unit tests for the context-based lookup. * iam: resolve the implicit user for standalone GetUser too The embedded surface fills UserName for GetUser; the standalone server skipped it, so an implicit GetUser on `weed iam` answered NoSuchEntity. Same verified-identity source as the other self-service actions. * iam: cover all implicit-username actions in the regression test |
||
|
|
593cfee85b |
s3tables: accept Puffin statistics files in a table's metadata/ (#11892)
Table and partition statistics are Puffin files that the table metadata lists in its statistics entries; orphan cleanup keeps them (#11651), and its test places one at metadata/stats-1.puffin. The table bucket layout rule accepted only *.stats among them, so an engine that names a statistics file *.puffin had the upload refused with 403 and could not write statistics at all. The metadata patterns now admit a *.puffin file directly under metadata/, with the same safe-character rule as the other catch-all patterns. Nothing else widens: a Puffin file in a subdirectory of metadata/, with a trailing extension, or outside <namespace>/<table>/ is still refused. Test: the new metadata/*.puffin cases in TestIcebergLayoutValidator_AcceptsRealWorldManifestNames and TestTableBucketFileValidator_PuffinStatisticsOnlyInTableMetadata fail on master ("invalid metadata file format") and pass with the change; the rejected cases pass on both. Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
 Daniel GarcíaandGitHub
|
03065155db |
filer.remote.sync: do not upload an entry whose only content is the remote object (#11890)
* filer.remote.sync: do not upload an entry whose local copy was dropped remote.uncache and the filer's eviction under storage pressure drop a synced entry's chunks and clear LastLocalSyncTsNs; remote.meta.sync does the same when it pulls a newer remote version. When the remote mtime falls in the entry's mtime second, which is the common case for a small file synced promptly, shouldSendToRemote falls back to the cleared stamp and reads the entry as dirty. The dropped chunks make the update a content change, so the chunkless entry was uploaded and a zero-byte object replaced the remote one, the entry's only content. Skip an update that leaves the entry at the same key with no local data and a sized RemoteEntry: the object at that key already is the content. The stamp stays cleared, so remote.cache and eviction still see the entry as uncached. Renames keep going through completeRemoteOnlyRename, metadata-only edits of a remote-only entry still reach the remote, and a rewrite in the remote mtime's second holds local data and is uploaded. * filer.remote.sync: upload a truncate that leaves the entry remote-only A FUSE truncate of a cached, remote-backed file to zero (truncateEntry, or SetAttr with size 0) drops the chunks and keeps the sized RemoteEntry, so the entry reads as remote-only and the dropped-copy skip left the remote object with its old content. Skip only while the entry's size is still the remote object's. Uncache and eviction keep the size and remote.meta.sync sets it from the object, so those stay skipped; a truncate changes the size and is uploaded, in the remote mtime's second too. |
||
 
|
f8cfea0dcf |
gcs: size the upload buffer from the entry instead of reserving the default 16 MiB chunk per writer (#11854)
Co-authored-by: james <james@medable.com> |
||
|
Jay HemnaniandGitHub
|
c90da999f2 | s3: answer a conditional write on a missing object with 404, not 412 (#11882) | ||
|
|
379abbe278 |
volume: repair torn tails on canDelete volumes; reject damaged 1-3 byte records (#11876)
- Go/Rust: run the unindexed .dat tail repair on noWriteCanDelete volumes too — deletes still append tombstone records, so the same crash window applies and they previously degraded to fully read-only. - Go readTailNeedle: reject a positive size of 1-3 for v2+ as corrupted before ReadBytes slices the body's 4-byte data-size field and panics; a torn last record is now cut instead of crashing the load. - Tests: TestVolumeLoadRepairsTailOnCanDeleteVolume and TestVolumeLoadCutsDamagedSmallSizeTailOnCanDeleteVolume (Go), and the Rust mirrors volume_load_repairs_tail_on_can_delete_volume / volume_load_cuts_damaged_small_size_tail. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
 
|
c601712cc4 |
README: describe OLake Go and fix its wiki link (#11880)
README: describe OLake Go and point its link at the renamed wiki page Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |
||
|
|
ab738bfd33 |
rust volume: repair an unindexed .dat tail at load (#11878)
* rust volume: repair an unindexed .dat tail at load Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: tests for the unindexed .dat tail repair Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: re-index a torn size 0 record as a live empty needle A delete and a write with no data append the identical size 0 record, so a recovered one cannot be told apart from the prior index state alone: the write path accepts an empty overwrite of a live key and indexes it with size 0, while a delete of the same key publishes a tombstone. Inferring a tombstone from prior live state turned a lost index row after an empty overwrite into a deletion -- the object read back not-found although the write was answered. Every re-indexed record is now indexed at its on-disk size, the entry publishWriteRequest writes for an empty write. The other misread is bounded: a recovered delete reads its key back as an empty object rather than not-found, and the size 0 record carries no data to resurrect. Review feedback on the Rust mirror (#11878): the same bug exists here; fixing both sides identically. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: re-index a torn size 0 record as a live empty needle Mirrors the Go fix in this pull request. A delete and a write with no data append the identical size 0 record, so resolve_empty_tail_records could not tell them apart from the prior index state: it treated a zero-size record of a live id as a tombstone, although the write path indexes the same bytes as a live size 0 entry. A crash after an empty overwrite of a live key therefore recovered the object as deleted. Every re-indexed record is now indexed at its on-disk size, the entry publish_write writes for an empty write. A recovered delete reads its key back as an empty object rather than not-found; the size 0 record carries no data to resurrect. Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: repair a .dat tail left past an empty .idx The writable index loader creates the .idx before the first needle append, so a crash after that append leaves a volume whose data nothing indexes. CheckVolumeDataIntegrity skipped an empty index outright, so the volume loaded writable and the surviving first needle stayed invisible while later writes appended past it. A zero-size index now runs the same repair with tailEntryPos -1: the walk starts at the first record past the superblock with no anchor needle and no append timestamp to compare against, gets the same complete-record validation and incomplete-tail policy, and re-verifies through the normal tail check once it wrote entries. A tail the repair cannot explain fails the check, so the volume loads read-only instead of writable on an unaccounted-for .dat. Review feedback on the Rust mirror (#11878): the same bug exists here; fixing both sides identically. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: repair a .dat tail left past an empty .idx Mirrors the Go fix in this pull request. The writable index loader creates the .idx before the first needle append, so a crash after that append leaves a volume whose data nothing indexes. Both the repair and the integrity check skipped an empty index, so the volume loaded writable and the surviving first needle stayed invisible while later writes appended past it. An empty .idx now runs the same repair with no anchor: the walk starts at the first record past the superblock, and a tail it cannot explain fails the load like any other repair failure, since every later check skips an empty index. Review: devin-ai-integration[bot] on #11878. Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: sync the .dat before publishing re-indexed .idx rows A power loss after the repaired .idx rows become durable but before the .dat tail records they name are flushed leaves the index pointing past the end of the data file; the next load then keeps the volume read-only until an operator steps in. Sync the .dat first so the rows never outlive the records they reference. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: sync the .dat before publishing re-indexed .idx rows Mirrors the Go fix: a power loss after the repaired .idx rows become durable but before the .dat tail records they name are flushed leaves the index pointing past the end of the data file. Sync the .dat first. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
222e96d6ab |
iceberg maintenance: merge bin files across parquet schema spellings (#11875)
* iceberg maintenance: compare parquet schemas by field id in bin-pack merge parquet.EqualNodes ignores Iceberg field ids, so a column dropped and\nre-added under the same name and type compared equal and was merged,\nfiling the new column's values under the dropped field id. Require the\nfield ids to agree wherever both sides carry one; the v3 layout's\nper-field check uses the same test.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: merge bin files whose schemas differ by field id A bin used to take the first file's Parquet schema as the writer's and\nrefuse every file that was not structurally identical, so a table never\ncompacted once files had a different column set or REQUIRED-vs-OPTIONAL\nrepetition. A file whose flat columns the writer's schema covers by\nfield id is converted into it (a column the file lacks reads as null);\na file that covers the writer's schema restarts the bin with it, so no\ncolumn is dropped. Equality-delete columns resolve on each file's own\nschema. Renames, type changes, nested columns and files without field\nids stay refused. The sorted and spatial merges are unchanged.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: test bin-pack merge across parquet schema spellings Regression coverage for the field-id merge: an added column in either\norder, REQUIRED vs OPTIONAL, equality deletes resolved per file, the\nre-added column refused on v1/v2 and v3, and the refusals that stay\n(rename, type change, one-sided columns, an added required column, no\nfield ids).\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve missing delete columns as null in bin-pack merge An equality delete on an optional column added after a data file was\nwritten no longer refuses the bin: the file reads the column as null,\nso a real delete key cannot match its rows and a null key deletes them.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: write bin-pack merges under the files' union schema The writer schema is now the union of the bin's file schemas by field\nid, computed in a schema-only pass before any row is written: a column\nstays REQUIRED only where every file has it and spells it so, and the\nmerge refuses only true name/type/id conflicts. Files that need a union\nneither side covers (an added column plus a REQUIRED/OPTIONAL\ndifference, disjoint added columns) now merge in a single pass instead\nof refusing, and progressively wider bins no longer reread and rewrite\nevery earlier file per restart.\n\nGenerated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: spell the re-added column's test file for codespell Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: resolve equality-delete columns by field id A column renamed after its data and delete files were written still carries its field id there; resolving it by id instead of the current name keeps a rename from turning every equality key null. The missing-column null-key fallback is now for data files only: an equality-delete file that lacks its equality column is refused, as the spec requires it to carry them. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iceberg maintenance: read only the parquet footer choosing a bin's writer schema The union-schema pass downloaded every input in full, then the merge downloaded each again. The footer suffices for the schema, and a chunked file's ReadAt pulls only the chunks the footer spans. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
a724f4296c |
s3, s3tables: create buckets and namespaces exclusively (#11872)
* filer_pb: add MkdirExclusive A taken name leaves the callback as a result, not an error, so filer failover does not count a lost race as a filer failure. * s3tables: create the entry with its attributes in one exclusive create CreateTableBucket and CreateNamespace checked that the name was free and then wrote the entry in several calls, so a second create in the window replaced the first caller's entry including its owner. The create now carries the marker, metadata and tags in one O_EXCL CreateEntry that answers ErrEntryAlreadyExists with the same 409. * s3: PutBucket uses an exclusive mkdir The same window let a plain PutBucket replace a table bucket created concurrently. The already-exists branch now answers from the entry itself, since the registry just cached the name as absent, and the admin auto-create path is exclusive too. * s3: read the winning entry when a create loses the race The concurrent-create branch called existingBucketError on a name the registry had just cached as absent, answering NoSuchBucket, and the auto-create fallback could assign an S3 owner to a table bucket it met there. The negative cache is cleared before the existing-bucket error is built, and a table bucket keeps its owner-less shape. * s3: fail an auto-create that met a table bucket A table bucket carries no S3 owner and must not gain one, and the upload that auto-create ran for must not continue into it either. * s3: fail an auto-create when the winning entry cannot be read An unreadable existing entry fell through to success, so the write continued into an unvalidated entry, possibly a table bucket. |
||
|
Chris LuandGitHub
|
f05d743459 |
s3: resolve versionId tagging requests to the version-tagging actions (#11874)
* s3: resolve versionId tagging requests to the version-tagging actions A request for an object's tags with versionId acted on a historical version but authorized against the current-object tagging action, so an explicit Deny on s3:*ObjectVersionTagging never applied. * iam: map PutObjectVersionTagging and DeleteObjectVersionTagging to the tagging action * s3: only resolve tagging when the query value is empty The tagging routes match ?tagging with an empty value, so a non-empty one is not a tagging request and would resolve a tagging grant onto the plain read or version read that actually serves it. * s3: keep resolving tagging for any query value The routes match ?tagging whenever the key is present, so tagging=x is a tagging request served by the tagging handlers and must resolve to the tagging actions. |
||
|
Chris LuandGitHub
|
03a75cbc52 |
s3tables: only the admin repairs a table bucket that has no owner metadata (#11873)
* s3tables: only the admin repairs a table bucket that has no owner metadata CreateNamespace wrote bucket metadata naming the caller as owner before any permission check ran, so an ordinary caller that found the marker without metadata was written in as the bucket's owner and passed every check after. A bucket in that state is answered with the operation's own denial; the admin or a trusted internal handler may still repair it. * s3tables: repair the marker-only bucket under the entry lock A creator finishing its own create between the missing-metadata check and the repair write would have its owner overwritten. The mutation now adopts whatever metadata is there and writes only while it is still absent. * s3tables: gate the repair on the Admin action, not an operation grant hasAdminAction also answers s3tables:*, so a non-admin identity holding the wildcard operation grant could pass the repair check and write itself in as the bucket's owner. |
||
 |
efa9d304d2 |
build(deps): bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /test/kafka/kafka-client-loadtest (#11879)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.18.0 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.18.0...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
 github-actions[bot]
|
b7f5d04428 | docs: regenerate star history chart | ||
|
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
ece2f7a7c2 |
iam: targeted writes for single-user IAM actions (#11877)
* s3api: persist single-identity IAM changes through the credential manager CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy, DeleteUserPolicy, SetUserStatus, UpdateUser and DeleteUser used to load the whole configuration, change one identity and then SaveConfiguration, rewriting and re-propagating every identity. They now write only the identity (and, for rename/delete, the groups and service accounts that referenced it) through the credential manager, which propagates that identity alone. Static identities and skipPersist keep the previous full-save/no-write contract. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * iamapi: persist single-identity IAM changes through the credential manager Same targeted-write change as the embedded IAM API for the standalone weed iam dispatcher: UpdateUser, DeleteUser, CreateAccessKey, DeleteAccessKey, UpdateAccessKey, PutUserPolicy and DeleteUserPolicy write one identity through the credential manager instead of saving the whole configuration. CreateAccessKey keeps the full save when the handler created a user the store does not know. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * credential/memory: re-key identity on UpdateUser rename UpdateUser stored the identity under the old username key even when identity.Name changed. Re-key it the way filer_etc does (delete old, save new) so the targeted rename path sees the moved record. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover targeted single-identity IAM writes Recording-store tests assert each rerouted action uses the single-identity credential store methods and never SaveConfiguration, in both the embedded S3 IAM dispatcher and the standalone iamapi dispatcher, plus the skipPersist and static-identity contracts and a memory-store rename case. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * credential: keep not-found and rename-collision semantics in stores IamGrpcStore.DeleteAccessKey translates gRPC NotFound to ErrAccessKeyNotFound so a repeat delete stays the no-op the IAM handlers advertise. MemoryStore.UpdateUser rejects a rename onto an existing user instead of overwriting it, matching RenameUser. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
610346ae8c |
iceberg: keep version-hint.text out of remove_orphans (#11867) (#11871)
Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
df89afbef4 |
volume: repair an unindexed .dat tail at load instead of loading read-only (#11870)
* volume: repair an unindexed .dat tail at load instead of loading read-only A needle write appends the needle to the .dat, then its entry to the .idx, then answers the client. A crash between the two appends leaves a .dat that ends past the last indexed needle; on the next start CheckVolumeDataIntegrity reports "actual N bytes expected M bytes" and the volume loads read-only, and a complete needle in that tail cannot be read because nothing indexes it. After a process crash those bytes belong to a write that was never answered; after a power loss a complete needle can survive in the .dat while its .idx entry did not, although the client was answered. So complete needles are re-indexed rather than cut. CheckVolumeDataIntegrity now repairs the tail before the existing check, which then runs on the repaired files as before. The repair: - acts only on a local, version 3 volume within MaxPossibleVolumeSize, with a whole number of .idx entries, that is about to load writable (neither noWriteOrDelete nor noWriteCanDelete is set); - requires the needle at the highest indexed offset to match its entry (id, size, whole record, CRC); - walks from the end of that needle to the real end of the file and gives every complete needle (header, body, checksum, timestamp and padding inside the file, CRC valid, append timestamp later than the needle before it) the .idx entry the volume server would have written, in .dat order; a size 0 record, which is how a delete is appended, is indexed as a tombstone; - cuts only an incomplete last record: fewer bytes than a header, a record running past the end of the file, or a last record whose CRC fails; - changes nothing otherwise (a damaged record with more data after it, a header with id 0 or a negative size, append timestamps going backwards), so the volume loads read-only exactly as before. The .idx entries are written and synced before the .dat is truncated and synced, so a crash during the repair leaves the needles indexed and the remainder still past them; the next load cuts it. One warning per repaired volume names the needles re-indexed and the bytes cut. Tests: volume_checking_tail_repair_test.go reloads volumes through the normal load path. Repaired: one and two complete unindexed needles, an unindexed delete, a partial header, a partial body, a complete needle followed by a partial one, a last needle with a bad CRC, a partial needle after an indexed tombstone; each reloads clean afterwards. Left read-only and unchanged: the last indexed needle with a wrong id or a bad CRC, a bad CRC followed by more data, zero bytes past the last needle, an index past the data, a volume persisted read-only, a partial .idx entry, and a clean volume. TestCheckVolumeDataIntegrityVerifiesDeletionTail now also checks that on a writable volume its 64 zero bytes are reported and kept. Fixes #11864 * volume: fail the integrity check when a tail repair cannot complete A repair that wrote the .idx or cut the .dat and then failed to sync only logged the error; the check that followed saw the repaired files and the volume loaded writable on a repair that may not be on disk. Any repair error other than an unexplained tail now fails the check, so the volume loads read-only, and the next load repairs again. A failed .idx append could leave part of an entry behind, which made every later load reject the .idx as a partial entry. The appended bytes are truncated away again before the error is returned. * volume: do not cut a tail record that could not be read The walk cut a last record whose read failed for any reason, so an I/O error (EIO) on a complete record erased it. Only a record whose bytes are damaged (needle.ErrorCorrupted: a CRC mismatch or a body that does not parse) is cut now; any other read error is returned with both files left as they are, and the volume loads read-only. * volume: treat a sized tail record with no data as damaged A needle with no data is written with size 0. A tail record whose header gives a body but whose data size field reads 0 parsed as an empty needle, and ReadBytes checks the CRC only when there is data, so the repair indexed it with its positive size and reads returned an empty body. Such a record is now damaged like one with a bad CRC: cut when it is the last record, otherwise the tail is left alone and the volume loads read-only. * volume: keep an unindexed empty write as an empty needle A delete and a write with no data both append a size 0 record, and the repair indexed every size 0 record in the tail as a tombstone, so an acknowledged empty write whose .idx entry was lost read back as deleted. A delete is only appended for a needle that is live, so a size 0 record of a needle that is absent or deleted (in the .idx and in the tail before it) is an empty write and is re-indexed with size 0. One of a live needle stays a tombstone, the reading weed fix and the backup's index regeneration already give it. The .idx is scanned only when the tail holds a size 0 record. * volume: cut torn-tail test records inside their body, not at +40 The partial-record cases cut the .dat 40 bytes into a record. A needle with up to 7 bytes of data is a 40-byte record, so newRandomNeedle's random length (1..1023) left it whole about once in 150 runs per case: the repair then correctly re-indexed it and cut nothing, and the test failed on the .dat size. The cut now lands halfway through the record's body, which is always past its header and short of its end. --------- Co-authored-by: Javier de la Torre <jatorre@lakehousebox.com> |
||
|
|
1253b97303 |
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.18.7 in /test/kafka/kafka-client-loadtest/tools (#11866)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
a1d8739464 |
build(deps): bump github.com/klauspost/compress from 1.17.0 to 1.18.7 in /test/kafka/simple-consumer (#11869)
build(deps): bump github.com/klauspost/compress Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.17.0 to 1.18.7. - [Release notes](https://github.com/klauspost/compress/releases) - [Commits](https://github.com/klauspost/compress/compare/v1.17.0...v1.18.7) --- updated-dependencies: - dependency-name: github.com/klauspost/compress dependency-version: 1.18.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
|
1cfd9bbb4c |
volume: run the durable sync outside the reader-visible lock (#11863)
* volume: run the durable sync outside the reader-visible lock Reads of a volume waited on that volume's fsync because the write lock was held across Sync()/sync_all. A new per-volume lock (Go dataFileSyncLock, Rust data_file_sync_lock) now serializes every .dat mutation and lifecycle op across append+publish+sync+rollback, while the fsync itself runs with dataFileAccessLock/the io write guard dropped — so readers proceed during a sync, and a failed sync's rollback still cannot clobber another write or a compaction swap. Same change in Go (volume_write.go syncWrite/processBatch, SyncToDisk) and Rust (write_needles_grouped/write_synced_run). Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: publish after sync; fix Close/CommitCompact lock order Address review on #11863: - Close took dataFileSyncLock before waiting on isCompactionInProgress while CommitCompact claims the flag then takes the lock — a deadlock. Close now claims the flag before taking the locks (flag -> sync -> access). - Go now publishes needle-map entries only after the .dat sync, matching Rust's ordering: appendWriteRequest/appendDeleteRequest stage the bytes, publishWriteRequest/publishDeleteRequest index them after Sync. A failed sync rolls back bytes no reader ever resolved — the streamed-read offset-invalidation window is gone. processBatch splits runs at repeated needle ids (Rust parity) so dedup/cookie checks stay sequential; the old snapshot/restore rollback machinery is removed. - Rust commit_compact (store-lock caller) takes data_file_sync_lock via try_lock -> WouldBlock before the write lease; commit_compact_held and the tier-down _held paths document/require the caller-held sync lock, and the two vol.instance().write_lock() grpc sites take it first. The swap can no longer land between a durable write's append and publish. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * volume: restore write watermarks when an inline fsync fails The syncWrite rollback truncated the .dat but left lastAppendAtNs, lastWriteAppendAtNs, lastWriteNeedleKey, lastWriteDeleted and lastModifiedTsSeconds describing the rejected write; recovery and compaction could read a watermark for a record that no longer exists. Snapshot and restore them on failure, as processBatchRun already does. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
0de6347b11 |
s3: mask anonymous misses behind AccessDenied unless the bucket is listable (#11857)
* s3: mask anonymous misses behind AccessDenied unless the bucket is listable Deferred anonymous GET/HEAD authorization only runs once an entry is selected, so misses returned 404 while existing keys returned 403 — an unauthenticated existence oracle for bucket names and object keys. Route not-found responses in Get/HeadObjectHandler through a check that only discloses 404 when anonymous listing is allowed (policy, identity or a public-read bucket); otherwise the miss is indistinguishable from a denied read, matching AWS and 4.48 behavior. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: mask missing-directory misses for anonymous reads too Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: check a fixed request shape when masking anonymous misses The miss check passed the caller's request through, so query parameters could retarget the resolved action (?policy) or satisfy an s3:prefix condition on an unrelated subtree. Each check now runs on a request with a fixed shape: a ListBucket scoped to the probed key and a plain read of the key, so the miss only shows when the caller could learn the same fact directly. * s3: mask conditional-header misses for anonymous reads An If-Match conditional on a missing key wrote NoSuchKey directly from processConditionalHeaders, before the handlers reached the masked miss path, so anonymous callers could still distinguish a missing key from a denied read. * s3: keep the versionId when masking anonymous misses The read leg must evaluate the action the caller's request resolves to: a versioned miss is s3:GetObjectVersion, and dropping versionId would let a plain GetObject grant disclose versions the caller may not read. --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
6936bbdc3e |
rust volume: score EC shard placement with the volume's own ratio (#11862)
The auto-select branches of ReceiveFile (is_ec_volume) and VolumeEcShardsCopy passed the build-default DATA_SHARDS_COUNT into find_ec_shard_target_location, so its free-slot maths scored every volume as 10+4 even when the volume's .vif declares a custom ratio (e.g. 6+2). A wider-ratio volume's shards could be steered onto a disk too full to hold them all, and a narrower one would be refused slots it actually has. Add Store::lookup_ec_data_shards(collection, vid), which answers from the mounted EC volume first and then from the .vif on whichever disk holds it (via read_ec_shard_config_across_dirs, so split -dir/-dir.idx and rebuild-staging layouts are covered), falling back to the default when no metadata exists yet. Mirrors the enterprise fix (seaweed-mono c22fa8a15). |
||
|
Chris LuandGitHub
|
4d06742664 |
Detect and quarantine duplicate volume ids (#11763) (#11828)
* master: detect and quarantine duplicate volume ids across layouts (#11763) * shell: volume.check.duplicate_ids audit command (#11763) * master: persist duplicate-vid quarantine across heartbeats (#11763) * master: quarantine all copies of a duplicated volume id Near-simultaneous registrations let each side quarantine only the other, so arrival order decided which copy stayed writable. A confirmed cross-layout collision means either copy could be the impostor, so flag every colliding copy read-only regardless of registration order. * topology: return a copy from VolumeLayout.Lookup Callers iterating the returned slice raced with registrations and removals replacing the location list in place. * master: release duplicate-vid quarantine when the conflict resolves A cross-node volume.move briefly shows the same id on two nodes under different disk layouts; permanent quarantine would pin a valid target read-only. Keep the first-reported state per quarantined copy and lift the marks once the conflicting copy unregisters, so only a persistent collision stays read-only. * shell: include ec shards in volume.check.duplicate_ids Ec-only volumes share the global id space. Compare shard collections against regular ones while allowing shards and regular copies of the same encoded volume. * master: keep quarantine release current and cover all removal paths Track the latest server-reported read-only flag while quarantined so a release restores the server s true state, and run the release check on collection deletion and data-node removal too, not only explicit volume unregistration. * shell: do not flag ec-only volumes as duplicates An ec-only volume has no regular copy, so its shard collections had nothing to compare against and always read as foreign. * master: quarantine every conflicting layout at the layout level (#11763) - collect all conflicting layouts, not just the first, so a vid spanning three layouts does not leave one writable - mark quarantine on the location list only: folding it into the stored report made the volume digest permanently disagree with the server and forced a full volume list every heartbeat - isAllWritable now honors the location read-only mark so a quarantined copy stays out of writables through resyncs and re-registration - re-check the conflict before keeping marks, and do not clear a node's quarantine while it still holds the vid in another layout - log the full layout keys on both sides of the conflict * master: push duplicate-vid quarantine to the volume server (#11763) A quarantined copy stayed writable on the volume server itself for a client holding an existing file id, since the master-only mark never reached the server's local write path. Quarantine now also calls VolumeMarkReadonly (can_delete so expiry still drains), is re-pushed if the server comes back reporting it writable, and VolumeMarkWritable runs when the conflict resolves and the server had reported writable. * master: retry unenforced quarantine marks and restore server writability on release (#11763) Dial the advertised grpc port, return mark errors, and keep a quarantine entry until the writable push succeeds so a released copy does not stay read-only on the server. * master: treat rp-only layout differences as an in-flight replication change (#11763) volume.configure.replication rolls through nodes one at a time, so a volume legitimately reports the old and new replica placement while it settles. Layouts differing only in rp within the same collection are a supported transition, not a reused id. * shell: group ec shards by collection in the duplicate id audit (#11763) EC shards of one volume legitimately span disk types when DiskTypePrefer spills, so grouping by disk type reports healthy volumes as duplicates. |
||
|
Chris LuandGitHub
|
2874bd05fb |
Freeze volume growth on topology id mismatch (#11763) (#11820)
* pb: ResumeVolumeGrowth rpc for operator-driven growth resume (#11763) * master: freeze volume growth on topology id mismatch (#11763) * shell: volume.growth.resume command (#11763) * master: replicate the growth freeze and gate it on veteran heartbeats (#11763) * master: block allocation while the topology id is being minted DeferAllocation only armed after the topology id persisted, so growth requests arriving during init ran ahead of the warmup window. Gate NextVolumeId on an init-pending flag spanning the whole mint. * master: stash reported topology ids until the local id is minted (#11763) * drain stashed topology id reports when the id arrives outside the mint path * persist the growth-freeze flag in the mirror file and raft snapshots * log hashicorp raft marshal and apply errors in applyGrowthFreeze |
||
|
|
1144bf15d4 |
rust volume: group-commit fsync uploads on the direct write path (#11860)
* rust volume: group-commit fsync uploads on the direct write path Each fsync=true upload used to run its own .dat/.idx sync when the write queue was off, which is the default. Route durable uploads through the per-volume write workers so a batch shares one sync, matching Go's writeNeedle2: non-fsync writes, and writes while the server is stopping, stay on the synchronous path. The queue is now always constructed and SEAWEED_WRITE_QUEUE is gone. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * rust volume: document post_handler write routing Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
8d2b417c69 |
iceberg, lance: vend read-only credentials to callers that may only read (#11859)
* s3tables: share the UpdateTable authorization gate Extract the validation, load, and permission check from handleUpdateTable into authorizeUpdateTable so the table an update was authorized against is the same snapshot the mutation writes to. Manager gains AuthorizeUpdateTable, mirroring AuthorizeCreateTable, so callers can ask the question without updating anything. * s3tables: share the UpdateView authorization gate Extract the lookup, policy load, and permission check from handleUpdateView into authorizeUpdateView, mirroring authorizeUpdateTable, and expose it as Manager.AuthorizeUpdateView for callers that need the answer without the update. * s3api: scope vended table credentials to read-only on request VendTableCredentials takes a readOnly flag and tablePrefixSessionPolicy emits only s3:GetObject for the table prefix when it is set, keeping the conditioned ListBucket and unconditioned GetBucketLocation statements that reads still need. * iceberg: vend read-only credentials when the caller may not write LoadTable and friends vend storage credentials only for the authority the caller holds: the vend now checks UpdateTable/UpdateView permission and a caller that may only read the catalog entry gets a read-only session. Paths that just passed a write gate (stage-create, view commit) keep the full session. * lance: vend read-only credentials when the caller may not write storage_options vends the same read-only session as the Iceberg catalog when the caller lacks UpdateTable on the table. * lance: keep write credentials for callers authorized to declare A Lance commit is a direct S3 write that never revisits the catalog, so a caller that just passed CreateTable still gets a writable session; the UpdateTable gate applies to describe, the read path. * iceberg, lance: log a failed write check instead of only reading it as denied |
||
|
|
7ed26deaae |
s3: list keys in byte order around directory markers (#11832)
* s3: list keys in byte order around directory markers * s3: order deferred dir blocks by slash name and hold quota * s3: fix deferred dir blocks at page and marker boundaries Review follow-ups on the marker byte-order change: - A marker inside an emitted "<name>/" block re-descended and re-listed its folded contents under delimiter "/" instead of skipping the block. - A prefix object's "<name>/" common prefix was emitted eagerly at the name position, outranking siblings like "a.txt" and repeating across pages; the plain key now emits alone at the name position and the block emits the common prefix when children exist. - A recursion starved on quota before listing a deferred subtree returned without truncation, dropping keys silently; the block now reports a resumable truncation when the subtree still holds entries. - Clone prefix-object entries with proto.Clone rather than copying the protobuf message by value. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * s3: close the hasEntries probe stream on return * s3: resume delimited listings at the dir block boundary A deferred directory block emits at its "<name>/" position, but emitDir left nextMarker at the bare name, which sorts inside the (name, name/) band and reads as "block pending". Every pass of the listing loop then re-emitted the same common prefix until maxKeys ran out (999 prefixes, IsTruncated), and a "<name>/" continuation token trimmed to "<name>" by adjustMarkerForDelimiter reopened the block and paged forever. Advance nextMarker to the block boundary the same way the recursive branch does, and keep a "/" -ending marker intact — it already names the bound past the block. --------- Co-authored-by: Devin <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
da0d423bd9 |
Add OLake to README and Admin UI; fix catalog_olake path-style docs (#11858)
* README: mention OLake in the Iceberg lakehouse section * catalog_olake: fix path-style prose; admin UI: list OLake as a REST catalog client * admin UI: restore generated file import block, keep only the string change --------- Co-authored-by: GAURI <174253719+gauriiiiiiiiiiii@users.noreply.github.com> |
||
|
|
2095c905ba |
server: give heartbeat-options test a topology (#11853)
heartbeatResponse now reads MaxVolumeId and TopologyId off ms.Topo, but the test built a MasterServer without one and panicked, failing the package for every PR. Generated with [Devin](https://devin.ai) Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Chris LuandGitHub
|
6b6a268903 |
volume balance: stop replica-fallback moves between equally full servers (#11851)
* volume balance: unify the per-move convergence guard Evaluate the post-move utilization comparison through serverUtilization so the guard also applies when servers lack MaxVolumeCount info (raw-count mode), where it previously did not run at all. * volume balance: restrict fallback destinations to improving servers The score-based fallback inside createBalanceTask could redirect a move to any valid server, including ones as full as the source — between equally full servers the move only relocates the imbalance and the next detection cycle reverses it (#11839). Exclude from eligibleTargets any server whose post-move utilization would exceed the source's, so a fallback destination satisfies the same convergence guard as the loop-chosen target. * volume balance: skip a volume when the chosen target violates placement When the detection loop's target resolves but cannot host the replica (it already holds one, or is out of the filtered scope), createBalanceTask now returns nil so the loop advances to the next volume — matching shell volume.balance, which continues to the next candidate instead of redirecting the move to a different server. The score-based fallback is kept for the case where the target cannot be resolved at all; its allowed set is already limited to servers the move would improve. * volume balance: regression test for replica-constrained targets (#11839) Reproduces the reported loop: the emptiest server holds the second rp=001 replica of the lowest-ID volumes on the fuller servers, so it is never a legal target for them. Detection must skip those volumes and send the movable ones to the under-utilized server, and re-running detection after applying the plan must never move the same volume twice. |