OLake (github.com/datazip-inc/olake) is a CDC and ingestion engine that writes
Apache Iceberg. It covers two paths none of the existing catalog suites reach.
It is a strict Java Iceberg client. OLake does not write Iceberg from Go — its
Go process spawns a Java sidecar over gRPC and writes through the official
Apache Iceberg library, because the Go library has no equality deletes and CDC
needs them. That makes it the client class weed/s3api/iceberg's metadata
compliance backfill exists to serve: the one that fails with "Cannot parse
missing long current-snapshot-id" when spec-required keys are omitted.
And it produces equality deletes. Its upsert path commits operation=overwrite
with an equality-delete file and a delete manifest. ClickHouse, Doris, Trino,
Spark and DuckDB all only append, so nothing else in this directory exercises
a delete manifest at all.
Six subtests: the destination check reaches SUCCEEDED and actually loads the
REST catalog; discover enumerates the source; a full sync commits a snapshot;
PyIceberg reads back what the Java writer committed; an update plus a re-sync
records an overwrite carrying equality deletes with a delete manifest in the
current snapshot; and the catalog does not rewrite manifests the official Java
writer produced, which gives the manifest-repair path a negative test to go
with the ClickHouse positive one.
What it deliberately does not assert is a delete-applied read. PyIceberg
refuses to scan a table carrying equality deletes (apache/iceberg#6568) while
reading its metadata fine, and an engine that can apply them costs a
multi-gigabyte image. Recording the commit correctly is the catalog's
contract; applying deletes on read is the engine's. The README says so, says
the read half was verified by hand once with StarRocks, and warns against
later "upgrading" this to a PyIceberg rows read — which would either fail or,
if PyIceberg ever starts skipping deletes instead of raising, pass by not
looking.
The workflow job asserts the suite actually ran rather than trusting a green
exit, for the same reason: at least one top-level PASS and zero SKIP.
No product change — the destination config is the generic catalog_type=rest
with the standard OAuth2 client-credentials flow, and s3_path_style is not even
set, since OLake turns it on itself whenever s3_endpoint is non-empty.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rust: a seaweed-common crate for the address and TLS helpers both crates carry
seaweed-volume and seaweed-worker are separate cargo trees with separate
lockfiles and no root manifest, so anything both of them need has had to be
written twice. Two of those copies are a correctness risk rather than a typing
cost, and this crate is where they stop being copies.
address.rs is the HTTP<->gRPC port rule: `host:port` means gRPC on port+10000,
`host:port.grpcPort` names it outright. The two copies had already drifted —
the worker's bracketed IPv6 literals, the volume server's did not — so the rule
lives here once, returning a typed AddressError whose Display text is the volume
server's original wording, with join_host_port public beside it. A test asserts
two of those messages in full rather than by substring, because the wording is
the contract its callers hand to a Status or an io::Error; the other three end
in a std ParseIntError message, which is std's to reword. The enum is
#[non_exhaustive] so a future variant is not a breaking change for either
consumer. The tests are both crates' cases together, plus the IPv6,
already-bracketed and normalisation cases neither copy covered on its own.
tls.rs is install_default_crypto_provider. Both binaries link aws-lc-rs and ring
transitively, so rustls cannot auto-select and tonic's client TLS panics on
first use; each binary has to pin one and it has to be the same one, which is
exactly the kind of choice that should not exist twice. It is safe to share
because `cargo tree -i rustls` resolves a single rustls in each tree (0.23.37 in
seaweed-volume, 0.23.43 in seaweed-worker) and cargo unifies all
semver-compatible `rustls = "0.23"` requirements into one crate per binary, so
this crate writes the same process-wide static its consumer reads. rustls is
already in both graphs — directly in the volume server, through tonic's
tls-aws-lc in seaweed-worker-core — so the dependency adds no crate to either.
rust-version is 1.91.1, the lower of the two consumers' floors, so depending on
this crate cannot raise either tree's MSRV; verified with
`cargo +1.91.1 check --all-targets`. The lockfile is committed even though this
is a library: CI builds it directly, so a committed lock is what makes those
runs reproducible and their caches stable.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* rust: take the address and TLS helpers from seaweed-common
Both public signatures are kept, so no caller outside the two wrapper files
changes. parse_grpc_address stays `Result<String, String>` and maps the typed
error through Display; server_to_grpc_address stays `Option<String>` and drops
it with .ok(). Their doc comments and the volume server's 13 call sites are
otherwise untouched.
Three behaviours change, each in the direction of the copy that was already
right:
- The volume server now brackets IPv6 literals. `::1:19333` used to come back as
`::1:29333`, which build_grpc_endpoint rejects with "invalid gRPC endpoint
http://::1:19333: invalid authority" — an IPv6 master or EC peer could not be
dialled at all. Two tests in grpc_client.rs pin it, one on the string and one
on the endpoint the string builds.
- The volume server now emits the *parsed* gRPC port of the dotted form instead
of the original text it had just validated, so `host:8080.018080` and
`host:8080.+18080` come back as `host:18080` rather than as authorities the
URI parser rejects. Same port either way; only malformed spellings change.
- The worker's dotted form now validates the HTTP port it discards.
`server_to_grpc_address("host:abc.18080")` used to answer Some("host:18080");
it now answers None, which is what the volume server's copy has always done.
install_default_crypto_provider becomes a re-export in both trees, so
`crate::security::tls::install_default_crypto_provider` and
`weed_lance_worker::tls::install_default_crypto_provider` still resolve. The
lance crate's `rustls = "0.23"` was its only direct use of rustls and goes away
with the body; seaweed-common states the same requirement, so neither the
resolved version nor the enabled features move in either lockfile.
The PEM test fixtures stay where they are. The two tests that use them are not
duplicates: the volume server's exercises build_grpc_endpoint, and the lance one
exists precisely because aws-lc-rs and ring are both linked in that crate's
graph. Only the literals are shared, and exporting test fixtures from a library
to dedupe two constants costs more than it saves.
A path dependency outside both trees means every build context that copies one
crate directory has to copy the other. The repo has one: the Rust source-build
stage of docker/Dockerfile.go_build, which now copies seaweed-common beside
seaweed-volume. Every workflow whose `paths:` filter keys on a crate directory
gains `seaweed-common/**` — the two Rust test workflows, rust_binaries_dev,
container_dev and performance. The tag- and dispatch-triggered ones
(rust_binaries_release, container_release_unified, container_latest) have no
`paths:` filter and need nothing.
The two Rust test workflows also run `cargo test` in seaweed-common, from their
unit-test job, because a path dependency is not a workspace member and neither
tree's own `cargo test` reaches it. Each step builds into its job's cached
target directory, and both cache keys now hash seaweed-common/Cargo.lock as well
so a change there invalidates the cache it would otherwise silently reuse.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docker: keep go_build working for BRANCH revisions without seaweed-common
The rust_builder stage copies seaweed-common unconditionally now that seaweed-volume path-depends on it, but BRANCH can name any revision — including ones that predate the crate. Create the directory in the builder stage so the COPY always has a source; an empty dir beside an old seaweed-volume is harmless.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
- README and values.yaml now describe the allowInsecureBind escape
hatch alongside the non-loopback bind guard
- remove a PR reference from the CI test comment
PR #11185 made `weed admin` refuse to bind a non-loopback address
without -adminPassword or mTLS. PR #11228 added -allowInsecureNoAuth
as an explicit opt-out for operators who restrict admin access some
other way (e.g. a NetworkPolicy plus an authenticating reverse proxy).
The chart's render-time guard added by #11236 (admin-statefulset.yaml,
seaweedfs.admin.authEnabled) predates -allowInsecureNoAuth and only
recognizes password-based auth, so there was no values.yaml path to
express that choice: the chart would fail(...) even though the binary
itself would start fine with a warning.
Add admin.allowInsecureBind (default false) to the seaweedfs.admin.authEnabled
helper's checks; when true it renders -allowInsecureNoAuth on the admin
command and satisfies the render guard alongside the existing
password-based checks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* s3api: add Snowflake s3compat API integration tests
Run the upstream snowflakedb/snowflake-s3compat-api-test-suite against a
local SeaweedFS server in CI. test/s3/snowflake/run.sh starts weed server
with S3 (-s3.autoCreateBucket=false so missing-bucket PUTs return
NoSuchBucket), prepares the fixtures the suite needs (versioned bucket,
deny-all-policy bucket, >1000-object prefix), clones the suite, patches
it to path-style addressing, and runs mvn -Dtest=S3CompatApiTest.
The suite also exposed that GetBucketLocation returned 404 NoSuchBucket
for a malformed bucket name; validate the name first and return
400 InvalidBucketName like AWS.
* test: harden snowflake s3compat runner per review
- Pin the upstream suite to a tested commit (SUITE_REV) instead of the
moving default branch
- Bind the test server to loopback only
- Require the AccessDenied error code when verifying the denied bucket
- Fix README so go install runs in a subshell
- checkout with persist-credentials: false
- Make the concurrency group unique per PR, and widen path filters to
the storage/operation/wdclient/cluster/pb packages the S3 stack uses
* test: advertise loopback ip for snowflake test server
-ip.bind 127.0.0.1 alone left the volume server advertising the host's
primary address, so chunk uploads were refused. Also set -ip 127.0.0.1
and disable the Iceberg/Lance listeners so the harness is loopback-only
and does not collide with other local services.
* telemetry/server: tidy module dependencies
* ci: make telemetry deploy work on Oracle Linux 7
* ci: install telemetry unit and logrotate on every deploy
* ci: abort telemetry deploy on install failures
* rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14
tonic 0.14 boxes the contents of tonic::Status, which is what made every
RPC path trip clippy's result_large_err; the allow for that lint goes in
the next commit. The prost codec moved out of tonic into tonic-prost and
tonic-prost-build, so both build scripts now call
tonic_prost_build::configure() and both crates depend on tonic-prost for
the generated code. The `tls` feature was split into a per-backend
feature; `tls-aws-lc` is the same backend both crates already install
through rustls::crypto::aws_lc_rs.
tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a
second axum and a second tower in each tree next to the 0.7 / 0.4 the
crates named themselves. Bumping them keeps one copy of each: axum 0.8
only changes the path-parameter syntax for the routes here (`/:vid` ->
`/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature
named explicitly for ServiceExt::oneshot (it used to arrive through
tonic's feature unification), and tower-http 0.6 is the matching
release.
Lock files move only through cargo's own resolution for the new
versions; no other dependency was refreshed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: drop the result_large_err allow now that tonic::Status is boxed
tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer
a large-Err type and clippy has nothing to say about it. Both crates
pass `cargo clippy --all-targets -- -D warnings` without the allow
(seaweed-volume in both feature sets), so the policy entry and its
comment go.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: drop the unused headers argument of try_expand_chunk_manifest
The parameter was already named `_headers`; nothing in the body reads it.
With it gone the function is under clippy's argument threshold and the
expect goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: pass EC peer reads an EcInterval instead of ten arguments
fetch_one_interval, read_remote_ec_shard_interval,
do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval
all took the same (vid, needle_id, shard_id, shard_offset, size,
expected_encode_ts_ns) tuple, and the two that reconstruct also took the
location map with the data/parity counts. Those are now EcInterval (Copy)
and EcShardMap (a borrow of the map plus the counts). The fan-out inside
recovery builds its per-shard request with `EcInterval { shard_id: sid,
..iv }`, which is the one place the old argument list was easy to get
wrong. Bodies destructure at the top, so the code below the signatures
is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun
encode_dat_file took the Reed-Solomon shape and three block sizes as five
loose integers; they are now one Copy struct, EcEncodeLayout, which is
what Go calls ECContext. The per-row and per-batch helpers took the same
six sinks and the offsets; they become methods on EncodeRun, which owns
the borrows for one run, so each call names only the offset and block
size that vary. The byte-level work is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments
write_dat_file_from_shards, its _with_dirs twin and the private
write_dat_file were three layers over one nine-argument signature. One
public function now takes a DatRebuild, whose shard_dirs is None when
every shard sits beside the .dat and Some(dirs) for the cross-disk
reconciled layout. The field docs carry what the function doc used to
say about the encode-time size and the block layout.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: split copy_file_from_source's fifteen arguments into two structs
CopyFileSpec is the per-file request (what to ask the source for, where
it lands, whether its bytes count as progress); CopyProgress is the
sender, throttler and report state that all three files of one
VolumeCopy share, held by &mut across the calls. The three production
call sites now read as the .dat/.idx/.vif literals they are, instead of
positional trues and falses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: create volumes from a VolumeSpec
Volume::new, DiskLocation::create_volume and Store::add_volume each
took the same five-value tail of Go's NewVolume argument list:
collection, replica placement, TTL, preallocation and needle version.
That tail is now VolumeSpec, a Copy struct whose Default is what almost
every test wanted anyway (empty collection, no replication, no TTL, no
preallocation, current version), so most of the 104 call sites shrink
to `&VolumeSpec::default()` or name the one field they set. The id,
directories, index kind and disk type stay positional because they
differ at every site.
Two imports that only test modules use moved into those modules, and
DiskLocation no longer imports ReplicaPlacement.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-worker: run cargo fmt
Layout only; no token in the workspace changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust-volume: run cargo fmt
Layout only; no token in the crate changes. Every earlier Rust PR here
formatted only the blocks it touched so as not to drown its diff in
this one, and this commit is that debt paid in a single place. rustfmt
needed two passes to settle one block in handlers.rs; the committed
form is the fixed point, so `cargo fmt --check` is clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out cargo fmt --check step to both Rust workflows
Same shape as the commented clippy step from #11312: the check is
written out so that making formatting a gate is a one-line uncomment,
and whether to do that stays a maintainer call.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* rust: apply clippy --fix to both crates
The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix`
on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both
manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion
clippy applied is within it: the collapsible_if sites become let chains
(1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87),
chunks_exact with a constant becomes as_chunks (1.88), repeat().take()
becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes
io::Error::other (1.74). The rest is redundant clones, borrows, casts,
closures and field names.
Nothing here changes behaviour. The three let_and_return sites in
needle_map.rs and store_ec.rs deserve a note: the `let result = ..;
result` shape was a deliberate edition-2021 workaround to drop a redb
guard before the table it borrows. Edition 2024 drops tail-expression
temporaries before locals, which is why clippy now flags it, and the
two comments that described the workaround say so instead.
Manual edits on top of the tool output: the blocks clippy rewrote are
re-indented the way rustfmt lays them out (only those blocks — the
crate is not rustfmt-clean and a whole-crate fmt would bury this diff),
the blank lines let_and_return left behind are removed, and the CRC
legacy_value test compares against a literal worked out from the
original shift formula rather than restating rotate_right.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust: clear the clippy warnings --fix cannot apply, and say why the rest stay
Hand fixes for the lints clippy only reports. Behaviour is unchanged
throughout; each rewrite is the one clippy names.
- needless_range_loop (7): index loops over shard vectors become
iterator loops. Where the old code indexed `v[..n]` the new loop
iterates `v[..n]` so an undersized vector still panics the same way.
- field_reassign_with_default (6): struct literals with `..Default`.
- redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes
`.is_err()`, which also releases the read guard at the end of the
condition instead of at the end of the block.
- manual_strip (2), manual_checked_ops, format_in_format_args,
redundant_locals, wrong_self_convention (to_vif takes self by value,
so it is into_vif; CompactEntry is Copy, so to_needle_value takes self).
- type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two
tuples that were spelled out inline.
- new_without_default: CompactNeedleMap gets a Default that calls new().
- suspicious_open_options: a test helper spells out `.truncate(false)`,
which is what `.create(true).write(true)` already did.
What stays, and the attribute that says so:
- too_many_arguments (10): `#[expect]` on each function. Folding 8–15
parameters into a struct is a design change, not a lint fix.
- await_holding_lock / readonly_write_lock: one test holds the store
write guard across a sleep on purpose, as a barrier that parks the
copy task at the mount block. `#[expect(.., reason = ..)]` records it.
- module_inception: needle/needle.rs mirrors the Go package layout.
Two lints become crate-wide policy in `[lints.clippy]`, with the reason
next to each: result_large_err, because every RPC path returns
tonic::Status (176 bytes) and boxing it would change every handler
signature; and needless_update, because `..Default::default()` on a
protobuf message literal is what lets a proto gain a field without
touching every constructor (all 11 sites are pb messages). The worker
workspace gets the same table and its members opt in with
`lints.workspace = true`; its generated plugin.rs also allows
large_enum_variant on prost's oneof enums.
Both crates are now clean under `cargo clippy --all-targets -- -D warnings`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: use the std APIs the 1.91 MSRV already pays for
The crate declares rust-version 1.91.1, so a few things the code still
worked around are plain std now. All of them come from the 1.85–1.91
release notes; nothing here needs a newer toolchain than the manifest
already requires.
- std::sync::LazyLock (1.80) replaces the lazy_static! block in
metrics.rs, and the lazy_static dependency goes. Every use site reads
the same through Deref, so no caller changes.
- Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)`
and `from_secs(v * 3600)` in the option parser and the shard-location
refresh TTLs. One difference for the parser: an absurd count that
overflows u64 seconds now panics in release builds too, where the
multiplication used to wrap.
- Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication
join handle.
- OsStr::display (1.87) replaces `to_string_lossy()` where the name was
only being formatted; the output is byte-identical.
- `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are
meant to be permanent, so a suppression that stops being needed
becomes a warning rather than lingering. Doing that found four that
already had: dead_code on ChunkManifest, base_name and last_io_error,
and too_many_arguments on read_from_data_shards, which is down to
seven parameters. Those attributes are deleted. The three allows that
depend on cfg (a unix-only mutation, a linux-only field set, a
profiling-only parameter) stay as allow, because expect would be
unfulfilled on the other platforms.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* ci: add a commented-out clippy step to both Rust workflows
Both crates are warning-free under `cargo clippy --all-targets
-D warnings` now. Whether that becomes a gate is a policy call, so the
step is present but commented out; uncommenting it is the whole change.
The comment points at the `[lints.clippy]` table where crate-wide
exceptions are recorded, so the gate does not become a reason to
sprinkle allows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU
* rust volume: guard parse_duration against overflow panics
Duration::from_mins/from_hours panic when the count overflows u64
seconds. Use checked_mul so an oversized CLI value falls back to the
parser default instead of crashing volume startup.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* fix: rebase on latest master before pushing star history chart
The daily star history workflow git push was rejected with a
non-fast-forward error because new commits landed on master between
the checkout and the push. Fetch full history (fetch-depth: 0) and
rebase the generated commit on top of the latest remote branch before
pushing so the workflow no longer fails when master has moved.
* fix: retry rebase-and-push to handle concurrent master updates
Address review feedback: a one-shot rebase still races if master advances
between the rebase and the push. Match the bounded retry loop used by
java_release.yml — push first, and on rejection rebase and retry up to
five times before failing.
* fix: serialize runs and ensure every rebase is followed by a push
Address review feedback:
- Devin (line 54-55): the old loop rebased after the 5th failed push but
never pushed the rebased commit. Restructure so each rebase (attempts
2-5) is followed by a push attempt, with a clear 5-attempt cap.
- Greptile: overlapping runs could conflict on the SVG during rebase.
Add a concurrency group (cancel-in-progress: true, matching the repo
convention) so only one chart regeneration runs at a time.
* fix: scope concurrency by ref and guard rebase against transient failures
Address review feedback:
- Devin (line 16): the global star-history concurrency group let a manual
run on another branch cancel an in-flight daily master update. Scope
the group by github.ref so only same-branch runs cancel each other.
- Greptile (line 58): git pull --rebase runs under the fail-fast shell,
so a transient fetch error or conflict aborted the whole step before
remaining attempts ran. Guard the rebase so a failure aborts the
in-progress rebase and continues to the next attempt instead.
PR #11236 added a render-time guard that fails the chart when admin.ip
is non-loopback (default 0.0.0.0) and admin auth is not configured,
since weed admin 4.46 refuses to bind a non-loopback address without
authentication. Several pre-existing helm_ci.yml test cases enable
admin.enabled=true as part of "everything on" renders without a
password, so helm template now exits non-zero and the Verify template
rendering step fails.
Add admin.secret.adminPassword to the four render calls that turn on
admin without auth (IAM gRPC opt-in, NetworkPolicy EVERYTHING, egress
without kubeApiServer.cidrs, and the license ALL_ON dict), using the
same key ci/admin-values.yaml already uses.
Allow normal post-tag proxy propagation before dispatching downstream releases, while preserving the check that prevents them from pinning the previous commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* rust volume: quick-repair redb on durable checkpoints
set_quick_repair(true) on the durable checkpoint transaction so an
OOM-killed volume server opens without a full-file repair scan.
* rust volume: optional redb insert_before bulk load
Behind redb-experimental-cursor (default off). Production binary
stays on sorted insert(). CI unit tests run both feature settings.
* rust volume: exercise insert_before across leaf splits
Replace the 5-key cfg clone with a 4000-key reverse-order rebuild
so CursorMut::insert_before hits page splits. CI runs the feature
only on storage::needle_map unit tests.
The star_history workflow referenced actions/setup-python@v8, which
does not exist, causing the workflow to fail at the "Set up job" step.
Pin to v7, matching the version used across the other workflows.
Change the matplotlib figure size from (10, 4) to (10, 6) so the
star history chart renders vertically longer in the README. The
regenerated note/star_history.svg reflects the new 5:3 aspect ratio
(720x432pt) instead of the previous flat 2.5:1 (720x288pt).
The "Mount and exercise" step left the weed.exe mini server and the final
WinFsp mount running when it exited. The next step's pwsh.exe then failed
with STATUS_DLL_INIT_FAILED (0xC0000142), failing a job whose actual test
step had passed. The same code passed on both the PR branch and the next
master run, so this was a transient launch failure — but it was caused by
an unclean environment and made fatal by a diagnostic step.
Tear down all weed.exe processes at the end of the test step so subsequent
steps launch into a clean environment, and mark the Logs step
continue-on-error so a diagnostic step can never fail the job on its own.
* docs(readme): replace star-history.com with self-generated chart
The star-history.com SVG is a third-party dependency that can rate
limit or go down. Replace it with a GitHub Action that fetches
stargazers via the REST API and renders an SVG with matplotlib,
committing note/star_history.svg weekly. The README references the
committed file directly, so the chart has no runtime dependency on
any external service.
* ci(star-history): run daily instead of weekly
* s3: HEAD with partNumber reports the part's size and range
HeadObject set its headers from the total object size and then only
validated the partNumber, so a client probing part 1 with HEAD got the
whole object's Content-Length and a 200 while the same GET returned the
part's size, a Content-Range and a 206.
Resolve the part's byte range before the headers are written, through the
range logic GetObject already used, and answer a partNumber HEAD as the
ranged HEAD that AWS documents.
* s3: answer an unsatisfiable partNumber with 416 InvalidPartNumber
GET and HEAD rejected a partNumber past the number of parts with 400
InvalidPart, the code for a missing part in CompleteMultipartUpload. AWS
answers a read of a part that does not exist with 416 InvalidPartNumber,
which lets a client probing for the part count tell the two apart.
The ceph suite pins RGW's 400 InvalidPart here, so the s3tests jobs patch
that expectation the way they already patch prefix ordering.
* s3: keep the whole-object checksum off a partNumber response
The stored checksum covers the whole object, so it is already withheld
from a ranged read. A partNumber HEAD now describes one part while the
request carries no Range header, so exclude it there too rather than
handing a client a checksum that does not match the bytes described.
* s3: resolve a partNumber against the parts the object records
Completion accepts ascending, not consecutive, part numbers, so the part
count is not the highest part number. Comparing the two rejected an
uploaded part 3 of a two-part object, and let a request for the absent
part 2 fall through to the positional chunk lookup and serve part 3's
bytes. Ask the recorded boundaries for the part instead, and keep the
count comparison for objects written before boundaries were stored.
* s3: apply a client Range within the part on HEAD too
GET narrowed the part by a Range sent alongside partNumber; HEAD reported
the whole part, so the two disagreed again for a request that carries
both. Move the narrowing into the shared range lookup so either verb
describes the same bytes.
2026-09-04 23:10:41 -07:00
Chris LuGitHubDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every image-signing job has failed since signing was added (#11129):
must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config
Cosign 3 turned on two defaults, not one. The action only disabled
--new-bundle-format to keep the .sig tag layout, but --use-signing-config
is still on, and cosign refuses that pairing because the signing-config
path has nowhere to write its verification material without a bundle.
Disabling it too falls back to the default Fulcio and Rekor URLs, the
same services the .sig layout always used. The verify step needs no
change: cosign verify looks for a referrer bundle first and falls back
to the .sig tag when there is none.
Generated with [Devin](https://devin.ai)
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: composite action that signs and verifies an image with cosign
Keyless, by digest, with a verification pass against the calling workflow's
own identity right after signing. Signatures use the .sig tag layout rather
than the OCI-referrer bundle cosign 3 writes by default, since that is what
the verifiers people run today read. Dependabot is pointed at the action so
the cosign-installer pin keeps moving.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign every variant on both registries
The merge job signs each variant's multi-arch index on GHCR and Docker Hub
once the tag exists, recursively so the platform images are covered too.
latest re-tags the same manifest and inherits the signature.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker dev: sign the dev image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: sign a latest rebuilt by hand
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: sign the foundationdb image
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: sign the per-version foundationdb and rocksdb builds
They push to the same repository as the releases, so an admission policy
that verifies chrislusf/seaweedfs would otherwise reject them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: document image signature verification
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin the actions the signing jobs newly run by commit
These run with registry credentials and the OIDC token that signs under
the repository's identity, so a retargeted tag upstream must not be able
to reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: pass the dispatch tag through env, not the script
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker: complete Kyverno policy, digest note, identity scope
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: keep the dispatch tag out of the manifest script too
The step predates signing, but the job now holds the OIDC identity.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* ci: pin every action in the jobs that sign
The jobs that hold the OIDC identity run these with registry credentials,
so a retargeted tag upstream must not reach them.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: copy and sign the digest the run created, pin the rest
crane copy and the signature both resolved the tag, which another
publisher could move between the two steps. The index digest is read once,
right after it is created, and the Docker Hub copy and both signatures use
it. The manual latest rebuild gets the same treatment. The actions in these
jobs are pinned to commits, crane to v0.22.0 by checksum, and the sparse
checkout no longer keeps the token.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker latest: the signing job checks out the workflow's own commit
The job only assembles and signs manifests, so nothing there needs the
source_ref checkout; the local signing action now comes from the same
revision as the workflow file that calls it.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* docker release: take the index digest from the create result
imagetools create writes the descriptor it pushed with --metadata-file
(buildx 0.32+, the runners ship 0.36), so the digest no longer comes from
re-resolving the tag even within the same step.
Claude-Session: https://claude.ai/code/session_01A5zMqzaUg1Snur4Yg8xJGa
* helm: values-driven labels on every ingress
Each ingress already takes annotations from values, but its labels were
a fixed block, so tools that select ingresses by label (ExternalDNS
label filters, for one) had nothing to key on. Every ingress block now
has a labels map rendered after the standard app.kubernetes.io labels,
including the Traefik IngressRouteTCP that shares the filer gRPC values.
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* helm ci: render check for ingress labels
Claude-Session: https://claude.ai/code/session_01L6eJGXtYkwe1W9QjGeUgr1
* filer: keep the sentinel when CreateEntry reports an update failure
CreateEntry flattened the error UpdateEntry wraps, so errors.Is stopped
matching and ErrExistingIsDirectory and ErrExistingIsFile never reached
the S3 mapper, which answered a retryable 500 instead.
* s3: let a key that is a prefix of other keys be an object
S3 keys are flat, so "a/b" and "a/b/c" are independent objects that
coexist in either write order. The filer stores a key as a path, so one
of them has to live on the directory the other is nested under.
Writing the nested key first refused the prefix key outright. Writing it
second promoted the file to a directory, which kept its data but lost the
key: an empty object left nothing to recognise it by and disappeared, and
one with data listed under a trailing slash it never had.
Mark the directory that carries such a key, and write the object onto it
when the path is already a directory. The mark makes an empty prefix
object visible to listings and readable by GET and HEAD, keeps the empty
folder cleaner off it, and lists it under the key it was written with.
Deleting the key strips the mark back off along with the data.
* filer: keep a TTL off a directory that stands for an object
An expired entry is deleted a row at a time, so expiring a directory
removes it and leaves everything under it unreachable. Promoting a file
to a directory carried its TTL across, and a promoted file is exactly the
one that has keys nested under it.
Drop the TTL on promotion, and leave one an older build wrote alone. The
lifecycle worker still expires the object, through the delete that leaves
the directory behind.
* s3: delete the null version of a key other keys are nested under
The routed delete cannot remove an entry that other keys live under, and
answered a retryable 500 rather than falling back to the lock path the
unversioned delete already falls back to. That path then looked the entry
up under the bucket with the whole key as its name, so the demote wrote it
back one directory too high and failed as not found.
Fall back on any non-precondition error, and split the key before deleting
it. Trailing-slash directory markers with children reach the same delete.
* filer: keep the sentinel when MkFile and Mkdir report a create failure
Same flattening one layer out: every mkFile caller lost the sentinel, so
a CopyObject onto a key that other keys are nested under answered a
retryable 500 where a PutObject of the same key answers 409.
* s3: copy and rename a key that other keys are nested under
Such a key is stored on the directory those keys live in, and copy and
rename both refused it: the source lookup maps every directory entry to
NoSuchKey, so a key a plain GET serves could not be copied or moved, and
the destination side refused it as a directory conflict.
The source is read through a view of the entry as the object it names.
The destination is written the way a PutObject of that key writes it. A
rename at either end copies the object's own data across and strips it off
the source key rather than going through AtomicRenameEntry, which moves a
directory by moving everything under it - the nested keys are not part of
what is being renamed.
* worker: compile plugin.proto with the protoc that ships with the build
seaweed-volume already does this: protoc-bin-vendored carries the binary, so
the build needs no package manager and every build sees the same version. An
explicit PROTOC still wins, which is what lets the lance crates - whose own
build scripts read the same variable - share it.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: point the worker builds at the vendored protoc
The jobs installed protobuf-compiler for lance's build scripts. They read
PROTOC, so pointing it at the binary protoc-bin-vendored already puts in the
registry serves them without a system package - one less apt call on the way
to a release, and the same protoc a developer's build uses.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say what the worker build needs from protoc
The lance crates' build scripts are the ones that need it, not ours, and they
take the same vendored binary.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* worker: name the binary weed-worker
It is the Rust side of `weed worker`, the way weed-volume is the Rust side of
`weed volume`, and lance is the first family of jobs it carries rather than
the only one it ever will. The crate keeps its own name: when a second family
arrives the bin target moves to a crate of its own, under this name.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: ship the Rust maintenance worker in the image
Lance table buckets need a worker that can read the format, and until now the
only way to get one was a Rust toolchain and a cargo build. It now sits at
/usr/bin/weed-worker beside the Rust volume server, reached as
`docker run chrislusf/seaweedfs worker-rust --admin host:23646` — the verb
mirrors volume-rust, so plain `worker` still runs the Go one.
Taken pre-built or not at all: the lance jobs pull in arrow and datafusion, far
too large a tree to compile inside the image build, so an architecture CI did
not build for gets the empty placeholder the entrypoint refuses to exec, the
way the Rust volume server already does.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build the Rust worker for the container images
The same native cross-compile the volume server uses, so the release, latest
and dev images all carry it on amd64 and arm64. The artifact holds both
binaries now, so it is named for that rather than for the volume server.
Only the release directory each job builds is cached: with a debug profile
beside it the worker's target/ reaches 24GB, against a 10GB cache budget.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: publish Rust worker binaries with the release
Linux amd64 and arm64 only: the worker runs beside the cluster it maintains,
and its dependency tree makes every extra target an expensive build.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: build and test the Rust workers on change
Nothing built seaweed-worker in CI, so the release and the container images
would have been the first place a break showed up. Tests run in release too,
rather than compiling lance, arrow and datafusion again in another profile.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docs: say how to get a released worker
Neither the image nor the release tarballs were mentioned; a toolchain and a
cargo build read as the only way in.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: install protoc for the Rust worker builds
lance's crates compile their own protos, and unlike seaweed-volume they do not
vendor a protoc to do it with, so every job that builds the worker failed at
lance-encoding's build script.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* ci: do not persist credentials in the worker release checkout
The upload step is handed a token explicitly; a cargo build script should not
find another one sitting in the checkout's git config.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* docker: keep the worker's argument boundaries
Unquoted $@ splits on whitespace and expands globs, so an argument carrying
either arrived as something else.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* helm: install chart CI against an image tag that exists
The release bumps appVersion on master well before the container build
publishes that tag, and the chart CI runs on the bump commit, so every
release turns it red with ImagePullBackOff. Resolve the tag first and
fall back to latest while the new one is still building.
* helm: run the chart CI when the workflow itself changes
* helm: bound the registry lookup in the chart CI
An unbounded curl can hold the job, and the log did not say why the tag
was rejected. Cap it and print the status.
* test: drive the Lance namespace with Spark
The counterpart of catalog_spark, which does this for the Iceberg REST
catalog. Spark is the engine most likely to be pointed at a lakehouse,
and it reaches the Lance catalog through the connector's DSV2 catalog -
org.lance.spark.LanceNamespaceSparkCatalog with impl=rest - over the same
routes every other client uses.
SHOW NAMESPACES -> ['`sparklance-lcephd80`.ml']
SHOW TABLES -> ['sparklance-lcephd80$ml$embeddings']
count -> 3
filtered -> [(2, 'two'), (3, 'three')]
count after a second commit -> 4
The second insert is there on purpose: a store that cannot order commits
fails on the second one, not the first.
Two things the run settled that were guesses beforehand. CREATE TABLE
works, because the connector declares through the namespace and writes the
data itself rather than pushing Arrow at the server. And SHOW TABLES
returns the namespace's own identifiers - bucket, namespace and name
joined by the delimiter - not bare Spark table names.
Credentials go under the catalog's storage.* prefix, which is handed to
lance as object_store options; a gateway without STS vends none, the same
trap the LanceDB suite documents.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame Spark for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: keep the ivy cache under the user's cache directory
It is mounted into a container running as root, so a shared temp path lets
another local user pre-create it and choose what Spark loads.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: assert the vector column's type, not only its name
A column that came back as array<double> or array<string> would still be
called vector and still pass.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: read the dataset off its location for real
The catalog being optional is the property that lets duckdb and pandas read
these tables; it was asserted in a comment and printed, never exercised.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the Spark Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: say the hosts in the README are placeholders
The suite passes dynamically allocated host.docker.internal ports.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
The db credential secret and the filer's WEED_MYSQL_* env were gated on
filer.enabled alone, so a filer on mongodb, redis, postgres or leveldb2 got a
generated mysql secret it never reads - kept forever by resource-policy: keep -
plus a mysql-db-host pointing nowhere.
Gate all of it on WEED_MYSQL_ENABLED, which is how the store is selected, plain
keys and secret-backed ones alike. An enable flag the chart cannot read - a
valueFrom, or one in secretExtraEnvironmentVars - counts as selected, so nothing
is dropped from a filer that is actually on mysql.
* s3: resolve ${VAR} in static config credentials from the environment
A deployment that keeps its S3 keys in a secret store had no way to hand
them to the gateway: -config takes a file, so the keys had to be written
into that file. Let a key in the static config name an environment
variable instead, and drop any credential whose reference stays unset so
the placeholder never becomes a usable key.
* helm: source the generated s3 identities from an existing secret
The only way to reuse credentials that already live in a Secret was to
hand-author the whole seaweedfs_s3_config JSON, since the literal keys in
values.yaml end up in git and a lookup-based keyRef renders empty under
helm template and Argo CD. Let s3.credentials.admin/read name a Secret and
its keys instead: the generated config references them as ${VAR} and the
gateway resolves them from the environment, so nothing is read from the
cluster at render time.
* s3: treat an empty environment value as an unresolved credential reference
A secret store can hand over a key that exists but is blank. Resolving it
would leave an access key whose signing secret is empty, so count it as
unresolved and drop the credential.
* helm: render the s3 secret when only the all-in-one auth flag is set
The all-in-one deployment mounts the s3 secret whenever any of the three
enableAuth flags is set, but the secret itself only rendered for the s3 and
filer flags, so allInOne.s3.enableAuth on its own left the pod waiting on a
secret nothing creates.
* helm ci: check the credential wiring on every workload that mounts it
The render check only looked at the standalone s3 deployment and only at
one of the four variables, so a helper that bound a variable to the wrong
secret key would still pass.
* helm: create the all-in-one s3 secret for every flag that mounts it
The all-in-one pod mounts the secret on any of the three enableAuth flags,
so keying its creation off allInOne.s3.enableAuth alone still left
filer.s3.enableAuth without filer.s3.enabled pointing at a secret nothing
creates. Mirror the deployment's own condition instead, and check each
flag renders both the mount and the secret.
* s3: reject a malformed credential reference instead of keying on it
A typo such as ${MY-VAR} matches no substitution, so it survived expansion
and the placeholder itself became the access key the gateway accepted.
Require every ${ in a static credential to open a well-formed reference.
* release: cut the whole release from the version bump workflow
The bump workflow stopped after pushing the version commit, and the rest was
manual: create the release, then run "Prepare release" in the csi-driver and
the operator. It now pushes the tag itself, which is what starts the binary,
container and helm workflows, creates the release with generated notes, and
dispatches the other two repositories, waiting for both.
Pushing the tag and reaching the other repositories both need RELEASE_PAT;
GITHUB_TOKEN raises no events that start workflows.
* release: tighten the release workflow after review
Check out master explicitly: a dispatch can select any branch, and the tag,
the commit and the release would then come off that branch while the
downstream job dispatches master.
Scope contents:write to the job that pushes; the downstream job talks to the
other repositories with RELEASE_PAT and needs nothing here.
Wait for the module proxy to serve the release commit as the tip before
dispatching, instead of priming it and hoping. The dispatched workflows pin
seaweedfs with `go get -u ...@latest`, so a stale tip means they release
against a pre-release commit, silently.
Identify the dispatched run by diffing the run list against the snapshot
taken before dispatching, rather than assuming the newest run is ours.
* release: wait on the downstream release, not on the run that makes it
A dispatched run cannot be told apart from a concurrent one: the API does not
report the inputs a run was dispatched with, so watching "the run that appeared
after mine" can watch someone else's and report their result as ours.
Wait for a release to appear in the downstream repository instead. That is the
thing being waited for, and it holds however many runs are in flight.
* test: read Lance tables from DuckDB
The LanceDB and Spark suites go through the catalog. DuckDB does not: its
lance extension reaches the data over S3 with no namespace involved, which
exercises the other half of the design - a table bucket's layout is a
valid Lance dataset directory, so a table stays readable when the catalog
is not in the path.
scan_rows=128
scan_columns=id,title,vector
filtered_rows=5
nearest=1,0,2
It also pins the one place the layout costs us. DuckDB's replacement scan
recognises a dataset by a .lance path suffix, and tables created through
this catalog deliberately have none: the catalog entry is the dataset
directory, a table name may not contain a dot, and a suffix would leak
into ARNs and policies. So __lance_scan is the way in, and the bare
SELECT ... FROM 's3://...' form does not see these tables.
The test asserts both halves - a suffixed path is read, a suffix-less one
is not - so if the extension ever recognises a bare directory, it fails
and says to update the documentation rather than leaving it wrong.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: require the catalog error from the suffix-less read
Any failure satisfied the old check - a missing extension, bad credentials,
an unreachable endpoint - so the assertion could pass without the
replacement scan ever classifying the path.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: verify the Lance table bucket was actually created
weed shell prints a command's own failure and still exits 0, so the harness
would go on to blame DuckDB for a bucket that was never made.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: bound the Docker probe
An unhealthy daemon makes docker version hang, and the probe runs before the
test has a timeout of its own.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: order the aggregates the assertions read
string_agg over an unordered relation may return the names, and the vector
search's ids, in any order, so the expectations could fail on a run where
nothing changed.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: do not persist credentials in the DuckDB Lance checkout
The job only uploads a log on failure; nothing in it pushes.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* lance worker: share the integration tests' scaffolding
The recorder that keeps what a handler sent, the config builder and the
storage-option fallback all lived inside compaction.rs, so a second test
binary would have had to copy them. They move to tests/common.
The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and
AWS_ENDPOINT_URL from the environment, defaulting to what it used before.
A harness can then point these tests at a gateway that checks what it is
given rather than one that accepts anything.
* lance worker: maintain one named table, for a harness to drive
Compacts and cleans up whatever WEED_LANCE_TABLE names, through the
handlers' own detect-then-execute path: a proposal the worker would not
have made is not one worth running.
The existing tests seed the tables they check. This one deliberately does
not, so a harness that has already written a table and knows what is in it
can have the real handlers maintain it and then read it back.
* test: take a table through its whole life, for Iceberg and Lance
Created in the catalog, filled by a real client, maintained by the worker,
read again, dropped. The step nothing was checking is the read after
maintenance: compaction once rewrote every dictionary-encoded column onto
a single value and shipped, because the maintenance tests were thorough
about sequence numbers, manifest entries and metadata versions and none of
them opened the parquet file the worker had just written.
So the assertion is a tally - row count, the cardinality of each
dictionary-encoded column, and an md5 over whole rows - taken before
maintenance and again after, required to be equal. The cardinalities name
the failure that happened; the digest catches a rewrite that keeps every
column's cardinality and hands the values to the wrong rows. A compaction
that merged nothing fails rather than passes, or the read afterwards is
checking a file the worker never wrote.
The Iceberg half runs two clients. DuckDB is the one the corruption was
reported against and the only one here that writes the deprecated
PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a
Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning
parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg
one, which is why both are here.
Lance maintenance lives in the Rust worker, so it runs there where cargo
is installed and through the two lance calls those handlers wrap where it
is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess.
* ci: run the table lifecycle tests
CI maintains the Lance table through the lance library rather than the
worker: a cold build of the lance crate costs more than the glue it would
be checking, and the worker's own tests cover its handlers.
The suite drives the Iceberg maintenance worker, so a change to it now
triggers this workflow too.
* test: let the lifecycle harness fail instead of skipping
Setup failures all exited zero, so a cluster that would not come up, or a
port allocation that lost, reported a green run for code nothing had
executed. That is the failure mode this whole directory exists to close,
and it was in the harness itself.
Only a checkout without a weed binary skips now, and it runs the tests so
each one says so rather than the package quietly passing. Everything else
fails.
The filer existence probe gets a deadline while I am here: it ran without
one, so an unresponsive filer would hang the suite past every timeout the
clients have.
* test: make the lifecycle checks check what they claim to
Three of them could pass without having looked.
The DuckDB skip matched "syntax error", "not implemented" and "Failed to
load" anywhere in the output, in any phase. A parse error in the SQL this
test generates, or a refusal from our own catalog, would have taken the
only coverage of the PLAIN_DICTIONARY encoding out of CI and left it
green. It now matches the extension failing to install, and only in the
phase that installs it. Everything past LOAD is ours and fails.
The digests covered id, category and value. Compaction rewrites the whole
row, so a defect confined to ts, or to a Lance vector, changed nothing
either side of maintenance. Every persisted column goes in now, ts as
microseconds so no timezone sits between the two runs.
The Lance drop check caught every exception as proof the dataset was
gone. pylance turns credential and transport failures into the same
ValueError, so it only accepts the message that means not found.
* docs: say up front which maintenance path the Lance half takes
The opening summary said the worker maintains both tables. It maintains
the Iceberg one always and the Lance one only where cargo is installed,
which is not what CI does.
* test: drive the Lance namespace with LanceDB
The Iceberg catalog is checked against Spark, Trino, ClickHouse, Doris,
Dremio and RisingWave. The Lance one had only its own reference client,
which is the same thing as checking it against ourselves.
LanceDB connects with connect_namespace("rest", ...), which speaks the
routes this catalog implements, so the suite exercises the protocol rather
than our idea of it: list the catalog, open a table through it, read the
schema, run a vector search and a filtered scan, create a table, and read
the same dataset straight off its URI with no catalog at all.
table_names -> ['lancedb-p0guidmm$ml$embeddings']
open_table -> 64 rows
search -> [1, 0, 2]
create_table -> 4 rows, listed by the catalog
direct read without the catalog -> 64 rows
Seeding is pylance, because the namespace records where a table lives and
does not carry its data. That split is the design rather than a limit of
the test.
One interop note the test encodes: a gateway without STS vends
storage_options carrying an endpoint and a region but no credentials, and
LanceDB uses what the namespace vends on some paths. The container gets
credentials in its environment as well, which is what a deployment without
STS would do.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
* test: pin the LanceDB client, and index before searching
Three from review.
The client's dependencies were unpinned, so an unrelated upstream release
could change what an old commit reproduces. Pinned to the versions this
suite was verified against; the client is as much the thing under test as
the server.
The search was called ANN and was not: without an index LanceDB scans.
The test now builds an IVF_PQ index over 1024 rows first, which is worth
more than the wording fix - an index writes into a directory of the table
that the S3 door has to admit, and that guard has refused a Lance
directory before. It builds, covers all 1024 rows, and searches.
The assertion moved with it. Demanding the exact nearest neighbour was
right for a brute-force scan and wrong for a quantized index, which
answered 0 as readily as 1; both are correct, so the check is now the
neighbourhood.
And the pushdown check accepted any failure. It now requires the refusal
to be the catalog's Unsupported and requires that nothing was left behind,
or, when the client falls back, that the table is complete.
Claude-Session: https://claude.ai/code/session_01Rkp1Mw5E89Jp6dzJFYiMrm
Four workflows ran apt to install libfuse3-dev before every FUSE job. Nothing
needs it: go-fuse implements the protocol in pure Go, no cgo in the tree
references fuse, and the package does not even provide the fusermount3 the
mount actually execs - fuse3 does, and it is already on the runner image, which
is why the setuid-repair step finds it.
So the step downloaded a dev package to build against headers no compiler ever
opened, and it is the step that has been hanging whenever the Ubuntu mirror
goes slow. Configuring /etc/fuse.conf is all that is left.
The e2e workflow sets defaults.run.working-directory: docker, so the call I
added resolved to docker/docker/apt-install and every FUSE Mount run has
failed with 'sudo: docker/apt-install: command not found' since it merged.
Every workflow that builds the Rust volume server first installed protoc
from a package manager - twelve steps across apt, brew and choco. That is
37s per job on a good day, and this week archive.ubuntu.com stalled long
enough for four jobs to burn their whole timeout without reaching a build.
protoc-bin-vendored ships the compiler as a build-dependency, so it now
arrives through the cargo registry the workflows already cache and there
is nothing left to install. cargo build works on a machine with no protoc
at all, which is worth as much locally as it is in CI.
It also pins the version. The apt protoc on ubuntu-22.04 is 3.12, old
enough to reject proto3 optional, which is why build.rs passes
--experimental_allow_proto3_optional; the vendored one is 31.1. The flag
stays, since it costs nothing and keeps a build against an older PROTOC
working, and an explicit PROTOC still overrides the vendored binary for
packagers who supply their own.
The e2e job overwrote the runner's sources.list with two azure-only lines and
installed fuse from it, so the same mirror outage that took out the image
builds failed the step outright - this time on the runner rather than inside
the container, where the image-side fallback cannot reach.
Install through the same helper, and widen its rewrite to match any archive
host so it works whether the pristine list came from the base image
(archive.ubuntu.com) or from a CI runner (azure.archive.ubuntu.com). Keeping
the runner's original list also restores the security and backports pockets,
which the hand-written two-line replacement dropped.
Verified against the outage itself: with the pristine list pointed at Azure,
the build logged the skip after Azure timed out for real and installed from
archive.ubuntu.com.
* mount: re-resolve volume locations after a failed chunk read
NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.
* test: FUSE integration tests for volume server failover
One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.
* test: report the filer's own view when append content mismatches
A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.
* test: wait for the reader mount to converge before comparing
A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.
* test: detect a failover cluster child that exited at startup
Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.
* test: read a file the killed volume server actually holds
Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.
* ci: stop persisting checkout credentials in the failover workflow
The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.
* test: discard the ignored errors errcheck flags in the failover harness
* test: resolve manifests when mapping a file to its volumes
A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.
* test: pin the stale-location recovery path with a primed reader
Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.
