4 Commits

Include renames

Author	SHA1	Message	Date
4 people	166af06a2b	rust: cargo fmt both crates, with a commented-out fmt --check CI step (#11329) ... * rust: migrate seaweed-volume and seaweed-worker to tonic 0.14 / prost 0.14 tonic 0.14 boxes the contents of tonic::Status, which is what made every RPC path trip clippy's result_large_err; the allow for that lint goes in the next commit. The prost codec moved out of tonic into tonic-prost and tonic-prost-build, so both build scripts now call tonic_prost_build::configure() and both crates depend on tonic-prost for the generated code. The `tls` feature was split into a per-backend feature; `tls-aws-lc` is the same backend both crates already install through rustls::crypto::aws_lc_rs. tonic 0.14 depends on axum 0.8 and tower 0.5, which would have left a second axum and a second tower in each tree next to the 0.7 / 0.4 the crates named themselves. Bumping them keeps one copy of each: axum 0.8 only changes the path-parameter syntax for the routes here (`/:vid` -> `/{vid}`, `/*path` -> `/{*path}`), tower 0.5 needs the `util` feature named explicitly for ServiceExt::oneshot (it used to arrive through tonic's feature unification), and tower-http 0.6 is the matching release. Lock files move only through cargo's own resolution for the new versions; no other dependency was refreshed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: drop the result_large_err allow now that tonic::Status is boxed tonic 0.14 stores Status behind a Box, so Result<_, Status> is no longer a large-Err type and clippy has nothing to say about it. Both crates pass `cargo clippy --all-targets -- -D warnings` without the allow (seaweed-volume in both feature sets), so the policy entry and its comment go. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: drop the unused headers argument of try_expand_chunk_manifest The parameter was already named `_headers`; nothing in the body reads it. With it gone the function is under clippy's argument threshold and the expect goes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: pass EC peer reads an EcInterval instead of ten arguments fetch_one_interval, read_remote_ec_shard_interval, do_read_remote_ec_shard_interval and recover_one_remote_ec_shard_interval all took the same (vid, needle_id, shard_id, shard_offset, size, expected_encode_ts_ns) tuple, and the two that reconstruct also took the location map with the data/parity counts. Those are now EcInterval (Copy) and EcShardMap (a borrow of the map plus the counts). The fan-out inside recovery builds its per-shard request with `EcInterval { shard_id: sid, ..iv }`, which is the one place the old argument list was easy to get wrong. Bodies destructure at the top, so the code below the signatures is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: give the EC encoder an EcEncodeLayout and an EncodeRun encode_dat_file took the Reed-Solomon shape and three block sizes as five loose integers; they are now one Copy struct, EcEncodeLayout, which is what Go calls ECContext. The per-row and per-batch helpers took the same six sinks and the offsets; they become methods on EncodeRun, which owns the borrows for one run, so each call names only the offset and block size that vary. The byte-level work is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: describe a .dat rebuild with DatRebuild instead of nine arguments write_dat_file_from_shards, its _with_dirs twin and the private write_dat_file were three layers over one nine-argument signature. One public function now takes a DatRebuild, whose shard_dirs is None when every shard sits beside the .dat and Some(dirs) for the cross-disk reconciled layout. The field docs carry what the function doc used to say about the encode-time size and the block layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: split copy_file_from_source's fifteen arguments into two structs CopyFileSpec is the per-file request (what to ask the source for, where it lands, whether its bytes count as progress); CopyProgress is the sender, throttler and report state that all three files of one VolumeCopy share, held by &mut across the calls. The three production call sites now read as the .dat/.idx/.vif literals they are, instead of positional trues and falses. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: create volumes from a VolumeSpec Volume::new, DiskLocation::create_volume and Store::add_volume each took the same five-value tail of Go's NewVolume argument list: collection, replica placement, TTL, preallocation and needle version. That tail is now VolumeSpec, a Copy struct whose Default is what almost every test wanted anyway (empty collection, no replication, no TTL, no preallocation, current version), so most of the 104 call sites shrink to `&VolumeSpec::default()` or name the one field they set. The id, directories, index kind and disk type stay positional because they differ at every site. Two imports that only test modules use moved into those modules, and DiskLocation no longer imports ReplicaPlacement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-worker: run cargo fmt Layout only; no token in the workspace changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust-volume: run cargo fmt Layout only; no token in the crate changes. Every earlier Rust PR here formatted only the blocks it touched so as not to drown its diff in this one, and this commit is that debt paid in a single place. rustfmt needed two passes to settle one block in handlers.rs; the committed form is the fixed point, so `cargo fmt --check` is clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out cargo fmt --check step to both Rust workflows Same shape as the commented clippy step from #11312: the check is written out so that making formatting a gate is a one-line uncomment, and whether to do that stays a maintainer call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2026-09-15 09:29:22 -07:00
4 people	adaf3534fa	rust: clippy-clean both crates and adopt the std APIs the 1.91 MSRV allows (#11312) ... * rust: apply clippy --fix to both crates The mechanical part of a clippy sweep: `cargo clippy --all-targets --fix` on seaweed-volume and the seaweed-worker workspace, hand-reviewed. Both manifests declare their MSRV (1.91.1 and 1.94.1), so every suggestion clippy applied is within it: the collapsible_if sites become let chains (1.88, edition 2024), `% n == 0` becomes is_multiple_of (1.87), chunks_exact with a constant becomes as_chunks (1.88), repeat().take() becomes repeat_n (1.82), and io::Error::new(Other, ..) becomes io::Error::other (1.74). The rest is redundant clones, borrows, casts, closures and field names. Nothing here changes behaviour. The three let_and_return sites in needle_map.rs and store_ec.rs deserve a note: the `let result = ..; result` shape was a deliberate edition-2021 workaround to drop a redb guard before the table it borrows. Edition 2024 drops tail-expression temporaries before locals, which is why clippy now flags it, and the two comments that described the workaround say so instead. Manual edits on top of the tool output: the blocks clippy rewrote are re-indented the way rustfmt lays them out (only those blocks — the crate is not rustfmt-clean and a whole-crate fmt would bury this diff), the blank lines let_and_return left behind are removed, and the CRC legacy_value test compares against a literal worked out from the original shift formula rather than restating rotate_right. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust: clear the clippy warnings --fix cannot apply, and say why the rest stay Hand fixes for the lints clippy only reports. Behaviour is unchanged throughout; each rewrite is the one clippy names. - needless_range_loop (7): index loops over shard vectors become iterator loops. Where the old code indexed `v[..n]` the new loop iterates `v[..n]` so an undersized vector still panics the same way. - field_reassign_with_default (6): struct literals with `..Default`. - redundant_pattern_matching (3): `if let Err(_) = guard.check()` becomes `.is_err()`, which also releases the read guard at the end of the condition instead of at the end of the block. - manual_strip (2), manual_checked_ops, format_in_format_args, redundant_locals, wrong_self_convention (to_vif takes self by value, so it is into_vif; CompactEntry is Copy, so to_needle_value takes self). - type_complexity (2): `OrphanShardLoad` and `RawNeedleEntry` name two tuples that were spelled out inline. - new_without_default: CompactNeedleMap gets a Default that calls new(). - suspicious_open_options: a test helper spells out `.truncate(false)`, which is what `.create(true).write(true)` already did. What stays, and the attribute that says so: - too_many_arguments (10): `#[expect]` on each function. Folding 8–15 parameters into a struct is a design change, not a lint fix. - await_holding_lock / readonly_write_lock: one test holds the store write guard across a sleep on purpose, as a barrier that parks the copy task at the mount block. `#[expect(.., reason = ..)]` records it. - module_inception: needle/needle.rs mirrors the Go package layout. Two lints become crate-wide policy in `[lints.clippy]`, with the reason next to each: result_large_err, because every RPC path returns tonic::Status (176 bytes) and boxing it would change every handler signature; and needless_update, because `..Default::default()` on a protobuf message literal is what lets a proto gain a field without touching every constructor (all 11 sites are pb messages). The worker workspace gets the same table and its members opt in with `lints.workspace = true`; its generated plugin.rs also allows large_enum_variant on prost's oneof enums. Both crates are now clean under `cargo clippy --all-targets -- -D warnings`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: use the std APIs the 1.91 MSRV already pays for The crate declares rust-version 1.91.1, so a few things the code still worked around are plain std now. All of them come from the 1.85–1.91 release notes; nothing here needs a newer toolchain than the manifest already requires. - std::sync::LazyLock (1.80) replaces the lazy_static! block in metrics.rs, and the lazy_static dependency goes. Every use site reads the same through Deref, so no caller changes. - Duration::from_mins / from_hours (1.91) replace `from_secs(v * 60)` and `from_secs(v * 3600)` in the option parser and the shard-location refresh TTLs. One difference for the parser: an absurd count that overflows u64 seconds now panics in release builds too, where the multiplication used to wrap. - Result::flatten (1.89) replaces `.and_then(|r| r)` on the replication join handle. - OsStr::display (1.87) replaces `to_string_lossy()` where the name was only being formatted; the output is byte-identical. - `#[allow]` becomes `#[expect]` (1.81) on the suppressions that are meant to be permanent, so a suppression that stops being needed becomes a warning rather than lingering. Doing that found four that already had: dead_code on ChunkManifest, base_name and last_io_error, and too_many_arguments on read_from_data_shards, which is down to seven parameters. Those attributes are deleted. The three allows that depend on cfg (a unix-only mutation, a linux-only field set, a profiling-only parameter) stay as allow, because expect would be unfulfilled on the other platforms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * ci: add a commented-out clippy step to both Rust workflows Both crates are warning-free under `cargo clippy --all-targets -D warnings` now. Whether that becomes a gate is a policy call, so the step is present but commented out; uncommenting it is the whole change. The comment points at the `[lints.clippy]` table where crate-wide exceptions are recorded, so the gate does not become a reason to sprinkle allows. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CjZY429aVU74SLDmo1wiuU * rust volume: guard parse_duration against overflow panics Duration::from_mins/from_hours panic when the count overflows u64 seconds. Use checked_mul so an oversized CLI value falls back to the parser default instead of crashing volume startup. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-14 11:29:29 -07:00
4 people	516e251f9e	rust volume: move the crate to edition 2024 (#11244) ... * rust volume: move the crate to edition 2024 Edition 2024 turns three things in this crate into hard errors, and changes drop order in a further 34 places without changing compilation. The compiler errors are fixed here; the silent changes were audited against `RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets` output captured before the flip, since edition 2024 stops reporting them. `std::env::set_var`/`remove_var` are unsafe as of 2024 because they race with concurrent readers. All six call sites are safe by construction rather than by assertion, and the SAFETY comments say why: the build script runs single-threaded before anything else in the process, and every test reaching the `config.rs` helpers holds `process_state_lock()` for the duration. The two `ref` bindings in handlers.rs sit in patterns that already borrow implicitly, so removing the modifier leaves both bindings at `&String`. On the 34 drop-order sites: no lock guard's scope is extended anywhere, and `volume.rs` has none. Most are moved-from `Option`/`Result` husks — `if let Some(v) = map.remove(&k)`, `while let Some(m) = stream.next().await` — where the value is moved into the binding and the temporary has nothing left to drop; where closing order actually matters these paths already call `v.close()`, `ec_vol.destroy()` or `drop(writer)` explicitly. Two sites get strictly better ordering: the metrics read guard in `run_metrics_push_loop` shrinks to the end of its initializer block (it never crossed an `.await` either way), and an EC test now closes the volume's descriptors before the `TempDir` removes the directory. No `rust-version` is declared. Edition 2024 needs rustc 1.85, but that is not the binding constraint — the dependency tree already requires 1.91.1 through the `aws-sdk-s3`/`aws-smithy-*` family, so `cargo +1.85 check` fails on the deps regardless. CI builds on `dtolnay/rust-toolchain@stable`. `vendor/reed-solomon-erasure` is a separate package and keeps edition 2021. Cargo.lock is unchanged despite edition 2024 implying resolver 3. Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features` 550 passed / 0 failed (the two feature sets produce an identical migration site list), `cargo build --release` clean. No automated test covers shutdown ordering, so the channel and runtime sites in `main.rs`, `write_queue.rs` and `grpc_server.rs` were read individually. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust volume: address edition-2024 review feedback Three fixes from review of the edition bump. Serialize the two environment-reading tests. The SAFETY comments on the `env::set_var`/`remove_var` helpers claim every test touching the environment holds `process_state_lock()`, but `test_resolve_config_defaults_dir_to_platform_temp_dir` and `test_resolve_config_index_accepts_redb_and_leveldb_aliases` called `resolve_config` — which reads HOME/USERPROFILE, SEAWEED_WRITE_QUEUE and the WEED_* set — without taking it. `set_var` is unsafe precisely because a concurrent *reader* is UB, not only a concurrent writer, so the comment was overclaiming. An audit of the module found exactly these two; every other environment-touching test already held the lock. The race predates edition 2024, which only made the requirement explicit. Declare `rust-version = "1.91.1"`. The edition needs 1.85, but that was never the binding constraint: `cargo +1.90 check --all-targets` fails on the `aws-sdk-s3`/`aws-smithy-*` family, and 1.91.1 checks clean. Declaring the verified floor turns a wall of per-dependency errors into one clear message. Cargo.lock is unchanged despite this making the resolver MSRV-aware. Update the README, which advertised "Rust 1.75+ (2021 edition)". 1.75 was already stale before this branch — the tree has needed 1.91 for a while. Verified: `cargo test` 551 passed / 0 failed, `cargo test --no-default-features` 550 passed / 0 failed, `cargo build --release` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust volume: state the exact MSRV patch release in the README The README said "Rust 1.91+", which reads as 1.91.0 and is wrong by one patch release: `cargo +1.91.0 check --all-targets` fails on the aws-sdk-s3 family, `cargo +1.91.1` passes. Say 1.91.1+, matching `rust-version` in Cargo.toml, and call out that the patch component is load-bearing so nobody installs 1.91.0 and hits the same wall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nty5Rj7ssMQdFxHHjZgDC * rust worker: move the workspace to edition 2024 Moves the seaweed-worker workspace (core, lance, sort) from edition 2021 to 2024, the same migration seaweed-volume just got in this branch. Edition 2024 turns exactly one thing in this workspace into a hard error. The baseline came from RUSTFLAGS='-W rust-2024-compatibility' cargo check --all-targets, run before the flip; unlike seaweed-volume's 34 silent + 8 hard sites, the worker reports only the one hard site and no tail_expr_drop_order or if_let_rescope sites at all. The worker is a much smaller crate and none of its expressions hold a guard or temporary whose drop order the edition changes, so there is nothing to audit on the silent side. Fixed (1 site): std::env::set_var is unsafe as of 2024 because it races with concurrent readers. The single call is in crates/core/build.rs, which sets PROTOC from protoc_bin_vendored the way seaweed-volume's build script does. A build script's main runs single-threaded before anything else in the process, so no other thread can be reading the environment concurrently; the SAFETY comment says so. There are no config.rs-style test helpers here -- the worker's tests do not mutate the environment -- so unlike the volume crate there are no process_state_lock() callers to audit. No redundant ref bindings to clean up: a grep for ref across the three crates finds none. MSRV: rust-version = "1.94.1", verified rather than inferred. Edition 2024 only needs 1.85, but the dependency tree needs more: lance's aws feature pulls in a newer cut of the same aws-sdk-*/aws-smithy-* family that sets seaweed-volume's 1.91.1 floor, and that newer cut requires 1.94.1. cargo +1.94.0 check --all-targets fails on that family; cargo +1.94.1 check --all-targets is clean. The worker's floor is therefore higher than the volume's, and moves with lance and the AWS SDK rather than with the edition. CI builds on dtolnay/rust-toolchain@stable, so nothing changes there. The edition is set once in [workspace.package] and inherited by each member via edition.workspace = true; rust-version is added the same way. The workspace keeps its explicit resolver = "2" -- edition 2024 would default to resolver 3, but the pin is deliberate and Cargo.lock is unchanged by this commit either way. The README gains a "Requires Rust 1.94.1+ (2024 edition)" line in its Building section, matching the one seaweed-volume's README now carries, and calling out that the patch release is load-bearing (1.94.0 does not build) so nobody installs 1.94.0 and hits the same wall. Verification: * cargo check --all-targets -- clean, zero warnings (default toolchain 1.97) * cargo +1.94.1 check --all-targets -- clean * cargo +1.94.0 check --all-targets -- fails on the AWS SDK, as claimed * cargo test --all-targets -- 40 passed, 0 failed (core 13, sort 11, lance lib 3, lance bin 2, compaction 6, lifecycle 1, sort integration 4) * Cargo.lock unchanged Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>	2026-09-09 10:54:06 -07:00
Chris LuandGitHub	9e34426a56	lance: a maintenance job that sorts a table by its declared fields (#11113) ... * lance: a maintenance job that sorts a table by its declared fields Lance appends fragments in write order and has no notion of a sorted table, so nothing but a rewrite establishes one, and nothing but another rewrite restores it once rows have been appended. lance_sort reads the order from the dataset's own configuration, falls back to the worker's, and rewrites the table in it. The spec and the marker live in crates/sort rather than in the job, because weed/worker/tasks/iceberg sorts too: two jobs that disagreed about what "id desc nulls-first" means would be two features wearing one name. The sort spills. lance builds its DataFusion runtime with a FairSpillPool and a disk manager, but only when LanceExecutionOptions::use_spilling is set, and that struct derives Default over a plain bool — so Scanner::try_into_stream, which fills its options with ..Default::default(), is precisely the path that does not spill. The job builds the plan with create_plan and executes it with spilling on and the operator's memory budget. The marker rides in the same commit as the data: Operation::Overwrite is the one operation carrying config values alongside fragments, so a sorted table and the record of its sorting cannot disagree. It records the version the sort read, not the one it wrote, which is not knowable while the marker is being assembled. Detection treats anything committed after the sort's own commit as data the sort did not produce — row counts alone cannot see a rewrite that leaves the count where it was, and such a table would look sorted forever. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: identify a sorted table by the files it wrote, not its version Review found three ways the version-based marker misjudges a table, and they share a cause: the version a sort produces is not knowable while the marker is being assembled, so the marker recorded the version it read and detection inferred the rest. A commit that rebases past a conflict lands on a different number, and the inference then reads a rewrite into an ordinary table — a full re-sort, and its indices, for nothing. Data file names do not have that problem. They are chosen before the commit, so the commit can carry them, and they do not change with the version it lands on. The marker now records how many files the sort wrote and a digest of their names, and detection asks whether the table still holds them: the same files means untouched, the same files followed by more means appended, anything else means the data was replaced. That also closes the hole the row threshold left. A replacement that grew the table by fewer rows than min_unsorted_rows read as sorted, however many rows had actually moved; the threshold now applies only where the sorted files are still in place, which is what it was for. A marker without a row count is stale rather than a zero to compare against, and deletes stop forcing a re-sort — they write a deletion file beside the data rather than rewriting it, and removing rows does not unsort the ones that remain. Sort fields are also compared exactly rather than case-folded. Arrow schemas are case-sensitive, so `id` and `ID` are two columns, and folding them together rejected a valid order. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y * lance: count the rows appended after a sort, not the table's net growth Review found that rows deleted from the sorted fragments hide appended rows one for one: the threshold compared the live row count against the count recorded at sort time, so 800 deletions and 300 appends read as a table that shrank, and a table where deletions keep pace with appends stays "sorted" with an unsorted tail forever. The fragments say it directly. The marker already records how many fragments the sort wrote, so the ones after that prefix are exactly what arrived since, and the manifest carries each fragment's live row count — physical rows less its deletions. Counting those is the arithmetic the threshold was always meant to do, and it needs no row count from the marker at all. A fragment whose length the manifest does not record cannot be counted, and a table that cannot be judged is one to sort rather than one to leave alone forever, so an uncountable appended fragment reads as stale. Claude-Session: https://claude.ai/code/session_015SZkLTUvd1svDu4xdr6Q3y	2026-09-02 21:25:50 -07:00