5 people 8ff2e0777e volume server: HTTP DELETE on a distributed EC volume (#11405) ...

* volume server: HTTP DELETE on a distributed EC volume

The delete handler validated the cookie with EcVolume::read_ec_shard_needle,
which reads only locally-mounted shards and errors "ec shard N not available
locally" for any interval held by a peer. Every Err was mapped to 500 and no
.ecj tombstone was appended, so on a standard 10+4 spread over 14 servers an
HTTP delete of an EC needle could not succeed. The GET path already goes
through read_ec_shard_needle_distributed.

Route the delete's read through the same distributed reader. It does a
local-first pass in its snapshot phase, so the all-shards-local case costs
what it did before, and no store guard is held across the await (the reader
takes its own; RwLockReadGuard is !Send).

Two smaller corrections fall out of the new return type:

  - the reader reports both "needle not in the index" and "volume vanished
    between the has_ec check and the snapshot" as Ok(None), which collapses
    the old Some(Ok(None)) and None arms into one 404;
  - an io::ErrorKind::NotFound now answers 404 rather than 500, matching the
    GET path. Telling a caller to retry a delete that can never succeed was
    half the bug.

The cookie check and its ordering before the journal append are unchanged.

Not addressed here: Rust journals the tombstone locally while Go routes it to
the primary shard holder. That is a separate behaviour change and belongs in
its own PR against the same issue-10 checkbox.

The regression test mounts 13 of 14 shards, leaving out the one holding the
needle's interval. The distributed reader seeds its Reed-Solomon buffers from
locally mounted siblings, so with >= 10 survivors it reconstructs with no peer
fan-out -- which makes the bug reproducible on a single node. Against the
unfixed handler the test fails with 500 vs 202.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* volume server: fail the delete when the EC volume unmounts mid-request

find_ec_volume_mut returning None used to fall through to a 202 with no
.ecj tombstone written, reporting success for a delete that did not
happen. Answer 404 like the other volume-vanished arms so the caller can
retry after a remount.

* volume server: forward EC needle deletes to a primary-shard holder

Mirror Go's doDeleteNeedleFromAtLeastOneRemoteEcShards: the tombstone is
journaled on one holder of the needle's primary data shard via
VolumeEcBlobDelete (or the local journal when this server holds the
shard), falling back to any other shard holder when the primary has
none. Journaling only on the node that received the DELETE scattered
tombstones across whichever server took the request.

* volume server: route BatchDelete EC deletes through the same forwarding

BatchDelete had the same local-journal divergence as HTTP DELETE, plus a
gap the old code admitted in a comment: the .ecx index cannot supply the
needle's cookie, so EC deletes ran with no cookie check at all. A
distributed read now fills the needle for every EC entry — matching Go's
DeleteEcShardNeedle, which reads and compares the fid cookie even when
skip_cookie_check is set — and the tombstone forwards via
delete_ec_shard_needle_distributed. A needle deleted between read and
journal reports 304 like Go's ErrorDeleted; a vanished volume reports
500 so the filer retries.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: chrislusf <chrislusf@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>

2026-09-20 23:26:31 -07:00

..

admin_auth_coverage.rs

rust: cargo fmt both crates, with a commented-out fmt --check CI step (#11329)

2026-09-15 09:29:22 -07:00

http_integration.rs

volume server: HTTP DELETE on a distributed EC volume (#11405)

2026-09-20 23:26:31 -07:00