rust volume: repair an unindexed .dat tail at load (#11878)

* rust volume: repair an unindexed .dat tail at load

Mirror Go's repairUnindexedDatTail (#11870): a crash between the .dat\nappend and the .idx append leaves a .dat tail no index entry covers,\nwhich sent the volume read-only on the next load and left acknowledged\nneedles unreadable. Before the index loads, re-index the complete\nneedles in the tail (a size 0 record as the tombstone or empty write\nresolveEmptyTailRecords reads it as) and cut only an incomplete last\nrecord. A tail an interrupted append does not explain is left as it is\nand the integrity check quarantines the volume read-only as before.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: tests for the unindexed .dat tail repair

Reload volumes through the real load path after dropping .idx entries\nand cutting or damaging the .dat, mirroring Go's\nvolume_checking_tail_repair_test.go: complete unindexed needles are\nre-indexed (deletes as tombstones, empty writes as empty needles), an\nincomplete last record is cut, and a tail an interrupted append does\nnot explain is left untouched with the volume read-only.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: re-index a torn size 0 record as a live empty needle

A delete and a write with no data append the identical size 0 record, so a
recovered one cannot be told apart from the prior index state alone: the
write path accepts an empty overwrite of a live key and indexes it with
size 0, while a delete of the same key publishes a tombstone. Inferring a
tombstone from prior live state turned a lost index row after an empty
overwrite into a deletion -- the object read back not-found although the
write was answered.

Every re-indexed record is now indexed at its on-disk size, the entry
publishWriteRequest writes for an empty write. The other misread is
bounded: a recovered delete reads its key back as an empty object rather
than not-found, and the size 0 record carries no data to resurrect.

Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: re-index a torn size 0 record as a live empty needle

Mirrors the Go fix in this pull request. A delete and a write with no data
append the identical size 0 record, so resolve_empty_tail_records could
not tell them apart from the prior index state: it treated a zero-size
record of a live id as a tombstone, although the write path indexes the
same bytes as a live size 0 entry. A crash after an empty overwrite of a
live key therefore recovered the object as deleted.

Every re-indexed record is now indexed at its on-disk size, the entry
publish_write writes for an empty write. A recovered delete reads its key
back as an empty object rather than not-found; the size 0 record carries
no data to resurrect.

Review: greptile-apps[bot] and devin-ai-integration[bot] on #11878.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: repair a .dat tail left past an empty .idx

The writable index loader creates the .idx before the first needle append,
so a crash after that append leaves a volume whose data nothing indexes.
CheckVolumeDataIntegrity skipped an empty index outright, so the volume
loaded writable and the surviving first needle stayed invisible while
later writes appended past it.

A zero-size index now runs the same repair with tailEntryPos -1: the walk
starts at the first record past the superblock with no anchor needle and
no append timestamp to compare against, gets the same complete-record
validation and incomplete-tail policy, and re-verifies through the normal
tail check once it wrote entries. A tail the repair cannot explain fails
the check, so the volume loads read-only instead of writable on an
unaccounted-for .dat.

Review feedback on the Rust mirror (#11878): the same bug exists here;
fixing both sides identically.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: repair a .dat tail left past an empty .idx

Mirrors the Go fix in this pull request. The writable index loader creates
the .idx before the first needle append, so a crash after that append
leaves a volume whose data nothing indexes. Both the repair and the
integrity check skipped an empty index, so the volume loaded writable and
the surviving first needle stayed invisible while later writes appended
past it.

An empty .idx now runs the same repair with no anchor: the walk starts at
the first record past the superblock, and a tail it cannot explain fails
the load like any other repair failure, since every later check skips an
empty index.

Review: devin-ai-integration[bot] on #11878.

Generated with [Devin](https://devin.ai)\n\nCo-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* volume: sync the .dat before publishing re-indexed .idx rows

A power loss after the repaired .idx rows become durable but before the
.dat tail records they name are flushed leaves the index pointing past
the end of the data file; the next load then keeps the volume read-only
until an operator steps in. Sync the .dat first so the rows never
outlive the records they reference.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* rust volume: sync the .dat before publishing re-indexed .idx rows

Mirrors the Go fix: a power loss after the repaired .idx rows become
durable but before the .dat tail records they name are flushed leaves
the index pointing past the end of the data file. Sync the .dat first.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>

seaweed-volume/src/storage/mod.rs

@@ -18,3 +18,4 @@ pub mod volume_idx_repair;
 pub mod volume_open;
 pub mod volume_report;
 pub mod volume_report_hash;
+pub mod volume_tail_repair;

seaweed-volume/src/storage/volume.rs

@@ -29,6 +29,7 @@ use crate::storage::needle_map::{CompactNeedleMap, NeedleMap, NeedleMapKind, Red
 use crate::storage::super_block::{ReplicaPlacement, SUPER_BLOCK_SIZE, SuperBlock};
 use crate::storage::types::*;
 use crate::storage::volume_open::open_volume_file;
+use crate::storage::volume_tail_repair::TailRepairError;
 
 // ============================================================================
 // Errors
@@ -1724,6 +1725,46 @@ impl Volume {
                 ),
             }
 
+            // A crash between the .dat append and the .idx append leaves a
+            // .dat tail no index entry covers. Re-index complete needles and
+            // cut an incomplete remainder before the index loads, so the
+            // recovered rows land in the needle map and inside its append
+            // offset. Mirrors Go running repairUnindexedDatTail inside
+            // CheckVolumeDataIntegrity ahead of the needle map load. Only a
+            // volume about to load writable is repaired.
+            if !self.has_remote_file() && self.no_write_flags() == (false, false) {
+                // An empty .idx gets the same repair: every check below
+                // skips an empty index on its own, so a tail the repair
+                // cannot explain there fails the load like in Go rather
+                // than warn and load writable.
+                let idx_empty = fs::metadata(self.file_name(".idx"))
+                    .map(|m| m.len() == 0)
+                    .unwrap_or(false);
+                match self.repair_unindexed_dat_tail(io) {
+                    Ok(()) => {}
+                    Err(e) if e.is_not_torn() && !idx_empty => warn!(
+                        volume_id = self.id.0,
+                        error = %e,
+                        "unindexed .dat tail not repaired"
+                    ),
+                    // The index refusing a read-write open is the same failure
+                    // Go reports opening it ahead of the check: it fails the
+                    // load rather than only the repair.
+                    Err(TailRepairError::OpenIndex(e)) => return Err(e),
+                    Err(e) => {
+                        // A repair that could not complete must not load the
+                        // volume writable on files it half-fixed: the check
+                        // fails as in Go, so the volume loads read-only.
+                        self.set_no_write_flag(NO_WRITE_OR_DELETE, true);
+                        warn!(
+                            volume_id = self.id.0,
+                            error = %e,
+                            "repair unindexed .dat tail"
+                        );
+                    }
+                }
+            }
+
             self.load_index(io)?;
 
             // Match Go: CheckVolumeDataIntegrity after loading index (volume_loading.go L154-159)
@@ -2082,7 +2123,7 @@ impl Volume {
         self.current_dat_file_size_io(&io)
     }
 
-    fn current_dat_file_size_io(&self, io: &VolumeIo) -> io::Result<u64> {
+    pub(crate) fn current_dat_file_size_io(&self, io: &VolumeIo) -> io::Result<u64> {
         if let Some(ref f) = io.dat_file {
             Ok(f.metadata()?.len())
         } else if let Some(ref remote_dat_file) = io.remote_dat_file {
@@ -2306,7 +2347,7 @@ impl Volume {
         self.read_needle_blob_unlocked(&io, offset, size)
     }
 
-    fn read_needle_blob_unlocked(
+    pub(crate) fn read_needle_blob_unlocked(
         &self,
         io: &VolumeIo,
         offset: i64,
@@ -2726,7 +2767,7 @@ impl Volume {
     /// Flush the .dat, the first half of a durable write. The .idx is flushed
     /// separately by flush_idx once the row is published; the two are split so
     /// nothing is indexed before the bytes it points at are down.
-    fn flush_dat(&self, io: &VolumeIo) -> io::Result<()> {
+    pub(crate) fn flush_dat(&self, io: &VolumeIo) -> io::Result<()> {
         #[cfg(test)]
         self.dat_syncs_for_test.fetch_add(1, Ordering::Relaxed);
         #[cfg(test)]
@@ -2778,7 +2819,7 @@ impl Volume {
     }
 
     /// Take an unflushed append back off the .dat after its sync failed.
-    fn truncate_dat(&self, io: &VolumeIo, len: u64) -> io::Result<()> {
+    pub(crate) fn truncate_dat(&self, io: &VolumeIo, len: u64) -> io::Result<()> {
         #[cfg(test)]
         if io.fail_truncate_for_test {
             return Err(io::Error::other("injected truncate failure"));
@@ -3512,7 +3553,7 @@ impl Volume {
     /// key-sorted .idx (e.g. legacy weed fix output) falls back to a single
     /// linear scan for the maximum offset. Returns -1 when no entry has a
     /// non-zero offset (a pre-allocated / all-zero index).
-    fn find_dat_tail_entry_offset(
+    pub(crate) fn find_dat_tail_entry_offset(
         &self,
         io: &VolumeIo,
         idx_file: &mut File,

weed/storage/volume_checking.go

@@ -145,7 +145,25 @@ func CheckVolumeDataIntegrity(v *Volume, indexFile *os.File) (lastAppendAtNs uin
 		return 0, fmt.Errorf("verifyIndexFileIntegrity %s failed: %v", indexFile.Name(), err)
 	}
 	if indexSize == 0 {
-		return 0, nil
+		if v.noWriteOrDelete || v.noWriteCanDelete {
+			return 0, nil
+		}
+		// An empty index can hide writes a crash stranded in the .dat: the
+		// writable index loader creates the .idx before the first needle
+		// append, so a crash after that append leaves a volume whose data
+		// nothing indexes — and every check below skips an empty index. The
+		// same repair runs from the first record past the superblock; a tail
+		// it cannot explain fails the check, since an unaccounted-for .dat
+		// must not load writable either.
+		if _, _, err = repairUnindexedDatTail(v, indexFile, 0, -1); err != nil {
+			return 0, fmt.Errorf("CheckVolumeDataIntegrity %s: repair unindexed .dat tail: %v", indexFile.Name(), err)
+		}
+		if indexSize, err = verifyIndexFileIntegrity(indexFile); err != nil {
+			return 0, fmt.Errorf("verifyIndexFileIntegrity %s after re-indexing failed: %v", indexFile.Name(), err)
+		}
+		if indexSize == 0 {
+			return 0, nil
+		}
 	}
 	// The deeper-than-tail structural check (every (offset + actual size)
 	// fits inside .dat) lives in volume.load(): it reads
@@ -258,14 +276,25 @@ var errTailNotTorn = errors.New("not a torn tail")
 // A write appends the needle to the .dat, then its entry to the .idx, then
 // answers the client. A process killed between the first two steps leaves a
 // needle, or part of one, past the end of the needle at the highest indexed
-// offset. After a power loss the order on disk is whatever the page cache
+// offset; a crash after the very first append leaves it past an index with
+// no entry at all, which tailEntryPos < 0 marks: the walk then starts at the
+// first record past the superblock with no anchor needle and no append
+// timestamp to compare against. After a power loss the order on disk is
+// whatever the page cache
 // flushed: a complete needle can survive whose .idx entry did not, although
 // the client was answered. So a complete needle is never dropped: every
 // complete needle in the tail (its header, body, checksum, timestamp and
 // padding inside the file, its CRC valid, its append timestamp later than the
-// needle before it) gets the .idx entry the volume server would have written
-// (a size 0 record as resolveEmptyTailRecords reads it), in .dat order, and
-// the .idx is synced. Only what follows the last complete
+// needle before it) gets the .idx entry the volume server would have written,
+// in .dat order; the .dat is synced first so the rows never become durable
+// ahead of the records they name, and the .idx is synced after. A size 0
+// record is indexed at size 0,
+// the same live entry the write path publishes for a write with no data. A
+// delete appends the identical record, so a recovered size 0 record can also
+// be a delete whose key then reads back as an empty object rather than
+// not-found -- the two cannot be told apart, and resurrecting an empty
+// placeholder beats a wrong tombstone, which drops an acknowledged write's
+// object entirely. Only what follows the last complete
 // needle and is itself incomplete is cut: fewer bytes than a header, a record
 // running past the end of the file, or a last record whose bytes are damaged
 // (its CRC fails or its body does not parse); the .dat is synced after the
@@ -293,18 +322,45 @@ func repairUnindexedDatTail(v *Volume, indexFile *os.File, indexSize int64, tail
 		// does not name its needle's position on its own
 		return 0, 0, nil
 	}
-	entry, err := readIndexEntryAtOffset(indexFile, tailEntryPos)
-	if err != nil {
-		return 0, 0, fmt.Errorf("read index entry at %d: %v", tailEntryPos, err)
+	var start int64
+	var lastAppendAtNs uint64
+	if tailEntryPos < 0 {
+		// The index holds no row at all, so no entry can anchor the walk:
+		// it starts at the first record past the superblock.
+		start = int64(v.SuperBlock.BlockSize())
+	} else {
+		entry, err := readIndexEntryAtOffset(indexFile, tailEntryPos)
+		if err != nil {
+			return 0, 0, fmt.Errorf("read index entry at %d: %v", tailEntryPos, err)
+		}
+		key, offset, size := idx.IdxFileEntry(entry)
+		if offset.IsZero() {
+			return 0, 0, nil
+		}
+		start = needleDiskEnd(offset, size, version)
+		if start >= datSize {
+			// the .dat ends at the last needle (clean), or the index references
+			// bytes the .dat does not have: not a torn tail, not ours to fix
+			return 0, 0, nil
+		}
+
+		// Anchor: the needle at the highest indexed offset must be the one its
+		// entry describes, whole and with a valid CRC.
+		onDiskSize := size
+		if size.IsDeleted() {
+			onDiskSize = 0
+		}
+		anchor, err := readTailNeedle(v.DataBackend, version, offset.ToActualOffset(), onDiskSize)
+		if err != nil {
+			return 0, 0, fmt.Errorf("last indexed needle %v at %d: %v: %w", key, offset.ToActualOffset(), err, errTailNotTorn)
+		}
+		if anchor.Id != key {
+			return 0, 0, fmt.Errorf("needle at %d is id %v, its index entry says id %v: %w", offset.ToActualOffset(), anchor.Id, key, errTailNotTorn)
+		}
+		lastAppendAtNs = anchor.AppendAtNs
 	}
-	key, offset, size := idx.IdxFileEntry(entry)
-	if offset.IsZero() {
-		return 0, 0, nil
-	}
-	start := needleDiskEnd(offset, size, version)
 	if start >= datSize {
-		// the .dat ends at the last needle (clean), or the index references
-		// bytes the .dat does not have: not a torn tail, not ours to fix
+		// no record past the superblock: nothing to index
 		return 0, 0, nil
 	}
 	// The backend rounds its size up to the needle alignment; the walk needs
@@ -314,28 +370,17 @@ func repairUnindexedDatTail(v *Volume, indexFile *os.File, indexSize int64, tail
 		fileSize = st.Size()
 	}
 
-	// Anchor: the needle at the highest indexed offset must be the one its
-	// entry describes, whole and with a valid CRC.
-	onDiskSize := size
-	if size.IsDeleted() {
-		onDiskSize = 0
-	}
-	anchor, err := readTailNeedle(v.DataBackend, version, offset.ToActualOffset(), onDiskSize)
-	if err != nil {
-		return 0, 0, fmt.Errorf("last indexed needle %v at %d: %v: %w", key, offset.ToActualOffset(), err, errTailNotTorn)
-	}
-	if anchor.Id != key {
-		return 0, 0, fmt.Errorf("needle at %d is id %v, its index entry says id %v: %w", offset.ToActualOffset(), anchor.Id, key, errTailNotTorn)
-	}
-
-	entries, cutAt, err := walkUnindexedDatTail(v.DataBackend, version, start, fileSize, anchor.AppendAtNs)
+	entries, cutAt, err := walkUnindexedDatTail(v.DataBackend, version, start, fileSize, lastAppendAtNs)
 	if err != nil {
 		return 0, 0, err
 	}
 	reindexed := len(entries) / types.NeedleMapEntrySize
 	if reindexed > 0 {
-		if err := resolveEmptyTailRecords(indexFile, entries); err != nil {
-			return 0, 0, err
+		// The new .idx rows must not become durable before the .dat records
+		// they name: a power loss between the two would leave the .idx
+		// pointing past the end of the .dat.
+		if err := v.DataBackend.Sync(); err != nil {
+			return 0, 0, fmt.Errorf("sync %s before re-indexing: %v", v.FileName(".dat"), err)
 		}
 		if err := appendIndexEntries(indexFile, entries, indexSize); err != nil {
 			return 0, 0, err
@@ -400,48 +445,6 @@ func readTailNeedle(datFile backend.BackendStorageFile, version needle.Version,
 	return n, nil
 }
 
-// resolveEmptyTailRecords turns the size 0 entries among the re-indexed ones
-// into what the volume server indexed them as. A delete and a write with no
-// data both append a size 0 record; a delete is indexed as a tombstone, an
-// empty write with size 0. A delete is only appended for a needle that is
-// live, so a size 0 record of a needle that is not live (absent or deleted in
-// the .idx and in the tail before it) is an empty write and keeps size 0. One
-// of a live needle is indexed as a tombstone, as weed fix and the backup's
-// index regeneration read a size 0 record.
-func resolveEmptyTailRecords(indexFile io.ReaderAt, entries []byte) error {
-	live := make(map[types.NeedleId]bool)
-	for i := 0; i < len(entries); i += types.NeedleMapEntrySize {
-		if key, _, size := idx.IdxFileEntry(entries[i : i+types.NeedleMapEntrySize]); size == 0 {
-			live[key] = false
-		}
-	}
-	if len(live) == 0 {
-		return nil
-	}
-	if err := idx.WalkIndexFile(indexFile, 0, func(key types.NeedleId, offset types.Offset, size types.Size) error {
-		if _, ok := live[key]; ok {
-			live[key] = !offset.IsZero() && !size.IsDeleted()
-		}
-		return nil
-	}); err != nil {
-		return fmt.Errorf("scan index: %v", err)
-	}
-	for i := 0; i < len(entries); i += types.NeedleMapEntrySize {
-		key, offset, size := idx.IdxFileEntry(entries[i : i+types.NeedleMapEntrySize])
-		wasLive, tracked := live[key]
-		if !tracked {
-			continue
-		}
-		if size == 0 && wasLive {
-			copy(entries[i:], needle_map.ToBytes(key, offset, types.TombstoneFileSize))
-			live[key] = false
-			continue
-		}
-		live[key] = true
-	}
-	return nil
-}
-
 // walkUnindexedDatTail walks the needles from start to fileSize and returns
 // the .idx entries of the complete ones, and the offset from which the rest is
 // an incomplete last record (fileSize when there is none). It fails with
@@ -480,8 +483,9 @@ func walkUnindexedDatTail(datFile backend.BackendStorageFile, version needle.Ver
 			return nil, 0, fmt.Errorf("record %v at %d was appended at %d, not after %d: %w", n.Id, at, n.AppendAtNs, lastAppendAtNs, errTailNotTorn)
 		}
 		lastAppendAtNs = n.AppendAtNs
-		// a size 0 record is a delete or an empty write: resolveEmptyTailRecords
-		// tells them apart
+		// A size 0 record is a delete or an empty write; the two append the
+		// identical record, so it is indexed at size 0 the way the write path
+		// indexes an empty write.
 		entries = append(entries, needle_map.ToBytes(n.Id, types.ToOffset(at), header.Size)...)
 		at = end
 	}

weed/storage/volume_checking_tail_repair_test.go

@@ -156,6 +156,20 @@ func assertNotReadable(t *testing.T, v *Volume, id uint64) {
 	}
 }
 
+// A recovered size 0 record indexes as a live empty needle: a delete and a
+// write with no data append the identical record, so the repair cannot tell
+// them apart and indexes the entry the write path would have written.
+func assertEmpty(t *testing.T, v *Volume, id uint64) {
+	t.Helper()
+	n := &needle.Needle{Id: types.Uint64ToNeedleId(id)}
+	if _, err := v.readNeedle(n, nil, nil); err != nil {
+		t.Fatalf("read needle %d: %v", id, err)
+	}
+	if len(n.Data) != 0 {
+		t.Fatalf("needle %d reads back %d bytes, want an empty needle", id, len(n.Data))
+	}
+}
+
 func writes(ids ...uint64) []tornOp {
 	ops := make([]tornOp, len(ids))
 	for i, id := range ids {
@@ -173,6 +187,7 @@ func TestVolumeLoadRepairsUnindexedDatTail(t *testing.T) {
 		wantSize  func(f *tornFixture) int64 // .dat size after the repair
 		reindexed int
 		readable  []uint64
+		empty     []uint64 // readable as an empty live needle
 		absent    []uint64
 	}{
 		{name: "one complete unindexed needle is re-indexed",
@@ -183,10 +198,14 @@ func TestVolumeLoadRepairsUnindexedDatTail(t *testing.T) {
 			ops: writes(1, 2, 3, 4, 5), unindexed: 2,
 			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 2,
 			readable: []uint64{1, 2, 3, 4, 5}},
-		{name: "an unindexed delete is re-indexed as a tombstone",
+		{name: "an unindexed delete is re-indexed as an empty live needle",
 			ops: append(writes(1, 2, 3), tornOp{id: 2, del: true}), unindexed: 1,
 			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 1,
-			readable: []uint64{1, 3}, absent: []uint64{2}},
+			readable: []uint64{1, 3}, empty: []uint64{2}},
+		{name: "an unindexed empty write of a live id stays an empty needle",
+			ops: append(writes(1, 2, 3), tornOp{id: 2, empty: true}), unindexed: 1,
+			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 1,
+			readable: []uint64{1, 3}, empty: []uint64{2}},
 		{name: "a partial header is cut",
 			ops: writes(1, 2, 3, 4), unindexed: 1,
 			damage:   func(t *testing.T, f *tornFixture) { f.truncateDat(t, f.starts[3]+7) },
@@ -223,11 +242,11 @@ func TestVolumeLoadRepairsUnindexedDatTail(t *testing.T) {
 		{name: "an unindexed write and delete of the same id are re-indexed in order",
 			ops: append(writes(1, 2, 3), tornOp{id: 3, del: true}), unindexed: 2,
 			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 2,
-			readable: []uint64{1, 2}, absent: []uint64{3}},
+			readable: []uint64{1, 2}, empty: []uint64{3}},
 		{name: "an unindexed empty write and its delete are re-indexed in order",
 			ops: append(writes(1, 2), tornOp{id: 3, empty: true}, tornOp{id: 3, del: true}), unindexed: 2,
 			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 2,
-			readable: []uint64{1, 2}, absent: []uint64{3}},
+			readable: []uint64{1, 2}, empty: []uint64{3}},
 		{name: "a partial needle after an indexed tombstone is cut",
 			ops: append(append(writes(1, 2, 3), tornOp{id: 2, del: true}), tornOp{id: 4}), unindexed: 1,
 			damage:   func(t *testing.T, f *tornFixture) { f.truncateDat(t, f.midRecord(4)) },
@@ -256,6 +275,9 @@ func TestVolumeLoadRepairsUnindexedDatTail(t *testing.T) {
 			for _, id := range tc.readable {
 				assertReadable(t, v, f.written[id])
 			}
+			for _, id := range tc.empty {
+				assertEmpty(t, v, id)
+			}
 			for _, id := range tc.absent {
 				assertNotReadable(t, v, id)
 			}
@@ -280,6 +302,132 @@ func TestVolumeLoadRepairsUnindexedDatTail(t *testing.T) {
 	}
 }
 
+// An empty index can hide writes a crash stranded in the .dat: the writable
+// index loader creates the .idx before the first needle append, so a crash
+// after that append leaves a volume whose data nothing indexes. The same
+// repair runs from the first record past the superblock.
+func TestVolumeLoadRepairsTailPastEmptyIndex(t *testing.T) {
+	cases := []struct {
+		name      string
+		ops       []tornOp
+		damage    func(t *testing.T, f *tornFixture)
+		wantSize  func(f *tornFixture) int64 // .dat size after the repair
+		reindexed int
+		readable  []uint64
+		empty     []uint64
+		absent    []uint64
+	}{
+		{name: "the only write is re-indexed",
+			ops:      writes(7),
+			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 1,
+			readable: []uint64{7}},
+		{name: "every write is re-indexed",
+			ops:      writes(1, 2, 3),
+			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 3,
+			readable: []uint64{1, 2, 3}},
+		{name: "an unindexed delete is re-indexed as an empty live needle",
+			ops:      append(writes(1, 2, 3), tornOp{id: 2, del: true}),
+			wantSize: func(f *tornFixture) int64 { return f.end }, reindexed: 4,
+			readable: []uint64{1, 3}, empty: []uint64{2}},
+		{name: "a complete needle is re-indexed and the partial one after it cut",
+			ops:      writes(1, 2),
+			damage:   func(t *testing.T, f *tornFixture) { f.truncateDat(t, f.midRecord(1)) },
+			wantSize: func(f *tornFixture) int64 { return f.starts[1] }, reindexed: 1,
+			readable: []uint64{1}, absent: []uint64{2}},
+		{name: "a last needle with a bad CRC is cut",
+			ops:      writes(1, 2),
+			damage:   func(t *testing.T, f *tornFixture) { f.flipDataByte(t, f.starts[1]) },
+			wantSize: func(f *tornFixture) int64 { return f.starts[1] }, reindexed: 1,
+			readable: []uint64{1}, absent: []uint64{2}},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			f := buildTornFixture(t, tc.ops, len(tc.ops))
+			if tc.damage != nil {
+				tc.damage(t, f)
+			}
+
+			v := newTornTailTestVolume(t, f.dir)
+			defer v.Close()
+			if v.noWriteOrDelete || v.noWriteCanDelete {
+				t.Fatal("volume with an unindexed .dat tail loaded read-only")
+			}
+			if got, want := fileSize(t, f.datName), tc.wantSize(f); got != want {
+				t.Fatalf(".dat is %d bytes after load, want %d", got, want)
+			}
+			if got, want := fileSize(t, f.idxName), int64(tc.reindexed)*types.NeedleMapEntrySize; got != want {
+				t.Fatalf(".idx is %d bytes after load, want %d (%d entries re-indexed)", got, want, tc.reindexed)
+			}
+			for _, id := range tc.readable {
+				assertReadable(t, v, f.written[id])
+			}
+			for _, id := range tc.empty {
+				assertEmpty(t, v, id)
+			}
+			for _, id := range tc.absent {
+				assertNotReadable(t, v, id)
+			}
+			v.Close()
+
+			// the repaired volume reloads clean, and keeps what it indexed
+			v2 := newTornTailTestVolume(t, f.dir)
+			defer v2.Close()
+			if v2.noWriteOrDelete {
+				t.Fatal("repaired volume loaded read-only on the next load")
+			}
+			for _, id := range tc.readable {
+				assertReadable(t, v2, f.written[id])
+			}
+			for _, id := range tc.empty {
+				assertEmpty(t, v2, id)
+			}
+		})
+	}
+}
+
+// An .dat an interrupted append does not explain, over an index with no rows
+// at all, loads the volume read-only: the checks that would quarantine it
+// otherwise all skip an empty index.
+func TestVolumeLoadKeepsUnexplainedTailPastEmptyIndexReadOnly(t *testing.T) {
+	cases := []struct {
+		name   string
+		damage func(t *testing.T, f *tornFixture)
+	}{
+		{name: "zero bytes where the first record would be (id 0)",
+			damage: func(t *testing.T, f *tornFixture) {
+				file, err := os.OpenFile(f.datName, os.O_RDWR|os.O_APPEND, 0644)
+				if err != nil {
+					t.Fatal(err)
+				}
+				defer file.Close()
+				if _, err := file.Write(make([]byte, 64)); err != nil {
+					t.Fatal(err)
+				}
+			}},
+		{name: "a damaged record more data follows",
+			damage: func(t *testing.T, f *tornFixture) { f.flipDataByte(t, f.starts[0]) }},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			f := buildTornFixture(t, writes(1, 2), 2)
+			tc.damage(t, f)
+			datBefore, idxBefore := fileSize(t, f.datName), fileSize(t, f.idxName)
+
+			v := newTornTailTestVolume(t, f.dir)
+			defer v.Close()
+			if !v.noWriteOrDelete {
+				t.Fatal("volume with an unexplained tail past an empty index loaded writable")
+			}
+			if got := fileSize(t, f.datName); got != datBefore {
+				t.Fatalf(".dat changed from %d to %d bytes", datBefore, got)
+			}
+			if got := fileSize(t, f.idxName); got != idxBefore {
+				t.Fatalf(".idx changed from %d to %d bytes", idxBefore, got)
+			}
+		})
+	}
+}
+
 // Every tail an interrupted append does not explain is left exactly as it is,
 // and the volume loads read-only as before.
 func TestVolumeLoadKeepsUnexplainedTailReadOnly(t *testing.T) {