  pingqiuandClaude Opus 4.6
|
ebe95b6e2e
|
fix: flusher OOM on multi-block writes + testrunner enhancements
Bug: flusher.go:336 allocated make([]byte, entryLen) per dirty block
instead of per unique WAL entry. A 4MB WriteLBA creates 1024 dirty map
entries (one per 4KB block), all sharing the same WAL offset. The flusher
read the full 4MB WAL entry 1024 times into separate buffers:
1024 × 4MB = 4GB per 4MB write → OOM on mkfs.ext4.
Root cause: flusher assumed 1:1 dirty-block-to-WAL-entry mapping.
WriteLBA supports multi-block writes but the flusher never deduplicated
shared WAL offsets.
Fix: deduplicate WAL reads by WalOffset in flushOnceLocked(). Multiple
dirty blocks from the same WAL entry share one read buffer and one
DecodeWALEntry call. Memory: O(WAL_entries × size) not O(blocks × size).
For a 4MB write: 4GB → 4MB.
Verified on hardware (m01/M02 25Gbps RoCE):
- Before: mkfs.ext4 → VS RSS 100MB→25GB → OOM killed
- After: mkfs.ext4 → VS RSS 129MB stable, mkfs succeeds
- pgbench TPC-B c=4: 1,248 TPS (RF=1, previously blocked by OOM)
Tests added:
- flusher_test.go: flush_multiblock_shared_wal_read (16 blocks share
one WAL offset, flush dedup verified)
- flusher_test.go: flush_multiblock_data_correct (3 mixed multi-block
writes, all data correct after flush)
- test/component/large_write_test.go: 7 component tests (single 4MB,
sequential mkfs sim, concurrent, mixed sizes, production volume,
flusher throughput 30s sustained)
- iscsi/large_write_mem_test.go: 2 iSCSI session memory tests (4MB
R2T flow, slow device)
Testrunner enhancements (same commit — all tested on hardware):
- discover_primary action: maps primary IP → topology node name,
supports alt_ips for multi-NIC (RoCE + management)
- NodeSpec.AltIPs field for multi-NIC node identification
- 5 new YAML scenarios: ec3, ec5, degraded sync_all/best_effort, pgbench
- All 13 hardware-verified scenarios PASS
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-04-02 14:24:10 -07:00 |
|
|
pingqiuandClaude Opus 4.6
|
785a7d7efd
|
feat: wire real pinner into flusher retention + real WAL scan executor (Phase 07 P1)
Pinner wired to real retention:
- NewPinner calls vol.SetV2RetentionFloor(p.MinWALRetentionFloor)
- Flusher.RetentionFloorFn() / SetRetentionFloorFn() exposed
- SetV2RetentionFloor chains with existing shipper retention floor
- Holds actually prevent WAL reclaim (not just tracked state)
Executor uses real WAL scan:
- BlockVol.ScanWALEntries(fromLSN, callback) wraps wal.ScanFrom
with real fd, walOffset, checkpointLSN
- Executor.StreamWALEntries uses ScanWALEntries (not stub)
- Reads real WAL entries, tracks highest LSN scanned
CommittedLSN mapping:
- Explicitly documented as interim V1 model (committed = checkpointed)
- Will diverge when V2 distributed commit separates from local flush
Carry-forward:
- TransferSnapshot/TransferFullBase/TruncateWAL: stubs (need extent I/O)
- Control intent from confirmed failover: deferred
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-03-30 20:01:46 -07:00 |
|