chrislu 0cdcb88081 feat: Identify root cause - data loss during buffer flush ...

Phase 6: Root Cause Discovered - NOT Disk Read Bug

After comprehensive debugging with server-side logging:

What We Found:
  ✅ Disk read works correctly (reads what exists on disk)
  ✅ Cache works correctly (caches what was read)
  ✅ Extraction works correctly (returns what's cached)
  ❌ DATA IS MISSING from both disk and memory!

The Evidence:
  Request offset: 1764
  Disk has: 1000-1763 (764 messages)
  Memory starts at: 1800
  Gap: 1764-1799 (36 messages) ← LOST!

Root Cause:
  Buffer flush logic creates GAPS in offset sequence
  Messages are lost when flushing from memory to disk
  bufferStartOffset jumps (1763 → 1800) instead of incrementing

Changes:
  - log_read_stateless.go: Simplified cache extraction to return empty for gaps
  - Removed complex invalidation/retry (data genuinely doesn't exist)

Test Results:
  Original: 87.9% delivery
  Cache invalidation attempt: 73.3% (cache thrashing)
  Gap handling: 82.1% (confirms data is missing)

Next: Fix buffer flush logic in log_buffer.go to prevent offset gaps

2025-10-17 12:42:43 -07:00

..

disk_buffer_cache.go

Add Kafka Gateway (#7231)

2025-10-13 18:05:17 -07:00

log_buffer_queryability_test.go

Add Kafka Gateway (#7231)

2025-10-13 18:05:17 -07:00

log_buffer_test.go

Add Kafka Gateway (#7231)

2025-10-13 18:05:17 -07:00

log_buffer.go

remove _schemas debug

2025-10-15 22:51:42 -07:00

log_read_integration_test.go

fix tests

2025-10-15 19:00:10 -07:00

log_read_stateless_test.go

log read stateless

2025-10-15 16:08:21 -07:00

log_read_stateless.go

feat: Identify root cause - data loss during buffer flush

2025-10-17 12:42:43 -07:00

log_read_test.go

Add Kafka Gateway (#7231)

2025-10-13 18:05:17 -07:00

log_read.go

Add Kafka Gateway (#7231)

2025-10-13 18:05:17 -07:00

sealed_buffer.go

Add Kafka Gateway (#7231)

2025-10-13 18:05:17 -07:00