mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-08-21 14:46:58 +00:00
chrislu
97be7c6aee
INSIGHT: User correctly pointed out: 'kafka gateway should just use the SMQ async offset committing' - we shouldn't manually create goroutines to wrap SMQ. REVISED APPROACH: 1. **In-memory commit** is the primary source of truth - Immediate response to client - Consumers rely on this for offset tracking - Fast < 1ms operation 2. **SMQ persistence** is best-effort for durability - Used for crash recovery when in-memory lost - Sync call (no manual goroutine wrapping) - If it fails, not fatal - in-memory is current state DESIGN: - In-memory: Authoritative, always succeeds (or client sees error) - SMQ storage: Durable, failure is logged but non-fatal - Auto-commit: Periodically pushes offsets to SMQ - Manual commit: Explicit confirmation of offset progress This matches Kafka semantics where: - Broker always knows current offsets in-memory - Persistent storage is for recovery scenarios - No artificial blocking on persistence EXPECTED BEHAVIOR: - Fast offset response (unblocked by SMQ writes) - Durable offset storage (via SMQ periodic persistence) - Correct offset recovery on restarts - No message loss or duplicates when offsets committed