mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-20 15:02:27 +00:00
Chris Lu
b763a5f6bf
* s3: add streaming reader interface for remote storage Add RemoteStorageStreamReader optional interface to support efficient streaming of large remote objects without buffering entire file in memory. This enables future stream-through caching where data can be served to clients while simultaneously writing to volume servers. Implement ReadFileAsStream() for S3, GCS, and Azure backends using their native streaming APIs. This provides the foundation for improving TTFB on large remote file access by serving data directly from remote storage while background cache operation populates local chunks. The streaming interface allows remote storage backends to return io.ReadCloser, enabling efficient memory usage for multi-GB objects compared to the current ReadFile() approach which buffers entire ranges in memory. * s3: adaptive timeout for remote object caching to improve TTFB Use size-aware cache polling timeout to balance cache-hit rate against time-to-first-byte: - Small files (<50MB): 10s timeout - more likely to complete caching before timeout, improving subsequent request performance - Medium files (50-500MB): 5s timeout - default balance - Large files (>500MB): 2s timeout - fail-fast to improve initial TTFB for very large downloads This reduces waiting time for large remote files while maintaining high cache-hit rate for smaller files that cache quickly. * s3: address code review feedback for stream-through cache - Move startBackgroundRemoteCache call after policy recheck to avoid cache side effects for denied requests (authorization first) - Make startBackgroundRemoteCache version-aware by accepting versionId parameter and using buildVersionedRemoteObjectPath - Add timeout (5 minutes) to background cache context to prevent goroutine pile-up if RPC stalls under load - Update cacheRemoteObjectForStreamingWithShortTimeout to return both entry and error, allowing callers to distinguish transient errors (timeout/cancellation) from permanent errors (not found, denied) - Update streamFromVolumeServers to handle permanent cache errors with appropriate HTTP status codes (404 for not found, 503 for transient)