* util: retry transient errors, not just the ones containing "transport"
util.Retry only retried when the error string contained "transport", so a
plain "read: connection reset by peer" from S3 got zero retries. Classify
the error instead: net timeouts, connection resets, and the throttling and
overload replies S3 and gRPC return are all worth another attempt, while a
cancelled or expired context is not.
* filer sync: hold the sync offset behind a failed event
A sync job that returned an error was logged and forgotten, and the
watermark advanced past it anyway. The offset is the durable resume point,
so the event was never replayed: for filer.remote.sync that left the file
present locally, absent on the remote, with no RemoteEntry and nothing to
retry it.
Pin the watermark at the oldest failed event. Later events keep flowing,
but the persisted offset stays behind the failure, so a restart replays it.
Chris LuandGitHub
652273301e