Files
scylladb/utils
Pavel Emelyanov 57af69e15f Merge 'Add retries to the S3 client' from Ernest Zaslavsky
1. Add `retry_strategy` interface and default implementation for exponential back-off retry strategy.
2. Add new S3 related errors, also introduce additional errors to describe pure http errors that has no additional information in the body.
3. Add retries to the s3 client, all retries are coordinated by an instance of `retry_strategy`. In a case of error also parse response body in attempt to retrieve additional and more focused error information as suggested by AWS. See https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html. Introduce `aws_exception` to carry the original `aws_error`.
4. Discard whatever exception is thrown in `abort_upload` when aborting multipart upload since we don't care about cleanly aborting it since there are other means to clean up dangling parts, for example `rclone cleanup` or S3 bucket's Lifecycle Management Policy.
5. Add tests to cover retries, and retry exhaustion. Also add tests for jumbo upload.
6. Add the S3 proxy which is used to randomly inject retryable S3 errors to test the "retry" part of the S3 client. Switch the `s3_test` to use the S3 proxy. `s3_tests` set afloat `put_object` problem that was causing segmentation when retrying, fixed.
7. Extend the `s3_test` to use both `minio` and `proxy` configurations.
8. Add parameter to the proxy to seed the error injection randomization to make it replayable.

fixes: #20611
fixes: #20613

Closes scylladb/scylladb#21054

* github.com:scylladb/scylladb:
  aws_errors: Make error messages more verbose.
  test: Make the minio proxy randomization re-playable
  test/boost/s3_test: add error injection scenarios to existing test suite
  test: Switch `s3_test` to use proxy
  test: Add more tests
  client: Stop returning error on `DELETE` in multipart upload abortion
  client: Fix sigsegv when retrying
  client: Add retries
  client: Adjust `map_s3_client_exception` to return exception instance
  aws_errors: Change aws_error::parse to return std::optional<>
  aws_errors: Add http errors mapping into aws_error
  client: Add aws_exception mapping
  aws_error: Add `aws_exeption` to carry original `aws_error`
  aws_errors: Add new error codes
  client: Introduce retry strategy
2024-11-11 08:35:55 +03:00
..
2023-09-11 14:38:46 +03:00
2023-12-02 22:37:22 +02:00
2024-01-18 12:50:06 +02:00
2024-02-01 14:50:17 +02:00
2024-05-27 17:34:38 +03:00
2024-10-22 06:54:46 +03:00
2023-12-02 22:37:22 +02:00
2023-12-02 22:37:22 +02:00
2024-04-30 09:07:22 +03:00