Fixes#1165
The signed chunk encoding with trailers should return api error for:
1. Invalid checksum - `(InvalidRequest) Value for x-amz-checksum-x trailing header is invalid.`
2. Incorrect checksum - `(BadDigest) The x you specified did not match the calculated checksum.`
Where `x` could be crc32, crc32c, sha1 ...
Closes#1159Fixes#1161
Implements signed chunk encoding with trailers in the gateway.
The signed encoding (both with and without trailers) is now handled by the `ChunkReader`.
Fixes the `ChunkReader` implementation to validate encoding headers byte by byte.
The chunk encoding with trailers follows the general signed chunk encoding pattern, but the final chunk includes the trailing signature (`x-amz-trailing-signature`) and the checksum header (`x-amz-checksum-x`, where `x` can be `crc32`, `crc32c`, `sha1`, `sha256`, or `crc64nvme`).
Adds validation for the `X-Amz-Trailer` header.
Fixes#1147
The final chunk header with 0 length, contains the last signature in signed chunk encoding implementation.
Added this last signature verification in the signed chunk encoding without trailers.
We were getting errors such as:
2025/02/07 19:24:28 Internal Error, write object data: write exceeds content length 87
whenever the chunk encoding did not have the correct chunk
signatures. The issue was that the chunk encoding reader
was reading from the underlying reader and then passing the full
buffer read back to the caller if the underlying reader returned
an error. This meant that we were not processing the chunk
headers within the buffer due to the higher level error, and
would possibly hand back the longer unprocessed chunk encoded
stream to the caller that was in turn trying to write to the
object file exceeding the content length limit.
Fixes#1056
An invalid chunk encoding, or parse errors leading to parsing
invalid data can lead to a server panic if the chunk header
remaining is determined to be larger than the max buffer size.
This was previously seen when the chunk trailer checksums were
used by the client without the support from the server side
for this encoding. Example panic:
panic: runtime error: slice bounds out of range [4088:1024]
goroutine 5 [running]:
github.com/versity/versitygw/s3api/utils.(*ChunkReader).parseChunkHeaderBytes(0xc0003c4280, {0xc0000e6000?, 0x3000?, 0x423525?})
/home/tester/s3api/utils/chunk-reader.go:242 +0x492
github.com/versity/versitygw/s3api/utils.(*ChunkReader).parseAndRemoveChunkInfo(0xc0003c4280, {0xc0000e6000, 0x3000, 0x8000})
/home/tester/s3api/utils/chunk-reader.go:170 +0x20b
github.com/versity/versitygw/s3api/utils.(*ChunkReader).Read(0xc0003c4280, {0xc0000e6000, 0xc0000b41e0?, 0x8000})
/home/tester/s3api/utils/chunk-reader.go:91 +0x11e
This fix will validate the data length before copying into the
temporary buffer to prevent a panic and instead just return
an error.
