Commit Graph

7 Commits

Author SHA1 Message Date
Calle Wilund
01f4dfed84 utils::gcp::object_storage: Add optional memory limits to up/download
Adds optional memory semaphore to limit the mem buffer usage in sink/source.
Note that we don't bookkeep exact, to avoid deadlock issues in higher layer.

In upload, we overlease on first buffer put to ensure we can at least fill
the desired 8M of buffers. We try to adjust when going over, but if we
fail, we fail, but at least will initiate upload -> soon release memory.
On next put, we try to grab multiples of 8M again, and so forth. Thus
potentially causing waiting for resources, without ending up not uploading
at least one active sink.

For download (source), we try to get lease for as much as we want to read,
but if we fail, we adjust this down to 256k and download anyway. Since this
will typically be released immediately, we at least don't overrun for long,
and again, avoid fully stopping, throttling rate instead.
2025-10-13 08:53:27 +00:00
Calle Wilund
bd1304972c utils::gcp::storage: Add merge objects operation
Allows merging 1-32 smaller files into a destination.
2025-10-13 08:53:27 +00:00
Calle Wilund
fd13ffd95d utils::gcp::object_storage: Make create_download_source return seekable_data_source
Since, given the nature of object storage API:s, it is no more complicated to
provide a reasonable implementation of a seekable, limited, interface,
give this back, which in turn means upper layers can provide easy read-only file
interfaces. Hint hint.
2025-10-13 08:53:26 +00:00
Calle Wilund
74578aaae2 utils::gcp::object_storage: Add timestamp to object listing 2025-10-13 08:53:26 +00:00
Calle Wilund
d0fe001518 utils::gcp::object_storage: Add paging support to list_objects
Allowing listing N entries at a time, with continuation.
2025-10-13 08:53:26 +00:00
Calle Wilund
9dde8806dd utils::gcp::object_storage: Add optional abort_source
Add forwarded abort_source to lengty ops
2025-10-13 08:53:25 +00:00
Calle Wilund
4a5b547a86 utils::gcp::object_storage: Add client for GCP object storage
Adds a minial client for GCP object storage operations:

* Create buckets
* Delete buckets
* List bucket content
* Copy/move bucket content
* Delete bucket content
* Upload bucket content
* Download bucket content
2025-09-01 18:03:44 +00:00