* pb: add multipart concurrency fields to RemoteConf and tier move requests
RemoteConf gains upload_concurrency/download_concurrency (0 = client
default); VolumeTierMoveDatToRemote/FromRemote requests gain a
concurrency field (0 = backend default).
* remote storage: honor RemoteConf upload/download concurrency in s3 and azure clients
s3 client: ReadFile passes conf download_concurrency to the downloader,
WriteFile uses upload_concurrency for the uploader; previously
hard-coded 1 upload / 5 download parts. 0 keeps defaults. Same for
azure client.
* storage: plumb concurrency through backend interface and tier upload/download
BackendStorage.CopyFile/DownloadFile take a concurrency hint (<=0 =
backend configured default); s3 backend reads
upload_concurrency/download_concurrency from scaffold config with
parseConcurrency fallback, rclone updated to the new signature. Tier
move gRPC handlers forward the request concurrency to the backend.
* shell: -upload_concurrency/-download_concurrency for remote.configure, -concurrent for volume.tier
remote.configure exposes upload/download concurrency persisted into
RemoteConf; volume.tier move/evict commands forward -concurrent to the
tier move requests. Documented in master-cloud.toml scaffold.
* test: cover concurrency propagation in remote tier integration test
* remote.configure: merge existing config on partial update
Load the stored RemoteConf before saving so a partial update (e.g. only
-upload_concurrency) preserves credentials, endpoints, and type instead
of replacing them with new-config defaults. Only treat a confirmed
ErrNotFound as a new configuration; propagate all other load errors so a
transient filer failure does not overwrite stored settings.
On a type transition, reset backend-specific fields to the destination
type's new-config defaults rather than inheriting the old backend's
empty values. Bound configured concurrency to a sane maximum.
* remote storage: honor configured download concurrency in S3 and Azure
ReadFileWithConcurrency now resolves a zero request override against the
client's configured download_concurrency (new downloadConcurrency()
helpers), so the remote-mount/cache read path honors
RemoteConf.DownloadConcurrency instead of the hard-coded default.
Azure also clamps the resolved value to math.MaxUint16 regardless of
whether the fallback was used, preventing uint16 wraparound when a
configured value exceeds 65535.
* shell: rename -concurrent to -concurrency and validate tier transfer bounds
Rename the -concurrent flag to -concurrency across volume.tier.upload,
volume.tier.download, and volume.tier.compact to match the proto field and
RemoteConf field names. Add validateTierConcurrency to reject values that
would wrap int32 or exceed a 1024 cap before constructing the request.
* server: clamp tier move concurrency in gRPC handlers
Add clampTierConcurrency to both VolumeTierMoveDatToRemote and
VolumeTierMoveDatFromRemote handlers so a direct gRPC caller cannot spawn
an unbounded number of network workers.
* trim verbose comments added with concurrency feature
Remove redundant doc comments on the backend interface, rclone backend,
s3_backend parseConcurrency, and test helpers that restated the obvious.
* remote.configure: apply type defaults before re-parse so explicit flags win
applyTypeDefaults ran after the second flag parse, overwriting explicit
destination flags (e.g. -s3.region=eu-west-1) with new-config defaults.
Move the type-transition default reset before the re-parse so user-supplied
flags override the destination defaults.
* remote.configure: only treat explicit -type as a type transition
The first parse defaults -type to s3, so a concurrency-only update on an
existing non-S3 config captured requestedType=s3 and wrongly triggered a
type transition, resetting the stored backend to S3. Use fs.Visit to
detect whether -type was explicitly supplied; an omitted -type keeps the
stored backend.
---------
Co-authored-by: Jack Meredith <9480542+jackusm@users.noreply.github.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
* tiering: stop a shared remote object being deleted while replicas still point at it
A remote-tiered volume's .dat content lives only in one cloud object that all
N replica .vif files point at. Deleting that object while destroying any one
replica, or before a downloaded replica is durable, bricks the survivors.
- volume.tier.move cleanup now deletes old replicas with keepRemoteData=true so
surviving replicas keep the shared object. Document why the alreadyPlaced
anchor needs no replica sync (same-object replicas are byte-identical).
- VolumeTierMoveDatFromRemote now fsyncs the downloaded .dat, fsyncs the
containing directory, trims the .vif (fsynced) and swaps to the local DiskFile
BEFORE deleting the remote object, on both the keep-remote and delete paths.
Only the final DeleteFile is gated by keep_remote_dat_file, so a keep-remote
download leaves the replica served from local disk rather than the shared
object, and a crash before delete merely leaks the object.
- volume.tier.download keeps the shared object for every replica except the
last, which deletes it.
- s3 and rclone download paths fsync the .dat before close.
* storage: swap the volume data backend under the data lock
The tier-download swap closed v.DataBackend and assigned the new local DiskFile
without holding dataFileAccessLock, racing concurrent reads/writes (use of a
closed file / nil deref). Add an exported Volume.SwapDataBackend that performs
the close-and-replace under the lock, and call it from the tier download.
* server: skip directory fsync on Windows in the tier download path
os.Open(dir).Sync() is unsupported on Windows and returns an error, which would
fail VolumeTierMoveDatFromRemote entirely there. Skip the directory fsync on
Windows, matching how the storage-side helper tolerates the unsupported case.
* shell: make multi-replica tier.download resilient to already-local replicas
If a multi-replica download is interrupted and retried, a replica made local
in the prior attempt returns "already on local disk", which aborted the whole
command and left the remaining remote replicas dangling. Treat that case as a
skip-and-continue so a retry completes the rest.
* server: assert downloaded .dat content, not just length, in the tier test
A length-only check passes even if the bytes are corrupted; compare the full
content of the local .dat against the original.
* Add Rclone storage backend
* Support templating the name of files stored via Rclone
* Enable Rclone accounting
* Remove redundant type conversion
* Provide progress information for Rclone download/upload operations
* Log error when Rclone can't instantiate filesystem
* Remove filename templating functionality for Rclone storage
To (maybe) be later reintroduced as a generic functionality for all
storage backends.
* Remove S3 specific check
* Move Rclone config initialisation to init() method
Chris LuandGitHub
7y-9andGitHub
