refactor appview to use envvars. move distribution configurations to code

2026-09-19 08:44:14 +00:00 · 2025-10-08 23:29:51 -05:00
parent fb040a72d3
commit b1e6857381
9 changed files with 1042 additions and 116 deletions
@@ -471,12 +471,30 @@ A company wants shared storage for their team:
 3. **In-memory cache** - Hold endpoint cache is in-memory (for production, use Redis)
 4. **Manual profile updates** - No UI for updating sailor profile (must use ATProto client)

+## Performance Optimization: S3 Presigned URLs
+
+**Status:** Planned implementation (see [PRESIGNED_URLS.md](./PRESIGNED_URLS.md))
+
+Currently, hold services act as proxies for blob data. With presigned URLs:
+
+- **Downloads:** Docker → S3 direct (via 307 redirect)
+- **Uploads:** Docker → AppView → S3 (via presigned URL)
+- **Hold service bandwidth:** Reduced by 99.98% (only orchestration)
+
+**Benefits:**
+- Hold services can run on minimal infrastructure ($5/month instances)
+- Direct S3 transfers at maximum speed
+- Scales to arbitrarily large images
+- Works with Storj, MinIO, Backblaze B2, Cloudflare R2
+
+See [PRESIGNED_URLS.md](./PRESIGNED_URLS.md) for complete technical details and implementation guide.
+
 ## Future Improvements

-1. **Automatic failover** - Multiple storage endpoints, fallback to default
-2. **Storage analytics** - Track usage per DID
-3. **Quota integration** - Optional quota tracking in storage service
-4. **Direct presigned URL support** - S3 native presigned URLs (bypass proxy)
+1. **S3 Presigned URLs** - Implement direct S3 URLs (see [PRESIGNED_URLS.md](./PRESIGNED_URLS.md))
+2. **Automatic failover** - Multiple storage endpoints, fallback to default
+3. **Storage analytics** - Track usage per DID
+4. **Quota integration** - Optional quota tracking in storage service
 5. **Profile management UI** - Web interface for users to manage their sailor profile
 6. **Distributed cache** - Redis/Memcached for hold endpoint cache in multi-instance deployments

@@ -0,0 +1,637 @@
+# S3 Presigned URLs Implementation
+
+## Overview
+
+Currently, ATCR's hold service acts as a proxy for all blob data, meaning every byte flows through the hold service when uploading or downloading container images. This document describes the implementation of **S3 presigned URLs** to eliminate this bottleneck, allowing direct data transfer between clients and S3-compatible storage.
+
+### Current Architecture (Proxy Mode)
+
+```
+Downloads:  Docker → AppView → Hold Service → S3 → Hold Service → AppView → Docker
+Uploads:    Docker → AppView → Hold Service → S3
+```
+
+**Problems:**
+- All blob data flows through hold service
+- Hold service bandwidth = total image bandwidth
+- Latency from extra hops
+- Hold service becomes bottleneck for large images
+
+### Target Architecture (Presigned URLs)
+
+```
+Downloads:  Docker → AppView (gets presigned URL) → S3 (direct download)
+Uploads:    Docker → AppView → S3 (via presigned URL)
+Move:       AppView → Hold Service → S3 (server-side CopyObject API)
+```
+
+**Benefits:**
+- ✅ Hold service only orchestrates (no data transfer)
+- ✅ Blob data never touches hold service
+- ✅ Direct S3 uploads/downloads at wire speed
+- ✅ Hold service can run on minimal resources
+- ✅ Works with all S3-compatible services
+
+## How Presigned URLs Work
+
+### For Downloads (GET)
+
+1. **Docker requests blob:** `GET /v2/alice/myapp/blobs/sha256:abc123`
+2. **AppView asks hold service:** `POST /get-presigned-url`
+   ```json
+   {"did": "did:plc:alice123", "digest": "sha256:abc123"}
+   ```
+3. **Hold service generates presigned URL:**
+   ```go
+   req, _ := s3Client.GetObjectRequest(&s3.GetObjectInput{
+       Bucket: "my-bucket",
+       Key:    "blobs/sha256/ab/abc123.../data",
+   })
+   url, _ := req.Presign(15 * time.Minute)
+   // Returns: https://gateway.storjshare.io/bucket/blobs/...?X-Amz-Signature=...
+   ```
+4. **AppView redirects Docker:** `HTTP 307 Location: <presigned-url>`
+5. **Docker downloads directly from S3** using the presigned URL
+
+**Data path:** Docker → S3 (direct)
+**Hold service bandwidth:** ~1KB (API request/response)
+
+### For Uploads (PUT)
+
+**Small blobs (< 5MB) using Put():**
+
+1. **Docker sends blob to AppView:** `PUT /v2/alice/myapp/blobs/uploads/{uuid}`
+2. **AppView asks hold service:** `POST /put-presigned-url`
+   ```json
+   {"did": "did:plc:alice123", "digest": "sha256:abc123", "size": 1024}
+   ```
+3. **Hold service generates presigned URL:**
+   ```go
+   req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{
+       Bucket: "my-bucket",
+       Key:    "blobs/sha256/ab/abc123.../data",
+   })
+   url, _ := req.Presign(15 * time.Minute)
+   ```
+4. **AppView uploads to S3** using presigned URL
+5. **AppView confirms to Docker:** `201 Created`
+
+**Data path:** Docker → AppView → S3 (via presigned URL)
+**Hold service bandwidth:** ~1KB (API request/response)
+
+### For Streaming Uploads (Create/Commit)
+
+**Large blobs (> 5MB) using streaming:**
+
+1. **Docker starts upload:** `POST /v2/alice/myapp/blobs/uploads/`
+2. **AppView creates upload session** with UUID
+3. **AppView gets presigned URL for temp location:**
+   ```json
+   POST /put-presigned-url
+   {"did": "...", "digest": "uploads/temp-{uuid}", "size": 0}
+   ```
+4. **Docker streams data:** `PATCH /v2/alice/myapp/blobs/uploads/{uuid}`
+5. **AppView streams to S3** using presigned URL to `uploads/temp-{uuid}/data`
+6. **Docker finalizes:** `PUT /v2/.../uploads/{uuid}?digest=sha256:abc123`
+7. **AppView requests move:** `POST /move?from=uploads/temp-{uuid}&to=sha256:abc123`
+8. **Hold service executes S3 server-side copy:**
+   ```go
+   s3.CopyObject(&s3.CopyObjectInput{
+       Bucket:     "my-bucket",
+       CopySource: "/my-bucket/uploads/temp-{uuid}/data",
+       Key:        "blobs/sha256/ab/abc123.../data",
+   })
+   s3.DeleteObject(&s3.DeleteObjectInput{
+       Key: "uploads/temp-{uuid}/data",
+   })
+   ```
+
+**Data path:** Docker → AppView → S3 (temp location)
+**Move path:** S3 internal copy (no data transfer!)
+**Hold service bandwidth:** ~2KB (presigned URL + CopyObject API)
+
+## Why the Temp → Final Move is Required
+
+This is **not an ATCR implementation detail** — it's required by the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push).
+
+### The Problem: Unknown Digest
+
+Docker doesn't know the blob's digest until **after** uploading:
+
+1. **Streaming data:** Can't buffer 5GB layer in memory to calculate digest first
+2. **Stdin pipes:** `docker build . | docker push` generates data on-the-fly
+3. **Chunked uploads:** Multiple PATCH requests, digest calculated as data streams
+
+### The Solution: Upload to Temp, Verify, Move
+
+**All OCI registries do this:**
+
+1. Client: `POST /v2/{name}/blobs/uploads/` → Get upload UUID
+2. Client: `PATCH /v2/{name}/blobs/uploads/{uuid}` → Stream data to temp location
+3. Client: `PUT /v2/{name}/blobs/uploads/{uuid}?digest=sha256:abc` → Provide digest
+4. Registry: Verify digest matches uploaded data
+5. Registry: Move `uploads/{uuid}` → `blobs/sha256/abc123...`
+
+**Docker Hub, GHCR, ECR, Harbor — all use this pattern.**
+
+### Why It's Efficient with S3
+
+**For S3, the move is a CopyObject API call:**
+
+```go
+// This happens INSIDE S3 servers - no data transfer!
+s3.CopyObject(&s3.CopyObjectInput{
+    Bucket:     "my-bucket",
+    CopySource: "/my-bucket/uploads/temp-12345/data",  // 5GB blob
+    Key:        "blobs/sha256/ab/abc123.../data",
+})
+// S3 copies internally, hold service only sends ~1KB API request
+```
+
+**For a 5GB layer:**
+- Hold service bandwidth: **~1KB** (API request/response)
+- S3 internal copy: Instant (metadata operation on S3 side)
+- No data leaves S3, no network transfer
+
+This is why the move operation is essentially free!
+
+## Implementation Details
+
+### 1. Add S3 Client to Hold Service
+
+**File: `cmd/hold/main.go`**
+
+Modify `HoldService` struct:
+```go
+type HoldService struct {
+    driver       storagedriver.StorageDriver
+    config       *Config
+    s3Client     *s3.S3      // NEW: S3 client for presigned URLs
+    bucket       string      // NEW: Bucket name
+    s3PathPrefix string      // NEW: Path prefix (if any)
+}
+```
+
+Add initialization function:
+```go
+func (s *HoldService) initS3Client() error {
+    if s.config.Storage.Type() != "s3" {
+        log.Printf("Storage driver is %s (not S3), presigned URLs disabled", s.config.Storage.Type())
+        return nil
+    }
+
+    params := s.config.Storage.Parameters()["s3"].(configuration.Parameters)
+
+    // Build AWS config
+    awsConfig := &aws.Config{
+        Region:      aws.String(params["region"].(string)),
+        Credentials: credentials.NewStaticCredentials(
+            params["accesskey"].(string),
+            params["secretkey"].(string),
+            "",
+        ),
+    }
+
+    // Add custom endpoint for S3-compatible services (Storj, MinIO, etc.)
+    if endpoint, ok := params["regionendpoint"].(string); ok && endpoint != "" {
+        awsConfig.Endpoint = aws.String(endpoint)
+        awsConfig.S3ForcePathStyle = aws.Bool(true) // Required for MinIO, Storj
+    }
+
+    sess, err := session.NewSession(awsConfig)
+    if err != nil {
+        return fmt.Errorf("failed to create AWS session: %w", err)
+    }
+
+    s.s3Client = s3.New(sess)
+    s.bucket = params["bucket"].(string)
+
+    log.Printf("S3 presigned URLs enabled for bucket: %s", s.bucket)
+    return nil
+}
+```
+
+Call during service initialization:
+```go
+func NewHoldService(cfg *Config) (*HoldService, error) {
+    // ... existing driver creation ...
+
+    service := &HoldService{
+        driver: driver,
+        config: cfg,
+    }
+
+    // Initialize S3 client for presigned URLs
+    if err := service.initS3Client(); err != nil {
+        log.Printf("WARNING: S3 presigned URLs disabled: %v", err)
+    }
+
+    return service, nil
+}
+```
+
+### 2. Implement Presigned URL Generation
+
+**For Downloads:**
+
+```go
+func (s *HoldService) getDownloadURL(ctx context.Context, digest string, did string) (string, error) {
+    path := blobPath(digest)
+
+    // Check if blob exists
+    if _, err := s.driver.Stat(ctx, path); err != nil {
+        return "", fmt.Errorf("blob not found: %w", err)
+    }
+
+    // If S3 client available, generate presigned URL
+    if s.s3Client != nil {
+        s3Key := strings.TrimPrefix(path, "/")
+
+        req, _ := s.s3Client.GetObjectRequest(&s3.GetObjectInput{
+            Bucket: aws.String(s.bucket),
+            Key:    aws.String(s3Key),
+        })
+
+        url, err := req.Presign(15 * time.Minute)
+        if err != nil {
+            log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
+            return s.getProxyDownloadURL(digest, did), nil
+        }
+
+        log.Printf("Generated presigned download URL for %s (expires in 15min)", digest)
+        return url, nil
+    }
+
+    // Fallback: return proxy URL
+    return s.getProxyDownloadURL(digest, did), nil
+}
+
+func (s *HoldService) getProxyDownloadURL(digest, did string) string {
+    return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
+}
+```
+
+**For Uploads:**
+
+```go
+func (s *HoldService) getUploadURL(ctx context.Context, digest string, size int64, did string) (string, error) {
+    path := blobPath(digest)
+
+    // If S3 client available, generate presigned URL
+    if s.s3Client != nil {
+        s3Key := strings.TrimPrefix(path, "/")
+
+        req, _ := s.s3Client.PutObjectRequest(&s3.PutObjectInput{
+            Bucket: aws.String(s.bucket),
+            Key:    aws.String(s3Key),
+        })
+
+        url, err := req.Presign(15 * time.Minute)
+        if err != nil {
+            log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
+            return s.getProxyUploadURL(digest, did), nil
+        }
+
+        log.Printf("Generated presigned upload URL for %s (expires in 15min)", digest)
+        return url, nil
+    }
+
+    // Fallback: return proxy URL
+    return s.getProxyUploadURL(digest, did), nil
+}
+
+func (s *HoldService) getProxyUploadURL(digest, did string) string {
+    return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
+}
+```
+
+### 3. No Changes Needed for Move Operation
+
+The existing `/move` endpoint already uses `driver.Move()`, which for S3:
+- Calls `s3.CopyObject()` (server-side copy)
+- Calls `s3.DeleteObject()` (delete source)
+- No data transfer through hold service!
+
+**File: `cmd/hold/main.go:296` (already exists, no changes needed)**
+
+```go
+func (s *HoldService) HandleMove(w http.ResponseWriter, r *http.Request) {
+    // ... existing auth and parsing ...
+
+    sourcePath := blobPath(fromPath)  // uploads/temp-{uuid}/data
+    destPath := blobPath(toDigest)    // blobs/sha256/ab/abc123.../data
+
+    // For S3, this does CopyObject + DeleteObject (server-side)
+    if err := s.driver.Move(ctx, sourcePath, destPath); err != nil {
+        // ... error handling ...
+    }
+}
+```
+
+### 4. AppView Changes (Optional Optimization)
+
+**File: `pkg/storage/proxy_blob_store.go:228`**
+
+Currently streams to hold service proxy URL. Could be optimized to use presigned URL:
+
+```go
+// In Create() - line 228
+go func() {
+    defer pipeReader.Close()
+
+    tempPath := fmt.Sprintf("uploads/temp-%s", writer.id)
+
+    // Try to get presigned URL for temp location
+    url, err := p.getUploadURL(ctx, digest.FromString(tempPath), 0)
+    if err != nil {
+        // Fallback to direct proxy URL
+        url = fmt.Sprintf("%s/blobs/%s?did=%s", p.storageEndpoint, tempPath, p.did)
+    }
+
+    req, err := http.NewRequestWithContext(uploadCtx, "PUT", url, pipeReader)
+    // ... rest unchanged
+}()
+```
+
+**Note:** This optimization is optional. The presigned URL will be returned by hold service's `getUploadURL()` anyway.
+
+## S3-Compatible Service Support
+
+### Storj
+
+```bash
+# .env file
+STORAGE_DRIVER=s3
+AWS_ACCESS_KEY_ID=your-storj-access-key
+AWS_SECRET_ACCESS_KEY=your-storj-secret-key
+S3_BUCKET=your-bucket-name
+S3_REGION=global
+S3_ENDPOINT=https://gateway.storjshare.io
+```
+
+**Presigned URL example:**
+```
+https://gateway.storjshare.io/your-bucket/blobs/sha256/ab/abc123.../data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Signature=...
+```
+
+### MinIO
+
+```bash
+STORAGE_DRIVER=s3
+AWS_ACCESS_KEY_ID=minioadmin
+AWS_SECRET_ACCESS_KEY=minioadmin
+S3_BUCKET=registry
+S3_REGION=us-east-1
+S3_ENDPOINT=http://minio.example.com:9000
+```
+
+### Backblaze B2
+
+```bash
+STORAGE_DRIVER=s3
+AWS_ACCESS_KEY_ID=your-b2-key-id
+AWS_SECRET_ACCESS_KEY=your-b2-application-key
+S3_BUCKET=your-bucket-name
+S3_REGION=us-west-002
+S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com
+```
+
+### Cloudflare R2
+
+```bash
+STORAGE_DRIVER=s3
+AWS_ACCESS_KEY_ID=your-r2-access-key-id
+AWS_SECRET_ACCESS_KEY=your-r2-secret-access-key
+S3_BUCKET=your-bucket-name
+S3_REGION=auto
+S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
+```
+
+**All these services support presigned URLs with AWS SDK v1!**
+
+## Performance Impact
+
+### Bandwidth Savings
+
+**Before (proxy mode):**
+- 5GB layer upload: Hold service receives 5GB, sends 5GB to S3 = **10GB** bandwidth
+- 5GB layer download: S3 sends 5GB to hold, hold sends 5GB to client = **10GB** bandwidth
+- **Total for push+pull: 20GB hold service bandwidth**
+
+**After (presigned URLs):**
+- 5GB layer upload: Hold generates URL (1KB), AppView → S3 direct (5GB), CopyObject API (1KB) = **~2KB** hold bandwidth
+- 5GB layer download: Hold generates URL (1KB), client → S3 direct = **~1KB** hold bandwidth
+- **Total for push+pull: ~3KB hold service bandwidth**
+
+**Savings: 99.98% reduction in hold service bandwidth!**
+
+### Latency Improvements
+
+**Before:**
+- Download: Client → AppView → Hold → S3 → Hold → AppView → Client (4 hops)
+- Upload: Client → AppView → Hold → S3 (3 hops)
+
+**After:**
+- Download: Client → AppView (redirect) → S3 (1 hop to data)
+- Upload: Client → AppView → S3 (2 hops)
+- Move: S3 internal (no network hops)
+
+### Resource Requirements
+
+**Before:**
+- Hold service needs bandwidth = sum of all image operations
+- For 100 concurrent 1GB pushes: 100GB/s bandwidth needed
+- Expensive, hard to scale
+
+**After:**
+- Hold service needs minimal CPU for presigned URL signing
+- For 100 concurrent 1GB pushes: ~100KB/s bandwidth needed (API traffic)
+- Can run on $5/month instance!
+
+## Security Considerations
+
+### Presigned URL Expiration
+
+- Default: **15 minutes** expiration
+- Presigned URL includes embedded credentials in query params
+- After expiry, URL becomes invalid (S3 rejects with 403)
+- No long-lived URLs floating around
+
+### Authorization Flow
+
+1. **AppView validates user** via ATProto OAuth
+2. **AppView passes DID to hold service** in presigned URL request
+3. **Hold service validates DID** (owner or crew member)
+4. **Hold service generates presigned URL** if authorized
+5. **Client uses presigned URL** directly with S3
+
+**Security boundary:** Hold service controls who gets presigned URLs, S3 validates the URLs.
+
+### Fallback Security
+
+If presigned URL generation fails:
+- Falls back to proxy URLs (existing behavior)
+- Still requires hold service authorization
+- Data flows through hold service (original security model)
+
+## Testing & Validation
+
+### Verify Presigned URLs are Used
+
+**1. Check hold service logs:**
+```bash
+docker logs atcr-hold | grep -i presigned
+# Should see: "Generated presigned download/upload URL for sha256:..."
+```
+
+**2. Monitor network traffic:**
+```bash
+# Before: Large data transfers to/from hold service
+docker stats atcr-hold
+
+# After: Minimal network usage on hold service
+docker stats atcr-hold
+```
+
+**3. Inspect redirect responses:**
+```bash
+# Should see 307 redirect to S3 URL
+curl -v http://appview:5000/v2/alice/myapp/blobs/sha256:abc123 \
+  -H "Authorization: Bearer $TOKEN"
+
+# Look for:
+# < HTTP/1.1 307 Temporary Redirect
+# < Location: https://gateway.storjshare.io/...?X-Amz-Signature=...
+```
+
+### Test Fallback Behavior
+
+**1. With filesystem driver (should use proxy URLs):**
+```bash
+STORAGE_DRIVER=filesystem docker-compose up atcr-hold
+# Logs should show: "Storage driver is filesystem (not S3), presigned URLs disabled"
+```
+
+**2. With S3 but invalid credentials (should fall back):**
+```bash
+AWS_ACCESS_KEY_ID=invalid docker-compose up atcr-hold
+# Logs should show: "WARN: Presigned URL generation failed, falling back to proxy"
+```
+
+### Bandwidth Monitoring
+
+**Track hold service bandwidth over time:**
+```bash
+# Install bandwidth monitoring
+docker exec atcr-hold apt-get update && apt-get install -y vnstat
+
+# Monitor
+docker exec atcr-hold vnstat -l
+```
+
+**Expected results:**
+- Before: Bandwidth correlates with image operations
+- After: Bandwidth stays minimal regardless of image operations
+
+## Migration Guide
+
+### For Existing ATCR Deployments
+
+**1. Update hold service code** (this implementation)
+
+**2. No configuration changes needed** if already using S3:
+```bash
+# Existing S3 config works automatically
+STORAGE_DRIVER=s3
+AWS_ACCESS_KEY_ID=...
+AWS_SECRET_ACCESS_KEY=...
+S3_BUCKET=...
+S3_ENDPOINT=...
+```
+
+**3. Restart hold service:**
+```bash
+docker-compose restart atcr-hold
+```
+
+**4. Verify in logs:**
+```
+S3 presigned URLs enabled for bucket: my-bucket
+```
+
+**5. Test with image push/pull:**
+```bash
+docker push atcr.io/alice/myapp:latest
+docker pull atcr.io/alice/myapp:latest
+```
+
+**6. Monitor bandwidth** to confirm reduction
+
+### Rollback Plan
+
+If issues arise:
+
+**Option 1: Disable presigned URLs via env var** (if we add this feature)
+```bash
+PRESIGNED_URLS_ENABLED=false docker-compose restart atcr-hold
+```
+
+**Option 2: Revert code changes** to previous hold service version
+
+The implementation has automatic fallbacks, so partial failures won't break functionality.
+
+## Future Enhancements
+
+### 1. Configurable Expiration
+
+Allow customizing presigned URL expiry:
+```bash
+PRESIGNED_URL_EXPIRY=30m  # Default: 15m
+```
+
+### 2. Presigned URL Caching
+
+Cache presigned URLs for frequently accessed blobs (with shorter TTL).
+
+### 3. CloudFront/CDN Integration
+
+For downloads, use CloudFront presigned URLs instead of direct S3:
+- Better global distribution
+- Lower egress costs
+- Faster downloads
+
+### 4. Multipart Upload Support
+
+For very large layers (>5GB), use presigned URLs with multipart upload:
+- Generate presigned URLs for each part
+- Client uploads parts directly to S3
+- Hold service finalizes multipart upload
+
+### 5. Metrics & Monitoring
+
+Track presigned URL usage:
+- Count of presigned URLs generated
+- Fallback rate (proxy vs presigned)
+- Bandwidth savings metrics
+
+## References
+
+- [OCI Distribution Specification - Push](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push)
+- [AWS SDK Go v1 - Presigned URLs](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html)
+- [Storj - Using Presigned URLs](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway/using-presigned-urls)
+- [MinIO - Presigned Upload via Browser](https://docs.min.io/community/minio-object-store/integrations/presigned-put-upload-via-browser.html)
+- [Cloudflare R2 - Presigned URLs](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)
+- [Backblaze B2 - S3 Compatible API](https://help.backblaze.com/hc/en-us/articles/360047815993-Does-the-B2-S3-Compatible-API-support-Pre-Signed-URLs)
+
+## Summary
+
+Implementing S3 presigned URLs transforms ATCR's hold service from a **data proxy** to a **lightweight orchestrator**:
+
+✅ **99.98% bandwidth reduction** for hold service
+✅ **Direct client → S3 transfers** for maximum speed
+✅ **Works with all S3-compatible services** (Storj, MinIO, R2, B2)
+✅ **OCI-compliant** temp → final move pattern
+✅ **Automatic fallback** to proxy mode for non-S3 drivers
+✅ **No breaking changes** to existing deployments
+
+This makes BYOS (Bring Your Own Storage) truly scalable and cost-effective, as users can run hold services on minimal infrastructure while serving arbitrarily large container images.