at-container-registry/docs/PRESIGNED_URLS.md

# S3 Presigned URLs Implementation

## Overview

Currently, ATCR's hold service acts as a proxy for all blob data, meaning every byte flows through the hold service when uploading or downloading container images. This document describes the implementation of **S3 presigned URLs** to eliminate this bottleneck, allowing direct data transfer between clients and S3-compatible storage.

### Current Architecture (Proxy Mode)

```
Downloads:  Docker → AppView → Hold Service → S3 → Hold Service → AppView → Docker
Uploads:    Docker → AppView → Hold Service → S3
```

**Problems:**
- All blob data flows through hold service
- Hold service bandwidth = total image bandwidth
- Latency from extra hops
- Hold service becomes bottleneck for large images

### Target Architecture (Presigned URLs)

```
Downloads:  Docker → AppView (gets presigned URL) → S3 (direct download)
Uploads:    Docker → AppView → S3 (via presigned URL)
Move:       AppView → Hold Service → S3 (server-side CopyObject API)
```

**Benefits:**
- ✅ Hold service only orchestrates (no data transfer)
- ✅ Blob data never touches hold service
- ✅ Direct S3 uploads/downloads at wire speed
- ✅ Hold service can run on minimal resources
- ✅ Works with all S3-compatible services

## How Presigned URLs Work

### For Downloads (GET)

1. **Docker requests blob:** `GET /v2/alice/myapp/blobs/sha256:abc123`
2. **AppView asks hold service:** `POST /get-presigned-url`
   ```json
   {"did": "did:plc:alice123", "digest": "sha256:abc123"}
   ```
3. **Hold service generates presigned URL:**
   ```go
   req, _ := s3Client.GetObjectRequest(&s3.GetObjectInput{
       Bucket: "my-bucket",
       Key:    "blobs/sha256/ab/abc123.../data",
   })
   url, _ := req.Presign(15 * time.Minute)
   // Returns: https://gateway.storjshare.io/bucket/blobs/...?X-Amz-Signature=...
   ```
4. **AppView redirects Docker:** `HTTP 307 Location: <presigned-url>`
5. **Docker downloads directly from S3** using the presigned URL

**Data path:** Docker → S3 (direct)
**Hold service bandwidth:** ~1KB (API request/response)

### For Uploads (PUT)

**Small blobs (< 5MB) using Put():**

1. **Docker sends blob to AppView:** `PUT /v2/alice/myapp/blobs/uploads/{uuid}`
2. **AppView asks hold service:** `POST /put-presigned-url`
   ```json
   {"did": "did:plc:alice123", "digest": "sha256:abc123", "size": 1024}
   ```
3. **Hold service generates presigned URL:**
   ```go
   req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{
       Bucket: "my-bucket",
       Key:    "blobs/sha256/ab/abc123.../data",
   })
   url, _ := req.Presign(15 * time.Minute)
   ```
4. **AppView uploads to S3** using presigned URL
5. **AppView confirms to Docker:** `201 Created`

**Data path:** Docker → AppView → S3 (via presigned URL)
**Hold service bandwidth:** ~1KB (API request/response)

### For Streaming Uploads (Create/Commit)

**Large blobs (> 5MB) using streaming:**

1. **Docker starts upload:** `POST /v2/alice/myapp/blobs/uploads/`
2. **AppView creates upload session** with UUID
3. **AppView gets presigned URL for temp location:**
   ```json
   POST /put-presigned-url
   {"did": "...", "digest": "uploads/temp-{uuid}", "size": 0}
   ```
4. **Docker streams data:** `PATCH /v2/alice/myapp/blobs/uploads/{uuid}`
5. **AppView streams to S3** using presigned URL to `uploads/temp-{uuid}/data`
6. **Docker finalizes:** `PUT /v2/.../uploads/{uuid}?digest=sha256:abc123`
7. **AppView requests move:** `POST /move?from=uploads/temp-{uuid}&to=sha256:abc123`
8. **Hold service executes S3 server-side copy:**
   ```go
   s3.CopyObject(&s3.CopyObjectInput{
       Bucket:     "my-bucket",
       CopySource: "/my-bucket/uploads/temp-{uuid}/data",
       Key:        "blobs/sha256/ab/abc123.../data",
   })
   s3.DeleteObject(&s3.DeleteObjectInput{
       Key: "uploads/temp-{uuid}/data",
   })
   ```

**Data path:** Docker → AppView → S3 (temp location)
**Move path:** S3 internal copy (no data transfer!)
**Hold service bandwidth:** ~2KB (presigned URL + CopyObject API)

### For Chunked Uploads (Multipart Upload)

**Large blobs with OCI chunked protocol (Docker PATCH requests):**

The OCI Distribution Spec uses chunked uploads via multiple PATCH requests. Single presigned URLs don't support this - we need **S3 Multipart Upload**.

1. **Docker starts upload:** `POST /v2/alice/myapp/blobs/uploads/`
2. **AppView initiates multipart:**
   ```json
   POST /start-multipart
   {"did": "...", "digest": "uploads/temp-{uuid}"}
   → Returns: {"upload_id": "xyz123"}
   ```
3. **Docker sends chunk 1:** `PATCH /v2/.../uploads/{uuid}` (5MB data)
4. **AppView gets part URL:**
   ```json
   POST /part-presigned-url
   {"did": "...", "digest": "uploads/temp-{uuid}", "upload_id": "xyz123", "part_number": 1}
   → Returns: {"url": "https://s3.../part?uploadId=xyz123&partNumber=1&..."}
   ```
5. **AppView uploads part 1** using presigned URL → Gets ETag
6. **Docker sends chunk 2:** `PATCH /v2/.../uploads/{uuid}` (5MB data)
7. **Repeat steps 4-5** for part 2 (and subsequent parts)
8. **Docker finalizes:** `PUT /v2/.../uploads/{uuid}?digest=sha256:abc123`
9. **AppView completes multipart:**
   ```json
   POST /complete-multipart
   {"did": "...", "digest": "uploads/temp-{uuid}", "upload_id": "xyz123",
    "parts": [{"part_number": 1, "etag": "..."}, {"part_number": 2, "etag": "..."}]}
   ```
10. **AppView requests move:** `POST /move?from=uploads/temp-{uuid}&to=sha256:abc123`
11. **Hold service executes S3 server-side copy** (same as above)

**Data path:** Docker → AppView (buffers 5MB) → S3 (via presigned URL per part)
**Each PATCH:** Independent, non-blocking, immediate response
**Hold service bandwidth:** ~1KB per part + ~1KB for completion

**Why This Fixes "Client Disconnected" Errors:**
- Previous implementation: Single presigned URL + pipe → PATCH blocks → Docker timeout
- New implementation: Each PATCH → separate part upload → immediate response → no blocking

## Why the Temp → Final Move is Required

This is **not an ATCR implementation detail** — it's required by the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push).

### The Problem: Unknown Digest

Docker doesn't know the blob's digest until **after** uploading:

1. **Streaming data:** Can't buffer 5GB layer in memory to calculate digest first
2. **Stdin pipes:** `docker build . | docker push` generates data on-the-fly
3. **Chunked uploads:** Multiple PATCH requests, digest calculated as data streams

### The Solution: Upload to Temp, Verify, Move

**All OCI registries do this:**

1. Client: `POST /v2/{name}/blobs/uploads/` → Get upload UUID
2. Client: `PATCH /v2/{name}/blobs/uploads/{uuid}` → Stream data to temp location
3. Client: `PUT /v2/{name}/blobs/uploads/{uuid}?digest=sha256:abc` → Provide digest
4. Registry: Verify digest matches uploaded data
5. Registry: Move `uploads/{uuid}` → `blobs/sha256/abc123...`

**Docker Hub, GHCR, ECR, Harbor — all use this pattern.**

### Why It's Efficient with S3

**For S3, the move is a CopyObject API call:**

```go
// This happens INSIDE S3 servers - no data transfer!
s3.CopyObject(&s3.CopyObjectInput{
    Bucket:     "my-bucket",
    CopySource: "/my-bucket/uploads/temp-12345/data",  // 5GB blob
    Key:        "blobs/sha256/ab/abc123.../data",
})
// S3 copies internally, hold service only sends ~1KB API request
```

**For a 5GB layer:**
- Hold service bandwidth: **~1KB** (API request/response)
- S3 internal copy: Instant (metadata operation on S3 side)
- No data leaves S3, no network transfer

This is why the move operation is essentially free!

## Implementation Details

### 1. Add S3 Client to Hold Service

**File: `cmd/hold/main.go`**

Modify `HoldService` struct:
```go
type HoldService struct {
    driver       storagedriver.StorageDriver
    config       *Config
    s3Client     *s3.S3      // NEW: S3 client for presigned URLs
    bucket       string      // NEW: Bucket name
    s3PathPrefix string      // NEW: Path prefix (if any)
}
```

Add initialization function:
```go
func (s *HoldService) initS3Client() error {
    if s.config.Storage.Type() != "s3" {
        log.Printf("Storage driver is %s (not S3), presigned URLs disabled", s.config.Storage.Type())
        return nil
    }

    params := s.config.Storage.Parameters()["s3"].(configuration.Parameters)

    // Build AWS config
    awsConfig := &aws.Config{
        Region:      aws.String(params["region"].(string)),
        Credentials: credentials.NewStaticCredentials(
            params["accesskey"].(string),
            params["secretkey"].(string),
            "",
        ),
    }

    // Add custom endpoint for S3-compatible services (Storj, MinIO, etc.)
    if endpoint, ok := params["regionendpoint"].(string); ok && endpoint != "" {
        awsConfig.Endpoint = aws.String(endpoint)
        awsConfig.S3ForcePathStyle = aws.Bool(true) // Required for MinIO, Storj
    }

    sess, err := session.NewSession(awsConfig)
    if err != nil {
        return fmt.Errorf("failed to create AWS session: %w", err)
    }

    s.s3Client = s3.New(sess)
    s.bucket = params["bucket"].(string)

    log.Printf("S3 presigned URLs enabled for bucket: %s", s.bucket)
    return nil
}
```

Call during service initialization:
```go
func NewHoldService(cfg *Config) (*HoldService, error) {
    // ... existing driver creation ...

    service := &HoldService{
        driver: driver,
        config: cfg,
    }

    // Initialize S3 client for presigned URLs
    if err := service.initS3Client(); err != nil {
        log.Printf("WARNING: S3 presigned URLs disabled: %v", err)
    }

    return service, nil
}
```

### 2. Implement Presigned URL Generation

**For Downloads:**

```go
func (s *HoldService) getDownloadURL(ctx context.Context, digest string, did string) (string, error) {
    path := blobPath(digest)

    // Check if blob exists
    if _, err := s.driver.Stat(ctx, path); err != nil {
        return "", fmt.Errorf("blob not found: %w", err)
    }

    // If S3 client available, generate presigned URL
    if s.s3Client != nil {
        s3Key := strings.TrimPrefix(path, "/")

        req, _ := s.s3Client.GetObjectRequest(&s3.GetObjectInput{
            Bucket: aws.String(s.bucket),
            Key:    aws.String(s3Key),
        })

        url, err := req.Presign(15 * time.Minute)
        if err != nil {
            log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
            return s.getProxyDownloadURL(digest, did), nil
        }

        log.Printf("Generated presigned download URL for %s (expires in 15min)", digest)
        return url, nil
    }

    // Fallback: return proxy URL
    return s.getProxyDownloadURL(digest, did), nil
}

func (s *HoldService) getProxyDownloadURL(digest, did string) string {
    return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
}
```

**For Uploads:**

```go
func (s *HoldService) getUploadURL(ctx context.Context, digest string, size int64, did string) (string, error) {
    path := blobPath(digest)

    // If S3 client available, generate presigned URL
    if s.s3Client != nil {
        s3Key := strings.TrimPrefix(path, "/")

        req, _ := s.s3Client.PutObjectRequest(&s3.PutObjectInput{
            Bucket: aws.String(s.bucket),
            Key:    aws.String(s3Key),
        })

        url, err := req.Presign(15 * time.Minute)
        if err != nil {
            log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
            return s.getProxyUploadURL(digest, did), nil
        }

        log.Printf("Generated presigned upload URL for %s (expires in 15min)", digest)
        return url, nil
    }

    // Fallback: return proxy URL
    return s.getProxyUploadURL(digest, did), nil
}

func (s *HoldService) getProxyUploadURL(digest, did string) string {
    return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
}
```

### 3. Multipart Upload Endpoints (Required for Chunked Uploads)

**File: `cmd/hold/main.go`**

#### Start Multipart Upload

```go
func (s *HoldService) HandleStartMultipart(w http.ResponseWriter, r *http.Request) {
    var req StartMultipartUploadRequest // {did, digest}

    // Validate DID authorization for WRITE
    if !s.isAuthorizedWrite(req.DID) {
        // Return 403 Forbidden
    }

    // Initiate S3 multipart upload
    result, err := s.s3Client.CreateMultipartUploadWithContext(ctx, &s3.CreateMultipartUploadInput{
        Bucket: aws.String(s.bucket),
        Key:    aws.String(s3Key),
    })

    // Return upload ID
    json.NewEncoder(w).Encode(StartMultipartUploadResponse{
        UploadID: *result.UploadId,
        ExpiresAt: time.Now().Add(24 * time.Hour),
    })
}
```

**Route:** `POST /start-multipart`

#### Get Part Presigned URL

```go
func (s *HoldService) HandleGetPartURL(w http.ResponseWriter, r *http.Request) {
    var req GetPartURLRequest // {did, digest, upload_id, part_number}

    // Generate presigned URL for specific part
    req, _ := s.s3Client.UploadPartRequest(&s3.UploadPartInput{
        Bucket:     aws.String(s.bucket),
        Key:        aws.String(s3Key),
        UploadId:   aws.String(uploadID),
        PartNumber: aws.Int64(int64(partNumber)),
    })

    url, err := req.Presign(15 * time.Minute)

    json.NewEncoder(w).Encode(GetPartURLResponse{URL: url})
}
```

**Route:** `POST /part-presigned-url`

#### Complete Multipart Upload

```go
func (s *HoldService) HandleCompleteMultipart(w http.ResponseWriter, r *http.Request) {
    var req CompleteMultipartRequest // {did, digest, upload_id, parts: [{part_number, etag}]}

    // Convert parts to S3 format
    s3Parts := make([]*s3.CompletedPart, len(req.Parts))
    for i, p := range req.Parts {
        s3Parts[i] = &s3.CompletedPart{
            PartNumber: aws.Int64(int64(p.PartNumber)),
            ETag:       aws.String(p.ETag),
        }
    }

    // Complete multipart upload
    _, err := s.s3Client.CompleteMultipartUploadWithContext(ctx, &s3.CompleteMultipartUploadInput{
        Bucket:   aws.String(s.bucket),
        Key:      aws.String(s3Key),
        UploadId: aws.String(uploadID),
        MultipartUpload: &s3.CompletedMultipartUpload{Parts: s3Parts},
    })
}
```

**Route:** `POST /complete-multipart`

#### Abort Multipart Upload

```go
func (s *HoldService) HandleAbortMultipart(w http.ResponseWriter, r *http.Request) {
    var req AbortMultipartRequest // {did, digest, upload_id}

    // Abort and cleanup parts
    _, err := s.s3Client.AbortMultipartUploadWithContext(ctx, &s3.AbortMultipartUploadInput{
        Bucket:   aws.String(s.bucket),
        Key:      aws.String(s3Key),
        UploadId: aws.String(uploadID),
    })
}
```

**Route:** `POST /abort-multipart`

### 4. Move Operation (No Changes)

The existing `/move` endpoint already uses `driver.Move()`, which for S3:
- Calls `s3.CopyObject()` (server-side copy)
- Calls `s3.DeleteObject()` (delete source)
- No data transfer through hold service!

**File: `cmd/hold/main.go:393` (already exists, no changes needed)**

```go
func (s *HoldService) HandleMove(w http.ResponseWriter, r *http.Request) {
    // ... existing auth and parsing ...

    sourcePath := blobPath(fromPath)  // uploads/temp-{uuid}/data
    destPath := blobPath(toDigest)    // blobs/sha256/ab/abc123.../data

    // For S3, this does CopyObject + DeleteObject (server-side)
    if err := s.driver.Move(ctx, sourcePath, destPath); err != nil {
        // ... error handling ...
    }
}
```

### 5. AppView Changes (Multipart Upload Implementation)

**File: `pkg/storage/proxy_blob_store.go:228`**

Currently streams to hold service proxy URL. Could be optimized to use presigned URL:

```go
// In Create() - line 228
go func() {
    defer pipeReader.Close()

    tempPath := fmt.Sprintf("uploads/temp-%s", writer.id)

    // Try to get presigned URL for temp location
    url, err := p.getUploadURL(ctx, digest.FromString(tempPath), 0)
    if err != nil {
        // Fallback to direct proxy URL
        url = fmt.Sprintf("%s/blobs/%s?did=%s", p.storageEndpoint, tempPath, p.did)
    }

    req, err := http.NewRequestWithContext(uploadCtx, "PUT", url, pipeReader)
    // ... rest unchanged
}()
```

**Note:** This optimization is optional. The presigned URL will be returned by hold service's `getUploadURL()` anyway.

## S3-Compatible Service Support

### Storj

```bash
# .env file
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-storj-access-key
AWS_SECRET_ACCESS_KEY=your-storj-secret-key
S3_BUCKET=your-bucket-name
S3_REGION=global
S3_ENDPOINT=https://gateway.storjshare.io
```

**Presigned URL example:**
```
https://gateway.storjshare.io/your-bucket/blobs/sha256/ab/abc123.../data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Signature=...
```

### MinIO

```bash
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET=registry
S3_REGION=us-east-1
S3_ENDPOINT=http://minio.example.com:9000
```

### Backblaze B2

```bash
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-b2-key-id
AWS_SECRET_ACCESS_KEY=your-b2-application-key
S3_BUCKET=your-bucket-name
S3_REGION=us-west-002
S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com
```

### Cloudflare R2

```bash
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-r2-access-key-id
AWS_SECRET_ACCESS_KEY=your-r2-secret-access-key
S3_BUCKET=your-bucket-name
S3_REGION=auto
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
```

**All these services support presigned URLs with AWS SDK v1!**

## Performance Impact

### Bandwidth Savings

**Before (proxy mode):**
- 5GB layer upload: Hold service receives 5GB, sends 5GB to S3 = **10GB** bandwidth
- 5GB layer download: S3 sends 5GB to hold, hold sends 5GB to client = **10GB** bandwidth
- **Total for push+pull: 20GB hold service bandwidth**

**After (presigned URLs):**
- 5GB layer upload: Hold generates URL (1KB), AppView → S3 direct (5GB), CopyObject API (1KB) = **~2KB** hold bandwidth
- 5GB layer download: Hold generates URL (1KB), client → S3 direct = **~1KB** hold bandwidth
- **Total for push+pull: ~3KB hold service bandwidth**

**Savings: 99.98% reduction in hold service bandwidth!**

### Latency Improvements

**Before:**
- Download: Client → AppView → Hold → S3 → Hold → AppView → Client (4 hops)
- Upload: Client → AppView → Hold → S3 (3 hops)

**After:**
- Download: Client → AppView (redirect) → S3 (1 hop to data)
- Upload: Client → AppView → S3 (2 hops)
- Move: S3 internal (no network hops)

### Resource Requirements

**Before:**
- Hold service needs bandwidth = sum of all image operations
- For 100 concurrent 1GB pushes: 100GB/s bandwidth needed
- Expensive, hard to scale

**After:**
- Hold service needs minimal CPU for presigned URL signing
- For 100 concurrent 1GB pushes: ~100KB/s bandwidth needed (API traffic)
- Can run on $5/month instance!

## Security Considerations

### Presigned URL Expiration

- Default: **15 minutes** expiration
- Presigned URL includes embedded credentials in query params
- After expiry, URL becomes invalid (S3 rejects with 403)
- No long-lived URLs floating around

### Authorization Flow

1. **AppView validates user** via ATProto OAuth
2. **AppView passes DID to hold service** in presigned URL request
3. **Hold service validates DID** (owner or crew member)
4. **Hold service generates presigned URL** if authorized
5. **Client uses presigned URL** directly with S3

**Security boundary:** Hold service controls who gets presigned URLs, S3 validates the URLs.

### Fallback Security

If presigned URL generation fails:
- Falls back to proxy URLs (existing behavior)
- Still requires hold service authorization
- Data flows through hold service (original security model)

## Testing & Validation

### Verify Presigned URLs are Used

**1. Check hold service logs:**
```bash
docker logs atcr-hold | grep -i presigned
# Should see: "Generated presigned download/upload URL for sha256:..."
```

**2. Monitor network traffic:**
```bash
# Before: Large data transfers to/from hold service
docker stats atcr-hold

# After: Minimal network usage on hold service
docker stats atcr-hold
```

**3. Inspect redirect responses:**
```bash
# Should see 307 redirect to S3 URL
curl -v http://appview:5000/v2/alice/myapp/blobs/sha256:abc123 \
  -H "Authorization: Bearer $TOKEN"

# Look for:
# < HTTP/1.1 307 Temporary Redirect
# < Location: https://gateway.storjshare.io/...?X-Amz-Signature=...
```

### Test Fallback Behavior

**1. With filesystem driver (should use proxy URLs):**
```bash
STORAGE_DRIVER=filesystem docker-compose up atcr-hold
# Logs should show: "Storage driver is filesystem (not S3), presigned URLs disabled"
```

**2. With S3 but invalid credentials (should fall back):**
```bash
AWS_ACCESS_KEY_ID=invalid docker-compose up atcr-hold
# Logs should show: "WARN: Presigned URL generation failed, falling back to proxy"
```

### Bandwidth Monitoring

**Track hold service bandwidth over time:**
```bash
# Install bandwidth monitoring
docker exec atcr-hold apt-get update && apt-get install -y vnstat

# Monitor
docker exec atcr-hold vnstat -l
```

**Expected results:**
- Before: Bandwidth correlates with image operations
- After: Bandwidth stays minimal regardless of image operations

## Migration Guide

### For Existing ATCR Deployments

**1. Update hold service code** (this implementation)

**2. No configuration changes needed** if already using S3:
```bash
# Existing S3 config works automatically
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
S3_BUCKET=...
S3_ENDPOINT=...
```

**3. Restart hold service:**
```bash
docker-compose restart atcr-hold
```

**4. Verify in logs:**
```
S3 presigned URLs enabled for bucket: my-bucket
```

**5. Test with image push/pull:**
```bash
docker push atcr.io/alice/myapp:latest
docker pull atcr.io/alice/myapp:latest
```

**6. Monitor bandwidth** to confirm reduction

### Rollback Plan

If issues arise:

**Option 1: Disable presigned URLs via env var** (if we add this feature)
```bash
PRESIGNED_URLS_ENABLED=false docker-compose restart atcr-hold
```

**Option 2: Revert code changes** to previous hold service version

The implementation has automatic fallbacks, so partial failures won't break functionality.

## Testing with DISABLE_PRESIGNED_URLS

### Environment Variable

Set `DISABLE_PRESIGNED_URLS=true` to force proxy/buffered mode even when S3 is configured.

**Use cases:**
- Testing proxy/buffered code paths with S3 storage
- Debugging multipart uploads in buffered mode
- Simulating S3 providers that don't support presigned URLs
- Verifying fallback behavior works correctly

### How It Works

When `DISABLE_PRESIGNED_URLS=true`:

**Single blob operations:**
- `getDownloadURL()` returns proxy URL instead of S3 presigned URL
- `getHeadURL()` returns proxy URL instead of S3 presigned HEAD URL
- `getUploadURL()` returns proxy URL instead of S3 presigned PUT URL
- Client uses `/blobs/{digest}` endpoints (proxy through hold service)

**Multipart uploads:**
- `StartMultipartUploadWithManager()` creates **Buffered** session instead of **S3Native**
- `GetPartUploadURL()` returns `/multipart-parts/{uploadID}/{partNumber}` instead of S3 presigned URL
- Parts are buffered in memory in the hold service
- `CompleteMultipartUploadWithManager()` assembles parts and writes via storage driver

### Testing Example

```bash
# Test S3 with forced proxy mode
export STORAGE_DRIVER=s3
export S3_BUCKET=my-bucket
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export DISABLE_PRESIGNED_URLS=true  # Force buffered/proxy mode

./bin/atcr-hold

# Push an image - should use proxy mode
docker push atcr.io/yourdid/test:latest

# Check logs for:
# "Presigned URLs disabled, using proxy URL"
# "Presigned URLs disabled (DISABLE_PRESIGNED_URLS=true), using buffered mode"
# "Stored part: uploadID=... part=1 size=..."
```

## Future Enhancements

### 1. Configurable Expiration

Allow customizing presigned URL expiry:
```bash
PRESIGNED_URL_EXPIRY=30m  # Default: 15m
```

### 2. Presigned URL Caching

Cache presigned URLs for frequently accessed blobs (with shorter TTL).

### 3. CloudFront/CDN Integration

For downloads, use CloudFront presigned URLs instead of direct S3:
- Better global distribution
- Lower egress costs
- Faster downloads

### 4. Multipart Upload Support

For very large layers (>5GB), use presigned URLs with multipart upload:
- Generate presigned URLs for each part
- Client uploads parts directly to S3
- Hold service finalizes multipart upload

### 5. Metrics & Monitoring

Track presigned URL usage:
- Count of presigned URLs generated
- Fallback rate (proxy vs presigned)
- Bandwidth savings metrics

## References

- [OCI Distribution Specification - Push](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push)
- [AWS SDK Go v1 - Presigned URLs](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html)
- [Storj - Using Presigned URLs](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway/using-presigned-urls)
- [MinIO - Presigned Upload via Browser](https://docs.min.io/community/minio-object-store/integrations/presigned-put-upload-via-browser.html)
- [Cloudflare R2 - Presigned URLs](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)
- [Backblaze B2 - S3 Compatible API](https://help.backblaze.com/hc/en-us/articles/360047815993-Does-the-B2-S3-Compatible-API-support-Pre-Signed-URLs)

## Summary

Implementing S3 presigned URLs transforms ATCR's hold service from a **data proxy** to a **lightweight orchestrator**:

✅ **99.98% bandwidth reduction** for hold service
✅ **Direct client → S3 transfers** for maximum speed
✅ **Works with all S3-compatible services** (Storj, MinIO, R2, B2)
✅ **OCI-compliant** temp → final move pattern
✅ **Automatic fallback** to proxy mode for non-S3 drivers
✅ **No breaking changes** to existing deployments

This makes BYOS (Bring Your Own Storage) truly scalable and cost-effective, as users can run hold services on minimal infrastructure while serving arbitrarily large container images.