at-container-registry/docs/PRESIGNED_URLS.md

S3 Presigned URLs Implementation

Overview

Currently, ATCR's hold service acts as a proxy for all blob data, meaning every byte flows through the hold service when uploading or downloading container images. This document describes the implementation of S3 presigned URLs to eliminate this bottleneck, allowing direct data transfer between clients and S3-compatible storage.

Current Architecture (Proxy Mode)

Downloads:  Docker → AppView → Hold Service → S3 → Hold Service → AppView → Docker
Uploads:    Docker → AppView → Hold Service → S3

Problems:

• All blob data flows through hold service
• Hold service bandwidth = total image bandwidth
• Latency from extra hops
• Hold service becomes bottleneck for large images

Target Architecture (Presigned URLs)

Downloads:  Docker → AppView (gets presigned URL) → S3 (direct download)
Uploads:    Docker → AppView → S3 (via presigned URL)
Move:       AppView → Hold Service → S3 (server-side CopyObject API)

Benefits:

• ✅ Hold service only orchestrates (no data transfer)
• ✅ Blob data never touches hold service
• ✅ Direct S3 uploads/downloads at wire speed
• ✅ Hold service can run on minimal resources
• ✅ Works with all S3-compatible services

How Presigned URLs Work

For Downloads (GET)

1. Docker requests blob: GET /v2/alice/myapp/blobs/sha256:abc123
2. AppView asks hold service: POST /get-presigned-url

   {"did": "did:plc:alice123", "digest": "sha256:abc123"}
   

3. Hold service generates presigned URL:

   req, _ := s3Client.GetObjectRequest(&s3.GetObjectInput{
       Bucket: "my-bucket",
       Key:    "blobs/sha256/ab/abc123.../data",
   })
   url, _ := req.Presign(15 * time.Minute)
   // Returns: https://gateway.storjshare.io/bucket/blobs/...?X-Amz-Signature=...
   

4. AppView redirects Docker: HTTP 307 Location: <presigned-url>
5. Docker downloads directly from S3 using the presigned URL

Data path: Docker → S3 (direct) Hold service bandwidth: ~1KB (API request/response)

For Uploads (PUT)

Small blobs (< 5MB) using Put():

1. Docker sends blob to AppView: PUT /v2/alice/myapp/blobs/uploads/{uuid}
2. AppView asks hold service: POST /put-presigned-url

   {"did": "did:plc:alice123", "digest": "sha256:abc123", "size": 1024}
   

3. Hold service generates presigned URL:

   req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{
       Bucket: "my-bucket",
       Key:    "blobs/sha256/ab/abc123.../data",
   })
   url, _ := req.Presign(15 * time.Minute)
   

4. AppView uploads to S3 using presigned URL
5. AppView confirms to Docker: 201 Created

Data path: Docker → AppView → S3 (via presigned URL) Hold service bandwidth: ~1KB (API request/response)

For Streaming Uploads (Create/Commit)

Large blobs (> 5MB) using streaming:

1. Docker starts upload: POST /v2/alice/myapp/blobs/uploads/
2. AppView creates upload session with UUID
3. AppView gets presigned URL for temp location:

   POST /put-presigned-url
   {"did": "...", "digest": "uploads/temp-{uuid}", "size": 0}
   

4. Docker streams data: PATCH /v2/alice/myapp/blobs/uploads/{uuid}
5. AppView streams to S3 using presigned URL to uploads/temp-{uuid}/data
6. Docker finalizes: PUT /v2/.../uploads/{uuid}?digest=sha256:abc123
7. AppView requests move: POST /move?from=uploads/temp-{uuid}&to=sha256:abc123
8. Hold service executes S3 server-side copy:

   s3.CopyObject(&s3.CopyObjectInput{
       Bucket:     "my-bucket",
       CopySource: "/my-bucket/uploads/temp-{uuid}/data",
       Key:        "blobs/sha256/ab/abc123.../data",
   })
   s3.DeleteObject(&s3.DeleteObjectInput{
       Key: "uploads/temp-{uuid}/data",
   })
   

Data path: Docker → AppView → S3 (temp location) Move path: S3 internal copy (no data transfer!) Hold service bandwidth: ~2KB (presigned URL + CopyObject API)

Why the Temp → Final Move is Required

This is not an ATCR implementation detail — it's required by the OCI Distribution Specification.

The Problem: Unknown Digest

Docker doesn't know the blob's digest until after uploading:

1. Streaming data: Can't buffer 5GB layer in memory to calculate digest first
2. Stdin pipes: docker build . | docker push generates data on-the-fly
3. Chunked uploads: Multiple PATCH requests, digest calculated as data streams

The Solution: Upload to Temp, Verify, Move

All OCI registries do this:

1. Client: POST /v2/{name}/blobs/uploads/ → Get upload UUID
2. Client: PATCH /v2/{name}/blobs/uploads/{uuid} → Stream data to temp location
3. Client: PUT /v2/{name}/blobs/uploads/{uuid}?digest=sha256:abc → Provide digest
4. Registry: Verify digest matches uploaded data
5. Registry: Move uploads/{uuid} → blobs/sha256/abc123...

Docker Hub, GHCR, ECR, Harbor — all use this pattern.

Why It's Efficient with S3

For S3, the move is a CopyObject API call:

// This happens INSIDE S3 servers - no data transfer!
s3.CopyObject(&s3.CopyObjectInput{
    Bucket:     "my-bucket",
    CopySource: "/my-bucket/uploads/temp-12345/data",  // 5GB blob
    Key:        "blobs/sha256/ab/abc123.../data",
})
// S3 copies internally, hold service only sends ~1KB API request

For a 5GB layer:

• Hold service bandwidth: ~1KB (API request/response)
• S3 internal copy: Instant (metadata operation on S3 side)
• No data leaves S3, no network transfer

This is why the move operation is essentially free!

Implementation Details

1. Add S3 Client to Hold Service

File: cmd/hold/main.go

Modify HoldService struct:

type HoldService struct {
    driver       storagedriver.StorageDriver
    config       *Config
    s3Client     *s3.S3      // NEW: S3 client for presigned URLs
    bucket       string      // NEW: Bucket name
    s3PathPrefix string      // NEW: Path prefix (if any)
}

Add initialization function:

func (s *HoldService) initS3Client() error {
    if s.config.Storage.Type() != "s3" {
        log.Printf("Storage driver is %s (not S3), presigned URLs disabled", s.config.Storage.Type())
        return nil
    }

    params := s.config.Storage.Parameters()["s3"].(configuration.Parameters)

    // Build AWS config
    awsConfig := &aws.Config{
        Region:      aws.String(params["region"].(string)),
        Credentials: credentials.NewStaticCredentials(
            params["accesskey"].(string),
            params["secretkey"].(string),
            "",
        ),
    }

    // Add custom endpoint for S3-compatible services (Storj, MinIO, etc.)
    if endpoint, ok := params["regionendpoint"].(string); ok && endpoint != "" {
        awsConfig.Endpoint = aws.String(endpoint)
        awsConfig.S3ForcePathStyle = aws.Bool(true) // Required for MinIO, Storj
    }

    sess, err := session.NewSession(awsConfig)
    if err != nil {
        return fmt.Errorf("failed to create AWS session: %w", err)
    }

    s.s3Client = s3.New(sess)
    s.bucket = params["bucket"].(string)

    log.Printf("S3 presigned URLs enabled for bucket: %s", s.bucket)
    return nil
}

Call during service initialization:

func NewHoldService(cfg *Config) (*HoldService, error) {
    // ... existing driver creation ...

    service := &HoldService{
        driver: driver,
        config: cfg,
    }

    // Initialize S3 client for presigned URLs
    if err := service.initS3Client(); err != nil {
        log.Printf("WARNING: S3 presigned URLs disabled: %v", err)
    }

    return service, nil
}

2. Implement Presigned URL Generation

For Downloads:

func (s *HoldService) getDownloadURL(ctx context.Context, digest string, did string) (string, error) {
    path := blobPath(digest)

    // Check if blob exists
    if _, err := s.driver.Stat(ctx, path); err != nil {
        return "", fmt.Errorf("blob not found: %w", err)
    }

    // If S3 client available, generate presigned URL
    if s.s3Client != nil {
        s3Key := strings.TrimPrefix(path, "/")

        req, _ := s.s3Client.GetObjectRequest(&s3.GetObjectInput{
            Bucket: aws.String(s.bucket),
            Key:    aws.String(s3Key),
        })

        url, err := req.Presign(15 * time.Minute)
        if err != nil {
            log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
            return s.getProxyDownloadURL(digest, did), nil
        }

        log.Printf("Generated presigned download URL for %s (expires in 15min)", digest)
        return url, nil
    }

    // Fallback: return proxy URL
    return s.getProxyDownloadURL(digest, did), nil
}

func (s *HoldService) getProxyDownloadURL(digest, did string) string {
    return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
}

For Uploads:

func (s *HoldService) getUploadURL(ctx context.Context, digest string, size int64, did string) (string, error) {
    path := blobPath(digest)

    // If S3 client available, generate presigned URL
    if s.s3Client != nil {
        s3Key := strings.TrimPrefix(path, "/")

        req, _ := s.s3Client.PutObjectRequest(&s3.PutObjectInput{
            Bucket: aws.String(s.bucket),
            Key:    aws.String(s3Key),
        })

        url, err := req.Presign(15 * time.Minute)
        if err != nil {
            log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
            return s.getProxyUploadURL(digest, did), nil
        }

        log.Printf("Generated presigned upload URL for %s (expires in 15min)", digest)
        return url, nil
    }

    // Fallback: return proxy URL
    return s.getProxyUploadURL(digest, did), nil
}

func (s *HoldService) getProxyUploadURL(digest, did string) string {
    return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
}

3. No Changes Needed for Move Operation

The existing /move endpoint already uses driver.Move(), which for S3:

• Calls s3.CopyObject() (server-side copy)
• Calls s3.DeleteObject() (delete source)
• No data transfer through hold service!

File: cmd/hold/main.go:296 (already exists, no changes needed)

func (s *HoldService) HandleMove(w http.ResponseWriter, r *http.Request) {
    // ... existing auth and parsing ...

    sourcePath := blobPath(fromPath)  // uploads/temp-{uuid}/data
    destPath := blobPath(toDigest)    // blobs/sha256/ab/abc123.../data

    // For S3, this does CopyObject + DeleteObject (server-side)
    if err := s.driver.Move(ctx, sourcePath, destPath); err != nil {
        // ... error handling ...
    }
}

4. AppView Changes (Optional Optimization)

File: pkg/storage/proxy_blob_store.go:228

Currently streams to hold service proxy URL. Could be optimized to use presigned URL:

// In Create() - line 228
go func() {
    defer pipeReader.Close()

    tempPath := fmt.Sprintf("uploads/temp-%s", writer.id)

    // Try to get presigned URL for temp location
    url, err := p.getUploadURL(ctx, digest.FromString(tempPath), 0)
    if err != nil {
        // Fallback to direct proxy URL
        url = fmt.Sprintf("%s/blobs/%s?did=%s", p.storageEndpoint, tempPath, p.did)
    }

    req, err := http.NewRequestWithContext(uploadCtx, "PUT", url, pipeReader)
    // ... rest unchanged
}()

Note: This optimization is optional. The presigned URL will be returned by hold service's getUploadURL() anyway.

S3-Compatible Service Support

Storj

# .env file
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-storj-access-key
AWS_SECRET_ACCESS_KEY=your-storj-secret-key
S3_BUCKET=your-bucket-name
S3_REGION=global
S3_ENDPOINT=https://gateway.storjshare.io

Presigned URL example:

https://gateway.storjshare.io/your-bucket/blobs/sha256/ab/abc123.../data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Signature=...

MinIO

STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET=registry
S3_REGION=us-east-1
S3_ENDPOINT=http://minio.example.com:9000

Backblaze B2

STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-b2-key-id
AWS_SECRET_ACCESS_KEY=your-b2-application-key
S3_BUCKET=your-bucket-name
S3_REGION=us-west-002
S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com

Cloudflare R2

STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-r2-access-key-id
AWS_SECRET_ACCESS_KEY=your-r2-secret-access-key
S3_BUCKET=your-bucket-name
S3_REGION=auto
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com

All these services support presigned URLs with AWS SDK v1!

Performance Impact

Bandwidth Savings

Before (proxy mode):

• 5GB layer upload: Hold service receives 5GB, sends 5GB to S3 = 10GB bandwidth
• 5GB layer download: S3 sends 5GB to hold, hold sends 5GB to client = 10GB bandwidth
• Total for push+pull: 20GB hold service bandwidth

After (presigned URLs):

• 5GB layer upload: Hold generates URL (1KB), AppView → S3 direct (5GB), CopyObject API (1KB) = ~2KB hold bandwidth
• 5GB layer download: Hold generates URL (1KB), client → S3 direct = ~1KB hold bandwidth
• Total for push+pull: ~3KB hold service bandwidth

Savings: 99.98% reduction in hold service bandwidth!

Latency Improvements

Before:

• Download: Client → AppView → Hold → S3 → Hold → AppView → Client (4 hops)
• Upload: Client → AppView → Hold → S3 (3 hops)

After:

• Download: Client → AppView (redirect) → S3 (1 hop to data)
• Upload: Client → AppView → S3 (2 hops)
• Move: S3 internal (no network hops)

Resource Requirements

Before:

• Hold service needs bandwidth = sum of all image operations
• For 100 concurrent 1GB pushes: 100GB/s bandwidth needed
• Expensive, hard to scale

After:

• Hold service needs minimal CPU for presigned URL signing
• For 100 concurrent 1GB pushes: ~100KB/s bandwidth needed (API traffic)
• Can run on $5/month instance!

Security Considerations

Presigned URL Expiration

• Default: 15 minutes expiration
• Presigned URL includes embedded credentials in query params
• After expiry, URL becomes invalid (S3 rejects with 403)
• No long-lived URLs floating around

Authorization Flow

1. AppView validates user via ATProto OAuth
2. AppView passes DID to hold service in presigned URL request
3. Hold service validates DID (owner or crew member)
4. Hold service generates presigned URL if authorized
5. Client uses presigned URL directly with S3

Security boundary: Hold service controls who gets presigned URLs, S3 validates the URLs.

Fallback Security

If presigned URL generation fails:

• Falls back to proxy URLs (existing behavior)
• Still requires hold service authorization
• Data flows through hold service (original security model)

Testing & Validation

Verify Presigned URLs are Used

1. Check hold service logs:

docker logs atcr-hold | grep -i presigned
# Should see: "Generated presigned download/upload URL for sha256:..."

2. Monitor network traffic:

# Before: Large data transfers to/from hold service
docker stats atcr-hold

# After: Minimal network usage on hold service
docker stats atcr-hold

3. Inspect redirect responses:

# Should see 307 redirect to S3 URL
curl -v http://appview:5000/v2/alice/myapp/blobs/sha256:abc123 \
  -H "Authorization: Bearer $TOKEN"

# Look for:
# < HTTP/1.1 307 Temporary Redirect
# < Location: https://gateway.storjshare.io/...?X-Amz-Signature=...

Test Fallback Behavior

1. With filesystem driver (should use proxy URLs):

STORAGE_DRIVER=filesystem docker-compose up atcr-hold
# Logs should show: "Storage driver is filesystem (not S3), presigned URLs disabled"

2. With S3 but invalid credentials (should fall back):

AWS_ACCESS_KEY_ID=invalid docker-compose up atcr-hold
# Logs should show: "WARN: Presigned URL generation failed, falling back to proxy"

Bandwidth Monitoring

Track hold service bandwidth over time:

# Install bandwidth monitoring
docker exec atcr-hold apt-get update && apt-get install -y vnstat

# Monitor
docker exec atcr-hold vnstat -l

Expected results:

• Before: Bandwidth correlates with image operations
• After: Bandwidth stays minimal regardless of image operations

Migration Guide

For Existing ATCR Deployments

1. Update hold service code (this implementation)

2. No configuration changes needed if already using S3:

# Existing S3 config works automatically
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
S3_BUCKET=...
S3_ENDPOINT=...

3. Restart hold service:

docker-compose restart atcr-hold

4. Verify in logs:

S3 presigned URLs enabled for bucket: my-bucket

5. Test with image push/pull:

docker push atcr.io/alice/myapp:latest
docker pull atcr.io/alice/myapp:latest

6. Monitor bandwidth to confirm reduction

Rollback Plan

If issues arise:

Option 1: Disable presigned URLs via env var (if we add this feature)

PRESIGNED_URLS_ENABLED=false docker-compose restart atcr-hold

Option 2: Revert code changes to previous hold service version

The implementation has automatic fallbacks, so partial failures won't break functionality.

Future Enhancements

1. Configurable Expiration

Allow customizing presigned URL expiry:

PRESIGNED_URL_EXPIRY=30m  # Default: 15m

2. Presigned URL Caching

Cache presigned URLs for frequently accessed blobs (with shorter TTL).

3. CloudFront/CDN Integration

For downloads, use CloudFront presigned URLs instead of direct S3:

• Better global distribution
• Lower egress costs
• Faster downloads

4. Multipart Upload Support

For very large layers (>5GB), use presigned URLs with multipart upload:

• Generate presigned URLs for each part
• Client uploads parts directly to S3
• Hold service finalizes multipart upload

5. Metrics & Monitoring

Track presigned URL usage:

• Count of presigned URLs generated
• Fallback rate (proxy vs presigned)
• Bandwidth savings metrics

References

• OCI Distribution Specification - Push
• AWS SDK Go v1 - Presigned URLs
• Storj - Using Presigned URLs
• MinIO - Presigned Upload via Browser
• Cloudflare R2 - Presigned URLs
• Backblaze B2 - S3 Compatible API

Summary

Implementing S3 presigned URLs transforms ATCR's hold service from a data proxy to a lightweight orchestrator:

✅ 99.98% bandwidth reduction for hold service ✅ Direct client → S3 transfers for maximum speed ✅ Works with all S3-compatible services (Storj, MinIO, R2, B2) ✅ OCI-compliant temp → final move pattern ✅ Automatic fallback to proxy mode for non-S3 drivers ✅ No breaking changes to existing deployments

This makes BYOS (Bring Your Own Storage) truly scalable and cost-effective, as users can run hold services on minimal infrastructure while serving arbitrarily large container images.