at-container-registry/docs/PRESIGNED_UPLOADS.md

Presigned Upload URLs Implementation Guide

Current Architecture (Proxy Mode)

Upload Flow Today

1. AppView receives blob upload request from Docker
2. ProxyBlobStore.Create() creates streaming upload via pipe
3. Data streams to Hold Service temp location: uploads/temp-{id}
4. Hold service uploads to S3 via storage driver
5. ProxyBlobWriter.Commit() moves blob: temp → final digest-based path
6. Hold service performs S3 Move operation

Why Uploads Don't Use Presigned URLs Today

• Create() doesn't know the blob digest upfront
• Presigned S3 URLs require the full object key (which includes digest)
• Current approach streams to temp location, calculates digest, then moves

Bandwidth Flow (Current)

Docker → AppView → Hold Service → S3/Storj
         (proxy)   (proxy)

All upload bandwidth flows through Hold Service.

---

Proposed Architecture (Presigned Uploads)

New Upload Flow

1. AppView receives blob upload request from Docker
2. ProxyBlobStore.Create() creates buffered upload writer
3. Data buffered in memory during Write() calls
4. ProxyBlobWriter.Commit() calculates digest from buffer
5. Request presigned PUT URL from Hold Service with digest
6. Upload buffered data directly to S3 via presigned URL
7. No move operation needed (uploaded to final path)

Bandwidth Flow (Presigned)

Docker → AppView → S3/Storj (direct via presigned URL)
         (buffer)

Hold Service only issues presigned URLs (minimal bandwidth)

---

Detailed Implementation

Phase 1: Add Buffering to ProxyBlobWriter

File: pkg/storage/proxy_blob_store.go

Changes to ProxyBlobWriter struct

type ProxyBlobWriter struct {
    store       *ProxyBlobStore
    options     distribution.CreateOptions

    // Remove pipe-based streaming
    // pipeWriter  *io.PipeWriter
    // pipeReader  *io.PipeReader
    // digestChan  chan string
    // uploadErr   chan error

    // Add buffering
    buffer      *bytes.Buffer  // In-memory buffer for blob data
    hasher      digest.Digester // Calculate digest while writing

    finalDigest string
    size        int64
    closed      bool
    id          string
    startedAt   time.Time
}

Rationale:

• Remove pipe mechanism (no longer streaming to temp)
• Add buffer to store blob data in memory
• Add hasher to calculate digest incrementally

Modify Create() method

Before (lines 208-312):

func (p *ProxyBlobStore) Create(ctx context.Context, options ...distribution.BlobCreateOption) (distribution.BlobWriter, error) {
    // Creates pipe and starts background goroutine for streaming
    pipeReader, pipeWriter := io.Pipe()
    // ... streams to temp location
}

After:

func (p *ProxyBlobStore) Create(ctx context.Context, options ...distribution.BlobCreateOption) (distribution.BlobWriter, error) {
    fmt.Printf("🔧 [proxy_blob_store/Create] Starting buffered upload for presigned URL\n")

    // Parse options
    var opts distribution.CreateOptions
    for _, option := range options {
        if err := option.Apply(&opts); err != nil {
            return nil, err
        }
    }

    // Create buffered writer
    writer := &ProxyBlobWriter{
        store:     p,
        options:   opts,
        buffer:    new(bytes.Buffer),
        hasher:    digest.Canonical.Digester(), // Usually SHA256
        id:        fmt.Sprintf("upload-%d", time.Now().UnixNano()),
        startedAt: time.Now(),
    }

    // Store in global uploads map for resume support
    globalUploadsMu.Lock()
    globalUploads[writer.id] = writer
    globalUploadsMu.Unlock()

    fmt.Printf("   Upload ID: %s\n", writer.id)
    fmt.Printf("   Repository: %s\n", p.repository)

    return writer, nil
}

Key Changes:

• No more pipe creation
• No background goroutine
• Initialize buffer and hasher
• Everything else stays synchronous

Modify Write() method

Before (lines 440-455):

func (w *ProxyBlobWriter) Write(p []byte) (int, error) {
    // Writes to pipe, streams to hold service
    n, err := w.pipeWriter.Write(p)
    w.size += int64(n)
    return n, nil
}

After:

func (w *ProxyBlobWriter) Write(p []byte) (int, error) {
    if w.closed {
        return 0, fmt.Errorf("writer closed")
    }

    // Write to buffer
    n, err := w.buffer.Write(p)
    if err != nil {
        return n, fmt.Errorf("failed to buffer data: %w", err)
    }

    // Update hasher for digest calculation
    w.hasher.Hash().Write(p)

    w.size += int64(n)

    // Memory pressure check (optional safety)
    if w.buffer.Len() > 500*1024*1024 { // 500MB limit
        return n, fmt.Errorf("blob too large for buffered upload: %d bytes", w.buffer.Len())
    }

    return n, nil
}

Key Changes:

• Write to in-memory buffer instead of pipe
• Update hasher incrementally (efficient)
• Add safety check for excessive memory usage
• No streaming to hold service yet

Modify Commit() method

Before (lines 493-548):

func (w *ProxyBlobWriter) Commit(ctx context.Context, desc distribution.Descriptor) (distribution.Descriptor, error) {
    // Close pipe, send digest to goroutine
    // Wait for temp upload
    // Move temp → final
}

After:

func (w *ProxyBlobWriter) Commit(ctx context.Context, desc distribution.Descriptor) (distribution.Descriptor, error) {
    if w.closed {
        return distribution.Descriptor{}, fmt.Errorf("writer closed")
    }
    w.closed = true

    // Remove from global uploads map
    globalUploadsMu.Lock()
    delete(globalUploads, w.id)
    globalUploadsMu.Unlock()

    // Calculate digest from buffered data
    calculatedDigest := w.hasher.Digest()

    // Verify digest matches if provided
    if desc.Digest != "" && desc.Digest != calculatedDigest {
        return distribution.Descriptor{}, fmt.Errorf(
            "digest mismatch: expected %s, got %s",
            desc.Digest, calculatedDigest,
        )
    }

    finalDigest := calculatedDigest
    if desc.Digest != "" {
        finalDigest = desc.Digest
    }

    fmt.Printf("📤 [ProxyBlobWriter.Commit] Uploading via presigned URL\n")
    fmt.Printf("   Digest: %s\n", finalDigest)
    fmt.Printf("   Size: %d bytes\n", w.size)
    fmt.Printf("   Buffered: %d bytes\n", w.buffer.Len())

    // Get presigned upload URL from hold service
    url, err := w.store.getUploadURL(ctx, finalDigest, w.size)
    if err != nil {
        return distribution.Descriptor{}, fmt.Errorf("failed to get presigned upload URL: %w", err)
    }

    fmt.Printf("   Presigned URL: %s\n", url)

    // Upload directly to S3 via presigned URL
    req, err := http.NewRequestWithContext(ctx, "PUT", url, bytes.NewReader(w.buffer.Bytes()))
    if err != nil {
        return distribution.Descriptor{}, fmt.Errorf("failed to create upload request: %w", err)
    }
    req.Header.Set("Content-Type", "application/octet-stream")
    req.ContentLength = w.size

    resp, err := w.store.httpClient.Do(req)
    if err != nil {
        return distribution.Descriptor{}, fmt.Errorf("presigned upload failed: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusCreated {
        bodyBytes, _ := io.ReadAll(resp.Body)
        return distribution.Descriptor{}, fmt.Errorf(
            "presigned upload failed: status %d, body: %s",
            resp.StatusCode, string(bodyBytes),
        )
    }

    fmt.Printf("✅ [ProxyBlobWriter.Commit] Upload successful\n")

    // Clear buffer to free memory
    w.buffer = nil

    return distribution.Descriptor{
        Digest:    finalDigest,
        Size:      w.size,
        MediaType: desc.MediaType,
    }, nil
}

Key Changes:

• Calculate digest from hasher (already computed incrementally)
• Verify digest if provided by client
• Get presigned upload URL with final digest
• Upload buffer contents directly to S3
• No temp location, no move operation
• Clear buffer to free memory immediately

Modify Cancel() method

Before (lines 551-572):

func (w *ProxyBlobWriter) Cancel(ctx context.Context) error {
    // Close pipe, cancel temp upload
}

After:

func (w *ProxyBlobWriter) Cancel(ctx context.Context) error {
    w.closed = true

    // Remove from global uploads map
    globalUploadsMu.Lock()
    delete(globalUploads, w.id)
    globalUploadsMu.Unlock()

    // Clear buffer to free memory
    w.buffer = nil

    fmt.Printf("[ProxyBlobWriter.Cancel] Upload cancelled: id=%s\n", w.id)
    return nil
}

Key Changes:

• Simply clear buffer
• No pipe cleanup needed
• No temp cleanup needed (nothing uploaded yet)

---

Phase 2: Update Hold Service (Optional Enhancement)

The current getUploadURL() implementation in cmd/hold/main.go (lines 528-587) already supports presigned uploads correctly. No changes needed unless you want to add additional logging.

Optional logging enhancement at line 547:

url, err := req.Presign(15 * time.Minute)
if err != nil {
    log.Printf("Failed to generate presigned upload URL: %v", err)
    return s.getProxyUploadURL(digest, did), nil
}

log.Printf("🔑 Generated presigned upload URL:")
log.Printf("   Digest: %s", digest)
log.Printf("   S3 Key: %s", s3Key)
log.Printf("   Size: %d bytes", size)
log.Printf("   URL length: %d chars", len(url))
log.Printf("   Expires: 15min")

return url, nil

---

Phase 3: Memory Management Considerations

Add Configuration for Max Buffer Size

File: pkg/storage/proxy_blob_store.go

Add constants at top of file:

const (
    maxChunkSize = 5 * 1024 * 1024 // 5MB (existing)

    // Maximum blob size for in-memory buffering
    // Blobs larger than this will fail (alternative: fallback to proxy mode)
    maxBufferedBlobSize = 500 * 1024 * 1024 // 500MB
)

Alternative: Disk-Based Buffering

For very large blobs, consider disk-based buffering:

type ProxyBlobWriter struct {
    // ... existing fields ...

    // Choose one:
    buffer     *bytes.Buffer      // Memory buffer (current)
    // OR
    tempFile   *os.File           // Disk buffer (for large blobs)
    bufferSize int64
}

Memory buffer (simple, fast):

• Pro: Fast, no disk I/O
• Con: Limited by available RAM
• Use for: Blobs < 500MB

Disk buffer (scalable):

• Pro: No memory limit
• Con: Slower, disk I/O overhead
• Use for: Blobs > 500MB

Hybrid Approach (Recommended)

const (
    memoryBufferThreshold = 50 * 1024 * 1024  // 50MB
)

func (w *ProxyBlobWriter) Write(p []byte) (int, error) {
    // If buffer exceeds threshold, switch to disk
    if w.buffer != nil && w.buffer.Len() > memoryBufferThreshold {
        return 0, fmt.Errorf("blob exceeds memory buffer threshold, disk buffering not implemented")
        // TODO: Implement disk buffering or fallback to proxy mode
    }

    // Otherwise use memory buffer
    // ... existing Write() logic ...
}

---

Optional Enhancement: Presigned HEAD URLs

Motivation

Currently HEAD requests (blob verification) are proxied through the Hold Service. This is fine because HEAD bandwidth is negligible (~300 bytes per request), but we can eliminate this round-trip by using presigned HEAD URLs.

Implementation

Step 1: Add getHeadURL() to Hold Service

File: cmd/hold/main.go

Add new function after getDownloadURL():

// getHeadURL generates a presigned HEAD URL for blob verification
func (s *HoldService) getHeadURL(ctx context.Context, digest string) (string, error) {
    // Check if blob exists first
    path := blobPath(digest)
    _, err := s.driver.Stat(ctx, path)
    if err != nil {
        return "", fmt.Errorf("blob not found: %w", err)
    }

    // If S3 client available, generate presigned HEAD URL
    if s.s3Client != nil {
        s3Key := strings.TrimPrefix(path, "/")
        if s.s3PathPrefix != "" {
            s3Key = s.s3PathPrefix + "/" + s3Key
        }

        // Generate presigned HEAD URL (method-specific!)
        req, _ := s.s3Client.HeadObjectRequest(&s3.HeadObjectInput{
            Bucket: aws.String(s.bucket),
            Key:    aws.String(s3Key),
        })

        log.Printf("🔍 [getHeadURL] Generating presigned HEAD URL:")
        log.Printf("   Digest: %s", digest)
        log.Printf("   S3 Key: %s", s3Key)

        url, err := req.Presign(15 * time.Minute)
        if err != nil {
            log.Printf("[getHeadURL] Presign failed: %v", err)
            // Fallback to proxy URL
            return s.getProxyHeadURL(digest), nil
        }

        log.Printf("✅ [getHeadURL] Presigned HEAD URL generated")
        return url, nil
    }

    // Fallback: return proxy URL
    return s.getProxyHeadURL(digest), nil
}

// getProxyHeadURL returns a proxy URL for HEAD requests
func (s *HoldService) getProxyHeadURL(digest string) string {
    // HEAD requests don't need DID in query string (read-only check)
    return fmt.Sprintf("%s/blobs/%s", s.config.Server.PublicURL, digest)
}

Step 2: Add HTTP endpoint for presigned HEAD URLs

File: cmd/hold/main.go

Add handler similar to HandleGetPresignedURL():

// HeadPresignedURLRequest represents a request for a presigned HEAD URL
type HeadPresignedURLRequest struct {
    DID    string `json:"did"`
    Digest string `json:"digest"`
}

// HeadPresignedURLResponse contains the presigned HEAD URL
type HeadPresignedURLResponse struct {
    URL       string    `json:"url"`
    ExpiresAt time.Time `json:"expires_at"`
}

// HandleHeadPresignedURL handles requests for HEAD URLs
func (s *HoldService) HandleHeadPresignedURL(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
        return
    }

    var req HeadPresignedURLRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, fmt.Sprintf("invalid request: %v", err), http.StatusBadRequest)
        return
    }

    // Validate DID authorization for READ
    if !s.isAuthorizedRead(req.DID) {
        if req.DID == "" {
            http.Error(w, "unauthorized: authentication required", http.StatusUnauthorized)
        } else {
            http.Error(w, "forbidden: access denied", http.StatusForbidden)
        }
        return
    }

    // Generate presigned HEAD URL
    ctx := context.Background()
    expiry := time.Now().Add(15 * time.Minute)

    url, err := s.getHeadURL(ctx, req.Digest)
    if err != nil {
        http.Error(w, fmt.Sprintf("failed to generate URL: %v", err), http.StatusInternalServerError)
        return
    }

    resp := HeadPresignedURLResponse{
        URL:       url,
        ExpiresAt: expiry,
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(resp)
}

Step 3: Register endpoint in main()

File: cmd/hold/main.go

In main() function, add route:

mux.HandleFunc("/head-presigned-url", service.HandleHeadPresignedURL)

Step 4: Update ProxyBlobStore.ServeBlob()

File: pkg/storage/proxy_blob_store.go

Modify HEAD handling (currently lines 197-224):

Before:

if r.Method == http.MethodHead {
    // Check if blob exists via hold service HEAD request
    url := fmt.Sprintf("%s/blobs/%s?did=%s", p.storageEndpoint, dgst.String(), p.did)
    req, err := http.NewRequestWithContext(ctx, "HEAD", url, nil)
    // ... proxy through hold service ...
}

After:

if r.Method == http.MethodHead {
    // Get presigned HEAD URL from hold service
    headURL, err := p.getHeadURL(ctx, dgst)
    if err != nil {
        return distribution.ErrBlobUnknown
    }

    // Redirect to presigned HEAD URL
    http.Redirect(w, r, headURL, http.StatusTemporaryRedirect)
    return nil
}

Step 5: Add getHeadURL() to ProxyBlobStore

File: pkg/storage/proxy_blob_store.go

Add after getDownloadURL():

// getHeadURL requests a presigned HEAD URL from the storage service
func (p *ProxyBlobStore) getHeadURL(ctx context.Context, dgst digest.Digest) (string, error) {
    reqBody := map[string]any{
        "did":    p.did,
        "digest": dgst.String(),
    }

    body, err := json.Marshal(reqBody)
    if err != nil {
        return "", err
    }

    url := fmt.Sprintf("%s/head-presigned-url", p.storageEndpoint)
    req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(body))
    if err != nil {
        return "", err
    }
    req.Header.Set("Content-Type", "application/json")

    resp, err := p.httpClient.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        return "", fmt.Errorf("failed to get HEAD URL: status %d", resp.StatusCode)
    }

    var result struct {
        URL string `json:"url"`
    }
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return "", err
    }

    return result.URL, nil
}

Presigned HEAD URLs: Trade-offs

Benefits:

• Offloads HEAD requests from Hold Service
• Docker verifies blobs directly against S3
• Slightly lower latency (one fewer hop)

Costs:

• Requires round-trip to get presigned HEAD URL
• More complex code
• Two HTTP requests instead of one proxy request

Bandwidth Analysis:

• Current: 1 HEAD request to Hold Service (~300 bytes)
• Presigned: 1 POST to get URL (~200 bytes) + 1 HEAD to S3 (~300 bytes)
• Net difference: Adds ~200 bytes per verification

Recommendation: Optional enhancement. The current proxied HEAD approach is simpler and bandwidth difference is negligible. Only implement if:

• Hold Service is becoming a bottleneck
• You want to minimize Hold Service load completely
• Latency of HEAD requests becomes noticeable

---

Testing & Validation

Test Plan for Presigned Uploads

1. Small Blob Upload (< 1MB)

# Build test image with small layers
echo "FROM scratch" > Dockerfile
echo "COPY small-file /" >> Dockerfile
dd if=/dev/urandom of=small-file bs=1024 count=512  # 512KB

docker build -t atcr.io/youruser/test:small .
docker push atcr.io/youruser/test:small

Expected behavior:

• Blob buffered in memory
• Presigned upload URL requested with correct digest
• Direct upload to S3 via presigned URL
• No temp location, no move operation

Verify in logs:

📤 [ProxyBlobWriter.Commit] Uploading via presigned URL
   Digest: sha256:...
   Size: 524288 bytes
   Presigned URL: https://gateway.storjshare.io/...
✅ [ProxyBlobWriter.Commit] Upload successful

2. Medium Blob Upload (10-50MB)

dd if=/dev/urandom of=medium-file bs=1048576 count=25  # 25MB

docker build -t atcr.io/youruser/test:medium .
docker push atcr.io/youruser/test:medium

Monitor memory usage:

# While push is running
docker stats atcr-appview

Should see ~25MB spike during buffer + upload.

3. Large Blob Upload (100-500MB)

dd if=/dev/urandom of=large-file bs=1048576 count=200  # 200MB

docker build -t atcr.io/youruser/test:large .
docker push atcr.io/youruser/test:large

Monitor:

• Memory usage (should see ~200MB spike)
• Upload completes successfully
• S3 shows blob in correct location

4. Concurrent Uploads

# Push multiple images in parallel
docker push atcr.io/youruser/test1:tag &
docker push atcr.io/youruser/test2:tag &
docker push atcr.io/youruser/test3:tag &
wait

Verify:

• All uploads complete successfully
• Memory usage peaks but doesn't OOM
• No data corruption (digests match)

5. Error Handling Tests

Test presigned URL failure:

• Temporarily break S3 credentials
• Verify graceful error message
• Check for memory leaks (buffer cleared on error)

Test digest mismatch:

• This shouldn't happen in practice, but verify error handling
• Buffer should be cleared even on error

Test network interruption:

• Kill network during upload
• Verify proper error propagation
• Check for hanging goroutines

Test Plan for Presigned HEAD URLs (Optional)

1. HEAD Request Redirect

# Pull image (triggers HEAD verification)
docker pull atcr.io/youruser/test:tag

Expected behavior:

• AppView redirects HEAD to presigned HEAD URL
• Docker follows redirect to S3
• S3 responds to HEAD request successfully

Verify in logs:

🔍 [getHeadURL] Generating presigned HEAD URL:
   Digest: sha256:...
✅ [getHeadURL] Presigned HEAD URL generated

2. Method Verification

# Manually verify presigned HEAD URL works
curl -I "presigned-head-url-here"

Should return 200 OK with Content-Length header.

# Verify it ONLY works with HEAD (not GET)
curl "presigned-head-url-here"

Should return 403 Forbidden (method mismatch).

---

Performance Comparison

Current Architecture (Proxy Mode)

Upload:

Client → AppView (stream) → Hold Service (stream) → S3
         ~0ms delay        ~0ms delay             ~100ms

• Total latency: ~100ms + upload time
• Bandwidth: All through Hold Service

Download:

Client → AppView (redirect) → S3 (presigned GET)
         ~5ms               ~50ms

• Total latency: ~55ms + download time
• Bandwidth: Direct from S3 ✅

Verification (HEAD):

Client → AppView (redirect) → Hold Service (proxy HEAD) → S3
         ~5ms                 ~10ms                      ~50ms

• Total latency: ~65ms
• Bandwidth: ~300 bytes through Hold Service

Presigned Upload Architecture

Upload:

Client → AppView (buffer) → S3 (presigned PUT)
         ~0ms              ~100ms

• Total latency: ~100ms + upload time (same)
• Bandwidth: Direct to S3 ✅
• Memory: +blob_size during buffer

Download: (unchanged)

Client → AppView (redirect) → S3 (presigned GET)

Verification (HEAD): (if presigned HEAD enabled)

Client → AppView (redirect) → S3 (presigned HEAD)
         ~5ms                ~50ms

• Total latency: ~55ms (10ms faster)
• Bandwidth: Direct to S3 ✅

---

Trade-offs Summary

Presigned Uploads

Aspect	Proxy Mode (Current)	Presigned URLs
Upload Bandwidth	Through Hold Service	Direct to S3 ✅
Hold Service Load	High (all upload traffic)	Low (only URL generation) ✅
Memory Usage	Low (streaming)	High (buffering) ⚠️
Disk Usage	None	Optional temp files for large blobs
Code Complexity	Simple ✅	Moderate
Max Blob Size	Unlimited ✅	Limited by memory (~500MB) ⚠️
Latency	Same	Same
Error Recovery	Simple (cancel stream)	More complex (clear buffer)

Presigned HEAD URLs

Aspect	Proxy Mode (Current)	Presigned HEAD
Bandwidth	300 bytes (negligible)	500 bytes (still negligible)
Hold Service Load	Low (HEAD is tiny)	Lower (but minimal gain)
Latency	65ms	55ms (10ms faster)
Code Complexity	Simple ✅	More complex
Reliability	High (fewer moving parts) ✅	Moderate (more failure modes)

---

Recommendations

Presigned Uploads

Implement if:

• ✅ Hold Service bandwidth is a concern
• ✅ You want to minimize Hold Service load
• ✅ Most blobs are < 100MB (typical Docker layers)
• ✅ AppView has sufficient memory (2-4GB+ RAM)

Skip if:

• ⚠️ Memory is constrained
• ⚠️ You regularly push very large layers (> 500MB)
• ⚠️ Current proxy mode is working fine
• ⚠️ Simplicity is priority

Presigned HEAD URLs

Implement if:

• ✅ You want complete S3 offloading
• ✅ You're already implementing presigned uploads
• ✅ Hold Service is CPU/bandwidth constrained

Skip if:

• ⚠️ Current HEAD proxying works fine (it does)
• ⚠️ You want to minimize code complexity
• ⚠️ 10ms latency difference doesn't matter

Suggested Approach

Phase 1: Implement presigned uploads first

• Bigger performance win (offloads upload bandwidth)
• More valuable for write-heavy workflows
• Test thoroughly with various blob sizes

Phase 2: Monitor and evaluate

• Check Hold Service load after presigned uploads
• Measure HEAD request impact
• Assess if presigned HEAD is worth the complexity

Phase 3: Optionally add presigned HEAD

• Only if Hold Service is still bottlenecked
• Or if you want feature completeness

---

Migration Path

Step 1: Feature Flag

Add configuration option to enable/disable presigned uploads:

// In AppView config
type Config struct {
    // ... existing fields ...

    UsePresignedUploads bool `yaml:"use_presigned_uploads"` // Default: false
}

Step 2: Gradual Rollout

1. Deploy with use_presigned_uploads: false (current behavior)
2. Test in staging with use_presigned_uploads: true
3. Roll out to production incrementally
4. Monitor memory usage and error rates

Step 3: Fallback Mechanism

If presigned upload fails, fallback to proxy mode:

func (w *ProxyBlobWriter) Commit(...) {
    // Try presigned upload
    url, err := w.store.getUploadURL(ctx, finalDigest, w.size)
    if err != nil {
        // Fallback: use proxy mode
        log.Printf("⚠️  Presigned upload unavailable, falling back to proxy")
        return w.proxyUpload(ctx, desc)
    }
    // ... presigned upload ...
}

---

Appendix: Memory Profiling

To monitor memory usage during development:

# Enable Go memory profiling
go tool pprof http://localhost:5000/debug/pprof/heap

# Or use runtime metrics
import "runtime"

var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf("Alloc = %v MB", m.Alloc / 1024 / 1024)

Monitor these metrics:

• Alloc: Current memory allocation
• TotalAlloc: Cumulative allocation (detect leaks)
• Sys: Total memory from OS
• NumGC: Garbage collection count

Expected behavior with presigned uploads:

• Memory spikes during Write() calls
• Memory drops after Commit() completes
• No memory leaks (TotalAlloc should plateau)

---

Questions for Decision

Before implementing, answer:

1. What's the typical size of your Docker layers?

   • < 50MB: Presigned uploads perfect fit
   • 50-200MB: Acceptable with memory monitoring

   • 200MB: Consider disk buffering or stick with proxy

2. What's your AppView's available memory?

   • 1GB: Skip presigned uploads
   • 2-4GB: Fine for typical workloads
   • 8GB+: No concerns

3. Is Hold Service bandwidth currently a problem?

   • No: Current proxy mode is fine
   • Yes: Presigned uploads will help significantly

4. How important is code simplicity?

   • Very: Stick with proxy mode
   • Moderate: Implement presigned uploads only
   • Low: Implement both presigned uploads and HEAD

5. What's your deployment model?

   • Single Hold Service: Bandwidth matters more
   • Multiple Hold Services: Less critical

---

Implementation Checklist

Presigned Uploads

• Modify ProxyBlobWriter struct (remove pipe, add buffer/hasher)
• Update Create() to initialize buffer
• Update Write() to buffer + hash data
• Update Commit() to upload via presigned URL
• Update Cancel() to clear buffer
• Add memory usage monitoring
• Add configuration flag
• Test with small blobs (< 1MB)
• Test with medium blobs (10-50MB)
• Test with large blobs (100-500MB)
• Test concurrent uploads
• Test error scenarios
• Update documentation
• Deploy to staging
• Monitor production rollout

Presigned HEAD URLs (Optional)

• Add getHeadURL() to Hold Service
• Add HandleHeadPresignedURL() endpoint
• Register /head-presigned-url route
• Add getHeadURL() to ProxyBlobStore
• Update ServeBlob() to redirect HEAD requests
• Test HEAD redirects
• Verify method-specific signatures
• Test with Docker pull operations
• Deploy to staging
• Monitor production rollout