refactor appview to use envvars. move distribution configurations to code

This commit is contained in:
Evan Jarrett
2025-10-08 23:29:51 -05:00
parent fb040a72d3
commit b1e6857381
9 changed files with 1042 additions and 116 deletions
+90
View File
@@ -0,0 +1,90 @@
# ATCR AppView Configuration
# Copy this file to .env.appview and fill in your values
# Load with: source .env.appview && ./bin/atcr-appview serve
# ==============================================================================
# Server Configuration
# ==============================================================================
# HTTP listen address (default: :5000)
ATCR_HTTP_ADDR=:5000
# Debug listen address (default: :5001)
# ATCR_DEBUG_ADDR=:5001
# Base URL for the AppView service (REQUIRED for production)
# Used to generate OAuth redirect URIs and JWT realms
# Development: Auto-detected from ATCR_HTTP_ADDR (e.g., http://127.0.0.1:5000)
# Production: Set to your public URL (e.g., https://atcr.io)
# ATCR_BASE_URL=http://127.0.0.1:5000
# Service name (used for JWT service/issuer fields)
# Default: Derived from base URL hostname, or "atcr.io"
# ATCR_SERVICE_NAME=atcr.io
# ==============================================================================
# Storage Configuration
# ==============================================================================
# Default hold service endpoint for users without their own storage (REQUIRED)
# Users with a sailor profile defaultHold setting will override this
# Docker: Use container name (http://atcr-hold:8080)
# Local dev: Use localhost (http://127.0.0.1:8080)
ATCR_DEFAULT_HOLD=http://127.0.0.1:8080
# ==============================================================================
# Authentication Configuration
# ==============================================================================
# Path to JWT signing private key (auto-generated if missing)
# Default: /var/lib/atcr/auth/private-key.pem
# ATCR_AUTH_KEY_PATH=/var/lib/atcr/auth/private-key.pem
# Path to JWT signing certificate (auto-generated if missing)
# Default: /var/lib/atcr/auth/private-key.crt
# ATCR_AUTH_CERT_PATH=/var/lib/atcr/auth/private-key.crt
# JWT token expiration in seconds (default: 300 = 5 minutes)
# ATCR_TOKEN_EXPIRATION=300
# ==============================================================================
# UI Configuration
# ==============================================================================
# Enable web UI (default: true)
# Set to "false" to disable web interface and run registry-only
ATCR_UI_ENABLED=true
# SQLite database path for UI data (sessions, stars, pull counts, etc.)
# Default: /var/lib/atcr/ui.db
# ATCR_UI_DATABASE_PATH=/var/lib/atcr/ui.db
# ==============================================================================
# Logging Configuration
# ==============================================================================
# Log level: debug, info, warn, error (default: info)
# ATCR_LOG_LEVEL=info
# Log formatter: text, json (default: text)
# ATCR_LOG_FORMATTER=text
# ==============================================================================
# Jetstream Configuration (ATProto event streaming)
# ==============================================================================
# Jetstream WebSocket URL for real-time ATProto events
# Default: wss://jetstream2.us-west.bsky.network/subscribe
# JETSTREAM_URL=wss://jetstream2.us-west.bsky.network/subscribe
# Enable backfill worker to sync historical records (default: false)
# Set to "true" to enable periodic syncing of ATProto records
# ATCR_BACKFILL_ENABLED=true
# ATProto relay endpoint for backfill sync API
# Default: https://relay1.us-east.bsky.network
# ATCR_RELAY_ENDPOINT=https://relay1.us-east.bsky.network
# Backfill interval (default: 1h)
# Examples: 30m, 1h, 2h, 24h
# ATCR_BACKFILL_INTERVAL=1h
+48 -23
View File
@@ -31,17 +31,26 @@ docker build -f Dockerfile.hold -t atcr.io/hold:latest .
# Or use docker-compose
docker-compose up -d
# Run locally (AppView)
export ATPROTO_DID=did:plc:your-did
export ATPROTO_ACCESS_TOKEN=your-token
./atcr-appview serve config/config.yml
# Run locally (AppView) - configure via env vars (see .env.appview.example)
export ATCR_HTTP_ADDR=:5000
export ATCR_DEFAULT_HOLD=http://127.0.0.1:8080
./bin/atcr-appview serve
# Run hold service (configure via env vars - see .env.example)
# Or use .env file:
cp .env.appview.example .env.appview
# Edit .env.appview with your settings
source .env.appview
./bin/atcr-appview serve
# Legacy mode (still supported):
# ./bin/atcr-appview serve config/config.yml
# Run hold service (configure via env vars - see .env.hold.example)
export HOLD_PUBLIC_URL=http://127.0.0.1:8080
export STORAGE_DRIVER=filesystem
export STORAGE_ROOT_DIR=/tmp/atcr-hold
export HOLD_OWNER=did:plc:your-did-here
./atcr-hold
./bin/atcr-hold
# Check logs for OAuth URL, visit in browser to complete registration
```
@@ -433,29 +442,45 @@ This ensures:
### Configuration
**AppView configuration** (`config/config.yml`):
- S3 bucket settings under `storage.s3`
- ATProto middleware under `middleware.repository`
- Name resolver under `middleware.registry`
- Default storage endpoint: `middleware.registry.options.default_storage_endpoint`
- Auth token signing keys and expiration
- Database path: `db.path` (SQLite database location)
- Jetstream endpoint: `jetstream.endpoint` (for ATProto event streaming)
**AppView configuration** (environment variables):
Both AppView and Hold service follow the same pattern: **zero config files, all configuration via environment variables**.
See `.env.appview.example` for all available options. Key environment variables:
**Server:**
- `ATCR_HTTP_ADDR` - HTTP listen address (default: `:5000`)
- `ATCR_BASE_URL` - Public URL for OAuth/JWT realm (auto-detected in dev)
- `ATCR_DEFAULT_HOLD` - Default hold endpoint for blob storage (REQUIRED)
**Authentication:**
- `ATCR_AUTH_KEY_PATH` - JWT signing key path (default: `/var/lib/atcr/auth/private-key.pem`)
- `ATCR_TOKEN_EXPIRATION` - JWT expiration in seconds (default: 300)
**UI:**
- `ATCR_UI_ENABLED` - Enable web interface (default: true)
- `ATCR_UI_DATABASE_PATH` - SQLite database path (default: `/var/lib/atcr/ui.db`)
**Jetstream:**
- `JETSTREAM_URL` - ATProto event stream URL
- `ATCR_BACKFILL_ENABLED` - Enable periodic sync (default: false)
**Legacy:** `config/config.yml` is still supported but deprecated. Use environment variables instead.
**Hold Service configuration** (environment variables):
- Storage driver config via env vars: `STORAGE_DRIVER`, `AWS_*`, `S3_*`
- Authorization: Based on PDS records (`hold.public`, crew records)
- Server settings: `HOLD_SERVER_ADDR`, `HOLD_PUBLIC_URL`, `HOLD_PUBLIC`
- Auto-registration: `HOLD_OWNER` (optional)
See `.env.hold.example` for all available options. Key environment variables:
- `HOLD_PUBLIC_URL` - Public URL of hold service (REQUIRED)
- `STORAGE_DRIVER` - Storage backend (s3, filesystem)
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` - S3 credentials
- `S3_BUCKET`, `S3_ENDPOINT` - S3 configuration
- `HOLD_PUBLIC` - Allow public reads (default: false)
- `HOLD_OWNER` - DID for auto-registration (optional)
**Credential Helper**:
- Token storage: `~/.atcr/oauth-token.json`
- Contains: access token, refresh token, DPoP key (PEM), DID, handle
Environment variables:
- `ATPROTO_DID`: DID for authentication with PDS (AppView only)
- `ATPROTO_ACCESS_TOKEN`: Access token for PDS operations (AppView only)
### Development Notes
**General:**
@@ -524,7 +549,7 @@ When writing tests:
- Queries in `pkg/appview/db/queries.go`
- Stores for OAuth, devices, sessions in separate files
- Run migrations automatically on startup
- Database path configurable via config.yml
- Database path configurable via `ATCR_UI_DATABASE_PATH` env var
**Adding web UI features**:
- Add handler in `pkg/appview/handlers/`
+2 -8
View File
@@ -31,18 +31,12 @@ WORKDIR /app
# Copy binary from builder
COPY --from=builder /build/atcr-appview .
# Copy default configuration
COPY config/config.yml /etc/atcr/config.yml
# Create directories for storage
RUN mkdir -p /var/lib/atcr/blobs /var/lib/atcr/auth
# Expose ports
EXPOSE 5000 5001
# Set environment variables
ENV ATCR_CONFIG=/etc/atcr/config.yml
# OCI image annotations
LABEL org.opencontainers.image.title="ATCR AppView" \
org.opencontainers.image.description="ATProto Container Registry - OCI-compliant registry using AT Protocol for manifest storage" \
@@ -53,6 +47,6 @@ LABEL org.opencontainers.image.title="ATCR AppView" \
org.opencontainers.image.version="0.1.0" \
io.atcr.icon="https://imgs.blue/evan.jarrett.net/1TpTNrRelfloN2emuWZDrWmPT0o93bAjEnozjD6UPgoVV9m4"
# Run the AppView
# Run the AppView (no config file - uses environment variables)
ENTRYPOINT ["/app/atcr-appview"]
CMD ["serve", "/etc/atcr/config.yml"]
CMD ["serve"]
+213
View File
@@ -0,0 +1,213 @@
package main
import (
"fmt"
"net/url"
"os"
"strconv"
"time"
"github.com/distribution/distribution/v3/configuration"
)
// loadConfigFromEnv builds a complete configuration from environment variables
// This follows the same pattern as the hold service (no config files, only env vars)
func loadConfigFromEnv() (*configuration.Configuration, error) {
config := &configuration.Configuration{}
// Version
config.Version = configuration.MajorMinorVersion(0, 1)
// Logging
config.Log = buildLogConfig()
// HTTP server
httpConfig, err := buildHTTPConfig()
if err != nil {
return nil, fmt.Errorf("failed to build HTTP config: %w", err)
}
config.HTTP = httpConfig
// Storage (fake in-memory placeholder - all real storage is proxied)
config.Storage = buildStorageConfig()
// Middleware (ATProto resolver)
defaultHold := os.Getenv("ATCR_DEFAULT_HOLD")
if defaultHold == "" {
return nil, fmt.Errorf("ATCR_DEFAULT_HOLD is required")
}
config.Middleware = buildMiddlewareConfig(defaultHold)
// Auth
baseURL := getBaseURL(httpConfig.Addr)
authConfig, err := buildAuthConfig(baseURL)
if err != nil {
return nil, fmt.Errorf("failed to build auth config: %w", err)
}
config.Auth = authConfig
// Health checks
config.Health = buildHealthConfig()
return config, nil
}
// buildLogConfig creates logging configuration from environment variables
func buildLogConfig() configuration.Log {
level := getEnvOrDefault("ATCR_LOG_LEVEL", "info")
formatter := getEnvOrDefault("ATCR_LOG_FORMATTER", "text")
return configuration.Log{
Level: configuration.Loglevel(level),
Formatter: formatter,
Fields: map[string]interface{}{
"service": "atcr-appview",
},
}
}
// buildHTTPConfig creates HTTP server configuration from environment variables
func buildHTTPConfig() (configuration.HTTP, error) {
addr := getEnvOrDefault("ATCR_HTTP_ADDR", ":5000")
debugAddr := getEnvOrDefault("ATCR_DEBUG_ADDR", ":5001")
return configuration.HTTP{
Addr: addr,
Headers: map[string][]string{
"X-Content-Type-Options": {"nosniff"},
},
Debug: configuration.Debug{
Addr: debugAddr,
},
}, nil
}
// buildStorageConfig creates a fake in-memory storage config
// This is required for distribution validation but is never actually used
// All storage is routed through middleware to ATProto (manifests) and hold services (blobs)
func buildStorageConfig() configuration.Storage {
storage := configuration.Storage{}
// Use in-memory storage as a placeholder
storage["inmemory"] = configuration.Parameters{}
// Disable upload purging
// NOTE: Must use map[interface{}]interface{} for uploadpurging (not configuration.Parameters)
// because distribution's validation code does a type assertion to map[interface{}]interface{}
storage["maintenance"] = configuration.Parameters{
"uploadpurging": map[interface{}]interface{}{
"enabled": false,
"age": 7 * 24 * time.Hour, // 168h
"interval": 24 * time.Hour, // 24h
"dryrun": false,
},
}
return storage
}
// buildMiddlewareConfig creates middleware configuration
func buildMiddlewareConfig(defaultHold string) map[string][]configuration.Middleware {
return map[string][]configuration.Middleware{
"registry": {
{
Name: "atproto-resolver",
Options: configuration.Parameters{
"default_storage_endpoint": defaultHold,
},
},
},
}
}
// buildAuthConfig creates authentication configuration from environment variables
func buildAuthConfig(baseURL string) (configuration.Auth, error) {
// Token configuration
privateKeyPath := getEnvOrDefault("ATCR_AUTH_KEY_PATH", "/var/lib/atcr/auth/private-key.pem")
certPath := getEnvOrDefault("ATCR_AUTH_CERT_PATH", "/var/lib/atcr/auth/private-key.crt")
// Token expiration in seconds (default: 5 minutes)
expirationStr := getEnvOrDefault("ATCR_TOKEN_EXPIRATION", "300")
expiration, err := strconv.Atoi(expirationStr)
if err != nil {
return configuration.Auth{}, fmt.Errorf("invalid ATCR_TOKEN_EXPIRATION: %w", err)
}
// Auto-derive service name from base URL or use env var
serviceName := getServiceName(baseURL)
// Auto-derive realm from base URL
realm := baseURL + "/auth/token"
return configuration.Auth{
"token": configuration.Parameters{
"realm": realm,
"service": serviceName,
"issuer": serviceName,
"rootcertbundle": certPath,
"privatekey": privateKeyPath,
"expiration": expiration,
},
}, nil
}
// buildHealthConfig creates health check configuration
func buildHealthConfig() configuration.Health {
return configuration.Health{
StorageDriver: configuration.StorageDriver{
Enabled: true,
Interval: 10 * time.Second,
Threshold: 3,
},
}
}
// getBaseURL determines the base URL for the service
// Priority: ATCR_BASE_URL env var, then derived from HTTP addr
func getBaseURL(httpAddr string) string {
baseURL := os.Getenv("ATCR_BASE_URL")
if baseURL != "" {
return baseURL
}
// Auto-detect from HTTP addr
if httpAddr[0] == ':' {
// Just a port, assume localhost
return fmt.Sprintf("http://127.0.0.1%s", httpAddr)
}
// Full address provided
return fmt.Sprintf("http://%s", httpAddr)
}
// getServiceName extracts service name from base URL or uses env var
func getServiceName(baseURL string) string {
// Check env var first
if serviceName := os.Getenv("ATCR_SERVICE_NAME"); serviceName != "" {
return serviceName
}
// Try to extract from base URL
parsed, err := url.Parse(baseURL)
if err == nil && parsed.Hostname() != "" {
hostname := parsed.Hostname()
// Strip localhost/127.0.0.1 and use default
if hostname == "localhost" || hostname == "127.0.0.1" {
return "atcr.io"
}
return hostname
}
// Default fallback
return "atcr.io"
}
// getEnvOrDefault gets an environment variable or returns a default value
func getEnvOrDefault(key, defaultValue string) string {
if val := os.Getenv(key); val != "" {
return val
}
return defaultValue
}
+12 -15
View File
@@ -60,11 +60,14 @@ func readOnlyAuthorizerCallback(action int, arg1, arg2, dbName string) int {
}
var serveCmd = &cobra.Command{
Use: "serve <config>",
Use: "serve",
Short: "Start the ATCR registry server",
Long: "Start the ATCR registry server with authentication endpoints",
Args: cobra.ExactArgs(1),
RunE: serveRegistry,
Long: `Start the ATCR registry server with authentication endpoints.
Configuration is loaded from environment variables.
See .env.appview.example for available environment variables.`,
Args: cobra.NoArgs,
RunE: serveRegistry,
}
func init() {
@@ -87,19 +90,13 @@ func init() {
}
func serveRegistry(cmd *cobra.Command, args []string) error {
configPath := args[0]
// Parse configuration
fp, err := os.Open(configPath)
// Load configuration from environment variables
fmt.Println("Loading configuration from environment variables...")
config, err := loadConfigFromEnv()
if err != nil {
return fmt.Errorf("failed to open config file: %w", err)
}
defer fp.Close()
config, err := configuration.Parse(fp)
if err != nil {
return fmt.Errorf("failed to parse configuration: %w", err)
return fmt.Errorf("failed to load config from environment: %w", err)
}
fmt.Println("Configuration loaded successfully from environment")
// Initialize UI database first (required for all stores)
fmt.Println("Initializing UI database...")
-57
View File
@@ -1,57 +0,0 @@
version: 0.1
log:
level: info
formatter: text
fields:
service: atcr-appview
# Storage is handled by external services:
# - Manifests/Tags -> ATProto PDS (user's personal data server)
# - Blobs/Layers -> Hold service (default or BYOS)
# The AppView should be stateless with no local storage
#
# NOTE: The storage section below is required for distribution config validation
# but is NOT actually used - all blob operations are routed through hold service
storage:
inmemory: {}
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
debug:
addr: :5001
middleware:
registry:
# Name resolution middleware
- name: atproto-resolver
options:
# Default hold service for blob storage
# Users without their own hold will use this endpoint
default_storage_endpoint: http://atcr-hold:8080
# Authentication - all endpoints on port 5000
auth:
token:
# Token service realm (where Docker gets tokens)
realm: http://127.0.0.1:5000/auth/token
service: atcr.io
issuer: atcr.io
expiration: 1800 # 30 minutes (in seconds)
# Certificate bundle for validating JWTs
rootcertbundle: /var/lib/atcr/auth/private-key.crt
# Private key for signing JWTs (used by custom auth handlers)
privatekey: /var/lib/atcr/auth/private-key.pem
# Token expiration in seconds (5 minutes)
expiration: 300
# Health check
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
+18 -9
View File
@@ -2,18 +2,27 @@ services:
atcr-appview:
build:
context: .
dockerfile: Dockerfile
dockerfile: Dockerfile.appview
image: atcr-appview:latest
container_name: atcr-appview
ports:
- "5000:5000"
# Optional: Load from .env.appview file (create from .env.appview.example)
# env_file:
# - .env.appview
environment:
- ATCR_UI_ENABLED=true
- ATCR_BACKFILL_ENABLED=true
# Server configuration
ATCR_HTTP_ADDR: :5000
ATCR_DEFAULT_HOLD: http://atcr-hold:8080
# UI configuration
ATCR_UI_ENABLED: true
ATCR_BACKFILL_ENABLED: true
# Logging
ATCR_LOG_LEVEL: info
volumes:
# Auth keys (JWT signing keys)
- atcr-auth:/var/lib/atcr/auth
# UI database (includes OAuth sessions, devices, and firehose cache)
# UI database (includes OAuth sessions, devices, and Jetstream cache)
- atcr-ui:/var/lib/atcr
restart: unless-stopped
dns:
@@ -22,11 +31,11 @@ services:
networks:
atcr-network:
ipv4_address: 172.28.0.2
# The AppView should be stateless - all storage is external:
# - Manifests/Tags -> ATProto PDS
# - Blobs/Layers -> Hold service
# - OAuth tokens -> Persistent volume (atcr-tokens)
# Future: Add read_only: true for production deployments
# The AppView is stateless - all storage is external:
# - Manifests/Tags -> ATProto PDS (via middleware)
# - Blobs/Layers -> Hold service (via ProxyBlobStore)
# - OAuth tokens -> SQLite database (atcr-ui volume)
# - No config.yml needed - all config via environment variables
atcr-hold:
env_file:
+22 -4
View File
@@ -471,12 +471,30 @@ A company wants shared storage for their team:
3. **In-memory cache** - Hold endpoint cache is in-memory (for production, use Redis)
4. **Manual profile updates** - No UI for updating sailor profile (must use ATProto client)
## Performance Optimization: S3 Presigned URLs
**Status:** Planned implementation (see [PRESIGNED_URLS.md](./PRESIGNED_URLS.md))
Currently, hold services act as proxies for blob data. With presigned URLs:
- **Downloads:** Docker → S3 direct (via 307 redirect)
- **Uploads:** Docker → AppView → S3 (via presigned URL)
- **Hold service bandwidth:** Reduced by 99.98% (only orchestration)
**Benefits:**
- Hold services can run on minimal infrastructure ($5/month instances)
- Direct S3 transfers at maximum speed
- Scales to arbitrarily large images
- Works with Storj, MinIO, Backblaze B2, Cloudflare R2
See [PRESIGNED_URLS.md](./PRESIGNED_URLS.md) for complete technical details and implementation guide.
## Future Improvements
1. **Automatic failover** - Multiple storage endpoints, fallback to default
2. **Storage analytics** - Track usage per DID
3. **Quota integration** - Optional quota tracking in storage service
4. **Direct presigned URL support** - S3 native presigned URLs (bypass proxy)
1. **S3 Presigned URLs** - Implement direct S3 URLs (see [PRESIGNED_URLS.md](./PRESIGNED_URLS.md))
2. **Automatic failover** - Multiple storage endpoints, fallback to default
3. **Storage analytics** - Track usage per DID
4. **Quota integration** - Optional quota tracking in storage service
5. **Profile management UI** - Web interface for users to manage their sailor profile
6. **Distributed cache** - Redis/Memcached for hold endpoint cache in multi-instance deployments
+637
View File
@@ -0,0 +1,637 @@
# S3 Presigned URLs Implementation
## Overview
Currently, ATCR's hold service acts as a proxy for all blob data, meaning every byte flows through the hold service when uploading or downloading container images. This document describes the implementation of **S3 presigned URLs** to eliminate this bottleneck, allowing direct data transfer between clients and S3-compatible storage.
### Current Architecture (Proxy Mode)
```
Downloads: Docker → AppView → Hold Service → S3 → Hold Service → AppView → Docker
Uploads: Docker → AppView → Hold Service → S3
```
**Problems:**
- All blob data flows through hold service
- Hold service bandwidth = total image bandwidth
- Latency from extra hops
- Hold service becomes bottleneck for large images
### Target Architecture (Presigned URLs)
```
Downloads: Docker → AppView (gets presigned URL) → S3 (direct download)
Uploads: Docker → AppView → S3 (via presigned URL)
Move: AppView → Hold Service → S3 (server-side CopyObject API)
```
**Benefits:**
- ✅ Hold service only orchestrates (no data transfer)
- ✅ Blob data never touches hold service
- ✅ Direct S3 uploads/downloads at wire speed
- ✅ Hold service can run on minimal resources
- ✅ Works with all S3-compatible services
## How Presigned URLs Work
### For Downloads (GET)
1. **Docker requests blob:** `GET /v2/alice/myapp/blobs/sha256:abc123`
2. **AppView asks hold service:** `POST /get-presigned-url`
```json
{"did": "did:plc:alice123", "digest": "sha256:abc123"}
```
3. **Hold service generates presigned URL:**
```go
req, _ := s3Client.GetObjectRequest(&s3.GetObjectInput{
Bucket: "my-bucket",
Key: "blobs/sha256/ab/abc123.../data",
})
url, _ := req.Presign(15 * time.Minute)
// Returns: https://gateway.storjshare.io/bucket/blobs/...?X-Amz-Signature=...
```
4. **AppView redirects Docker:** `HTTP 307 Location: <presigned-url>`
5. **Docker downloads directly from S3** using the presigned URL
**Data path:** Docker → S3 (direct)
**Hold service bandwidth:** ~1KB (API request/response)
### For Uploads (PUT)
**Small blobs (< 5MB) using Put():**
1. **Docker sends blob to AppView:** `PUT /v2/alice/myapp/blobs/uploads/{uuid}`
2. **AppView asks hold service:** `POST /put-presigned-url`
```json
{"did": "did:plc:alice123", "digest": "sha256:abc123", "size": 1024}
```
3. **Hold service generates presigned URL:**
```go
req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{
Bucket: "my-bucket",
Key: "blobs/sha256/ab/abc123.../data",
})
url, _ := req.Presign(15 * time.Minute)
```
4. **AppView uploads to S3** using presigned URL
5. **AppView confirms to Docker:** `201 Created`
**Data path:** Docker → AppView → S3 (via presigned URL)
**Hold service bandwidth:** ~1KB (API request/response)
### For Streaming Uploads (Create/Commit)
**Large blobs (> 5MB) using streaming:**
1. **Docker starts upload:** `POST /v2/alice/myapp/blobs/uploads/`
2. **AppView creates upload session** with UUID
3. **AppView gets presigned URL for temp location:**
```json
POST /put-presigned-url
{"did": "...", "digest": "uploads/temp-{uuid}", "size": 0}
```
4. **Docker streams data:** `PATCH /v2/alice/myapp/blobs/uploads/{uuid}`
5. **AppView streams to S3** using presigned URL to `uploads/temp-{uuid}/data`
6. **Docker finalizes:** `PUT /v2/.../uploads/{uuid}?digest=sha256:abc123`
7. **AppView requests move:** `POST /move?from=uploads/temp-{uuid}&to=sha256:abc123`
8. **Hold service executes S3 server-side copy:**
```go
s3.CopyObject(&s3.CopyObjectInput{
Bucket: "my-bucket",
CopySource: "/my-bucket/uploads/temp-{uuid}/data",
Key: "blobs/sha256/ab/abc123.../data",
})
s3.DeleteObject(&s3.DeleteObjectInput{
Key: "uploads/temp-{uuid}/data",
})
```
**Data path:** Docker → AppView → S3 (temp location)
**Move path:** S3 internal copy (no data transfer!)
**Hold service bandwidth:** ~2KB (presigned URL + CopyObject API)
## Why the Temp → Final Move is Required
This is **not an ATCR implementation detail** — it's required by the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push).
### The Problem: Unknown Digest
Docker doesn't know the blob's digest until **after** uploading:
1. **Streaming data:** Can't buffer 5GB layer in memory to calculate digest first
2. **Stdin pipes:** `docker build . | docker push` generates data on-the-fly
3. **Chunked uploads:** Multiple PATCH requests, digest calculated as data streams
### The Solution: Upload to Temp, Verify, Move
**All OCI registries do this:**
1. Client: `POST /v2/{name}/blobs/uploads/` → Get upload UUID
2. Client: `PATCH /v2/{name}/blobs/uploads/{uuid}` → Stream data to temp location
3. Client: `PUT /v2/{name}/blobs/uploads/{uuid}?digest=sha256:abc` → Provide digest
4. Registry: Verify digest matches uploaded data
5. Registry: Move `uploads/{uuid}` → `blobs/sha256/abc123...`
**Docker Hub, GHCR, ECR, Harbor — all use this pattern.**
### Why It's Efficient with S3
**For S3, the move is a CopyObject API call:**
```go
// This happens INSIDE S3 servers - no data transfer!
s3.CopyObject(&s3.CopyObjectInput{
Bucket: "my-bucket",
CopySource: "/my-bucket/uploads/temp-12345/data", // 5GB blob
Key: "blobs/sha256/ab/abc123.../data",
})
// S3 copies internally, hold service only sends ~1KB API request
```
**For a 5GB layer:**
- Hold service bandwidth: **~1KB** (API request/response)
- S3 internal copy: Instant (metadata operation on S3 side)
- No data leaves S3, no network transfer
This is why the move operation is essentially free!
## Implementation Details
### 1. Add S3 Client to Hold Service
**File: `cmd/hold/main.go`**
Modify `HoldService` struct:
```go
type HoldService struct {
driver storagedriver.StorageDriver
config *Config
s3Client *s3.S3 // NEW: S3 client for presigned URLs
bucket string // NEW: Bucket name
s3PathPrefix string // NEW: Path prefix (if any)
}
```
Add initialization function:
```go
func (s *HoldService) initS3Client() error {
if s.config.Storage.Type() != "s3" {
log.Printf("Storage driver is %s (not S3), presigned URLs disabled", s.config.Storage.Type())
return nil
}
params := s.config.Storage.Parameters()["s3"].(configuration.Parameters)
// Build AWS config
awsConfig := &aws.Config{
Region: aws.String(params["region"].(string)),
Credentials: credentials.NewStaticCredentials(
params["accesskey"].(string),
params["secretkey"].(string),
"",
),
}
// Add custom endpoint for S3-compatible services (Storj, MinIO, etc.)
if endpoint, ok := params["regionendpoint"].(string); ok && endpoint != "" {
awsConfig.Endpoint = aws.String(endpoint)
awsConfig.S3ForcePathStyle = aws.Bool(true) // Required for MinIO, Storj
}
sess, err := session.NewSession(awsConfig)
if err != nil {
return fmt.Errorf("failed to create AWS session: %w", err)
}
s.s3Client = s3.New(sess)
s.bucket = params["bucket"].(string)
log.Printf("S3 presigned URLs enabled for bucket: %s", s.bucket)
return nil
}
```
Call during service initialization:
```go
func NewHoldService(cfg *Config) (*HoldService, error) {
// ... existing driver creation ...
service := &HoldService{
driver: driver,
config: cfg,
}
// Initialize S3 client for presigned URLs
if err := service.initS3Client(); err != nil {
log.Printf("WARNING: S3 presigned URLs disabled: %v", err)
}
return service, nil
}
```
### 2. Implement Presigned URL Generation
**For Downloads:**
```go
func (s *HoldService) getDownloadURL(ctx context.Context, digest string, did string) (string, error) {
path := blobPath(digest)
// Check if blob exists
if _, err := s.driver.Stat(ctx, path); err != nil {
return "", fmt.Errorf("blob not found: %w", err)
}
// If S3 client available, generate presigned URL
if s.s3Client != nil {
s3Key := strings.TrimPrefix(path, "/")
req, _ := s.s3Client.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String(s.bucket),
Key: aws.String(s3Key),
})
url, err := req.Presign(15 * time.Minute)
if err != nil {
log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
return s.getProxyDownloadURL(digest, did), nil
}
log.Printf("Generated presigned download URL for %s (expires in 15min)", digest)
return url, nil
}
// Fallback: return proxy URL
return s.getProxyDownloadURL(digest, did), nil
}
func (s *HoldService) getProxyDownloadURL(digest, did string) string {
return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
}
```
**For Uploads:**
```go
func (s *HoldService) getUploadURL(ctx context.Context, digest string, size int64, did string) (string, error) {
path := blobPath(digest)
// If S3 client available, generate presigned URL
if s.s3Client != nil {
s3Key := strings.TrimPrefix(path, "/")
req, _ := s.s3Client.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String(s.bucket),
Key: aws.String(s3Key),
})
url, err := req.Presign(15 * time.Minute)
if err != nil {
log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
return s.getProxyUploadURL(digest, did), nil
}
log.Printf("Generated presigned upload URL for %s (expires in 15min)", digest)
return url, nil
}
// Fallback: return proxy URL
return s.getProxyUploadURL(digest, did), nil
}
func (s *HoldService) getProxyUploadURL(digest, did string) string {
return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
}
```
### 3. No Changes Needed for Move Operation
The existing `/move` endpoint already uses `driver.Move()`, which for S3:
- Calls `s3.CopyObject()` (server-side copy)
- Calls `s3.DeleteObject()` (delete source)
- No data transfer through hold service!
**File: `cmd/hold/main.go:296` (already exists, no changes needed)**
```go
func (s *HoldService) HandleMove(w http.ResponseWriter, r *http.Request) {
// ... existing auth and parsing ...
sourcePath := blobPath(fromPath) // uploads/temp-{uuid}/data
destPath := blobPath(toDigest) // blobs/sha256/ab/abc123.../data
// For S3, this does CopyObject + DeleteObject (server-side)
if err := s.driver.Move(ctx, sourcePath, destPath); err != nil {
// ... error handling ...
}
}
```
### 4. AppView Changes (Optional Optimization)
**File: `pkg/storage/proxy_blob_store.go:228`**
Currently streams to hold service proxy URL. Could be optimized to use presigned URL:
```go
// In Create() - line 228
go func() {
defer pipeReader.Close()
tempPath := fmt.Sprintf("uploads/temp-%s", writer.id)
// Try to get presigned URL for temp location
url, err := p.getUploadURL(ctx, digest.FromString(tempPath), 0)
if err != nil {
// Fallback to direct proxy URL
url = fmt.Sprintf("%s/blobs/%s?did=%s", p.storageEndpoint, tempPath, p.did)
}
req, err := http.NewRequestWithContext(uploadCtx, "PUT", url, pipeReader)
// ... rest unchanged
}()
```
**Note:** This optimization is optional. The presigned URL will be returned by hold service's `getUploadURL()` anyway.
## S3-Compatible Service Support
### Storj
```bash
# .env file
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-storj-access-key
AWS_SECRET_ACCESS_KEY=your-storj-secret-key
S3_BUCKET=your-bucket-name
S3_REGION=global
S3_ENDPOINT=https://gateway.storjshare.io
```
**Presigned URL example:**
```
https://gateway.storjshare.io/your-bucket/blobs/sha256/ab/abc123.../data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Signature=...
```
### MinIO
```bash
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
S3_BUCKET=registry
S3_REGION=us-east-1
S3_ENDPOINT=http://minio.example.com:9000
```
### Backblaze B2
```bash
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-b2-key-id
AWS_SECRET_ACCESS_KEY=your-b2-application-key
S3_BUCKET=your-bucket-name
S3_REGION=us-west-002
S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com
```
### Cloudflare R2
```bash
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=your-r2-access-key-id
AWS_SECRET_ACCESS_KEY=your-r2-secret-access-key
S3_BUCKET=your-bucket-name
S3_REGION=auto
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
```
**All these services support presigned URLs with AWS SDK v1!**
## Performance Impact
### Bandwidth Savings
**Before (proxy mode):**
- 5GB layer upload: Hold service receives 5GB, sends 5GB to S3 = **10GB** bandwidth
- 5GB layer download: S3 sends 5GB to hold, hold sends 5GB to client = **10GB** bandwidth
- **Total for push+pull: 20GB hold service bandwidth**
**After (presigned URLs):**
- 5GB layer upload: Hold generates URL (1KB), AppView → S3 direct (5GB), CopyObject API (1KB) = **~2KB** hold bandwidth
- 5GB layer download: Hold generates URL (1KB), client → S3 direct = **~1KB** hold bandwidth
- **Total for push+pull: ~3KB hold service bandwidth**
**Savings: 99.98% reduction in hold service bandwidth!**
### Latency Improvements
**Before:**
- Download: Client → AppView → Hold → S3 → Hold → AppView → Client (4 hops)
- Upload: Client → AppView → Hold → S3 (3 hops)
**After:**
- Download: Client → AppView (redirect) → S3 (1 hop to data)
- Upload: Client → AppView → S3 (2 hops)
- Move: S3 internal (no network hops)
### Resource Requirements
**Before:**
- Hold service needs bandwidth = sum of all image operations
- For 100 concurrent 1GB pushes: 100GB/s bandwidth needed
- Expensive, hard to scale
**After:**
- Hold service needs minimal CPU for presigned URL signing
- For 100 concurrent 1GB pushes: ~100KB/s bandwidth needed (API traffic)
- Can run on $5/month instance!
## Security Considerations
### Presigned URL Expiration
- Default: **15 minutes** expiration
- Presigned URL includes embedded credentials in query params
- After expiry, URL becomes invalid (S3 rejects with 403)
- No long-lived URLs floating around
### Authorization Flow
1. **AppView validates user** via ATProto OAuth
2. **AppView passes DID to hold service** in presigned URL request
3. **Hold service validates DID** (owner or crew member)
4. **Hold service generates presigned URL** if authorized
5. **Client uses presigned URL** directly with S3
**Security boundary:** Hold service controls who gets presigned URLs, S3 validates the URLs.
### Fallback Security
If presigned URL generation fails:
- Falls back to proxy URLs (existing behavior)
- Still requires hold service authorization
- Data flows through hold service (original security model)
## Testing & Validation
### Verify Presigned URLs are Used
**1. Check hold service logs:**
```bash
docker logs atcr-hold | grep -i presigned
# Should see: "Generated presigned download/upload URL for sha256:..."
```
**2. Monitor network traffic:**
```bash
# Before: Large data transfers to/from hold service
docker stats atcr-hold
# After: Minimal network usage on hold service
docker stats atcr-hold
```
**3. Inspect redirect responses:**
```bash
# Should see 307 redirect to S3 URL
curl -v http://appview:5000/v2/alice/myapp/blobs/sha256:abc123 \
-H "Authorization: Bearer $TOKEN"
# Look for:
# < HTTP/1.1 307 Temporary Redirect
# < Location: https://gateway.storjshare.io/...?X-Amz-Signature=...
```
### Test Fallback Behavior
**1. With filesystem driver (should use proxy URLs):**
```bash
STORAGE_DRIVER=filesystem docker-compose up atcr-hold
# Logs should show: "Storage driver is filesystem (not S3), presigned URLs disabled"
```
**2. With S3 but invalid credentials (should fall back):**
```bash
AWS_ACCESS_KEY_ID=invalid docker-compose up atcr-hold
# Logs should show: "WARN: Presigned URL generation failed, falling back to proxy"
```
### Bandwidth Monitoring
**Track hold service bandwidth over time:**
```bash
# Install bandwidth monitoring
docker exec atcr-hold apt-get update && apt-get install -y vnstat
# Monitor
docker exec atcr-hold vnstat -l
```
**Expected results:**
- Before: Bandwidth correlates with image operations
- After: Bandwidth stays minimal regardless of image operations
## Migration Guide
### For Existing ATCR Deployments
**1. Update hold service code** (this implementation)
**2. No configuration changes needed** if already using S3:
```bash
# Existing S3 config works automatically
STORAGE_DRIVER=s3
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
S3_BUCKET=...
S3_ENDPOINT=...
```
**3. Restart hold service:**
```bash
docker-compose restart atcr-hold
```
**4. Verify in logs:**
```
S3 presigned URLs enabled for bucket: my-bucket
```
**5. Test with image push/pull:**
```bash
docker push atcr.io/alice/myapp:latest
docker pull atcr.io/alice/myapp:latest
```
**6. Monitor bandwidth** to confirm reduction
### Rollback Plan
If issues arise:
**Option 1: Disable presigned URLs via env var** (if we add this feature)
```bash
PRESIGNED_URLS_ENABLED=false docker-compose restart atcr-hold
```
**Option 2: Revert code changes** to previous hold service version
The implementation has automatic fallbacks, so partial failures won't break functionality.
## Future Enhancements
### 1. Configurable Expiration
Allow customizing presigned URL expiry:
```bash
PRESIGNED_URL_EXPIRY=30m # Default: 15m
```
### 2. Presigned URL Caching
Cache presigned URLs for frequently accessed blobs (with shorter TTL).
### 3. CloudFront/CDN Integration
For downloads, use CloudFront presigned URLs instead of direct S3:
- Better global distribution
- Lower egress costs
- Faster downloads
### 4. Multipart Upload Support
For very large layers (>5GB), use presigned URLs with multipart upload:
- Generate presigned URLs for each part
- Client uploads parts directly to S3
- Hold service finalizes multipart upload
### 5. Metrics & Monitoring
Track presigned URL usage:
- Count of presigned URLs generated
- Fallback rate (proxy vs presigned)
- Bandwidth savings metrics
## References
- [OCI Distribution Specification - Push](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push)
- [AWS SDK Go v1 - Presigned URLs](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html)
- [Storj - Using Presigned URLs](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway/using-presigned-urls)
- [MinIO - Presigned Upload via Browser](https://docs.min.io/community/minio-object-store/integrations/presigned-put-upload-via-browser.html)
- [Cloudflare R2 - Presigned URLs](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)
- [Backblaze B2 - S3 Compatible API](https://help.backblaze.com/hc/en-us/articles/360047815993-Does-the-B2-S3-Compatible-API-support-Pre-Signed-URLs)
## Summary
Implementing S3 presigned URLs transforms ATCR's hold service from a **data proxy** to a **lightweight orchestrator**:
**99.98% bandwidth reduction** for hold service
**Direct client → S3 transfers** for maximum speed
**Works with all S3-compatible services** (Storj, MinIO, R2, B2)
**OCI-compliant** temp → final move pattern
**Automatic fallback** to proxy mode for non-S3 drivers
**No breaking changes** to existing deployments
This makes BYOS (Bring Your Own Storage) truly scalable and cost-effective, as users can run hold services on minimal infrastructure while serving arbitrarily large container images.