mirror of
https://tangled.org/evan.jarrett.net/at-container-registry
synced 2026-08-29 12:17:00 +00:00
refactor appview to use envvars. move distribution configurations to code
This commit is contained in:
@@ -0,0 +1,90 @@
|
||||
# ATCR AppView Configuration
|
||||
# Copy this file to .env.appview and fill in your values
|
||||
# Load with: source .env.appview && ./bin/atcr-appview serve
|
||||
|
||||
# ==============================================================================
|
||||
# Server Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# HTTP listen address (default: :5000)
|
||||
ATCR_HTTP_ADDR=:5000
|
||||
|
||||
# Debug listen address (default: :5001)
|
||||
# ATCR_DEBUG_ADDR=:5001
|
||||
|
||||
# Base URL for the AppView service (REQUIRED for production)
|
||||
# Used to generate OAuth redirect URIs and JWT realms
|
||||
# Development: Auto-detected from ATCR_HTTP_ADDR (e.g., http://127.0.0.1:5000)
|
||||
# Production: Set to your public URL (e.g., https://atcr.io)
|
||||
# ATCR_BASE_URL=http://127.0.0.1:5000
|
||||
|
||||
# Service name (used for JWT service/issuer fields)
|
||||
# Default: Derived from base URL hostname, or "atcr.io"
|
||||
# ATCR_SERVICE_NAME=atcr.io
|
||||
|
||||
# ==============================================================================
|
||||
# Storage Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# Default hold service endpoint for users without their own storage (REQUIRED)
|
||||
# Users with a sailor profile defaultHold setting will override this
|
||||
# Docker: Use container name (http://atcr-hold:8080)
|
||||
# Local dev: Use localhost (http://127.0.0.1:8080)
|
||||
ATCR_DEFAULT_HOLD=http://127.0.0.1:8080
|
||||
|
||||
# ==============================================================================
|
||||
# Authentication Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# Path to JWT signing private key (auto-generated if missing)
|
||||
# Default: /var/lib/atcr/auth/private-key.pem
|
||||
# ATCR_AUTH_KEY_PATH=/var/lib/atcr/auth/private-key.pem
|
||||
|
||||
# Path to JWT signing certificate (auto-generated if missing)
|
||||
# Default: /var/lib/atcr/auth/private-key.crt
|
||||
# ATCR_AUTH_CERT_PATH=/var/lib/atcr/auth/private-key.crt
|
||||
|
||||
# JWT token expiration in seconds (default: 300 = 5 minutes)
|
||||
# ATCR_TOKEN_EXPIRATION=300
|
||||
|
||||
# ==============================================================================
|
||||
# UI Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# Enable web UI (default: true)
|
||||
# Set to "false" to disable web interface and run registry-only
|
||||
ATCR_UI_ENABLED=true
|
||||
|
||||
# SQLite database path for UI data (sessions, stars, pull counts, etc.)
|
||||
# Default: /var/lib/atcr/ui.db
|
||||
# ATCR_UI_DATABASE_PATH=/var/lib/atcr/ui.db
|
||||
|
||||
# ==============================================================================
|
||||
# Logging Configuration
|
||||
# ==============================================================================
|
||||
|
||||
# Log level: debug, info, warn, error (default: info)
|
||||
# ATCR_LOG_LEVEL=info
|
||||
|
||||
# Log formatter: text, json (default: text)
|
||||
# ATCR_LOG_FORMATTER=text
|
||||
|
||||
# ==============================================================================
|
||||
# Jetstream Configuration (ATProto event streaming)
|
||||
# ==============================================================================
|
||||
|
||||
# Jetstream WebSocket URL for real-time ATProto events
|
||||
# Default: wss://jetstream2.us-west.bsky.network/subscribe
|
||||
# JETSTREAM_URL=wss://jetstream2.us-west.bsky.network/subscribe
|
||||
|
||||
# Enable backfill worker to sync historical records (default: false)
|
||||
# Set to "true" to enable periodic syncing of ATProto records
|
||||
# ATCR_BACKFILL_ENABLED=true
|
||||
|
||||
# ATProto relay endpoint for backfill sync API
|
||||
# Default: https://relay1.us-east.bsky.network
|
||||
# ATCR_RELAY_ENDPOINT=https://relay1.us-east.bsky.network
|
||||
|
||||
# Backfill interval (default: 1h)
|
||||
# Examples: 30m, 1h, 2h, 24h
|
||||
# ATCR_BACKFILL_INTERVAL=1h
|
||||
@@ -31,17 +31,26 @@ docker build -f Dockerfile.hold -t atcr.io/hold:latest .
|
||||
# Or use docker-compose
|
||||
docker-compose up -d
|
||||
|
||||
# Run locally (AppView)
|
||||
export ATPROTO_DID=did:plc:your-did
|
||||
export ATPROTO_ACCESS_TOKEN=your-token
|
||||
./atcr-appview serve config/config.yml
|
||||
# Run locally (AppView) - configure via env vars (see .env.appview.example)
|
||||
export ATCR_HTTP_ADDR=:5000
|
||||
export ATCR_DEFAULT_HOLD=http://127.0.0.1:8080
|
||||
./bin/atcr-appview serve
|
||||
|
||||
# Run hold service (configure via env vars - see .env.example)
|
||||
# Or use .env file:
|
||||
cp .env.appview.example .env.appview
|
||||
# Edit .env.appview with your settings
|
||||
source .env.appview
|
||||
./bin/atcr-appview serve
|
||||
|
||||
# Legacy mode (still supported):
|
||||
# ./bin/atcr-appview serve config/config.yml
|
||||
|
||||
# Run hold service (configure via env vars - see .env.hold.example)
|
||||
export HOLD_PUBLIC_URL=http://127.0.0.1:8080
|
||||
export STORAGE_DRIVER=filesystem
|
||||
export STORAGE_ROOT_DIR=/tmp/atcr-hold
|
||||
export HOLD_OWNER=did:plc:your-did-here
|
||||
./atcr-hold
|
||||
./bin/atcr-hold
|
||||
# Check logs for OAuth URL, visit in browser to complete registration
|
||||
```
|
||||
|
||||
@@ -433,29 +442,45 @@ This ensures:
|
||||
|
||||
### Configuration
|
||||
|
||||
**AppView configuration** (`config/config.yml`):
|
||||
- S3 bucket settings under `storage.s3`
|
||||
- ATProto middleware under `middleware.repository`
|
||||
- Name resolver under `middleware.registry`
|
||||
- Default storage endpoint: `middleware.registry.options.default_storage_endpoint`
|
||||
- Auth token signing keys and expiration
|
||||
- Database path: `db.path` (SQLite database location)
|
||||
- Jetstream endpoint: `jetstream.endpoint` (for ATProto event streaming)
|
||||
**AppView configuration** (environment variables):
|
||||
|
||||
Both AppView and Hold service follow the same pattern: **zero config files, all configuration via environment variables**.
|
||||
|
||||
See `.env.appview.example` for all available options. Key environment variables:
|
||||
|
||||
**Server:**
|
||||
- `ATCR_HTTP_ADDR` - HTTP listen address (default: `:5000`)
|
||||
- `ATCR_BASE_URL` - Public URL for OAuth/JWT realm (auto-detected in dev)
|
||||
- `ATCR_DEFAULT_HOLD` - Default hold endpoint for blob storage (REQUIRED)
|
||||
|
||||
**Authentication:**
|
||||
- `ATCR_AUTH_KEY_PATH` - JWT signing key path (default: `/var/lib/atcr/auth/private-key.pem`)
|
||||
- `ATCR_TOKEN_EXPIRATION` - JWT expiration in seconds (default: 300)
|
||||
|
||||
**UI:**
|
||||
- `ATCR_UI_ENABLED` - Enable web interface (default: true)
|
||||
- `ATCR_UI_DATABASE_PATH` - SQLite database path (default: `/var/lib/atcr/ui.db`)
|
||||
|
||||
**Jetstream:**
|
||||
- `JETSTREAM_URL` - ATProto event stream URL
|
||||
- `ATCR_BACKFILL_ENABLED` - Enable periodic sync (default: false)
|
||||
|
||||
**Legacy:** `config/config.yml` is still supported but deprecated. Use environment variables instead.
|
||||
|
||||
**Hold Service configuration** (environment variables):
|
||||
- Storage driver config via env vars: `STORAGE_DRIVER`, `AWS_*`, `S3_*`
|
||||
- Authorization: Based on PDS records (`hold.public`, crew records)
|
||||
- Server settings: `HOLD_SERVER_ADDR`, `HOLD_PUBLIC_URL`, `HOLD_PUBLIC`
|
||||
- Auto-registration: `HOLD_OWNER` (optional)
|
||||
|
||||
See `.env.hold.example` for all available options. Key environment variables:
|
||||
- `HOLD_PUBLIC_URL` - Public URL of hold service (REQUIRED)
|
||||
- `STORAGE_DRIVER` - Storage backend (s3, filesystem)
|
||||
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` - S3 credentials
|
||||
- `S3_BUCKET`, `S3_ENDPOINT` - S3 configuration
|
||||
- `HOLD_PUBLIC` - Allow public reads (default: false)
|
||||
- `HOLD_OWNER` - DID for auto-registration (optional)
|
||||
|
||||
**Credential Helper**:
|
||||
- Token storage: `~/.atcr/oauth-token.json`
|
||||
- Contains: access token, refresh token, DPoP key (PEM), DID, handle
|
||||
|
||||
Environment variables:
|
||||
- `ATPROTO_DID`: DID for authentication with PDS (AppView only)
|
||||
- `ATPROTO_ACCESS_TOKEN`: Access token for PDS operations (AppView only)
|
||||
|
||||
### Development Notes
|
||||
|
||||
**General:**
|
||||
@@ -524,7 +549,7 @@ When writing tests:
|
||||
- Queries in `pkg/appview/db/queries.go`
|
||||
- Stores for OAuth, devices, sessions in separate files
|
||||
- Run migrations automatically on startup
|
||||
- Database path configurable via config.yml
|
||||
- Database path configurable via `ATCR_UI_DATABASE_PATH` env var
|
||||
|
||||
**Adding web UI features**:
|
||||
- Add handler in `pkg/appview/handlers/`
|
||||
|
||||
@@ -31,18 +31,12 @@ WORKDIR /app
|
||||
# Copy binary from builder
|
||||
COPY --from=builder /build/atcr-appview .
|
||||
|
||||
# Copy default configuration
|
||||
COPY config/config.yml /etc/atcr/config.yml
|
||||
|
||||
# Create directories for storage
|
||||
RUN mkdir -p /var/lib/atcr/blobs /var/lib/atcr/auth
|
||||
|
||||
# Expose ports
|
||||
EXPOSE 5000 5001
|
||||
|
||||
# Set environment variables
|
||||
ENV ATCR_CONFIG=/etc/atcr/config.yml
|
||||
|
||||
# OCI image annotations
|
||||
LABEL org.opencontainers.image.title="ATCR AppView" \
|
||||
org.opencontainers.image.description="ATProto Container Registry - OCI-compliant registry using AT Protocol for manifest storage" \
|
||||
@@ -53,6 +47,6 @@ LABEL org.opencontainers.image.title="ATCR AppView" \
|
||||
org.opencontainers.image.version="0.1.0" \
|
||||
io.atcr.icon="https://imgs.blue/evan.jarrett.net/1TpTNrRelfloN2emuWZDrWmPT0o93bAjEnozjD6UPgoVV9m4"
|
||||
|
||||
# Run the AppView
|
||||
# Run the AppView (no config file - uses environment variables)
|
||||
ENTRYPOINT ["/app/atcr-appview"]
|
||||
CMD ["serve", "/etc/atcr/config.yml"]
|
||||
CMD ["serve"]
|
||||
@@ -0,0 +1,213 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/distribution/distribution/v3/configuration"
|
||||
)
|
||||
|
||||
// loadConfigFromEnv builds a complete configuration from environment variables
|
||||
// This follows the same pattern as the hold service (no config files, only env vars)
|
||||
func loadConfigFromEnv() (*configuration.Configuration, error) {
|
||||
config := &configuration.Configuration{}
|
||||
|
||||
// Version
|
||||
config.Version = configuration.MajorMinorVersion(0, 1)
|
||||
|
||||
// Logging
|
||||
config.Log = buildLogConfig()
|
||||
|
||||
// HTTP server
|
||||
httpConfig, err := buildHTTPConfig()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to build HTTP config: %w", err)
|
||||
}
|
||||
config.HTTP = httpConfig
|
||||
|
||||
// Storage (fake in-memory placeholder - all real storage is proxied)
|
||||
config.Storage = buildStorageConfig()
|
||||
|
||||
// Middleware (ATProto resolver)
|
||||
defaultHold := os.Getenv("ATCR_DEFAULT_HOLD")
|
||||
if defaultHold == "" {
|
||||
return nil, fmt.Errorf("ATCR_DEFAULT_HOLD is required")
|
||||
}
|
||||
config.Middleware = buildMiddlewareConfig(defaultHold)
|
||||
|
||||
// Auth
|
||||
baseURL := getBaseURL(httpConfig.Addr)
|
||||
authConfig, err := buildAuthConfig(baseURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to build auth config: %w", err)
|
||||
}
|
||||
config.Auth = authConfig
|
||||
|
||||
// Health checks
|
||||
config.Health = buildHealthConfig()
|
||||
|
||||
return config, nil
|
||||
}
|
||||
|
||||
// buildLogConfig creates logging configuration from environment variables
|
||||
func buildLogConfig() configuration.Log {
|
||||
level := getEnvOrDefault("ATCR_LOG_LEVEL", "info")
|
||||
formatter := getEnvOrDefault("ATCR_LOG_FORMATTER", "text")
|
||||
|
||||
return configuration.Log{
|
||||
Level: configuration.Loglevel(level),
|
||||
Formatter: formatter,
|
||||
Fields: map[string]interface{}{
|
||||
"service": "atcr-appview",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// buildHTTPConfig creates HTTP server configuration from environment variables
|
||||
func buildHTTPConfig() (configuration.HTTP, error) {
|
||||
addr := getEnvOrDefault("ATCR_HTTP_ADDR", ":5000")
|
||||
debugAddr := getEnvOrDefault("ATCR_DEBUG_ADDR", ":5001")
|
||||
|
||||
return configuration.HTTP{
|
||||
Addr: addr,
|
||||
Headers: map[string][]string{
|
||||
"X-Content-Type-Options": {"nosniff"},
|
||||
},
|
||||
Debug: configuration.Debug{
|
||||
Addr: debugAddr,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// buildStorageConfig creates a fake in-memory storage config
|
||||
// This is required for distribution validation but is never actually used
|
||||
// All storage is routed through middleware to ATProto (manifests) and hold services (blobs)
|
||||
func buildStorageConfig() configuration.Storage {
|
||||
storage := configuration.Storage{}
|
||||
|
||||
// Use in-memory storage as a placeholder
|
||||
storage["inmemory"] = configuration.Parameters{}
|
||||
|
||||
// Disable upload purging
|
||||
// NOTE: Must use map[interface{}]interface{} for uploadpurging (not configuration.Parameters)
|
||||
// because distribution's validation code does a type assertion to map[interface{}]interface{}
|
||||
storage["maintenance"] = configuration.Parameters{
|
||||
"uploadpurging": map[interface{}]interface{}{
|
||||
"enabled": false,
|
||||
"age": 7 * 24 * time.Hour, // 168h
|
||||
"interval": 24 * time.Hour, // 24h
|
||||
"dryrun": false,
|
||||
},
|
||||
}
|
||||
|
||||
return storage
|
||||
}
|
||||
|
||||
// buildMiddlewareConfig creates middleware configuration
|
||||
func buildMiddlewareConfig(defaultHold string) map[string][]configuration.Middleware {
|
||||
return map[string][]configuration.Middleware{
|
||||
"registry": {
|
||||
{
|
||||
Name: "atproto-resolver",
|
||||
Options: configuration.Parameters{
|
||||
"default_storage_endpoint": defaultHold,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// buildAuthConfig creates authentication configuration from environment variables
|
||||
func buildAuthConfig(baseURL string) (configuration.Auth, error) {
|
||||
// Token configuration
|
||||
privateKeyPath := getEnvOrDefault("ATCR_AUTH_KEY_PATH", "/var/lib/atcr/auth/private-key.pem")
|
||||
certPath := getEnvOrDefault("ATCR_AUTH_CERT_PATH", "/var/lib/atcr/auth/private-key.crt")
|
||||
|
||||
// Token expiration in seconds (default: 5 minutes)
|
||||
expirationStr := getEnvOrDefault("ATCR_TOKEN_EXPIRATION", "300")
|
||||
expiration, err := strconv.Atoi(expirationStr)
|
||||
if err != nil {
|
||||
return configuration.Auth{}, fmt.Errorf("invalid ATCR_TOKEN_EXPIRATION: %w", err)
|
||||
}
|
||||
|
||||
// Auto-derive service name from base URL or use env var
|
||||
serviceName := getServiceName(baseURL)
|
||||
|
||||
// Auto-derive realm from base URL
|
||||
realm := baseURL + "/auth/token"
|
||||
|
||||
return configuration.Auth{
|
||||
"token": configuration.Parameters{
|
||||
"realm": realm,
|
||||
"service": serviceName,
|
||||
"issuer": serviceName,
|
||||
"rootcertbundle": certPath,
|
||||
"privatekey": privateKeyPath,
|
||||
"expiration": expiration,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// buildHealthConfig creates health check configuration
|
||||
func buildHealthConfig() configuration.Health {
|
||||
return configuration.Health{
|
||||
StorageDriver: configuration.StorageDriver{
|
||||
Enabled: true,
|
||||
Interval: 10 * time.Second,
|
||||
Threshold: 3,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// getBaseURL determines the base URL for the service
|
||||
// Priority: ATCR_BASE_URL env var, then derived from HTTP addr
|
||||
func getBaseURL(httpAddr string) string {
|
||||
baseURL := os.Getenv("ATCR_BASE_URL")
|
||||
if baseURL != "" {
|
||||
return baseURL
|
||||
}
|
||||
|
||||
// Auto-detect from HTTP addr
|
||||
if httpAddr[0] == ':' {
|
||||
// Just a port, assume localhost
|
||||
return fmt.Sprintf("http://127.0.0.1%s", httpAddr)
|
||||
}
|
||||
|
||||
// Full address provided
|
||||
return fmt.Sprintf("http://%s", httpAddr)
|
||||
}
|
||||
|
||||
// getServiceName extracts service name from base URL or uses env var
|
||||
func getServiceName(baseURL string) string {
|
||||
// Check env var first
|
||||
if serviceName := os.Getenv("ATCR_SERVICE_NAME"); serviceName != "" {
|
||||
return serviceName
|
||||
}
|
||||
|
||||
// Try to extract from base URL
|
||||
parsed, err := url.Parse(baseURL)
|
||||
if err == nil && parsed.Hostname() != "" {
|
||||
hostname := parsed.Hostname()
|
||||
|
||||
// Strip localhost/127.0.0.1 and use default
|
||||
if hostname == "localhost" || hostname == "127.0.0.1" {
|
||||
return "atcr.io"
|
||||
}
|
||||
|
||||
return hostname
|
||||
}
|
||||
|
||||
// Default fallback
|
||||
return "atcr.io"
|
||||
}
|
||||
|
||||
// getEnvOrDefault gets an environment variable or returns a default value
|
||||
func getEnvOrDefault(key, defaultValue string) string {
|
||||
if val := os.Getenv(key); val != "" {
|
||||
return val
|
||||
}
|
||||
return defaultValue
|
||||
}
|
||||
+12
-15
@@ -60,11 +60,14 @@ func readOnlyAuthorizerCallback(action int, arg1, arg2, dbName string) int {
|
||||
}
|
||||
|
||||
var serveCmd = &cobra.Command{
|
||||
Use: "serve <config>",
|
||||
Use: "serve",
|
||||
Short: "Start the ATCR registry server",
|
||||
Long: "Start the ATCR registry server with authentication endpoints",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: serveRegistry,
|
||||
Long: `Start the ATCR registry server with authentication endpoints.
|
||||
|
||||
Configuration is loaded from environment variables.
|
||||
See .env.appview.example for available environment variables.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: serveRegistry,
|
||||
}
|
||||
|
||||
func init() {
|
||||
@@ -87,19 +90,13 @@ func init() {
|
||||
}
|
||||
|
||||
func serveRegistry(cmd *cobra.Command, args []string) error {
|
||||
configPath := args[0]
|
||||
|
||||
// Parse configuration
|
||||
fp, err := os.Open(configPath)
|
||||
// Load configuration from environment variables
|
||||
fmt.Println("Loading configuration from environment variables...")
|
||||
config, err := loadConfigFromEnv()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to open config file: %w", err)
|
||||
}
|
||||
defer fp.Close()
|
||||
|
||||
config, err := configuration.Parse(fp)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to parse configuration: %w", err)
|
||||
return fmt.Errorf("failed to load config from environment: %w", err)
|
||||
}
|
||||
fmt.Println("Configuration loaded successfully from environment")
|
||||
|
||||
// Initialize UI database first (required for all stores)
|
||||
fmt.Println("Initializing UI database...")
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
version: 0.1
|
||||
log:
|
||||
level: info
|
||||
formatter: text
|
||||
fields:
|
||||
service: atcr-appview
|
||||
|
||||
# Storage is handled by external services:
|
||||
# - Manifests/Tags -> ATProto PDS (user's personal data server)
|
||||
# - Blobs/Layers -> Hold service (default or BYOS)
|
||||
# The AppView should be stateless with no local storage
|
||||
#
|
||||
# NOTE: The storage section below is required for distribution config validation
|
||||
# but is NOT actually used - all blob operations are routed through hold service
|
||||
storage:
|
||||
inmemory: {}
|
||||
|
||||
http:
|
||||
addr: :5000
|
||||
headers:
|
||||
X-Content-Type-Options: [nosniff]
|
||||
debug:
|
||||
addr: :5001
|
||||
|
||||
middleware:
|
||||
registry:
|
||||
# Name resolution middleware
|
||||
- name: atproto-resolver
|
||||
options:
|
||||
# Default hold service for blob storage
|
||||
# Users without their own hold will use this endpoint
|
||||
default_storage_endpoint: http://atcr-hold:8080
|
||||
|
||||
# Authentication - all endpoints on port 5000
|
||||
auth:
|
||||
token:
|
||||
# Token service realm (where Docker gets tokens)
|
||||
realm: http://127.0.0.1:5000/auth/token
|
||||
service: atcr.io
|
||||
issuer: atcr.io
|
||||
expiration: 1800 # 30 minutes (in seconds)
|
||||
|
||||
# Certificate bundle for validating JWTs
|
||||
rootcertbundle: /var/lib/atcr/auth/private-key.crt
|
||||
|
||||
# Private key for signing JWTs (used by custom auth handlers)
|
||||
privatekey: /var/lib/atcr/auth/private-key.pem
|
||||
|
||||
# Token expiration in seconds (5 minutes)
|
||||
expiration: 300
|
||||
|
||||
# Health check
|
||||
health:
|
||||
storagedriver:
|
||||
enabled: true
|
||||
interval: 10s
|
||||
threshold: 3
|
||||
+18
-9
@@ -2,18 +2,27 @@ services:
|
||||
atcr-appview:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
dockerfile: Dockerfile.appview
|
||||
image: atcr-appview:latest
|
||||
container_name: atcr-appview
|
||||
ports:
|
||||
- "5000:5000"
|
||||
# Optional: Load from .env.appview file (create from .env.appview.example)
|
||||
# env_file:
|
||||
# - .env.appview
|
||||
environment:
|
||||
- ATCR_UI_ENABLED=true
|
||||
- ATCR_BACKFILL_ENABLED=true
|
||||
# Server configuration
|
||||
ATCR_HTTP_ADDR: :5000
|
||||
ATCR_DEFAULT_HOLD: http://atcr-hold:8080
|
||||
# UI configuration
|
||||
ATCR_UI_ENABLED: true
|
||||
ATCR_BACKFILL_ENABLED: true
|
||||
# Logging
|
||||
ATCR_LOG_LEVEL: info
|
||||
volumes:
|
||||
# Auth keys (JWT signing keys)
|
||||
- atcr-auth:/var/lib/atcr/auth
|
||||
# UI database (includes OAuth sessions, devices, and firehose cache)
|
||||
# UI database (includes OAuth sessions, devices, and Jetstream cache)
|
||||
- atcr-ui:/var/lib/atcr
|
||||
restart: unless-stopped
|
||||
dns:
|
||||
@@ -22,11 +31,11 @@ services:
|
||||
networks:
|
||||
atcr-network:
|
||||
ipv4_address: 172.28.0.2
|
||||
# The AppView should be stateless - all storage is external:
|
||||
# - Manifests/Tags -> ATProto PDS
|
||||
# - Blobs/Layers -> Hold service
|
||||
# - OAuth tokens -> Persistent volume (atcr-tokens)
|
||||
# Future: Add read_only: true for production deployments
|
||||
# The AppView is stateless - all storage is external:
|
||||
# - Manifests/Tags -> ATProto PDS (via middleware)
|
||||
# - Blobs/Layers -> Hold service (via ProxyBlobStore)
|
||||
# - OAuth tokens -> SQLite database (atcr-ui volume)
|
||||
# - No config.yml needed - all config via environment variables
|
||||
|
||||
atcr-hold:
|
||||
env_file:
|
||||
|
||||
+22
-4
@@ -471,12 +471,30 @@ A company wants shared storage for their team:
|
||||
3. **In-memory cache** - Hold endpoint cache is in-memory (for production, use Redis)
|
||||
4. **Manual profile updates** - No UI for updating sailor profile (must use ATProto client)
|
||||
|
||||
## Performance Optimization: S3 Presigned URLs
|
||||
|
||||
**Status:** Planned implementation (see [PRESIGNED_URLS.md](./PRESIGNED_URLS.md))
|
||||
|
||||
Currently, hold services act as proxies for blob data. With presigned URLs:
|
||||
|
||||
- **Downloads:** Docker → S3 direct (via 307 redirect)
|
||||
- **Uploads:** Docker → AppView → S3 (via presigned URL)
|
||||
- **Hold service bandwidth:** Reduced by 99.98% (only orchestration)
|
||||
|
||||
**Benefits:**
|
||||
- Hold services can run on minimal infrastructure ($5/month instances)
|
||||
- Direct S3 transfers at maximum speed
|
||||
- Scales to arbitrarily large images
|
||||
- Works with Storj, MinIO, Backblaze B2, Cloudflare R2
|
||||
|
||||
See [PRESIGNED_URLS.md](./PRESIGNED_URLS.md) for complete technical details and implementation guide.
|
||||
|
||||
## Future Improvements
|
||||
|
||||
1. **Automatic failover** - Multiple storage endpoints, fallback to default
|
||||
2. **Storage analytics** - Track usage per DID
|
||||
3. **Quota integration** - Optional quota tracking in storage service
|
||||
4. **Direct presigned URL support** - S3 native presigned URLs (bypass proxy)
|
||||
1. **S3 Presigned URLs** - Implement direct S3 URLs (see [PRESIGNED_URLS.md](./PRESIGNED_URLS.md))
|
||||
2. **Automatic failover** - Multiple storage endpoints, fallback to default
|
||||
3. **Storage analytics** - Track usage per DID
|
||||
4. **Quota integration** - Optional quota tracking in storage service
|
||||
5. **Profile management UI** - Web interface for users to manage their sailor profile
|
||||
6. **Distributed cache** - Redis/Memcached for hold endpoint cache in multi-instance deployments
|
||||
|
||||
|
||||
@@ -0,0 +1,637 @@
|
||||
# S3 Presigned URLs Implementation
|
||||
|
||||
## Overview
|
||||
|
||||
Currently, ATCR's hold service acts as a proxy for all blob data, meaning every byte flows through the hold service when uploading or downloading container images. This document describes the implementation of **S3 presigned URLs** to eliminate this bottleneck, allowing direct data transfer between clients and S3-compatible storage.
|
||||
|
||||
### Current Architecture (Proxy Mode)
|
||||
|
||||
```
|
||||
Downloads: Docker → AppView → Hold Service → S3 → Hold Service → AppView → Docker
|
||||
Uploads: Docker → AppView → Hold Service → S3
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- All blob data flows through hold service
|
||||
- Hold service bandwidth = total image bandwidth
|
||||
- Latency from extra hops
|
||||
- Hold service becomes bottleneck for large images
|
||||
|
||||
### Target Architecture (Presigned URLs)
|
||||
|
||||
```
|
||||
Downloads: Docker → AppView (gets presigned URL) → S3 (direct download)
|
||||
Uploads: Docker → AppView → S3 (via presigned URL)
|
||||
Move: AppView → Hold Service → S3 (server-side CopyObject API)
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- ✅ Hold service only orchestrates (no data transfer)
|
||||
- ✅ Blob data never touches hold service
|
||||
- ✅ Direct S3 uploads/downloads at wire speed
|
||||
- ✅ Hold service can run on minimal resources
|
||||
- ✅ Works with all S3-compatible services
|
||||
|
||||
## How Presigned URLs Work
|
||||
|
||||
### For Downloads (GET)
|
||||
|
||||
1. **Docker requests blob:** `GET /v2/alice/myapp/blobs/sha256:abc123`
|
||||
2. **AppView asks hold service:** `POST /get-presigned-url`
|
||||
```json
|
||||
{"did": "did:plc:alice123", "digest": "sha256:abc123"}
|
||||
```
|
||||
3. **Hold service generates presigned URL:**
|
||||
```go
|
||||
req, _ := s3Client.GetObjectRequest(&s3.GetObjectInput{
|
||||
Bucket: "my-bucket",
|
||||
Key: "blobs/sha256/ab/abc123.../data",
|
||||
})
|
||||
url, _ := req.Presign(15 * time.Minute)
|
||||
// Returns: https://gateway.storjshare.io/bucket/blobs/...?X-Amz-Signature=...
|
||||
```
|
||||
4. **AppView redirects Docker:** `HTTP 307 Location: <presigned-url>`
|
||||
5. **Docker downloads directly from S3** using the presigned URL
|
||||
|
||||
**Data path:** Docker → S3 (direct)
|
||||
**Hold service bandwidth:** ~1KB (API request/response)
|
||||
|
||||
### For Uploads (PUT)
|
||||
|
||||
**Small blobs (< 5MB) using Put():**
|
||||
|
||||
1. **Docker sends blob to AppView:** `PUT /v2/alice/myapp/blobs/uploads/{uuid}`
|
||||
2. **AppView asks hold service:** `POST /put-presigned-url`
|
||||
```json
|
||||
{"did": "did:plc:alice123", "digest": "sha256:abc123", "size": 1024}
|
||||
```
|
||||
3. **Hold service generates presigned URL:**
|
||||
```go
|
||||
req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{
|
||||
Bucket: "my-bucket",
|
||||
Key: "blobs/sha256/ab/abc123.../data",
|
||||
})
|
||||
url, _ := req.Presign(15 * time.Minute)
|
||||
```
|
||||
4. **AppView uploads to S3** using presigned URL
|
||||
5. **AppView confirms to Docker:** `201 Created`
|
||||
|
||||
**Data path:** Docker → AppView → S3 (via presigned URL)
|
||||
**Hold service bandwidth:** ~1KB (API request/response)
|
||||
|
||||
### For Streaming Uploads (Create/Commit)
|
||||
|
||||
**Large blobs (> 5MB) using streaming:**
|
||||
|
||||
1. **Docker starts upload:** `POST /v2/alice/myapp/blobs/uploads/`
|
||||
2. **AppView creates upload session** with UUID
|
||||
3. **AppView gets presigned URL for temp location:**
|
||||
```json
|
||||
POST /put-presigned-url
|
||||
{"did": "...", "digest": "uploads/temp-{uuid}", "size": 0}
|
||||
```
|
||||
4. **Docker streams data:** `PATCH /v2/alice/myapp/blobs/uploads/{uuid}`
|
||||
5. **AppView streams to S3** using presigned URL to `uploads/temp-{uuid}/data`
|
||||
6. **Docker finalizes:** `PUT /v2/.../uploads/{uuid}?digest=sha256:abc123`
|
||||
7. **AppView requests move:** `POST /move?from=uploads/temp-{uuid}&to=sha256:abc123`
|
||||
8. **Hold service executes S3 server-side copy:**
|
||||
```go
|
||||
s3.CopyObject(&s3.CopyObjectInput{
|
||||
Bucket: "my-bucket",
|
||||
CopySource: "/my-bucket/uploads/temp-{uuid}/data",
|
||||
Key: "blobs/sha256/ab/abc123.../data",
|
||||
})
|
||||
s3.DeleteObject(&s3.DeleteObjectInput{
|
||||
Key: "uploads/temp-{uuid}/data",
|
||||
})
|
||||
```
|
||||
|
||||
**Data path:** Docker → AppView → S3 (temp location)
|
||||
**Move path:** S3 internal copy (no data transfer!)
|
||||
**Hold service bandwidth:** ~2KB (presigned URL + CopyObject API)
|
||||
|
||||
## Why the Temp → Final Move is Required
|
||||
|
||||
This is **not an ATCR implementation detail** — it's required by the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push).
|
||||
|
||||
### The Problem: Unknown Digest
|
||||
|
||||
Docker doesn't know the blob's digest until **after** uploading:
|
||||
|
||||
1. **Streaming data:** Can't buffer 5GB layer in memory to calculate digest first
|
||||
2. **Stdin pipes:** `docker build . | docker push` generates data on-the-fly
|
||||
3. **Chunked uploads:** Multiple PATCH requests, digest calculated as data streams
|
||||
|
||||
### The Solution: Upload to Temp, Verify, Move
|
||||
|
||||
**All OCI registries do this:**
|
||||
|
||||
1. Client: `POST /v2/{name}/blobs/uploads/` → Get upload UUID
|
||||
2. Client: `PATCH /v2/{name}/blobs/uploads/{uuid}` → Stream data to temp location
|
||||
3. Client: `PUT /v2/{name}/blobs/uploads/{uuid}?digest=sha256:abc` → Provide digest
|
||||
4. Registry: Verify digest matches uploaded data
|
||||
5. Registry: Move `uploads/{uuid}` → `blobs/sha256/abc123...`
|
||||
|
||||
**Docker Hub, GHCR, ECR, Harbor — all use this pattern.**
|
||||
|
||||
### Why It's Efficient with S3
|
||||
|
||||
**For S3, the move is a CopyObject API call:**
|
||||
|
||||
```go
|
||||
// This happens INSIDE S3 servers - no data transfer!
|
||||
s3.CopyObject(&s3.CopyObjectInput{
|
||||
Bucket: "my-bucket",
|
||||
CopySource: "/my-bucket/uploads/temp-12345/data", // 5GB blob
|
||||
Key: "blobs/sha256/ab/abc123.../data",
|
||||
})
|
||||
// S3 copies internally, hold service only sends ~1KB API request
|
||||
```
|
||||
|
||||
**For a 5GB layer:**
|
||||
- Hold service bandwidth: **~1KB** (API request/response)
|
||||
- S3 internal copy: Instant (metadata operation on S3 side)
|
||||
- No data leaves S3, no network transfer
|
||||
|
||||
This is why the move operation is essentially free!
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add S3 Client to Hold Service
|
||||
|
||||
**File: `cmd/hold/main.go`**
|
||||
|
||||
Modify `HoldService` struct:
|
||||
```go
|
||||
type HoldService struct {
|
||||
driver storagedriver.StorageDriver
|
||||
config *Config
|
||||
s3Client *s3.S3 // NEW: S3 client for presigned URLs
|
||||
bucket string // NEW: Bucket name
|
||||
s3PathPrefix string // NEW: Path prefix (if any)
|
||||
}
|
||||
```
|
||||
|
||||
Add initialization function:
|
||||
```go
|
||||
func (s *HoldService) initS3Client() error {
|
||||
if s.config.Storage.Type() != "s3" {
|
||||
log.Printf("Storage driver is %s (not S3), presigned URLs disabled", s.config.Storage.Type())
|
||||
return nil
|
||||
}
|
||||
|
||||
params := s.config.Storage.Parameters()["s3"].(configuration.Parameters)
|
||||
|
||||
// Build AWS config
|
||||
awsConfig := &aws.Config{
|
||||
Region: aws.String(params["region"].(string)),
|
||||
Credentials: credentials.NewStaticCredentials(
|
||||
params["accesskey"].(string),
|
||||
params["secretkey"].(string),
|
||||
"",
|
||||
),
|
||||
}
|
||||
|
||||
// Add custom endpoint for S3-compatible services (Storj, MinIO, etc.)
|
||||
if endpoint, ok := params["regionendpoint"].(string); ok && endpoint != "" {
|
||||
awsConfig.Endpoint = aws.String(endpoint)
|
||||
awsConfig.S3ForcePathStyle = aws.Bool(true) // Required for MinIO, Storj
|
||||
}
|
||||
|
||||
sess, err := session.NewSession(awsConfig)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create AWS session: %w", err)
|
||||
}
|
||||
|
||||
s.s3Client = s3.New(sess)
|
||||
s.bucket = params["bucket"].(string)
|
||||
|
||||
log.Printf("S3 presigned URLs enabled for bucket: %s", s.bucket)
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
Call during service initialization:
|
||||
```go
|
||||
func NewHoldService(cfg *Config) (*HoldService, error) {
|
||||
// ... existing driver creation ...
|
||||
|
||||
service := &HoldService{
|
||||
driver: driver,
|
||||
config: cfg,
|
||||
}
|
||||
|
||||
// Initialize S3 client for presigned URLs
|
||||
if err := service.initS3Client(); err != nil {
|
||||
log.Printf("WARNING: S3 presigned URLs disabled: %v", err)
|
||||
}
|
||||
|
||||
return service, nil
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Implement Presigned URL Generation
|
||||
|
||||
**For Downloads:**
|
||||
|
||||
```go
|
||||
func (s *HoldService) getDownloadURL(ctx context.Context, digest string, did string) (string, error) {
|
||||
path := blobPath(digest)
|
||||
|
||||
// Check if blob exists
|
||||
if _, err := s.driver.Stat(ctx, path); err != nil {
|
||||
return "", fmt.Errorf("blob not found: %w", err)
|
||||
}
|
||||
|
||||
// If S3 client available, generate presigned URL
|
||||
if s.s3Client != nil {
|
||||
s3Key := strings.TrimPrefix(path, "/")
|
||||
|
||||
req, _ := s.s3Client.GetObjectRequest(&s3.GetObjectInput{
|
||||
Bucket: aws.String(s.bucket),
|
||||
Key: aws.String(s3Key),
|
||||
})
|
||||
|
||||
url, err := req.Presign(15 * time.Minute)
|
||||
if err != nil {
|
||||
log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
|
||||
return s.getProxyDownloadURL(digest, did), nil
|
||||
}
|
||||
|
||||
log.Printf("Generated presigned download URL for %s (expires in 15min)", digest)
|
||||
return url, nil
|
||||
}
|
||||
|
||||
// Fallback: return proxy URL
|
||||
return s.getProxyDownloadURL(digest, did), nil
|
||||
}
|
||||
|
||||
func (s *HoldService) getProxyDownloadURL(digest, did string) string {
|
||||
return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
|
||||
}
|
||||
```
|
||||
|
||||
**For Uploads:**
|
||||
|
||||
```go
|
||||
func (s *HoldService) getUploadURL(ctx context.Context, digest string, size int64, did string) (string, error) {
|
||||
path := blobPath(digest)
|
||||
|
||||
// If S3 client available, generate presigned URL
|
||||
if s.s3Client != nil {
|
||||
s3Key := strings.TrimPrefix(path, "/")
|
||||
|
||||
req, _ := s.s3Client.PutObjectRequest(&s3.PutObjectInput{
|
||||
Bucket: aws.String(s.bucket),
|
||||
Key: aws.String(s3Key),
|
||||
})
|
||||
|
||||
url, err := req.Presign(15 * time.Minute)
|
||||
if err != nil {
|
||||
log.Printf("WARN: Presigned URL generation failed, falling back to proxy: %v", err)
|
||||
return s.getProxyUploadURL(digest, did), nil
|
||||
}
|
||||
|
||||
log.Printf("Generated presigned upload URL for %s (expires in 15min)", digest)
|
||||
return url, nil
|
||||
}
|
||||
|
||||
// Fallback: return proxy URL
|
||||
return s.getProxyUploadURL(digest, did), nil
|
||||
}
|
||||
|
||||
func (s *HoldService) getProxyUploadURL(digest, did string) string {
|
||||
return fmt.Sprintf("%s/blobs/%s?did=%s", s.config.Server.PublicURL, digest, did)
|
||||
}
|
||||
```
|
||||
|
||||
### 3. No Changes Needed for Move Operation
|
||||
|
||||
The existing `/move` endpoint already uses `driver.Move()`, which for S3:
|
||||
- Calls `s3.CopyObject()` (server-side copy)
|
||||
- Calls `s3.DeleteObject()` (delete source)
|
||||
- No data transfer through hold service!
|
||||
|
||||
**File: `cmd/hold/main.go:296` (already exists, no changes needed)**
|
||||
|
||||
```go
|
||||
func (s *HoldService) HandleMove(w http.ResponseWriter, r *http.Request) {
|
||||
// ... existing auth and parsing ...
|
||||
|
||||
sourcePath := blobPath(fromPath) // uploads/temp-{uuid}/data
|
||||
destPath := blobPath(toDigest) // blobs/sha256/ab/abc123.../data
|
||||
|
||||
// For S3, this does CopyObject + DeleteObject (server-side)
|
||||
if err := s.driver.Move(ctx, sourcePath, destPath); err != nil {
|
||||
// ... error handling ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4. AppView Changes (Optional Optimization)
|
||||
|
||||
**File: `pkg/storage/proxy_blob_store.go:228`**
|
||||
|
||||
Currently streams to hold service proxy URL. Could be optimized to use presigned URL:
|
||||
|
||||
```go
|
||||
// In Create() - line 228
|
||||
go func() {
|
||||
defer pipeReader.Close()
|
||||
|
||||
tempPath := fmt.Sprintf("uploads/temp-%s", writer.id)
|
||||
|
||||
// Try to get presigned URL for temp location
|
||||
url, err := p.getUploadURL(ctx, digest.FromString(tempPath), 0)
|
||||
if err != nil {
|
||||
// Fallback to direct proxy URL
|
||||
url = fmt.Sprintf("%s/blobs/%s?did=%s", p.storageEndpoint, tempPath, p.did)
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(uploadCtx, "PUT", url, pipeReader)
|
||||
// ... rest unchanged
|
||||
}()
|
||||
```
|
||||
|
||||
**Note:** This optimization is optional. The presigned URL will be returned by hold service's `getUploadURL()` anyway.
|
||||
|
||||
## S3-Compatible Service Support
|
||||
|
||||
### Storj
|
||||
|
||||
```bash
|
||||
# .env file
|
||||
STORAGE_DRIVER=s3
|
||||
AWS_ACCESS_KEY_ID=your-storj-access-key
|
||||
AWS_SECRET_ACCESS_KEY=your-storj-secret-key
|
||||
S3_BUCKET=your-bucket-name
|
||||
S3_REGION=global
|
||||
S3_ENDPOINT=https://gateway.storjshare.io
|
||||
```
|
||||
|
||||
**Presigned URL example:**
|
||||
```
|
||||
https://gateway.storjshare.io/your-bucket/blobs/sha256/ab/abc123.../data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Signature=...
|
||||
```
|
||||
|
||||
### MinIO
|
||||
|
||||
```bash
|
||||
STORAGE_DRIVER=s3
|
||||
AWS_ACCESS_KEY_ID=minioadmin
|
||||
AWS_SECRET_ACCESS_KEY=minioadmin
|
||||
S3_BUCKET=registry
|
||||
S3_REGION=us-east-1
|
||||
S3_ENDPOINT=http://minio.example.com:9000
|
||||
```
|
||||
|
||||
### Backblaze B2
|
||||
|
||||
```bash
|
||||
STORAGE_DRIVER=s3
|
||||
AWS_ACCESS_KEY_ID=your-b2-key-id
|
||||
AWS_SECRET_ACCESS_KEY=your-b2-application-key
|
||||
S3_BUCKET=your-bucket-name
|
||||
S3_REGION=us-west-002
|
||||
S3_ENDPOINT=https://s3.us-west-002.backblazeb2.com
|
||||
```
|
||||
|
||||
### Cloudflare R2
|
||||
|
||||
```bash
|
||||
STORAGE_DRIVER=s3
|
||||
AWS_ACCESS_KEY_ID=your-r2-access-key-id
|
||||
AWS_SECRET_ACCESS_KEY=your-r2-secret-access-key
|
||||
S3_BUCKET=your-bucket-name
|
||||
S3_REGION=auto
|
||||
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
|
||||
```
|
||||
|
||||
**All these services support presigned URLs with AWS SDK v1!**
|
||||
|
||||
## Performance Impact
|
||||
|
||||
### Bandwidth Savings
|
||||
|
||||
**Before (proxy mode):**
|
||||
- 5GB layer upload: Hold service receives 5GB, sends 5GB to S3 = **10GB** bandwidth
|
||||
- 5GB layer download: S3 sends 5GB to hold, hold sends 5GB to client = **10GB** bandwidth
|
||||
- **Total for push+pull: 20GB hold service bandwidth**
|
||||
|
||||
**After (presigned URLs):**
|
||||
- 5GB layer upload: Hold generates URL (1KB), AppView → S3 direct (5GB), CopyObject API (1KB) = **~2KB** hold bandwidth
|
||||
- 5GB layer download: Hold generates URL (1KB), client → S3 direct = **~1KB** hold bandwidth
|
||||
- **Total for push+pull: ~3KB hold service bandwidth**
|
||||
|
||||
**Savings: 99.98% reduction in hold service bandwidth!**
|
||||
|
||||
### Latency Improvements
|
||||
|
||||
**Before:**
|
||||
- Download: Client → AppView → Hold → S3 → Hold → AppView → Client (4 hops)
|
||||
- Upload: Client → AppView → Hold → S3 (3 hops)
|
||||
|
||||
**After:**
|
||||
- Download: Client → AppView (redirect) → S3 (1 hop to data)
|
||||
- Upload: Client → AppView → S3 (2 hops)
|
||||
- Move: S3 internal (no network hops)
|
||||
|
||||
### Resource Requirements
|
||||
|
||||
**Before:**
|
||||
- Hold service needs bandwidth = sum of all image operations
|
||||
- For 100 concurrent 1GB pushes: 100GB/s bandwidth needed
|
||||
- Expensive, hard to scale
|
||||
|
||||
**After:**
|
||||
- Hold service needs minimal CPU for presigned URL signing
|
||||
- For 100 concurrent 1GB pushes: ~100KB/s bandwidth needed (API traffic)
|
||||
- Can run on $5/month instance!
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Presigned URL Expiration
|
||||
|
||||
- Default: **15 minutes** expiration
|
||||
- Presigned URL includes embedded credentials in query params
|
||||
- After expiry, URL becomes invalid (S3 rejects with 403)
|
||||
- No long-lived URLs floating around
|
||||
|
||||
### Authorization Flow
|
||||
|
||||
1. **AppView validates user** via ATProto OAuth
|
||||
2. **AppView passes DID to hold service** in presigned URL request
|
||||
3. **Hold service validates DID** (owner or crew member)
|
||||
4. **Hold service generates presigned URL** if authorized
|
||||
5. **Client uses presigned URL** directly with S3
|
||||
|
||||
**Security boundary:** Hold service controls who gets presigned URLs, S3 validates the URLs.
|
||||
|
||||
### Fallback Security
|
||||
|
||||
If presigned URL generation fails:
|
||||
- Falls back to proxy URLs (existing behavior)
|
||||
- Still requires hold service authorization
|
||||
- Data flows through hold service (original security model)
|
||||
|
||||
## Testing & Validation
|
||||
|
||||
### Verify Presigned URLs are Used
|
||||
|
||||
**1. Check hold service logs:**
|
||||
```bash
|
||||
docker logs atcr-hold | grep -i presigned
|
||||
# Should see: "Generated presigned download/upload URL for sha256:..."
|
||||
```
|
||||
|
||||
**2. Monitor network traffic:**
|
||||
```bash
|
||||
# Before: Large data transfers to/from hold service
|
||||
docker stats atcr-hold
|
||||
|
||||
# After: Minimal network usage on hold service
|
||||
docker stats atcr-hold
|
||||
```
|
||||
|
||||
**3. Inspect redirect responses:**
|
||||
```bash
|
||||
# Should see 307 redirect to S3 URL
|
||||
curl -v http://appview:5000/v2/alice/myapp/blobs/sha256:abc123 \
|
||||
-H "Authorization: Bearer $TOKEN"
|
||||
|
||||
# Look for:
|
||||
# < HTTP/1.1 307 Temporary Redirect
|
||||
# < Location: https://gateway.storjshare.io/...?X-Amz-Signature=...
|
||||
```
|
||||
|
||||
### Test Fallback Behavior
|
||||
|
||||
**1. With filesystem driver (should use proxy URLs):**
|
||||
```bash
|
||||
STORAGE_DRIVER=filesystem docker-compose up atcr-hold
|
||||
# Logs should show: "Storage driver is filesystem (not S3), presigned URLs disabled"
|
||||
```
|
||||
|
||||
**2. With S3 but invalid credentials (should fall back):**
|
||||
```bash
|
||||
AWS_ACCESS_KEY_ID=invalid docker-compose up atcr-hold
|
||||
# Logs should show: "WARN: Presigned URL generation failed, falling back to proxy"
|
||||
```
|
||||
|
||||
### Bandwidth Monitoring
|
||||
|
||||
**Track hold service bandwidth over time:**
|
||||
```bash
|
||||
# Install bandwidth monitoring
|
||||
docker exec atcr-hold apt-get update && apt-get install -y vnstat
|
||||
|
||||
# Monitor
|
||||
docker exec atcr-hold vnstat -l
|
||||
```
|
||||
|
||||
**Expected results:**
|
||||
- Before: Bandwidth correlates with image operations
|
||||
- After: Bandwidth stays minimal regardless of image operations
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### For Existing ATCR Deployments
|
||||
|
||||
**1. Update hold service code** (this implementation)
|
||||
|
||||
**2. No configuration changes needed** if already using S3:
|
||||
```bash
|
||||
# Existing S3 config works automatically
|
||||
STORAGE_DRIVER=s3
|
||||
AWS_ACCESS_KEY_ID=...
|
||||
AWS_SECRET_ACCESS_KEY=...
|
||||
S3_BUCKET=...
|
||||
S3_ENDPOINT=...
|
||||
```
|
||||
|
||||
**3. Restart hold service:**
|
||||
```bash
|
||||
docker-compose restart atcr-hold
|
||||
```
|
||||
|
||||
**4. Verify in logs:**
|
||||
```
|
||||
S3 presigned URLs enabled for bucket: my-bucket
|
||||
```
|
||||
|
||||
**5. Test with image push/pull:**
|
||||
```bash
|
||||
docker push atcr.io/alice/myapp:latest
|
||||
docker pull atcr.io/alice/myapp:latest
|
||||
```
|
||||
|
||||
**6. Monitor bandwidth** to confirm reduction
|
||||
|
||||
### Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
|
||||
**Option 1: Disable presigned URLs via env var** (if we add this feature)
|
||||
```bash
|
||||
PRESIGNED_URLS_ENABLED=false docker-compose restart atcr-hold
|
||||
```
|
||||
|
||||
**Option 2: Revert code changes** to previous hold service version
|
||||
|
||||
The implementation has automatic fallbacks, so partial failures won't break functionality.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### 1. Configurable Expiration
|
||||
|
||||
Allow customizing presigned URL expiry:
|
||||
```bash
|
||||
PRESIGNED_URL_EXPIRY=30m # Default: 15m
|
||||
```
|
||||
|
||||
### 2. Presigned URL Caching
|
||||
|
||||
Cache presigned URLs for frequently accessed blobs (with shorter TTL).
|
||||
|
||||
### 3. CloudFront/CDN Integration
|
||||
|
||||
For downloads, use CloudFront presigned URLs instead of direct S3:
|
||||
- Better global distribution
|
||||
- Lower egress costs
|
||||
- Faster downloads
|
||||
|
||||
### 4. Multipart Upload Support
|
||||
|
||||
For very large layers (>5GB), use presigned URLs with multipart upload:
|
||||
- Generate presigned URLs for each part
|
||||
- Client uploads parts directly to S3
|
||||
- Hold service finalizes multipart upload
|
||||
|
||||
### 5. Metrics & Monitoring
|
||||
|
||||
Track presigned URL usage:
|
||||
- Count of presigned URLs generated
|
||||
- Fallback rate (proxy vs presigned)
|
||||
- Bandwidth savings metrics
|
||||
|
||||
## References
|
||||
|
||||
- [OCI Distribution Specification - Push](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push)
|
||||
- [AWS SDK Go v1 - Presigned URLs](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html)
|
||||
- [Storj - Using Presigned URLs](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway/using-presigned-urls)
|
||||
- [MinIO - Presigned Upload via Browser](https://docs.min.io/community/minio-object-store/integrations/presigned-put-upload-via-browser.html)
|
||||
- [Cloudflare R2 - Presigned URLs](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)
|
||||
- [Backblaze B2 - S3 Compatible API](https://help.backblaze.com/hc/en-us/articles/360047815993-Does-the-B2-S3-Compatible-API-support-Pre-Signed-URLs)
|
||||
|
||||
## Summary
|
||||
|
||||
Implementing S3 presigned URLs transforms ATCR's hold service from a **data proxy** to a **lightweight orchestrator**:
|
||||
|
||||
✅ **99.98% bandwidth reduction** for hold service
|
||||
✅ **Direct client → S3 transfers** for maximum speed
|
||||
✅ **Works with all S3-compatible services** (Storj, MinIO, R2, B2)
|
||||
✅ **OCI-compliant** temp → final move pattern
|
||||
✅ **Automatic fallback** to proxy mode for non-S3 drivers
|
||||
✅ **No breaking changes** to existing deployments
|
||||
|
||||
This makes BYOS (Bring Your Own Storage) truly scalable and cost-effective, as users can run hold services on minimal infrastructure while serving arbitrarily large container images.
|
||||
Reference in New Issue
Block a user