# ATCR Quota System This document describes ATCR's storage quota implementation using ATProto records for per-user layer tracking. ## Table of Contents - [Overview](#overview) - [Quota Model](#quota-model) - [Layer Record Schema](#layer-record-schema) - [Quota Calculation](#quota-calculation) - [Push Flow](#push-flow) - [Delete Flow](#delete-flow) - [Garbage Collection](#garbage-collection) - [Configuration](#configuration) - [Quota API Endpoints](#quota-api-endpoints) - [Future Enhancements](#future-enhancements) ## Overview ATCR implements per-user storage quotas to: 1. **Limit storage consumption** on shared hold services 2. **Provide transparency** (show users their storage usage) 3. **Enable fair billing** (users pay for what they use) **Key principle:** Users pay for layers they reference, deduplicated per-user. If you push the same layer in multiple images, you only pay once. ### Example Scenario ``` Alice pushes myapp:v1 (layers A, B, C - each 100MB) → Creates 3 layer records in hold's PDS → Alice's quota: 300MB (3 unique layers) Alice pushes myapp:v2 (layers A, B, D) → Creates 3 more layer records (A, B again, plus D) → Alice's quota: 400MB (4 unique layers: A, B, C, D) → Layers A, B appear twice in records but deduplicated in quota calc Bob pushes his-app:latest (layers A, E) → Creates 2 layer records for Bob → Bob's quota: 200MB (2 unique layers: A, E) → Layer A shared with Alice in S3, but Bob pays for his own usage Physical S3 storage: 500MB (A, B, C, D, E - deduplicated globally) Alice's quota: 400MB Bob's quota: 200MB ``` ## Quota Model ### Everyone Pays for What They Upload Each user is charged for all unique layers they reference, regardless of whether those layers exist in S3 from other users' uploads. **Why this model?** - **Simple mental model**: "I pushed 500MB of layers, I use 500MB of quota" - **Predictable**: Your quota doesn't change based on others' actions - **Clean deletion**: Delete manifest → layer records removed → quota freed - **No cross-user dependencies**: Users are isolated **Trade-off:** - Total claimed storage can exceed physical S3 storage - This is acceptable - deduplication is an operational benefit for ATCR, not a billing feature ### ATProto-Native Storage Layer tracking uses ATProto records stored in the hold's embedded PDS: - **Collection**: `io.atcr.hold.layer` - **Repository**: Hold's DID (e.g., `did:web:hold01.atcr.io`) - **Records**: One per manifest-layer relationship (TID-based keys) This approach: - Keeps quota data in ATProto (no separate database) - Enables standard ATProto sync/query mechanisms - Provides full audit trail of layer usage ## Layer Record Schema ### LayerRecord ```go // pkg/atproto/lexicon.go type LayerRecord struct { Type string `json:"$type"` // "io.atcr.hold.layer" Digest string `json:"digest"` // Layer digest (sha256:abc123...) Size int64 `json:"size"` // Size in bytes MediaType string `json:"mediaType"` // e.g., "application/vnd.oci.image.layer.v1.tar+gzip" Manifest string `json:"manifest"` // at://did:plc:alice/io.atcr.manifest/abc123 UserDID string `json:"userDid"` // User's DID for quota grouping CreatedAt string `json:"createdAt"` // ISO 8601 timestamp } ``` ### Record Key Records use TID (timestamp-based ID) as the rkey. This means: - Multiple records can exist for the same layer (from different manifests) - Deduplication happens at query time, not storage time - Simple append-only writes on manifest push ### Example Records ``` Manifest A (layers X, Y, Z) → creates 3 records Manifest B (layers X, W) → creates 2 records io.atcr.hold.layer collection: ┌──────────────┬────────┬──────┬───────────────────────────────────┬─────────────────┐ │ rkey (TID) │ digest │ size │ manifest │ userDid │ ├──────────────┼────────┼──────┼───────────────────────────────────┼─────────────────┤ │ 3jui7...001 │ X │ 100 │ at://did:plc:alice/.../manifestA │ did:plc:alice │ │ 3jui7...002 │ Y │ 200 │ at://did:plc:alice/.../manifestA │ did:plc:alice │ │ 3jui7...003 │ Z │ 150 │ at://did:plc:alice/.../manifestA │ did:plc:alice │ │ 3jui7...004 │ X │ 100 │ at://did:plc:alice/.../manifestB │ did:plc:alice │ ← duplicate digest │ 3jui7...005 │ W │ 300 │ at://did:plc:alice/.../manifestB │ did:plc:alice │ └──────────────┴────────┴──────┴───────────────────────────────────┴─────────────────┘ ``` ## Quota Calculation ### Query: User's Unique Storage ```sql -- Calculate quota by deduplicating layers SELECT SUM(size) FROM ( SELECT DISTINCT digest, size FROM io.atcr.hold.layer WHERE userDid = ? ) ``` Using the example above: - Layer X appears twice but counted once: 100 - Layers Y, Z, W counted once each: 200 + 150 + 300 - **Total: 750 bytes** ### Implementation Usage is computed **hold-side**, where the layer records actually live. There is no `QuotaManager` type — the pieces are: - **`quota.Manager`** (`pkg/hold/quota/config.go`) resolves a crew tier name to a byte limit (tier → limit only). It does not compute usage. `Manager.IsEnabled()` reports whether any tiers are configured. - **`recordsIndex.QuotaForDID(collection, userDID)`** runs the deduplicating SQL aggregation over the denormalized `digest`/`size` columns in the records index, returning `(uniqueBlobs, totalSize)`. - **`HoldPDS.GetQuotaForUser`** (`pkg/hold/pds/layer.go`) wraps `QuotaForDID` and returns a `QuotaStats`. - **`HoldPDS.GetQuotaForUserWithTier`** (`pkg/hold/pds/layer.go`) layers the tier limit on top: captain (owner) is always unlimited; otherwise it looks up the crew member's tier and asks `quota.Manager` for the limit. ```go // pkg/hold/pds/layer.go // QuotaStats represents storage quota information for a user. type QuotaStats struct { UserDID string `json:"userDid"` UniqueBlobs int `json:"uniqueBlobs"` TotalSize int64 `json:"totalSize"` Limit *int64 `json:"limit,omitempty"` // nil = unlimited Tier string `json:"tier,omitempty"` // e.g. "deckhand", "bosun" } // GetQuotaForUser deduplicates layer records via SQL aggregation. func (p *HoldPDS) GetQuotaForUser(ctx context.Context, userDID string) (*QuotaStats, error) // GetQuotaForUserWithTier adds the tier-resolved Limit (captain = unlimited). func (p *HoldPDS) GetQuotaForUserWithTier(ctx context.Context, userDID string, quotaMgr *quota.Manager) (*QuotaStats, error) ``` `QuotaStats` is the JSON shape returned by the public `io.atcr.hold.getQuota` endpoint, which the appview calls during the push gate (see [Push Flow](#push-flow)). ## Push Flow ### Where Quota Is Enforced Quota is enforced in the appview's **push authorizer at `/auth/token` time**, before the push begins — not inside the manifest store. When a Docker client requests a push token, the authorizer (`pkg/appview/authgate/push_authorizer.go`, `checkQuota`) calls `atproto.FetchQuotaStats` against the hold's public `io.atcr.hold.getQuota` endpoint and rejects the token request when the user's current usage already meets or exceeds their limit. ``` ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Client │ │ AppView │ │ Hold │ │ (Docker) │ │ /auth/ │ │ getQuota │ │ │ │ token │ │ │ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ 1. Request push token│ │ ├─────────────────────>│ │ │ │ 2. checkQuota: │ │ │ FetchQuotaStats │ │ ├─────────────────────>│ │ │ 3. {totalSize,limit} │ │ │<─────────────────────┤ │ │ │ │ │ 4. if limit != nil │ │ │ && totalSize >= │ │ │ limit → reject │ │ │ (denied: quota) │ │ │ │ │ 5. 200 token / 401 │ │ │<─────────────────────┤ │ │ │ │ │ 6. Push blobs + manifest (if token granted) │ ├─────────────────────>│ ... │ ``` ### Behavior - **Check is current-usage vs. limit, evaluated before the push.** The condition is `stats.TotalSize >= *stats.Limit`. It does **not** pre-add the incoming layer sizes; a push is allowed to start whenever existing usage is strictly below the limit, even if that push will push the user over. Overage is caught on the *next* push attempt. - **Unlimited tiers skip the check.** `stats.Limit` is `nil` for captains and for holds with no quota tiers configured, in which case no limit is enforced. - **Fails open on errors.** If the hold's `getQuota` endpoint is unreachable or returns an error, `checkQuota` logs a warning and **allows the push**. This is deliberate: the endpoint is public/unauthenticated, and a hold outage should not produce spurious `denied: quota` errors unrelated to the user's actual usage. ### Implementation ```go // pkg/appview/authgate/push_authorizer.go func (a *Authorizer) checkQuota(ctx context.Context, userDID, holdDID string) error { stats, err := atproto.FetchQuotaStats(ctx, a.httpClient, holdDID, userDID) if err != nil { // Fail open: a hold outage must not block unrelated pushes. slog.Warn("push gate: quota call failed; allowing push", "did", userDID, "error", err) return nil } if stats.Limit != nil && stats.TotalSize >= *stats.Limit { return fmt.Errorf("quota exceeded: %s / %s used by %s. Delete images to free space", formatGB(stats.TotalSize), formatGB(*stats.Limit), a.identityLabel(ctx, userDID)) } return nil } ``` Layer records that back the usage figure are created hold-side as part of the blob upload / push pipeline, not by the appview's manifest store. ## Delete Flow ### Manifest Deletion When a user deletes a manifest: ``` ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ User │ │ AppView │ │ Hold │ │ User PDS │ │ UI │ │ │ │ Service │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ │ DELETE manifest │ │ │ ├─────────────────────>│ │ │ │ │ │ │ │ │ 1. Delete manifest │ │ │ │ from user's PDS │ │ │ ├──────────────────────┼─────────────────────>│ │ │ │ │ │ │ 2. Delete layer │ │ │ │ records for this │ │ │ │ manifest │ │ │ ├─────────────────────>│ │ │ │ │ 3. Remove records │ │ │ │ where manifest │ │ │ │ == deleted URI │ │ │ │ │ │ 4. 204 No Content │ │ │ │<─────────────────────┤ │ │ ``` ### Hold Service: Delete Layer Record The deletion primitive is hold-side: `HoldPDS.DeleteLayerRecord` in `pkg/hold/pds/layer.go`. It removes a single layer record by its rkey from both the repo (the MST/CAR store) and the records index (the index delete is best-effort — failures are logged, since a backfill resync will reconcile it). ```go // pkg/hold/pds/layer.go // DeleteLayerRecord deletes a layer record by rkey from the repo (MST) // and the records index. func (p *HoldPDS) DeleteLayerRecord(ctx context.Context, rkey string) error ``` To remove all layer records for a deleted manifest, callers identify the rkeys whose records reference that manifest and call `DeleteLayerRecord` for each. Orphan cleanup also happens automatically during garbage collection (see below), so a missed deletion is eventually reconciled rather than leaking quota permanently. ### Quota After Deletion After deleting a manifest: - Layer records for that manifest are removed - Quota recalculated with `SELECT DISTINCT` query - If layer was only in deleted manifest → quota decreases - If layer exists in other manifests → quota unchanged (still deduplicated) ## Garbage Collection ### Orphaned Blobs Orphaned blobs accumulate when: 1. Manifest push fails after blobs uploaded 2. Quota exceeded - manifest rejected 3. User deletes manifest - blobs may no longer be referenced ### GC Process GC is implemented in `pkg/hold/gc/gc.go`. The public entrypoint `GarbageCollector.Run` takes a single-run lock and delegates to `doRun`, which works in multiple phases (see `(*GarbageCollector).doRun`): 1. **Analyze records** (`analyzeRecords`) — build the set of referenced layer digests, identify orphaned layer records (rkeys), and find manifest layers missing a record. 2. **Reconcile missing records** — create layer records for referenced layers that lost their record, so usage accounting stays correct. 3. **Delete orphaned layer records** (`deleteOrphanedRecords`) — remove layer records no longer referenced by any manifest. 4. **Delete orphaned blobs** (`deleteOrphanedBlobs`) — walk S3 and delete blobs not in the referenced set. A read-only `Preview` / `doPreview` path runs phases 1 and the blob scan without deleting anything, for the admin panel. A grace period (`gcGracePeriod`, 7 days) protects recently created records from collection. ### GC Schedule GC is toggled by a single config field; the interval is **not** configurable — it is the hardcoded `gcInterval` constant (24h) in `pkg/hold/gc/config.go`. ```yaml # config-hold.yaml gc: enabled: true # env: GC_ENABLED ``` There is no `GC_INTERVAL` setting. ## Configuration ### Enabling Quotas Quotas are **enabled by the presence of quota tiers** in the hold config — there is no `QUOTA_ENABLED` flag and no global `QUOTA_DEFAULT_LIMIT`. `quota.Manager.IsEnabled()` returns true whenever a config with at least one tier was loaded; with no tiers, every user is unlimited. ```yaml # config-hold.yaml quota: tiers: - name: deckhand quota: 5GB - name: bosun quota: 50GB scan_on_push: true - name: quartermaster quota: 100GB scan_on_push: true defaults: new_crew_tier: deckhand gc: enabled: true # env: GC_ENABLED (interval is a hardcoded 24h, not configurable) ``` The only quota/GC environment variable is `GC_ENABLED` (bound to `gc.enabled`). Tier limits are human-readable sizes (`5GB`, `50GB`, `1TB`) parsed by `quota.ParseHumanBytes`. ### Quota Limits by Bytes | Size | Bytes | |------|-------| | 1 GB | 1073741824 | | 5 GB | 5368709120 | | 10 GB | 10737418240 | | 50 GB | 53687091200 | | 100 GB | 107374182400 | ## Quota API Endpoints The per-user quota endpoint is **implemented**. It is public (no auth) and is what the appview's push gate calls (see [Push Flow](#push-flow)). ``` GET /xrpc/io.atcr.hold.getQuota?userDid={did} → {"userDid": "...", "uniqueBlobs": 10, "totalSize": 1073741824, "limit": ..., "tier": "..."} ``` - Endpoint constant: `HoldGetQuota` in `pkg/atproto/endpoints.go` - Handler: `HandleGetQuota`, registered in `pkg/hold/pds/xrpc.go` - Client helper: `atproto.FetchQuotaStats` in `pkg/atproto/quota.go` ## Future Enhancements ### 1. Quota Breakdown Endpoint ``` GET /xrpc/io.atcr.hold.getQuotaBreakdown - Storage by repository (not yet implemented) ``` ### 2. Quota Alerts - Warning thresholds at 80%, 90%, 95% - Email/webhook notifications - Grace period before hard enforcement ### 3. Tier-Based Quotas (Implemented) ATCR uses quota tiers to limit storage per crew member, configured via `quotas.yaml`: ```yaml # quotas.yaml tiers: deckhand: # Entry-level crew quota: 5GB bosun: # Mid-level crew quota: 50GB quartermaster: # High-level crew quota: 100GB defaults: new_crew_tier: deckhand # Default tier for new crew members ``` | Tier | Limit | Description | |------|-------|-------------| | deckhand | 5 GB | Entry-level crew member | | bosun | 50 GB | Mid-level crew member | | quartermaster | 100 GB | Senior crew member | | owner (captain) | Unlimited | Hold owner always has unlimited | **Tier Resolution:** 1. If user is captain (owner) → unlimited 2. If crew member has explicit tier → use that tier's limit 3. If crew member has no tier → use `defaults.new_crew_tier` 4. If default tier not found → unlimited **Crew Record Example:** ```json { "$type": "io.atcr.hold.crew", "member": "did:plc:alice123", "role": "writer", "permissions": ["blob:write"], "tier": "bosun", "addedAt": "2026-01-04T12:00:00Z" } ``` ### 4. Rate Limiting Pull rate limits (Docker Hub style): - Anonymous: 100 pulls per 6 hours per IP - Authenticated: 200 pulls per 6 hours - Paid: Unlimited ### 5. Quota Purchasing - Stripe integration for additional storage - $0.10/GB/month pricing (industry standard) ## References - **Harbor Quotas:** https://goharbor.io/docs/1.10/administration/configure-project-quotas/ - **ATProto Spec:** https://atproto.com/specs/record - **OCI Distribution Spec:** https://github.com/opencontainers/distribution-spec --- **Document Version:** 2.1 **Last Updated:** 2026-06-11 **Model:** Per-user layer tracking with ATProto records