at-container-registry/docs/QUOTAS.md

ATCR Quota System

This document describes ATCR's storage quota implementation using ATProto records for per-user layer tracking.

Table of Contents

• Overview
• Quota Model
• Layer Record Schema
• Quota Calculation
• Push Flow
• Delete Flow
• Garbage Collection
• Configuration
• Quota API Endpoints
• Future Enhancements

Overview

ATCR implements per-user storage quotas to:

1. Limit storage consumption on shared hold services
2. Provide transparency (show users their storage usage)
3. Enable fair billing (users pay for what they use)

Key principle: Users pay for layers they reference, deduplicated per-user. If you push the same layer in multiple images, you only pay once.

Example Scenario

Alice pushes myapp:v1 (layers A, B, C - each 100MB)
→ Creates 3 layer records in hold's PDS
→ Alice's quota: 300MB (3 unique layers)

Alice pushes myapp:v2 (layers A, B, D)
→ Creates 3 more layer records (A, B again, plus D)
→ Alice's quota: 400MB (4 unique layers: A, B, C, D)
→ Layers A, B appear twice in records but deduplicated in quota calc

Bob pushes his-app:latest (layers A, E)
→ Creates 2 layer records for Bob
→ Bob's quota: 200MB (2 unique layers: A, E)
→ Layer A shared with Alice in S3, but Bob pays for his own usage

Physical S3 storage: 500MB (A, B, C, D, E - deduplicated globally)
Alice's quota: 400MB
Bob's quota: 200MB

Quota Model

Everyone Pays for What They Upload

Each user is charged for all unique layers they reference, regardless of whether those layers exist in S3 from other users' uploads.

Why this model?

• Simple mental model: "I pushed 500MB of layers, I use 500MB of quota"
• Predictable: Your quota doesn't change based on others' actions
• Clean deletion: Delete manifest → layer records removed → quota freed
• No cross-user dependencies: Users are isolated

Trade-off:

• Total claimed storage can exceed physical S3 storage
• This is acceptable - deduplication is an operational benefit for ATCR, not a billing feature

ATProto-Native Storage

Layer tracking uses ATProto records stored in the hold's embedded PDS:

• Collection: io.atcr.hold.layer
• Repository: Hold's DID (e.g., did:web:hold01.atcr.io)
• Records: One per manifest-layer relationship (TID-based keys)

This approach:

• Keeps quota data in ATProto (no separate database)
• Enables standard ATProto sync/query mechanisms
• Provides full audit trail of layer usage

Layer Record Schema

LayerRecord

// pkg/atproto/lexicon.go

type LayerRecord struct {
    Type      string `json:"$type"`     // "io.atcr.hold.layer"
    Digest    string `json:"digest"`    // Layer digest (sha256:abc123...)
    Size      int64  `json:"size"`      // Size in bytes
    MediaType string `json:"mediaType"` // e.g., "application/vnd.oci.image.layer.v1.tar+gzip"
    Manifest  string `json:"manifest"`  // at://did:plc:alice/io.atcr.manifest/abc123
    UserDID   string `json:"userDid"`   // User's DID for quota grouping
    CreatedAt string `json:"createdAt"` // ISO 8601 timestamp
}

Record Key

Records use TID (timestamp-based ID) as the rkey. This means:

• Multiple records can exist for the same layer (from different manifests)
• Deduplication happens at query time, not storage time
• Simple append-only writes on manifest push

Example Records

Manifest A (layers X, Y, Z) → creates 3 records
Manifest B (layers X, W)    → creates 2 records

io.atcr.hold.layer collection:
┌──────────────┬────────┬──────┬───────────────────────────────────┬─────────────────┐
│ rkey (TID)   │ digest │ size │ manifest                          │ userDid         │
├──────────────┼────────┼──────┼───────────────────────────────────┼─────────────────┤
│ 3jui7...001  │ X      │ 100  │ at://did:plc:alice/.../manifestA  │ did:plc:alice   │
│ 3jui7...002  │ Y      │ 200  │ at://did:plc:alice/.../manifestA  │ did:plc:alice   │
│ 3jui7...003  │ Z      │ 150  │ at://did:plc:alice/.../manifestA  │ did:plc:alice   │
│ 3jui7...004  │ X      │ 100  │ at://did:plc:alice/.../manifestB  │ did:plc:alice   │  ← duplicate digest
│ 3jui7...005  │ W      │ 300  │ at://did:plc:alice/.../manifestB  │ did:plc:alice   │
└──────────────┴────────┴──────┴───────────────────────────────────┴─────────────────┘

Quota Calculation

Query: User's Unique Storage

-- Calculate quota by deduplicating layers
SELECT SUM(size) FROM (
    SELECT DISTINCT digest, size
    FROM io.atcr.hold.layer
    WHERE userDid = ?
)

Using the example above:

• Layer X appears twice but counted once: 100
• Layers Y, Z, W counted once each: 200 + 150 + 300
• Total: 750 bytes

Implementation

Usage is computed hold-side, where the layer records actually live. There is no QuotaManager type — the pieces are:

• quota.Manager (pkg/hold/quota/config.go) resolves a crew tier name to a byte limit (tier → limit only). It does not compute usage. Manager.IsEnabled() reports whether any tiers are configured.
• recordsIndex.QuotaForDID(collection, userDID) runs the deduplicating SQL aggregation over the denormalized digest/size columns in the records index, returning (uniqueBlobs, totalSize).
• HoldPDS.GetQuotaForUser (pkg/hold/pds/layer.go) wraps QuotaForDID and returns a QuotaStats.
• HoldPDS.GetQuotaForUserWithTier (pkg/hold/pds/layer.go) layers the tier limit on top: captain (owner) is always unlimited; otherwise it looks up the crew member's tier and asks quota.Manager for the limit.

// pkg/hold/pds/layer.go

// QuotaStats represents storage quota information for a user.
type QuotaStats struct {
    UserDID     string `json:"userDid"`
    UniqueBlobs int    `json:"uniqueBlobs"`
    TotalSize   int64  `json:"totalSize"`
    Limit       *int64 `json:"limit,omitempty"` // nil = unlimited
    Tier        string `json:"tier,omitempty"`  // e.g. "deckhand", "bosun"
}

// GetQuotaForUser deduplicates layer records via SQL aggregation.
func (p *HoldPDS) GetQuotaForUser(ctx context.Context, userDID string) (*QuotaStats, error)

// GetQuotaForUserWithTier adds the tier-resolved Limit (captain = unlimited).
func (p *HoldPDS) GetQuotaForUserWithTier(ctx context.Context, userDID string, quotaMgr *quota.Manager) (*QuotaStats, error)

QuotaStats is the JSON shape returned by the public io.atcr.hold.getQuota endpoint, which the appview calls during the push gate (see Push Flow).

Push Flow

Where Quota Is Enforced

Quota is enforced in the appview's push authorizer at /auth/token time, before the push begins — not inside the manifest store. When a Docker client requests a push token, the authorizer (pkg/appview/authgate/push_authorizer.go, checkQuota) calls atproto.FetchQuotaStats against the hold's public io.atcr.hold.getQuota endpoint and rejects the token request when the user's current usage already meets or exceeds their limit.

┌──────────┐          ┌──────────┐          ┌──────────┐
│  Client  │          │ AppView  │          │   Hold   │
│ (Docker) │          │ /auth/   │          │ getQuota │
│          │          │  token   │          │          │
└──────────┘          └──────────┘          └──────────┘
     │                      │                      │
     │ 1. Request push token│                      │
     ├─────────────────────>│                      │
     │                      │ 2. checkQuota:       │
     │                      │    FetchQuotaStats   │
     │                      ├─────────────────────>│
     │                      │ 3. {totalSize,limit} │
     │                      │<─────────────────────┤
     │                      │                      │
     │                      │ 4. if limit != nil   │
     │                      │    && totalSize >=   │
     │                      │    limit → reject     │
     │                      │    (denied: quota)    │
     │                      │                      │
     │ 5. 200 token  /  401 │                      │
     │<─────────────────────┤                      │
     │                      │                      │
     │ 6. Push blobs + manifest (if token granted) │
     ├─────────────────────>│ ...                  │

Behavior

• Check is current-usage vs. limit, evaluated before the push. The condition is stats.TotalSize >= *stats.Limit. It does not pre-add the incoming layer sizes; a push is allowed to start whenever existing usage is strictly below the limit, even if that push will push the user over. Overage is caught on the next push attempt.
• Unlimited tiers skip the check. stats.Limit is nil for captains and for holds with no quota tiers configured, in which case no limit is enforced.
• Fails open on errors. If the hold's getQuota endpoint is unreachable or returns an error, checkQuota logs a warning and allows the push. This is deliberate: the endpoint is public/unauthenticated, and a hold outage should not produce spurious denied: quota errors unrelated to the user's actual usage.

Implementation

// pkg/appview/authgate/push_authorizer.go

func (a *Authorizer) checkQuota(ctx context.Context, userDID, holdDID string) error {
    stats, err := atproto.FetchQuotaStats(ctx, a.httpClient, holdDID, userDID)
    if err != nil {
        // Fail open: a hold outage must not block unrelated pushes.
        slog.Warn("push gate: quota call failed; allowing push", "did", userDID, "error", err)
        return nil
    }

    if stats.Limit != nil && stats.TotalSize >= *stats.Limit {
        return fmt.Errorf("quota exceeded: %s / %s used by %s. Delete images to free space",
            formatGB(stats.TotalSize), formatGB(*stats.Limit), a.identityLabel(ctx, userDID))
    }
    return nil
}

Layer records that back the usage figure are created hold-side as part of the blob upload / push pipeline, not by the appview's manifest store.

Delete Flow

Manifest Deletion

When a user deletes a manifest:

┌──────────┐          ┌──────────┐          ┌──────────┐          ┌──────────┐
│   User   │          │ AppView  │          │   Hold   │          │ User PDS │
│    UI    │          │          │          │ Service  │          │          │
└──────────┘          └──────────┘          └──────────┘          └──────────┘
     │                      │                      │                      │
     │ DELETE manifest      │                      │                      │
     ├─────────────────────>│                      │                      │
     │                      │                      │                      │
     │                      │ 1. Delete manifest   │                      │
     │                      │    from user's PDS   │                      │
     │                      ├──────────────────────┼─────────────────────>│
     │                      │                      │                      │
     │                      │ 2. Delete layer      │                      │
     │                      │    records for this  │                      │
     │                      │    manifest          │                      │
     │                      ├─────────────────────>│                      │
     │                      │                      │ 3. Remove records    │
     │                      │                      │    where manifest    │
     │                      │                      │    == deleted URI    │
     │                      │                      │                      │
     │ 4. 204 No Content    │                      │                      │
     │<─────────────────────┤                      │                      │

Hold Service: Delete Layer Record

The deletion primitive is hold-side: HoldPDS.DeleteLayerRecord in pkg/hold/pds/layer.go. It removes a single layer record by its rkey from both the repo (the MST/CAR store) and the records index (the index delete is best-effort — failures are logged, since a backfill resync will reconcile it).

// pkg/hold/pds/layer.go

// DeleteLayerRecord deletes a layer record by rkey from the repo (MST)
// and the records index.
func (p *HoldPDS) DeleteLayerRecord(ctx context.Context, rkey string) error

To remove all layer records for a deleted manifest, callers identify the rkeys whose records reference that manifest and call DeleteLayerRecord for each. Orphan cleanup also happens automatically during garbage collection (see below), so a missed deletion is eventually reconciled rather than leaking quota permanently.

Quota After Deletion

After deleting a manifest:

• Layer records for that manifest are removed
• Quota recalculated with SELECT DISTINCT query
• If layer was only in deleted manifest → quota decreases
• If layer exists in other manifests → quota unchanged (still deduplicated)

Garbage Collection

Orphaned Blobs

Orphaned blobs accumulate when:

1. Manifest push fails after blobs uploaded
2. Quota exceeded - manifest rejected
3. User deletes manifest - blobs may no longer be referenced

GC Process

GC is implemented in pkg/hold/gc/gc.go. The public entrypoint GarbageCollector.Run takes a single-run lock and delegates to doRun, which works in multiple phases (see (*GarbageCollector).doRun):

1. Analyze records (analyzeRecords) — build the set of referenced layer digests, identify orphaned layer records (rkeys), and find manifest layers missing a record.
2. Reconcile missing records — create layer records for referenced layers that lost their record, so usage accounting stays correct.
3. Delete orphaned layer records (deleteOrphanedRecords) — remove layer records no longer referenced by any manifest.
4. Delete orphaned blobs (deleteOrphanedBlobs) — walk S3 and delete blobs not in the referenced set.

A read-only Preview / doPreview path runs phases 1 and the blob scan without deleting anything, for the admin panel. A grace period (gcGracePeriod, 7 days) protects recently created records from collection.

GC Schedule

GC is toggled by a single config field; the interval is not configurable — it is the hardcoded gcInterval constant (24h) in pkg/hold/gc/config.go.

# config-hold.yaml
gc:
  enabled: true   # env: GC_ENABLED

There is no GC_INTERVAL setting.

Configuration

Enabling Quotas

Quotas are enabled by the presence of quota tiers in the hold config — there is no QUOTA_ENABLED flag and no global QUOTA_DEFAULT_LIMIT. quota.Manager.IsEnabled() returns true whenever a config with at least one tier was loaded; with no tiers, every user is unlimited.

# config-hold.yaml

quota:
  tiers:
    - name: deckhand
      quota: 5GB
    - name: bosun
      quota: 50GB
      scan_on_push: true
    - name: quartermaster
      quota: 100GB
      scan_on_push: true
  defaults:
    new_crew_tier: deckhand

gc:
  enabled: true   # env: GC_ENABLED (interval is a hardcoded 24h, not configurable)

The only quota/GC environment variable is GC_ENABLED (bound to gc.enabled). Tier limits are human-readable sizes (5GB, 50GB, 1TB) parsed by quota.ParseHumanBytes.

Quota Limits by Bytes

Size	Bytes
1 GB	1073741824
5 GB	5368709120
10 GB	10737418240
50 GB	53687091200
100 GB	107374182400

Quota API Endpoints

The per-user quota endpoint is implemented. It is public (no auth) and is what the appview's push gate calls (see Push Flow).

GET /xrpc/io.atcr.hold.getQuota?userDid={did}
    → {"userDid": "...", "uniqueBlobs": 10, "totalSize": 1073741824, "limit": ..., "tier": "..."}

• Endpoint constant: HoldGetQuota in pkg/atproto/endpoints.go
• Handler: HandleGetQuota, registered in pkg/hold/pds/xrpc.go
• Client helper: atproto.FetchQuotaStats in pkg/atproto/quota.go

Future Enhancements

1. Quota Breakdown Endpoint

GET /xrpc/io.atcr.hold.getQuotaBreakdown  - Storage by repository (not yet implemented)

2. Quota Alerts

• Warning thresholds at 80%, 90%, 95%
• Email/webhook notifications
• Grace period before hard enforcement

3. Tier-Based Quotas (Implemented)

ATCR uses quota tiers to limit storage per crew member, configured via quotas.yaml:

# quotas.yaml
tiers:
  deckhand:        # Entry-level crew
    quota: 5GB
  bosun:           # Mid-level crew
    quota: 50GB
  quartermaster:   # High-level crew
    quota: 100GB

defaults:
  new_crew_tier: deckhand  # Default tier for new crew members

Tier	Limit	Description
deckhand	5 GB	Entry-level crew member
bosun	50 GB	Mid-level crew member
quartermaster	100 GB	Senior crew member
owner (captain)	Unlimited	Hold owner always has unlimited

Tier Resolution:

1. If user is captain (owner) → unlimited
2. If crew member has explicit tier → use that tier's limit
3. If crew member has no tier → use defaults.new_crew_tier
4. If default tier not found → unlimited

Crew Record Example:

{
  "$type": "io.atcr.hold.crew",
  "member": "did:plc:alice123",
  "role": "writer",
  "permissions": ["blob:write"],
  "tier": "bosun",
  "addedAt": "2026-01-04T12:00:00Z"
}

4. Rate Limiting

Pull rate limits (Docker Hub style):

• Anonymous: 100 pulls per 6 hours per IP
• Authenticated: 200 pulls per 6 hours
• Paid: Unlimited

5. Quota Purchasing

• Stripe integration for additional storage
• $0.10/GB/month pricing (industry standard)

References

• Harbor Quotas: https://goharbor.io/docs/1.10/administration/configure-project-quotas/
• ATProto Spec: https://atproto.com/specs/record
• OCI Distribution Spec: https://github.com/opencontainers/distribution-spec

---

Document Version: 2.1 Last Updated: 2026-06-11 Model: Per-user layer tracking with ATProto records