at-container-registry/docs/QUOTAS.md

# ATCR Quota System

This document describes ATCR's storage quota implementation using ATProto records for per-user layer tracking.

## Table of Contents

- [Overview](#overview)
- [Quota Model](#quota-model)
- [Layer Record Schema](#layer-record-schema)
- [Quota Calculation](#quota-calculation)
- [Push Flow](#push-flow)
- [Delete Flow](#delete-flow)
- [Garbage Collection](#garbage-collection)
- [Configuration](#configuration)
- [Quota API Endpoints](#quota-api-endpoints)
- [Future Enhancements](#future-enhancements)

## Overview

ATCR implements per-user storage quotas to:
1. **Limit storage consumption** on shared hold services
2. **Provide transparency** (show users their storage usage)
3. **Enable fair billing** (users pay for what they use)

**Key principle:** Users pay for layers they reference, deduplicated per-user. If you push the same layer in multiple images, you only pay once.

### Example Scenario

```
Alice pushes myapp:v1 (layers A, B, C - each 100MB)
→ Creates 3 layer records in hold's PDS
→ Alice's quota: 300MB (3 unique layers)

Alice pushes myapp:v2 (layers A, B, D)
→ Creates 3 more layer records (A, B again, plus D)
→ Alice's quota: 400MB (4 unique layers: A, B, C, D)
→ Layers A, B appear twice in records but deduplicated in quota calc

Bob pushes his-app:latest (layers A, E)
→ Creates 2 layer records for Bob
→ Bob's quota: 200MB (2 unique layers: A, E)
→ Layer A shared with Alice in S3, but Bob pays for his own usage

Physical S3 storage: 500MB (A, B, C, D, E - deduplicated globally)
Alice's quota: 400MB
Bob's quota: 200MB
```

## Quota Model

### Everyone Pays for What They Upload

Each user is charged for all unique layers they reference, regardless of whether those layers exist in S3 from other users' uploads.

**Why this model?**
- **Simple mental model**: "I pushed 500MB of layers, I use 500MB of quota"
- **Predictable**: Your quota doesn't change based on others' actions
- **Clean deletion**: Delete manifest → layer records removed → quota freed
- **No cross-user dependencies**: Users are isolated

**Trade-off:**
- Total claimed storage can exceed physical S3 storage
- This is acceptable - deduplication is an operational benefit for ATCR, not a billing feature

### ATProto-Native Storage

Layer tracking uses ATProto records stored in the hold's embedded PDS:
- **Collection**: `io.atcr.hold.layer`
- **Repository**: Hold's DID (e.g., `did:web:hold01.atcr.io`)
- **Records**: One per manifest-layer relationship (TID-based keys)

This approach:
- Keeps quota data in ATProto (no separate database)
- Enables standard ATProto sync/query mechanisms
- Provides full audit trail of layer usage

## Layer Record Schema

### LayerRecord

```go
// pkg/atproto/lexicon.go

type LayerRecord struct {
    Type      string `json:"$type"`     // "io.atcr.hold.layer"
    Digest    string `json:"digest"`    // Layer digest (sha256:abc123...)
    Size      int64  `json:"size"`      // Size in bytes
    MediaType string `json:"mediaType"` // e.g., "application/vnd.oci.image.layer.v1.tar+gzip"
    Manifest  string `json:"manifest"`  // at://did:plc:alice/io.atcr.manifest/abc123
    UserDID   string `json:"userDid"`   // User's DID for quota grouping
    CreatedAt string `json:"createdAt"` // ISO 8601 timestamp
}
```

### Record Key

Records use TID (timestamp-based ID) as the rkey. This means:
- Multiple records can exist for the same layer (from different manifests)
- Deduplication happens at query time, not storage time
- Simple append-only writes on manifest push

### Example Records

```
Manifest A (layers X, Y, Z) → creates 3 records
Manifest B (layers X, W)    → creates 2 records

io.atcr.hold.layer collection:
┌──────────────┬────────┬──────┬───────────────────────────────────┬─────────────────┐
│ rkey (TID)   │ digest │ size │ manifest                          │ userDid         │
├──────────────┼────────┼──────┼───────────────────────────────────┼─────────────────┤
│ 3jui7...001  │ X      │ 100  │ at://did:plc:alice/.../manifestA  │ did:plc:alice   │
│ 3jui7...002  │ Y      │ 200  │ at://did:plc:alice/.../manifestA  │ did:plc:alice   │
│ 3jui7...003  │ Z      │ 150  │ at://did:plc:alice/.../manifestA  │ did:plc:alice   │
│ 3jui7...004  │ X      │ 100  │ at://did:plc:alice/.../manifestB  │ did:plc:alice   │  ← duplicate digest
│ 3jui7...005  │ W      │ 300  │ at://did:plc:alice/.../manifestB  │ did:plc:alice   │
└──────────────┴────────┴──────┴───────────────────────────────────┴─────────────────┘
```

## Quota Calculation

### Query: User's Unique Storage

```sql
-- Calculate quota by deduplicating layers
SELECT SUM(size) FROM (
    SELECT DISTINCT digest, size
    FROM io.atcr.hold.layer
    WHERE userDid = ?
)
```

Using the example above:
- Layer X appears twice but counted once: 100
- Layers Y, Z, W counted once each: 200 + 150 + 300
- **Total: 750 bytes**

### Implementation

Usage is computed **hold-side**, where the layer records actually live. There is no
`QuotaManager` type — the pieces are:

- **`quota.Manager`** (`pkg/hold/quota/config.go`) resolves a crew tier name to a
  byte limit (tier → limit only). It does not compute usage. `Manager.IsEnabled()`
  reports whether any tiers are configured.
- **`recordsIndex.QuotaForDID(collection, userDID)`** runs the deduplicating SQL
  aggregation over the denormalized `digest`/`size` columns in the records index,
  returning `(uniqueBlobs, totalSize)`.
- **`HoldPDS.GetQuotaForUser`** (`pkg/hold/pds/layer.go`) wraps `QuotaForDID` and
  returns a `QuotaStats`.
- **`HoldPDS.GetQuotaForUserWithTier`** (`pkg/hold/pds/layer.go`) layers the tier
  limit on top: captain (owner) is always unlimited; otherwise it looks up the
  crew member's tier and asks `quota.Manager` for the limit.

```go
// pkg/hold/pds/layer.go

// QuotaStats represents storage quota information for a user.
type QuotaStats struct {
    UserDID     string `json:"userDid"`
    UniqueBlobs int    `json:"uniqueBlobs"`
    TotalSize   int64  `json:"totalSize"`
    Limit       *int64 `json:"limit,omitempty"` // nil = unlimited
    Tier        string `json:"tier,omitempty"`  // e.g. "deckhand", "bosun"
}

// GetQuotaForUser deduplicates layer records via SQL aggregation.
func (p *HoldPDS) GetQuotaForUser(ctx context.Context, userDID string) (*QuotaStats, error)

// GetQuotaForUserWithTier adds the tier-resolved Limit (captain = unlimited).
func (p *HoldPDS) GetQuotaForUserWithTier(ctx context.Context, userDID string, quotaMgr *quota.Manager) (*QuotaStats, error)
```

`QuotaStats` is the JSON shape returned by the public `io.atcr.hold.getQuota`
endpoint, which the appview calls during the push gate (see [Push Flow](#push-flow)).

## Push Flow

### Where Quota Is Enforced

Quota is enforced in the appview's **push authorizer at `/auth/token` time**, before
the push begins — not inside the manifest store. When a Docker client requests a push
token, the authorizer (`pkg/appview/authgate/push_authorizer.go`, `checkQuota`) calls
`atproto.FetchQuotaStats` against the hold's public `io.atcr.hold.getQuota` endpoint
and rejects the token request when the user's current usage already meets or exceeds
their limit.

```
┌──────────┐          ┌──────────┐          ┌──────────┐
│  Client  │          │ AppView  │          │   Hold   │
│ (Docker) │          │ /auth/   │          │ getQuota │
│          │          │  token   │          │          │
└──────────┘          └──────────┘          └──────────┘
     │                      │                      │
     │ 1. Request push token│                      │
     ├─────────────────────>│                      │
     │                      │ 2. checkQuota:       │
     │                      │    FetchQuotaStats   │
     │                      ├─────────────────────>│
     │                      │ 3. {totalSize,limit} │
     │                      │<─────────────────────┤
     │                      │                      │
     │                      │ 4. if limit != nil   │
     │                      │    && totalSize >=   │
     │                      │    limit → reject     │
     │                      │    (denied: quota)    │
     │                      │                      │
     │ 5. 200 token  /  401 │                      │
     │<─────────────────────┤                      │
     │                      │                      │
     │ 6. Push blobs + manifest (if token granted) │
     ├─────────────────────>│ ...                  │
```

### Behavior

- **Check is current-usage vs. limit, evaluated before the push.** The condition is
  `stats.TotalSize >= *stats.Limit`. It does **not** pre-add the incoming layer sizes;
  a push is allowed to start whenever existing usage is strictly below the limit, even
  if that push will push the user over. Overage is caught on the *next* push attempt.
- **Unlimited tiers skip the check.** `stats.Limit` is `nil` for captains and for holds
  with no quota tiers configured, in which case no limit is enforced.
- **Fails open on errors.** If the hold's `getQuota` endpoint is unreachable or returns
  an error, `checkQuota` logs a warning and **allows the push**. This is deliberate: the
  endpoint is public/unauthenticated, and a hold outage should not produce spurious
  `denied: quota` errors unrelated to the user's actual usage.

### Implementation

```go
// pkg/appview/authgate/push_authorizer.go

func (a *Authorizer) checkQuota(ctx context.Context, userDID, holdDID string) error {
    stats, err := atproto.FetchQuotaStats(ctx, a.httpClient, holdDID, userDID)
    if err != nil {
        // Fail open: a hold outage must not block unrelated pushes.
        slog.Warn("push gate: quota call failed; allowing push", "did", userDID, "error", err)
        return nil
    }

    if stats.Limit != nil && stats.TotalSize >= *stats.Limit {
        return fmt.Errorf("quota exceeded: %s / %s used by %s. Delete images to free space",
            formatGB(stats.TotalSize), formatGB(*stats.Limit), a.identityLabel(ctx, userDID))
    }
    return nil
}
```

Layer records that back the usage figure are created hold-side as part of the blob
upload / push pipeline, not by the appview's manifest store.

## Delete Flow

### Manifest Deletion

When a user deletes a manifest:

```
┌──────────┐          ┌──────────┐          ┌──────────┐          ┌──────────┐
│   User   │          │ AppView  │          │   Hold   │          │ User PDS │
│    UI    │          │          │          │ Service  │          │          │
└──────────┘          └──────────┘          └──────────┘          └──────────┘
     │                      │                      │                      │
     │ DELETE manifest      │                      │                      │
     ├─────────────────────>│                      │                      │
     │                      │                      │                      │
     │                      │ 1. Delete manifest   │                      │
     │                      │    from user's PDS   │                      │
     │                      ├──────────────────────┼─────────────────────>│
     │                      │                      │                      │
     │                      │ 2. Delete layer      │                      │
     │                      │    records for this  │                      │
     │                      │    manifest          │                      │
     │                      ├─────────────────────>│                      │
     │                      │                      │ 3. Remove records    │
     │                      │                      │    where manifest    │
     │                      │                      │    == deleted URI    │
     │                      │                      │                      │
     │ 4. 204 No Content    │                      │                      │
     │<─────────────────────┤                      │                      │
```

### Hold Service: Delete Layer Record

The deletion primitive is hold-side: `HoldPDS.DeleteLayerRecord` in
`pkg/hold/pds/layer.go`. It removes a single layer record by its rkey from both the
repo (the MST/CAR store) and the records index (the index delete is best-effort —
failures are logged, since a backfill resync will reconcile it).

```go
// pkg/hold/pds/layer.go

// DeleteLayerRecord deletes a layer record by rkey from the repo (MST)
// and the records index.
func (p *HoldPDS) DeleteLayerRecord(ctx context.Context, rkey string) error
```

To remove all layer records for a deleted manifest, callers identify the rkeys whose
records reference that manifest and call `DeleteLayerRecord` for each. Orphan cleanup
also happens automatically during garbage collection (see below), so a missed deletion
is eventually reconciled rather than leaking quota permanently.

### Quota After Deletion

After deleting a manifest:
- Layer records for that manifest are removed
- Quota recalculated with `SELECT DISTINCT` query
- If layer was only in deleted manifest → quota decreases
- If layer exists in other manifests → quota unchanged (still deduplicated)

## Garbage Collection

### Orphaned Blobs

Orphaned blobs accumulate when:
1. Manifest push fails after blobs uploaded
2. Quota exceeded - manifest rejected
3. User deletes manifest - blobs may no longer be referenced

### GC Process

GC is implemented in `pkg/hold/gc/gc.go`. The public entrypoint
`GarbageCollector.Run` takes a single-run lock and delegates to `doRun`, which works
in multiple phases (see `(*GarbageCollector).doRun`):

1. **Analyze records** (`analyzeRecords`) — build the set of referenced layer digests,
   identify orphaned layer records (rkeys), and find manifest layers missing a record.
2. **Reconcile missing records** — create layer records for referenced layers that lost
   their record, so usage accounting stays correct.
3. **Delete orphaned layer records** (`deleteOrphanedRecords`) — remove layer records no
   longer referenced by any manifest.
4. **Delete orphaned blobs** (`deleteOrphanedBlobs`) — walk S3 and delete blobs not in the
   referenced set.

A read-only `Preview` / `doPreview` path runs phases 1 and the blob scan without deleting
anything, for the admin panel. A grace period (`gcGracePeriod`, 7 days) protects recently
created records from collection.

### GC Schedule

GC is toggled by a single config field; the interval is **not** configurable — it is the
hardcoded `gcInterval` constant (24h) in `pkg/hold/gc/config.go`.

```yaml
# config-hold.yaml
gc:
  enabled: true   # env: GC_ENABLED
```

There is no `GC_INTERVAL` setting.

## Configuration

### Enabling Quotas

Quotas are **enabled by the presence of quota tiers** in the hold config — there is no
`QUOTA_ENABLED` flag and no global `QUOTA_DEFAULT_LIMIT`. `quota.Manager.IsEnabled()`
returns true whenever a config with at least one tier was loaded; with no tiers, every
user is unlimited.

```yaml
# config-hold.yaml

quota:
  tiers:
    - name: deckhand
      quota: 5GB
    - name: bosun
      quota: 50GB
      scan_on_push: true
    - name: quartermaster
      quota: 100GB
      scan_on_push: true
  defaults:
    new_crew_tier: deckhand

gc:
  enabled: true   # env: GC_ENABLED (interval is a hardcoded 24h, not configurable)
```

The only quota/GC environment variable is `GC_ENABLED` (bound to `gc.enabled`). Tier
limits are human-readable sizes (`5GB`, `50GB`, `1TB`) parsed by `quota.ParseHumanBytes`.

### Quota Limits by Bytes

| Size | Bytes |
|------|-------|
| 1 GB | 1073741824 |
| 5 GB | 5368709120 |
| 10 GB | 10737418240 |
| 50 GB | 53687091200 |
| 100 GB | 107374182400 |

## Quota API Endpoints

The per-user quota endpoint is **implemented**. It is public (no auth) and is what the
appview's push gate calls (see [Push Flow](#push-flow)).

```
GET /xrpc/io.atcr.hold.getQuota?userDid={did}
    → {"userDid": "...", "uniqueBlobs": 10, "totalSize": 1073741824, "limit": ..., "tier": "..."}
```

- Endpoint constant: `HoldGetQuota` in `pkg/atproto/endpoints.go`
- Handler: `HandleGetQuota`, registered in `pkg/hold/pds/xrpc.go`
- Client helper: `atproto.FetchQuotaStats` in `pkg/atproto/quota.go`

## Future Enhancements

### 1. Quota Breakdown Endpoint

```
GET /xrpc/io.atcr.hold.getQuotaBreakdown  - Storage by repository (not yet implemented)
```

### 2. Quota Alerts

- Warning thresholds at 80%, 90%, 95%
- Email/webhook notifications
- Grace period before hard enforcement

### 3. Tier-Based Quotas (Implemented)

ATCR uses quota tiers to limit storage per crew member, configured via `quotas.yaml`:

```yaml
# quotas.yaml
tiers:
  deckhand:        # Entry-level crew
    quota: 5GB
  bosun:           # Mid-level crew
    quota: 50GB
  quartermaster:   # High-level crew
    quota: 100GB

defaults:
  new_crew_tier: deckhand  # Default tier for new crew members
```

| Tier | Limit | Description |
|------|-------|-------------|
| deckhand | 5 GB | Entry-level crew member |
| bosun | 50 GB | Mid-level crew member |
| quartermaster | 100 GB | Senior crew member |
| owner (captain) | Unlimited | Hold owner always has unlimited |

**Tier Resolution:**
1. If user is captain (owner) → unlimited
2. If crew member has explicit tier → use that tier's limit
3. If crew member has no tier → use `defaults.new_crew_tier`
4. If default tier not found → unlimited

**Crew Record Example:**
```json
{
  "$type": "io.atcr.hold.crew",
  "member": "did:plc:alice123",
  "role": "writer",
  "permissions": ["blob:write"],
  "tier": "bosun",
  "addedAt": "2026-01-04T12:00:00Z"
}
```

### 4. Rate Limiting

Pull rate limits (Docker Hub style):
- Anonymous: 100 pulls per 6 hours per IP
- Authenticated: 200 pulls per 6 hours
- Paid: Unlimited

### 5. Quota Purchasing

- Stripe integration for additional storage
- $0.10/GB/month pricing (industry standard)

## References

- **Harbor Quotas:** https://goharbor.io/docs/1.10/administration/configure-project-quotas/
- **ATProto Spec:** https://atproto.com/specs/record
- **OCI Distribution Spec:** https://github.com/opencontainers/distribution-spec

---

**Document Version:** 2.1
**Last Updated:** 2026-06-11
**Model:** Per-user layer tracking with ATProto records