Files
at-container-registry/docs/HOLD_DISCOVERY.md

1722 lines
53 KiB
Markdown

# Hold Discovery
This document describes how AppView discovers available holds and presents them to users for selection.
## TL;DR
**Problem:** Users currently enter hold URLs manually in a text field. They don't know what holds exist or which ones they can access.
**Solution:**
1. Subscribe to Jetstream for `io.atcr.hold.captain` and `io.atcr.hold.crew` collections
2. Cache discovered holds and crew memberships in SQLite
3. Replace the text input with a dropdown showing available holds grouped by access level
**Key Changes:**
- New table: `hold_crew_members` (hold_did, member_did, rkey, permissions, ...)
- Jetstream collections: `io.atcr.hold.captain`, `io.atcr.hold.crew`
- Settings UI: Text input → `<select>` dropdown with optgroups
- Form field: `hold_endpoint` (URL) → `hold_did` (DID)
**Hold Categories in Dropdown:**
| Group | Who Can Use |
|-------|-------------|
| Your Holds | User is captain (owner) |
| Crew Member | User has explicit crew record |
| Open Registration | `allowAllCrew=true` |
| Public Holds | `public=true` |
## Overview
Users need to select a "default hold" for blob storage. The AppView must discover available holds and determine which ones each user can access. This enables a dropdown in user settings showing:
- Holds the user owns (captain)
- Holds where the user is a crew member
- Holds that allow all crew members (open registration)
- Public holds (anyone can read/write)
## Architecture
### Discovery Sources
Hold discovery leverages the ATProto network infrastructure:
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Hold Service │────▶│ Relay │────▶│ Jetstream │
│ (embedded PDS) │ │ (BGS/bigsky) │ │ │
└─────────────────┘ └─────────────────┘ └────────┬────────┘
┌─────────────────┐
│ AppView │
│ (subscriber) │
└────────┬────────┘
┌─────────────────┐
│ SQLite │
│ (cache) │
└─────────────────┘
```
1. **Hold services** run embedded PDSes that store captain and crew records
2. **Relays** crawl hold PDSes after `request-crawl.sh` is run
3. **Jetstream** streams record events filtered by collection
4. **AppView** subscribes to Jetstream and caches records in SQLite
### Record Types
Two ATProto record collections are relevant for discovery:
#### `io.atcr.hold.captain`
Singleton record (rkey: `self`) in each hold's embedded PDS describing the hold:
```json
{
"$type": "io.atcr.hold.captain",
"ownerDid": "did:plc:abc123",
"public": false,
"allowAllCrew": true,
"deployedAt": "2025-01-07T12:00:00Z",
"region": "us-east-1",
"provider": "fly.io"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `ownerDid` | string | DID of the hold owner (captain) |
| `public` | boolean | If true, anyone can read and write blobs |
| `allowAllCrew` | boolean | If true, any authenticated user can self-register as crew |
| `deployedAt` | string | ISO 8601 timestamp of deployment |
| `region` | string | Optional geographic region identifier |
| `provider` | string | Optional hosting provider name |
#### `io.atcr.hold.crew`
One record per crew member in the hold's embedded PDS:
```json
{
"$type": "io.atcr.hold.crew",
"memberDid": "did:plc:xyz789",
"role": "contributor",
"permissions": ["blob:read", "blob:write"],
"tier": "standard",
"addedAt": "2025-01-07T12:00:00Z"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `memberDid` | string | DID of the crew member |
| `role` | string | Human-readable role name |
| `permissions` | string[] | Permission grants: `blob:read`, `blob:write`, `crew:admin` |
| `tier` | string | Optional tier for quota management |
| `addedAt` | string | ISO 8601 timestamp when added |
**Record key derivation:** Crew records use a deterministic rkey based on the member's DID:
```go
func CrewRecordKey(memberDID string) string {
hash := sha256.Sum256([]byte(memberDID))
return base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(hash[:16])
}
```
This enables O(1) lookup of a specific member's crew record.
## Data Model
### Database Schema
Add to `pkg/appview/db/schema.sql`:
```sql
-- Cached hold captain records from Jetstream
-- Primary discovery source for available holds
CREATE TABLE IF NOT EXISTS hold_captain_records (
did TEXT PRIMARY KEY, -- Hold's DID (did:web:hold01.atcr.io)
owner_did TEXT NOT NULL, -- Captain's DID
public INTEGER NOT NULL DEFAULT 0, -- 1 if public hold
allow_all_crew INTEGER NOT NULL DEFAULT 0, -- 1 if open registration
deployed_at TEXT, -- ISO 8601 deployment timestamp
region TEXT, -- Geographic region
provider TEXT, -- Hosting provider
endpoint TEXT, -- Resolved HTTP endpoint (cached)
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_hold_captain_owner ON hold_captain_records(owner_did);
CREATE INDEX IF NOT EXISTS idx_hold_captain_public ON hold_captain_records(public);
CREATE INDEX IF NOT EXISTS idx_hold_captain_allow_all ON hold_captain_records(allow_all_crew);
-- Cached hold crew memberships from Jetstream
-- Enables reverse lookup: "which holds is user X a member of?"
CREATE TABLE IF NOT EXISTS hold_crew_members (
hold_did TEXT NOT NULL, -- Hold's DID
member_did TEXT NOT NULL, -- Crew member's DID
rkey TEXT NOT NULL, -- ATProto record key (for delete handling)
role TEXT, -- Human-readable role
permissions TEXT, -- JSON array of permissions
tier TEXT, -- Optional quota tier
added_at TEXT, -- ISO 8601 timestamp
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
PRIMARY KEY (hold_did, member_did),
FOREIGN KEY (hold_did) REFERENCES hold_captain_records(did) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_hold_crew_member ON hold_crew_members(member_did);
CREATE INDEX IF NOT EXISTS idx_hold_crew_hold ON hold_crew_members(hold_did);
CREATE INDEX IF NOT EXISTS idx_hold_crew_rkey ON hold_crew_members(hold_did, rkey);
```
### Migration
Add to `pkg/appview/db/migrations/`:
```yaml
# 006_hold_discovery.yaml
id: 006_hold_discovery
description: Add hold crew members table for discovery
up: |
CREATE TABLE IF NOT EXISTS hold_crew_members (
hold_did TEXT NOT NULL,
member_did TEXT NOT NULL,
rkey TEXT NOT NULL,
role TEXT,
permissions TEXT,
tier TEXT,
added_at TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
PRIMARY KEY (hold_did, member_did),
FOREIGN KEY (hold_did) REFERENCES hold_captain_records(did) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_hold_crew_member ON hold_crew_members(member_did);
CREATE INDEX IF NOT EXISTS idx_hold_crew_hold ON hold_crew_members(hold_did);
CREATE INDEX IF NOT EXISTS idx_hold_crew_rkey ON hold_crew_members(hold_did, rkey);
down: |
DROP INDEX IF EXISTS idx_hold_crew_rkey;
DROP INDEX IF EXISTS idx_hold_crew_hold;
DROP INDEX IF EXISTS idx_hold_crew_member;
DROP TABLE IF EXISTS hold_crew_members;
```
## Jetstream Integration
### Subscription Configuration
Update the Jetstream worker to subscribe to hold collections:
```go
// pkg/appview/jetstream/worker.go
var wantedCollections = []string{
"io.atcr.manifest",
"io.atcr.tag",
"io.atcr.hold.stats",
"io.atcr.hold.captain", // NEW: Hold discovery
"io.atcr.hold.crew", // NEW: Crew membership discovery
}
```
### Event Processing
Add processors for captain and crew records:
```go
// pkg/appview/jetstream/processor.go
func (p *Processor) ProcessEvent(evt *Event) error {
switch evt.Collection {
case "io.atcr.manifest":
return p.ProcessManifest(evt)
case "io.atcr.tag":
return p.ProcessTag(evt)
case "io.atcr.hold.stats":
return p.ProcessStats(evt)
case "io.atcr.hold.captain":
return p.ProcessCaptain(evt)
case "io.atcr.hold.crew":
return p.ProcessCrew(evt)
default:
return nil
}
}
func (p *Processor) ProcessCaptain(evt *Event) error {
// The repo DID IS the hold DID (hold's embedded PDS)
holdDID := evt.DID
if evt.Operation == "delete" {
return p.db.DeleteCaptainRecord(holdDID)
}
var record atproto.CaptainRecord
if err := json.Unmarshal(evt.Record, &record); err != nil {
return fmt.Errorf("unmarshal captain record: %w", err)
}
// Resolve hold DID to HTTP endpoint for caching
endpoint, err := p.resolver.ResolveHoldURL(holdDID)
if err != nil {
// Log but don't fail - endpoint can be resolved later
log.Warn().Err(err).Str("did", holdDID).Msg("failed to resolve hold endpoint")
}
// Verify this is actually a hold by checking /.well-known/did.json
// for #atcr_hold service type
if !p.verifyHoldService(holdDID, endpoint) {
log.Debug().Str("did", holdDID).Msg("skipping non-hold captain record")
return nil
}
return p.db.UpsertCaptainRecord(holdDID, &db.CaptainRecord{
DID: holdDID,
OwnerDID: record.OwnerDID,
Public: record.Public,
AllowAllCrew: record.AllowAllCrew,
DeployedAt: record.DeployedAt,
Region: record.Region,
Provider: record.Provider,
Endpoint: endpoint,
})
}
func (p *Processor) ProcessCrew(evt *Event) error {
// The repo DID IS the hold DID (hold's embedded PDS)
holdDID := evt.DID
if evt.Operation == "delete" {
// Need to determine member DID from rkey or record
// For delete events, we may not have the record body
return p.db.DeleteCrewMemberByRkey(holdDID, evt.Rkey)
}
var record atproto.CrewRecord
if err := json.Unmarshal(evt.Record, &record); err != nil {
return fmt.Errorf("unmarshal crew record: %w", err)
}
// Verify the hold exists in our captain records
// If not, this crew record is for an unknown hold - skip it
if _, err := p.db.GetCaptainRecord(holdDID); err != nil {
log.Debug().Str("hold", holdDID).Msg("skipping crew record for unknown hold")
return nil
}
permissionsJSON, _ := json.Marshal(record.Permissions)
return p.db.UpsertCrewMember(holdDID, &db.CrewMember{
HoldDID: holdDID,
MemberDID: record.MemberDID,
Role: record.Role,
Permissions: string(permissionsJSON),
Tier: record.Tier,
AddedAt: record.AddedAt,
})
}
func (p *Processor) verifyHoldService(did, endpoint string) bool {
// Fetch /.well-known/did.json and check for #atcr_hold service
didDoc, err := p.resolver.ResolveDIDDocument(did)
if err != nil {
return false
}
for _, svc := range didDoc.Service {
if svc.ID == did+"#atcr_hold" || svc.Type == "AtcrHold" {
return true
}
}
return false
}
```
### Hold Service Verification
Before caching a captain record, verify the DID document contains the `#atcr_hold` service:
```go
// pkg/atproto/resolver.go
type DIDDocument struct {
ID string `json:"id"`
Service []Service `json:"service"`
// ... other fields
}
type Service struct {
ID string `json:"id"`
Type string `json:"type"`
ServiceEndpoint string `json:"serviceEndpoint"`
}
func (r *Resolver) HasHoldService(did string) (bool, string, error) {
doc, err := r.ResolveDIDDocument(did)
if err != nil {
return false, "", err
}
for _, svc := range doc.Service {
// Check for #atcr_hold fragment or AtcrHold type
if strings.HasSuffix(svc.ID, "#atcr_hold") || svc.Type == "AtcrHold" {
return true, svc.ServiceEndpoint, nil
}
}
return false, "", nil
}
```
## Backfill Strategy
### Initial Backfill
For holds that existed before AppView started listening to Jetstream, use the existing backfill mechanism:
```go
// pkg/appview/jetstream/backfill.go
func (b *Backfiller) BackfillHolds(ctx context.Context) error {
// List all repos from relay that have io.atcr.hold.captain collection
repos, err := b.listReposWithCollection(ctx, "io.atcr.hold.captain")
if err != nil {
return err
}
for _, repo := range repos {
// Fetch captain record
captain, err := b.fetchRecord(ctx, repo.DID, "io.atcr.hold.captain", "self")
if err != nil {
log.Warn().Err(err).Str("did", repo.DID).Msg("failed to fetch captain record")
continue
}
// Verify it's a hold service
hasService, endpoint, _ := b.resolver.HasHoldService(repo.DID)
if !hasService {
continue
}
// Upsert captain record
if err := b.db.UpsertCaptainRecord(repo.DID, captain); err != nil {
log.Warn().Err(err).Str("did", repo.DID).Msg("failed to upsert captain record")
continue
}
// Fetch and upsert all crew records for this hold
if err := b.backfillCrewRecords(ctx, repo.DID); err != nil {
log.Warn().Err(err).Str("did", repo.DID).Msg("failed to backfill crew records")
}
}
return nil
}
func (b *Backfiller) backfillCrewRecords(ctx context.Context, holdDID string) error {
// List all records in io.atcr.hold.crew collection
records, err := b.listRecords(ctx, holdDID, "io.atcr.hold.crew")
if err != nil {
return err
}
for _, record := range records {
var crew atproto.CrewRecord
if err := json.Unmarshal(record.Value, &crew); err != nil {
continue
}
permissionsJSON, _ := json.Marshal(crew.Permissions)
if err := b.db.UpsertCrewMember(holdDID, &db.CrewMember{
HoldDID: holdDID,
MemberDID: crew.MemberDID,
Role: crew.Role,
Permissions: string(permissionsJSON),
Tier: crew.Tier,
AddedAt: crew.AddedAt,
}); err != nil {
log.Warn().Err(err).Msg("failed to upsert crew member")
}
}
return nil
}
```
### Listing Repos by Collection
Query the relay for repos that have a specific collection:
```go
func (b *Backfiller) listReposWithCollection(ctx context.Context, collection string) ([]Repo, error) {
// Use com.atproto.sync.listRepos to get all repos
// Then filter to those with the target collection
//
// Note: This is O(n) over all repos on the relay.
// For efficiency, could maintain a separate index or use
// Jetstream historical replay if available.
var repos []Repo
cursor := ""
for {
resp, err := b.client.SyncListRepos(ctx, cursor, 1000)
if err != nil {
return nil, err
}
for _, repo := range resp.Repos {
// Check if repo has the collection by attempting to list records
records, err := b.client.RepoListRecords(ctx, repo.DID, collection, "", 1)
if err == nil && len(records.Records) > 0 {
repos = append(repos, Repo{DID: repo.DID})
}
}
if resp.Cursor == nil || *resp.Cursor == "" {
break
}
cursor = *resp.Cursor
}
return repos, nil
}
```
### Bootstrap Configuration
For known holds that may not yet be on relays, support a bootstrap list in configuration:
```bash
# Environment variable
ATCR_BOOTSTRAP_HOLDS="did:web:hold01.atcr.io,did:web:hold02.atcr.io"
```
```go
func (b *Backfiller) BackfillBootstrapHolds(ctx context.Context, holdDIDs []string) error {
for _, did := range holdDIDs {
// Verify it's a hold
hasService, endpoint, err := b.resolver.HasHoldService(did)
if err != nil || !hasService {
log.Warn().Str("did", did).Msg("bootstrap hold is not a valid hold service")
continue
}
// Fetch captain record directly from hold's PDS
captain, err := b.fetchCaptainFromHold(ctx, did, endpoint)
if err != nil {
log.Warn().Err(err).Str("did", did).Msg("failed to fetch captain from hold")
continue
}
if err := b.db.UpsertCaptainRecord(did, captain); err != nil {
log.Warn().Err(err).Str("did", did).Msg("failed to upsert bootstrap captain")
continue
}
// Also backfill crew records
if err := b.backfillCrewFromHold(ctx, did, endpoint); err != nil {
log.Warn().Err(err).Str("did", did).Msg("failed to backfill bootstrap crew")
}
}
return nil
}
func (b *Backfiller) fetchCaptainFromHold(ctx context.Context, did, endpoint string) (*db.CaptainRecord, error) {
// GET {endpoint}/xrpc/com.atproto.repo.getRecord?repo={did}&collection=io.atcr.hold.captain&rkey=self
url := fmt.Sprintf("%s/xrpc/com.atproto.repo.getRecord?repo=%s&collection=io.atcr.hold.captain&rkey=self",
endpoint, did)
resp, err := http.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
var result struct {
Value atproto.CaptainRecord `json:"value"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return nil, err
}
return &db.CaptainRecord{
DID: did,
OwnerDID: result.Value.OwnerDID,
Public: result.Value.Public,
AllowAllCrew: result.Value.AllowAllCrew,
DeployedAt: result.Value.DeployedAt,
Region: result.Value.Region,
Provider: result.Value.Provider,
Endpoint: endpoint,
}, nil
}
```
## Database Queries
### Hold Store Functions
Add to `pkg/appview/db/hold_store.go`:
```go
// CrewMember represents a cached crew membership
type CrewMember struct {
HoldDID string
MemberDID string
Role string
Permissions string // JSON array
Tier string
AddedAt string
CreatedAt string
UpdatedAt string
}
// UpsertCrewMember inserts or updates a crew member record
func UpsertCrewMember(db *sql.DB, holdDID string, member *CrewMember) error {
_, err := db.Exec(`
INSERT INTO hold_crew_members (hold_did, member_did, role, permissions, tier, added_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, datetime('now'))
ON CONFLICT(hold_did, member_did) DO UPDATE SET
role = excluded.role,
permissions = excluded.permissions,
tier = excluded.tier,
added_at = excluded.added_at,
updated_at = datetime('now')
`, holdDID, member.MemberDID, member.Role, member.Permissions, member.Tier, member.AddedAt)
return err
}
// DeleteCrewMember removes a crew member record
func DeleteCrewMember(db *sql.DB, holdDID, memberDID string) error {
_, err := db.Exec(`
DELETE FROM hold_crew_members WHERE hold_did = ? AND member_did = ?
`, holdDID, memberDID)
return err
}
// DeleteCrewMemberByRkey removes a crew member by rkey (for delete events)
func DeleteCrewMemberByRkey(db *sql.DB, holdDID, rkey string) error {
// We need to find the member by rkey hash
// This is tricky because we store member_did, not rkey
// Option 1: Store rkey in the table
// Option 2: Iterate and check (slow)
// Option 3: Store both member_did and rkey
// For now, we'll need to add rkey to the schema
_, err := db.Exec(`
DELETE FROM hold_crew_members WHERE hold_did = ? AND rkey = ?
`, holdDID, rkey)
return err
}
// AvailableHold represents a hold available to a user
type AvailableHold struct {
DID string
OwnerDID string
Public bool
AllowAllCrew bool
Region string
Provider string
Endpoint string
Membership string // "owner", "crew", "eligible", "public"
Permissions []string // nil if not crew
}
// GetAvailableHolds returns all holds available to a user
func GetAvailableHolds(db *sql.DB, userDID string) ([]AvailableHold, error) {
rows, err := db.Query(`
SELECT
h.did,
h.owner_did,
h.public,
h.allow_all_crew,
h.region,
h.provider,
h.endpoint,
CASE
WHEN h.owner_did = ?1 THEN 'owner'
WHEN c.member_did IS NOT NULL THEN 'crew'
WHEN h.allow_all_crew = 1 THEN 'eligible'
WHEN h.public = 1 THEN 'public'
ELSE 'none'
END as membership,
c.permissions
FROM hold_captain_records h
LEFT JOIN hold_crew_members c
ON h.did = c.hold_did AND c.member_did = ?1
WHERE h.public = 1
OR h.allow_all_crew = 1
OR h.owner_did = ?1
OR c.member_did IS NOT NULL
ORDER BY
CASE
WHEN h.owner_did = ?1 THEN 0
WHEN c.member_did IS NOT NULL THEN 1
WHEN h.allow_all_crew = 1 THEN 2
ELSE 3
END,
h.did
`, userDID)
if err != nil {
return nil, err
}
defer rows.Close()
var holds []AvailableHold
for rows.Next() {
var h AvailableHold
var permissionsJSON sql.NullString
err := rows.Scan(
&h.DID,
&h.OwnerDID,
&h.Public,
&h.AllowAllCrew,
&h.Region,
&h.Provider,
&h.Endpoint,
&h.Membership,
&permissionsJSON,
)
if err != nil {
return nil, err
}
if permissionsJSON.Valid {
json.Unmarshal([]byte(permissionsJSON.String), &h.Permissions)
}
holds = append(holds, h)
}
return holds, rows.Err()
}
// GetHoldsOwnedBy returns holds owned by a specific DID
func GetHoldsOwnedBy(db *sql.DB, ownerDID string) ([]CaptainRecord, error) {
rows, err := db.Query(`
SELECT did, owner_did, public, allow_all_crew, deployed_at, region, provider, endpoint
FROM hold_captain_records
WHERE owner_did = ?
ORDER BY deployed_at DESC
`, ownerDID)
if err != nil {
return nil, err
}
defer rows.Close()
var holds []CaptainRecord
for rows.Next() {
var h CaptainRecord
err := rows.Scan(&h.DID, &h.OwnerDID, &h.Public, &h.AllowAllCrew,
&h.DeployedAt, &h.Region, &h.Provider, &h.Endpoint)
if err != nil {
return nil, err
}
holds = append(holds, h)
}
return holds, rows.Err()
}
// GetCrewMemberships returns all holds where a user is a crew member
func GetCrewMemberships(db *sql.DB, memberDID string) ([]CrewMember, error) {
rows, err := db.Query(`
SELECT hold_did, member_did, role, permissions, tier, added_at
FROM hold_crew_members
WHERE member_did = ?
ORDER BY added_at DESC
`, memberDID)
if err != nil {
return nil, err
}
defer rows.Close()
var memberships []CrewMember
for rows.Next() {
var m CrewMember
err := rows.Scan(&m.HoldDID, &m.MemberDID, &m.Role, &m.Permissions, &m.Tier, &m.AddedAt)
if err != nil {
return nil, err
}
memberships = append(memberships, m)
}
return memberships, rows.Err()
}
```
## UI Integration
### Current State
The settings page (`pkg/appview/templates/pages/settings.html`) currently has a **text input field** for the default hold:
```html
<!-- Current implementation (to be replaced) -->
<section class="settings-section">
<h2>Default Hold</h2>
<p>Current: <strong id="current-hold">{{ if .Profile.DefaultHold }}{{ .Profile.DefaultHold }}{{ else }}Not set{{ end }}</strong></p>
<form hx-post="/api/profile/default-hold" ...>
<div class="form-group">
<label for="hold-endpoint">Hold Endpoint:</label>
<input type="text"
id="hold-endpoint"
name="hold_endpoint"
value="{{ .Profile.DefaultHold }}"
placeholder="https://hold.example.com" />
<small>Leave empty to use AppView default storage</small>
</div>
<button type="submit" class="btn-primary">Save</button>
</form>
</section>
```
**Problems with the current approach:**
1. **Users must know hold URLs** - Requires users to manually find and copy hold endpoint URLs
2. **No validation** - Users can enter invalid or inaccessible URLs
3. **No discovery** - Users don't know what holds are available to them
4. **Poor UX** - Text input is error-prone and unfriendly
5. **No membership visibility** - Users can't see which holds they're crew on
### Proposed Change: Dropdown with Discovered Holds
Replace the text input with a `<select>` dropdown populated from the hold discovery cache:
```html
<!-- New implementation -->
<section class="settings-section">
<h2>Default Hold</h2>
<p class="help-text">
Select where your container images will be stored. Holds are organized by your access level.
</p>
<form hx-post="/api/profile/default-hold"
hx-target="#hold-status"
hx-swap="innerHTML"
id="hold-form">
<div class="form-group">
<label for="default-hold">Storage Hold:</label>
<select id="default-hold" name="hold_did" class="form-select">
<option value="">AppView Default ({{ .DefaultHoldDisplayName }})</option>
{{if .OwnedHolds}}
<optgroup label="Your Holds">
{{range .OwnedHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} ({{.Region}}){{end}}
</option>
{{end}}
</optgroup>
{{end}}
{{if .CrewHolds}}
<optgroup label="Crew Member">
{{range .CrewHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} ({{.Region}}){{end}}
{{if not .HasWritePermission}}[read-only]{{end}}
</option>
{{end}}
</optgroup>
{{end}}
{{if .EligibleHolds}}
<optgroup label="Open Registration">
{{range .EligibleHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} ({{.Region}}){{end}}
</option>
{{end}}
</optgroup>
{{end}}
{{if .PublicHolds}}
<optgroup label="Public Holds">
{{range .PublicHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} ({{.Region}}){{end}}
</option>
{{end}}
</optgroup>
{{end}}
</select>
<small>Your images will be stored on the selected hold</small>
</div>
<button type="submit" class="btn-primary">Save</button>
</form>
<div id="hold-status"></div>
<!-- Hold details panel (shows when hold selected) -->
<div id="hold-details" class="hold-details" style="display: none;">
<h3>Hold Details</h3>
<dl>
<dt>DID:</dt>
<dd id="hold-did"></dd>
<dt>Provider:</dt>
<dd id="hold-provider"></dd>
<dt>Region:</dt>
<dd id="hold-region"></dd>
<dt>Your Access:</dt>
<dd id="hold-access"></dd>
</dl>
</div>
</section>
```
### Dropdown Option Groups
The dropdown organizes holds into logical groups based on user's relationship:
| Group | Description | Access Level |
|-------|-------------|--------------|
| **Your Holds** | Holds where user is the captain (owner) | Full control |
| **Crew Member** | Holds where user has explicit crew membership | Based on permissions |
| **Open Registration** | Holds with `allowAllCrew=true` | Can self-register |
| **Public Holds** | Holds with `public=true` | Anyone can use |
### Visual Indicators
Each option should show relevant context:
```
┌─ Storage Hold: ─────────────────────────────────────┐
│ ▼ hold01.atcr.io (us-east) │
├─────────────────────────────────────────────────────┤
│ AppView Default (hold01.atcr.io) │
│ ───────────────────────────────────── │
│ Your Holds │
│ my-hold.fly.dev (us-west) │
│ ───────────────────────────────────── │
│ Crew Member │
│ team-hold.company.com (eu-central) │
│ shared-hold.org (asia-pacific) [read-only] │
│ ───────────────────────────────────── │
│ Open Registration │
│ community-hold.dev (us-east) │
│ ───────────────────────────────────── │
│ Public Holds │
│ public-hold.example.com (global) │
└─────────────────────────────────────────────────────┘
```
### Form Submission Change
The form now submits `hold_did` (a DID) instead of `hold_endpoint` (a URL):
**Before:**
```
POST /api/profile/default-hold
Content-Type: application/x-www-form-urlencoded
hold_endpoint=https://hold01.atcr.io
```
**After:**
```
POST /api/profile/default-hold
Content-Type: application/x-www-form-urlencoded
hold_did=did:web:hold01.atcr.io
```
The `UpdateDefaultHoldHandler` needs to be updated to accept DIDs:
```go
// pkg/appview/handlers/settings.go
func (h *UpdateDefaultHoldHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
user := middleware.GetUser(r)
if user == nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
// Accept DID (new) or endpoint (legacy/fallback)
holdDID := r.FormValue("hold_did")
if holdDID == "" {
// Fallback for legacy form submissions
holdDID = r.FormValue("hold_endpoint")
}
// Validate the hold DID if provided
if holdDID != "" {
// Check it's in our discovered holds cache
captain, err := h.DB.GetCaptainRecord(holdDID)
if err != nil {
http.Error(w, "Unknown hold: "+holdDID, http.StatusBadRequest)
return
}
// Verify user has access to this hold
available, err := db.GetAvailableHolds(h.DB, user.DID)
if err != nil {
http.Error(w, "Failed to check hold access", http.StatusInternalServerError)
return
}
hasAccess := false
for _, h := range available {
if h.DID == holdDID {
hasAccess = true
break
}
}
if !hasAccess {
http.Error(w, "You don't have access to this hold", http.StatusForbidden)
return
}
}
// ... rest of profile update logic
}
```
### Settings Handler
Update the settings handler to include available holds:
```go
// pkg/appview/handlers/settings.go
func (h *Handler) SettingsPage(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
userDID := auth.GetDID(ctx)
// Get user's current profile
profile, err := h.storage.GetProfile(ctx, userDID)
if err != nil {
// Handle error
}
// Get available holds for dropdown
availableHolds, err := db.GetAvailableHolds(h.db, userDID)
if err != nil {
// Handle error
}
data := SettingsPageData{
Profile: profile,
AvailableHolds: availableHolds,
CurrentHoldDID: profile.DefaultHold,
}
h.renderTemplate(w, "settings.html", data)
}
```
### Settings Template
```html
<!-- pkg/appview/templates/pages/settings.html -->
<div class="settings-section">
<h2>Default Hold</h2>
<p class="help-text">
Select where your container images will be stored by default.
</p>
<form method="POST" action="/settings/hold">
<select name="defaultHold" id="defaultHold" class="form-select">
<option value="">-- Select a Hold --</option>
{{if .OwnedHolds}}
<optgroup label="Your Holds">
{{range .OwnedHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}} (Owner)
{{if .Region}} - {{.Region}}{{end}}
</option>
{{end}}
</optgroup>
{{end}}
{{if .CrewHolds}}
<optgroup label="Crew Member">
{{range .CrewHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} - {{.Region}}{{end}}
</option>
{{end}}
</optgroup>
{{end}}
{{if .EligibleHolds}}
<optgroup label="Open Registration">
{{range .EligibleHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} - {{.Region}}{{end}}
</option>
{{end}}
</optgroup>
{{end}}
{{if .PublicHolds}}
<optgroup label="Public Holds">
{{range .PublicHolds}}
<option value="{{.DID}}" {{if eq $.CurrentHoldDID .DID}}selected{{end}}>
{{.DisplayName}}
{{if .Region}} - {{.Region}}{{end}}
</option>
{{end}}
</optgroup>
{{end}}
</select>
<button type="submit" class="btn btn-primary">Save</button>
</form>
</div>
```
### Template Data Preparation
```go
// pkg/appview/handlers/settings.go
type SettingsPageData struct {
Profile *atproto.SailorProfile
CurrentHoldDID string
OwnedHolds []HoldDisplay
CrewHolds []HoldDisplay
EligibleHolds []HoldDisplay
PublicHolds []HoldDisplay
}
type HoldDisplay struct {
DID string
DisplayName string // Derived from DID or endpoint
Region string
Provider string
Permissions []string
}
func (h *Handler) prepareSettingsData(userDID string, holds []db.AvailableHold, currentHold string) SettingsPageData {
data := SettingsPageData{
CurrentHoldDID: currentHold,
}
for _, hold := range holds {
display := HoldDisplay{
DID: hold.DID,
DisplayName: deriveDisplayName(hold.DID, hold.Endpoint),
Region: hold.Region,
Provider: hold.Provider,
Permissions: hold.Permissions,
}
switch hold.Membership {
case "owner":
data.OwnedHolds = append(data.OwnedHolds, display)
case "crew":
data.CrewHolds = append(data.CrewHolds, display)
case "eligible":
data.EligibleHolds = append(data.EligibleHolds, display)
case "public":
data.PublicHolds = append(data.PublicHolds, display)
}
}
return data
}
func deriveDisplayName(did, endpoint string) string {
// For did:web, extract the domain
if strings.HasPrefix(did, "did:web:") {
return strings.TrimPrefix(did, "did:web:")
}
// For did:plc, use the endpoint hostname if available
if endpoint != "" {
if u, err := url.Parse(endpoint); err == nil {
return u.Host
}
}
// Fallback to truncated DID
if len(did) > 20 {
return did[:20] + "..."
}
return did
}
```
### CSS Styles
Add styles for the hold dropdown and details panel:
```css
/* pkg/appview/templates/pages/settings.html - add to <style> section */
/* Hold Selection Styles */
.form-select {
width: 100%;
padding: 0.75rem;
font-size: 1rem;
border: 1px solid var(--border);
border-radius: 4px;
background: var(--bg);
color: var(--fg);
cursor: pointer;
}
.form-select:focus {
outline: none;
border-color: var(--primary);
box-shadow: 0 0 0 2px var(--primary-bg);
}
.form-select optgroup {
font-weight: bold;
color: var(--fg-muted);
padding-top: 0.5rem;
}
.form-select option {
padding: 0.5rem;
font-weight: normal;
color: var(--fg);
}
/* Hold Details Panel */
.hold-details {
margin-top: 1rem;
padding: 1rem;
background: var(--code-bg);
border-radius: 4px;
border: 1px solid var(--border);
}
.hold-details h3 {
margin-top: 0;
margin-bottom: 0.75rem;
font-size: 0.9rem;
color: var(--fg-muted);
text-transform: uppercase;
letter-spacing: 0.05em;
}
.hold-details dl {
display: grid;
grid-template-columns: auto 1fr;
gap: 0.5rem 1rem;
margin: 0;
}
.hold-details dt {
color: var(--fg-muted);
font-weight: 500;
}
.hold-details dd {
margin: 0;
font-family: monospace;
}
/* Access Level Badges */
.access-badge {
display: inline-block;
padding: 0.125rem 0.5rem;
border-radius: 4px;
font-size: 0.85rem;
font-weight: 500;
}
.access-owner {
background: #fef3c7;
color: #92400e;
}
.access-crew {
background: #dcfce7;
color: #166534;
}
.access-eligible {
background: #e0e7ff;
color: #3730a3;
}
.access-public {
background: #f3f4f6;
color: #374151;
}
/* Read-only indicator */
.read-only-indicator {
color: var(--warning);
font-size: 0.85rem;
margin-left: 0.25rem;
}
```
### JavaScript Interaction
Add JavaScript to show hold details when selection changes:
```html
<!-- Add to settings.html <script> section -->
<script>
(function() {
// Hold selection and details display
const holdSelect = document.getElementById('default-hold');
const holdDetails = document.getElementById('hold-details');
// Hold data embedded from server (JSON in data attribute or inline)
const holdData = {{ .HoldDataJSON }};
if (holdSelect) {
holdSelect.addEventListener('change', function() {
const selectedDID = this.value;
if (!selectedDID || !holdData[selectedDID]) {
holdDetails.style.display = 'none';
return;
}
const hold = holdData[selectedDID];
document.getElementById('hold-did').textContent = hold.did;
document.getElementById('hold-provider').textContent = hold.provider || 'Unknown';
document.getElementById('hold-region').textContent = hold.region || 'Global';
// Set access level with badge
const accessEl = document.getElementById('hold-access');
const accessClass = 'access-' + hold.membership;
const accessLabel = {
'owner': 'Owner (Full Control)',
'crew': 'Crew Member',
'eligible': 'Open Registration',
'public': 'Public Access'
}[hold.membership] || hold.membership;
accessEl.innerHTML = `<span class="access-badge ${accessClass}">${accessLabel}</span>`;
// Show permissions for crew members
if (hold.membership === 'crew' && hold.permissions) {
const perms = hold.permissions.join(', ');
accessEl.innerHTML += `<br><small>Permissions: ${perms}</small>`;
}
holdDetails.style.display = 'block';
});
// Trigger on page load if a hold is already selected
if (holdSelect.value) {
holdSelect.dispatchEvent(new Event('change'));
}
}
})();
</script>
```
### Server-Side Hold Data
The handler needs to serialize hold data for the JavaScript:
```go
// pkg/appview/handlers/settings.go
import "encoding/json"
type HoldDataEntry struct {
DID string `json:"did"`
DisplayName string `json:"displayName"`
Provider string `json:"provider"`
Region string `json:"region"`
Membership string `json:"membership"`
Permissions []string `json:"permissions,omitempty"`
}
func (h *SettingsHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// ... existing code ...
// Get available holds
availableHolds, err := db.GetAvailableHolds(h.DB, user.DID)
if err != nil {
slog.Error("Failed to get available holds", "error", err)
availableHolds = []db.AvailableHold{}
}
// Build hold data map for JavaScript
holdDataMap := make(map[string]HoldDataEntry)
for _, hold := range availableHolds {
holdDataMap[hold.DID] = HoldDataEntry{
DID: hold.DID,
DisplayName: deriveDisplayName(hold.DID, hold.Endpoint),
Provider: hold.Provider,
Region: hold.Region,
Membership: hold.Membership,
Permissions: hold.Permissions,
}
}
holdDataJSON, _ := json.Marshal(holdDataMap)
data := SettingsPageData{
// ... existing fields ...
HoldDataJSON: template.JS(holdDataJSON), // Safe for embedding in <script>
}
// ... render template ...
}
```
### Empty State Handling
When no holds are discovered yet, show a helpful message:
```html
{{if and (not .OwnedHolds) (not .CrewHolds) (not .EligibleHolds) (not .PublicHolds)}}
<div class="empty-holds-notice">
<p>
<i data-lucide="info"></i>
No holds discovered yet. Using AppView default storage.
</p>
<p class="help-text">
Holds are discovered automatically via the ATProto network.
If you've deployed your own hold, make sure it has requested a relay crawl.
</p>
</div>
{{else}}
<!-- Show the dropdown -->
{{end}}
```
### Refresh Button
Allow users to manually trigger hold refresh:
```html
<div class="hold-actions">
<button type="button"
class="btn-secondary"
hx-post="/api/holds/refresh"
hx-target="#hold-refresh-status"
hx-swap="innerHTML">
<i data-lucide="refresh-cw"></i> Refresh Holds
</button>
<span id="hold-refresh-status"></span>
</div>
```
```go
// pkg/appview/handlers/settings.go
func (h *RefreshHoldsHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
user := middleware.GetUser(r)
if user == nil {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
// Trigger async refresh of hold cache
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := h.Backfiller.RefreshAllHolds(ctx); err != nil {
slog.Error("Failed to refresh holds", "error", err)
}
}()
w.Header().Set("Content-Type", "text/html")
w.Write([]byte(`<span class="success">Refreshing... reload page in a moment</span>`))
}
```
## Cache Invalidation
### Real-time Updates via Jetstream
Jetstream events automatically update the cache:
- **Captain record created/updated**: Upsert to `hold_captain_records`
- **Captain record deleted**: Delete from `hold_captain_records` (cascades to crew)
- **Crew record created/updated**: Upsert to `hold_crew_members`
- **Crew record deleted**: Delete from `hold_crew_members`
### Manual Refresh
For cases where Jetstream may be delayed or missed events:
```go
// pkg/appview/handlers/settings.go
func (h *Handler) RefreshHoldCache(w http.ResponseWriter, r *http.Request) {
holdDID := r.URL.Query().Get("did")
if holdDID == "" {
http.Error(w, "missing did parameter", http.StatusBadRequest)
return
}
// Verify it's a hold service
hasService, endpoint, err := h.resolver.HasHoldService(holdDID)
if err != nil || !hasService {
http.Error(w, "invalid hold DID", http.StatusBadRequest)
return
}
// Fetch and update captain record
captain, err := h.backfiller.fetchCaptainFromHold(r.Context(), holdDID, endpoint)
if err != nil {
http.Error(w, "failed to fetch captain record", http.StatusInternalServerError)
return
}
if err := h.db.UpsertCaptainRecord(holdDID, captain); err != nil {
http.Error(w, "failed to update cache", http.StatusInternalServerError)
return
}
// Also refresh crew records
if err := h.backfiller.backfillCrewFromHold(r.Context(), holdDID, endpoint); err != nil {
log.Warn().Err(err).Str("did", holdDID).Msg("failed to refresh crew records")
}
http.Redirect(w, r, "/settings", http.StatusSeeOther)
}
```
### TTL-based Refresh
Optionally, run periodic refresh of cached records:
```go
// pkg/appview/jetstream/backfill.go
func (b *Backfiller) RefreshStaleHolds(ctx context.Context, maxAge time.Duration) error {
// Find holds not updated recently
rows, err := b.db.Query(`
SELECT did, endpoint FROM hold_captain_records
WHERE updated_at < datetime('now', ?)
`, fmt.Sprintf("-%d seconds", int(maxAge.Seconds())))
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var did, endpoint string
if err := rows.Scan(&did, &endpoint); err != nil {
continue
}
// Refresh this hold's data
if err := b.refreshHold(ctx, did, endpoint); err != nil {
log.Warn().Err(err).Str("did", did).Msg("failed to refresh stale hold")
}
}
return rows.Err()
}
```
## Security Considerations
### Trust Model
- **Captain records are authoritative**: The hold's embedded PDS is the source of truth
- **Crew records are authoritative**: Same as captain records
- **Cache is for performance**: Always validate against source for sensitive operations
- **No user-provided data**: All data comes from Jetstream or direct PDS queries
### Access Control
- **Read access**: Any authenticated user can view available holds
- **Write access**: Only hold owners can modify captain records
- **Crew management**: Only hold owners and crew admins can add/remove crew
### Data Validation
```go
func validateCaptainRecord(record *atproto.CaptainRecord) error {
if record.OwnerDID == "" {
return errors.New("owner DID is required")
}
if !strings.HasPrefix(record.OwnerDID, "did:") {
return errors.New("invalid owner DID format")
}
return nil
}
func validateCrewRecord(record *atproto.CrewRecord) error {
if record.MemberDID == "" {
return errors.New("member DID is required")
}
if !strings.HasPrefix(record.MemberDID, "did:") {
return errors.New("invalid member DID format")
}
for _, perm := range record.Permissions {
if !isValidPermission(perm) {
return fmt.Errorf("invalid permission: %s", perm)
}
}
return nil
}
func isValidPermission(perm string) bool {
valid := map[string]bool{
"blob:read": true,
"blob:write": true,
"crew:admin": true,
}
return valid[perm]
}
```
## Implementation Checklist
### Phase 1: Database Schema
- [ ] Add `hold_crew_members` table to `pkg/appview/db/schema.sql`
- [ ] Create migration file `pkg/appview/db/migrations/006_hold_discovery.yaml`
- [ ] Verify `rkey` column included for delete event handling
- [ ] Run migration on dev/staging databases
- [ ] Verify foreign key cascade works correctly
### Phase 2: Jetstream Integration
- [ ] Add `io.atcr.hold.captain` to wanted collections in `pkg/appview/jetstream/worker.go`
- [ ] Add `io.atcr.hold.crew` to wanted collections
- [ ] Implement `ProcessCaptain` function in `pkg/appview/jetstream/processor.go`
- [ ] Implement `ProcessCrew` function
- [ ] Add hold service verification (`#atcr_hold` check via DID document)
- [ ] Handle delete events for captain records (cascade to crew)
- [ ] Handle delete events for crew records (by rkey lookup)
- [ ] Test with local hold service connected to local relay
### Phase 3: Backfill
- [ ] Implement `BackfillHolds` function in `pkg/appview/jetstream/backfill.go`
- [ ] Implement `backfillCrewRecords` function
- [ ] Implement `listReposWithCollection` helper
- [ ] Add `ATCR_BOOTSTRAP_HOLDS` environment variable support
- [ ] Implement `BackfillBootstrapHolds` function
- [ ] Implement `fetchCaptainFromHold` direct fetch
- [ ] Test backfill with production relay
- [ ] Add backfill command to CLI (optional)
### Phase 4: Database Queries
- [ ] Implement `UpsertCrewMember` in `pkg/appview/db/hold_store.go`
- [ ] Implement `DeleteCrewMember(holdDID, memberDID)`
- [ ] Implement `DeleteCrewMemberByRkey(holdDID, rkey)`
- [ ] Implement `GetAvailableHolds(userDID)` with membership categorization
- [ ] Implement `GetHoldsOwnedBy(ownerDID)`
- [ ] Implement `GetCrewMemberships(memberDID)`
- [ ] Add unit tests for all queries
### Phase 5: UI Integration - Settings Handler
- [ ] Add `DB *sql.DB` field to `SettingsHandler` struct
- [ ] Call `db.GetAvailableHolds()` in handler
- [ ] Create `SettingsPageData` struct with hold lists
- [ ] Implement `prepareSettingsData` helper function
- [ ] Implement `deriveDisplayName(did, endpoint)` helper
- [ ] Create `HoldDataEntry` struct for JSON serialization
- [ ] Serialize hold data to JSON for JavaScript
### Phase 6: UI Integration - Template Changes
- [ ] Replace text input with `<select>` dropdown in `settings.html`
- [ ] Add `<optgroup>` sections: Your Holds, Crew Member, Open Registration, Public
- [ ] Add `[read-only]` indicator for crew without write permission
- [ ] Add hold details panel (`#hold-details` div)
- [ ] Add empty state notice when no holds discovered
- [ ] Add "Refresh Holds" button
- [ ] Update form to submit `hold_did` instead of `hold_endpoint`
### Phase 7: UI Integration - Styles & JavaScript
- [ ] Add `.form-select` styles for dropdown
- [ ] Add `.hold-details` styles for details panel
- [ ] Add `.access-badge` styles (owner, crew, eligible, public)
- [ ] Add JavaScript for hold selection change handler
- [ ] Show hold details on selection change
- [ ] Display permissions for crew members
- [ ] Handle initial page load with pre-selected hold
### Phase 8: Form Handler Updates
- [ ] Update `UpdateDefaultHoldHandler` to accept `hold_did` parameter
- [ ] Add fallback for legacy `hold_endpoint` parameter
- [ ] Validate hold DID exists in cache
- [ ] Verify user has access to selected hold
- [ ] Return appropriate error for unknown/inaccessible holds
- [ ] Add `RefreshHoldsHandler` for manual refresh button
### Phase 9: Testing
- [ ] Unit tests for database queries
- [ ] Unit tests for Jetstream processors
- [ ] Integration test: discover hold via Jetstream
- [ ] Integration test: backfill existing holds
- [ ] E2E test: settings page displays holds
- [ ] E2E test: change default hold via dropdown
- [ ] E2E test: verify push uses new default hold
### Phase 10: Cache Management & Monitoring
- [ ] Implement `RefreshStaleHolds` for TTL-based refresh (optional)
- [ ] Add Prometheus metrics for cache operations
- [ ] Monitor cache hit/miss rates
- [ ] Add logging for discovery events
- [ ] Document operational procedures
## Future Enhancements
### Hold Search
Add search/filter capabilities:
```sql
SELECT * FROM hold_captain_records
WHERE region LIKE ?
OR provider LIKE ?
ORDER BY ...
```
### Hold Recommendations
Suggest holds based on:
- Geographic proximity (region matching)
- Provider preference
- Existing crew memberships
### Hold Statistics
Display usage information:
- Storage used
- Number of images
- Number of crew members
- Uptime/availability
### Hold Comparison
Side-by-side comparison of:
- Storage limits
- Supported features
- Geographic regions
- Pricing (if applicable)