at-container-registry/docs/DEVELOPMENT.md

Development Workflow for ATCR

Goal

Run the ATCR services (AppView, Hold, Labeler) locally with hot reload so that Go, template, CSS, and JS changes show up after a fast incremental rebuild instead of a full production image rebuild.

The mechanism is Air (github.com/air-verse/air) running inside a development container. Air watches the mounted source tree and rebuilds the relevant binary on change. Production images are unaffected — they use the multi-stage Dockerfile.appview / Dockerfile.hold / Dockerfile.scanner builds with embedded assets.

How It Works

All UI assets are embedded into the binary via //go:embed in pkg/appview/ui.go (//go:embed public and //go:embed templates/**/*.html). There is no filesystem-vs-embed toggle and no ATCR_DEV_MODE switch — the binary always serves embedded assets. Hot reload therefore works by having Air rebuild the binary whenever a watched file changes, not by reading templates off disk at request time.

When Air rebuilds the AppView binary it runs a pre_cmd of go generate ./pkg/appview/.... The //go:generate directive in pkg/appview/ui.go shells out to npm run build:appview, which regenerates the CSS bundle (pkg/appview/public/css/style.css), the JS bundle (pkg/appview/public/js/bundle.min.js), and the icon sprite (pkg/appview/public/icons.svg) before they are re-embedded into the new binary.

Do not run npm run css:build / npm run js:build manually. The go generate step driven by Air handles asset builds. Editing a source asset (CSS/JS/template) and saving triggers an Air rebuild, which regenerates and re-embeds the assets automatically.

Architecture Flow

┌─────────────────────────────────────────────────────┐
│ Your editor                                          │
│ Edit: *.go, templates/*.html, src/css/*, src/js/*   │
└─────────────────┬───────────────────────────────────┘
                  │ (files saved to disk)
                  ▼
┌─────────────────────────────────────────────────────┐
│ Volume mount (docker-compose.yml)                    │
│   volumes:                                           │
│     - .:/app:z   (entire codebase mounted)           │
└─────────────────┬───────────────────────────────────┘
                  │ (changes appear in container)
                  ▼
┌─────────────────────────────────────────────────────┐
│ Container (mirror.gcr.io/library/golang:1.26.2)     │
│                                                      │
│  ┌──────────────────────────────────────┐           │
│  │ Air (github.com/air-verse/air)        │           │
│  │ poll = true, poll_interval = 500      │           │
│  │ Watches: *.go *.html *.css *.js       │           │
│  │                                       │           │
│  │ On change:                            │           │
│  │   1. pre_cmd: go generate (npm build) │           │
│  │   2. cmd: go build → ./tmp/atcr-*     │           │
│  │   3. restart binary (entrypoint)      │           │
│  └──────────────────────────────────────┘           │
│                 │                                    │
│                 ▼                                    │
│  ATCR AppView (serves embedded assets from           │
│  the freshly built binary)                           │
└─────────────────────────────────────────────────────┘

Polling (poll = true, poll_interval = 500) is required: inotify/fsnotify events do not propagate reliably across Docker bind mounts, so Air polls the mounted tree every 500ms instead.

Files Involved

File	Purpose
Dockerfile.dev	Single dev image used by all three services. golang:1.26.2-trixie base with Air, Node/npm, and SQLite installed. Source comes from a volume mount, not COPY. Accepts an AIR_CONFIG build arg to select which .air.*.toml to run.
docker-compose.yml	The dev compose file (this is the primary compose file — there is no separate docker-compose.dev.yml). Defines atcr-appview, atcr-hold, atcr-labeler, and victorialogs, all on a fixed 172.28.0.0/24 network.
.air.toml	AppView Air config (default AIR_CONFIG).
.air.hold.toml	Hold Air config (selected via AIR_CONFIG=.air.hold.toml).
.air.labeler.toml	Labeler Air config (selected via AIR_CONFIG=.air.labeler.toml).

Dockerfile.dev

# Development image with Air hot reload
FROM mirror.gcr.io/library/golang:1.26.2-trixie

ARG AIR_CONFIG=.air.toml

ENV DEBIAN_FRONTEND=noninteractive
ENV AIR_CONFIG=${AIR_CONFIG}

RUN apt-get update && \
    apt-get install -y --no-install-recommends sqlite3 libsqlite3-dev curl nodejs npm && \
    rm -rf /var/lib/apt/lists/* && \
    go install github.com/air-verse/air@latest

WORKDIR /app

# Copy go.mod first for layer caching
COPY go.mod go.sum ./
RUN go mod download

# For development: source mounted as volume, Air handles builds
CMD ["sh", "-c", "air -c ${AIR_CONFIG}"]

Note the Air install path is github.com/air-verse/air@latest. The old github.com/cosmtrek/air module is archived and must not be used.

.air.toml (AppView)

This is the real file — keep it in sync rather than copying a hand-written version. Load-bearing settings:

root = "."
tmp_dir = "tmp"

[build]
# Use polling for Docker volume mounts (inotify doesn't work across mounts)
poll = true
poll_interval = 500
# Pre-build: generate assets if missing (each string is a shell command)
pre_cmd = ["go generate ./pkg/appview/..."]
cmd = "go build -tags billing -buildvcs=false -o ./tmp/atcr-appview ./cmd/appview"
entrypoint = ["./tmp/atcr-appview", "serve", "--config", "config-appview.example.yaml"]
include_ext = ["go", "html", "css", "js"]
exclude_dir = ["bin", "tmp", "vendor", "deploy", "docs", ".git", "dist", "node_modules", "scanner", "pkg/hold", "pkg/labeler"]
exclude_regex = ["_test\\.go$", "cbor_gen\\.go$", "\\.min\\.js$", "public/css/style\\.css$", "public/icons\\.svg$"]
delay = 3000
stop_on_error = true
send_interrupt = true
kill_delay = 3000

Key points that differ from a naive config:

• poll = true / poll_interval = 500 — needed for Docker bind mounts.
• pre_cmd runs go generate ./pkg/appview/..., which regenerates and re-embeds CSS/JS/icons before the build.
• cmd builds with -tags billing (AppView dev runs with billing support) and -buildvcs=false.
• entrypoint is the full argv for the built binary: it runs serve --config config-appview.example.yaml. (The example config is the dev base config; env vars in docker-compose.yml override it.)
• The exclude_regex deliberately ignores the generated asset outputs (*.min.js, public/css/style.css, public/icons.svg) so regeneration does not trigger an infinite rebuild loop.

.air.hold.toml and .air.labeler.toml are analogous: they build ./cmd/hold / ./cmd/labeler, generate ./pkg/hold/... (the labeler has no generate step), and exclude the other services' packages from watching.

Configuration via Environment Variables

docker-compose.yml sets a base config file per service via the Air entrypoint (config-appview.example.yaml, config-hold.example.yaml, config-labeler.example.yaml) and overrides specific values with environment variables. Viper maps env var names from the YAML path, prefixed with the service prefix and joined with _.

Real AppView env vars (note these are the Viper-mapped names, not invented shorthand):

Env var	Maps to
ATCR_SERVER_ADDR	server.addr (listen address, e.g. :5000)
ATCR_SERVER_BASE_URL	server.base_url
ATCR_SERVER_MANAGED_HOLDS	server.managed_holds — comma-separated DID list; the first entry is the default blob-storage hold. Viper splits on commas.
ATCR_AUTH_CERT_PATH	auth.cert_path
ATCR_JETSTREAM_BACKFILL_ENABLED	jetstream.backfill_enabled
ATCR_LABELER_DID	labeler.did
ATCR_SERVER_TEST_MODE	server.test_mode
ATCR_LOG_LEVEL	log.level

There is no ATCR_DEV_MODE variable anywhere in the codebase. Likewise ATCR_HTTP_ADDR, ATCR_BASE_URL, ATCR_DEFAULT_HOLD_DID, ATCR_AUTH_KEY_PATH, and ATCR_BACKFILL_ENABLED are not real — use the Viper-mapped names above.

Hold and Labeler use the HOLD_ and LABELER_ prefixes respectively (e.g. HOLD_SERVER_PUBLIC_URL, HOLD_SERVER_APPVIEW_DID, LABELER_LABELER_PUBLIC_URL). See docker-compose.yml for the dev values.

S3/Storj credentials and shared secrets are loaded from an external ../atcr-secrets.env file referenced via env_file: in docker-compose.yml.

Usage

Start the dev environment

docker-compose.yml is the dev compose file, so no -f flag is needed:

# Build and start everything (appview, hold, labeler, victorialogs)
docker compose up --build

# Or in the background
docker compose up -d

# Tail a single service
docker compose logs -f atcr-appview

Services bind to fixed ports on the host:

• AppView: http://localhost:5000
• Hold: http://localhost:8080
• Labeler: http://localhost:5002
• Victoria Logs: http://localhost:9428

On a clean start you should see Air bootstrap, run the pre-build generate step, build, and launch the binary, e.g.:

atcr-appview  | watching .
atcr-appview  | !exclude tmp
atcr-appview  | running pre_cmd: go generate ./pkg/appview/...
atcr-appview  | building...
atcr-appview  | running...
atcr-appview  | <appview startup logs: server listening on :5000 ...>

Daily workflow

• Edit Go code → save → Air rebuilds (go build) and restarts the binary in a few seconds.
• Edit a template (pkg/appview/templates/**/*.html) → save → Air rebuilds so the new template is re-embedded.
• Edit CSS source (pkg/appview/src/css/main.css) → save → the pre_cmd go generate regenerates pkg/appview/public/css/style.css via npm run build:appview, then the binary rebuilds.
• Edit JS source (pkg/appview/src/js/main.js) → save → go generate regenerates pkg/appview/public/js/bundle.min.js, then the binary rebuilds.

Important asset-source vs. generated-output distinctions:

You edit (source)	Do NOT edit (generated)
pkg/appview/src/css/main.css	pkg/appview/public/css/style.css
pkg/appview/src/js/main.js	pkg/appview/public/js/bundle.min.js
icon references in templates	pkg/appview/public/icons.svg

Refresh the browser after the rebuild completes.

Stop the dev environment

# Stop containers
docker compose down

# Stop and wipe volumes (fresh DB / PDS / labeler state)
docker compose down -v

Local Development (No Docker)

For a tighter loop you can run a single service on the host. The make dev target runs the AppView under Air using .air.toml:

make dev

make dev ensures Air is installed (go install github.com/air-verse/air@latest), builds the generated assets, and runs air -c .air.toml.

You can also run Air directly, or skip hot reload entirely:

# Air, AppView config
air -c .air.toml

# No hot reload — build and run once
go build -tags billing -o bin/atcr-appview ./cmd/appview
./bin/atcr-appview serve --config config-appview.example.yaml

Running on the host requires a working toolchain for the build: Go 1.26.2 (see go.work), Node/npm (for the go generate asset step), and SQLite headers. Override config values with the ATCR_* env vars listed above, or edit your local config file.

Production Builds (Unchanged)

Production images use the multi-stage Dockerfiles and embed all assets at compile time:

make docker            # build appview + hold + scanner images
make docker-appview    # just the appview image

These do not involve Air, do not bind-mount source, and serve embedded assets exactly as the dev binary does — the only difference is that the dev container rebuilds on change.

Troubleshooting

Air not rebuilding

docker compose logs atcr-appview
# Confirm Air is running and polling. poll=true is required for bind mounts;
# without it, saved files are never detected.
docker compose restart atcr-appview

Confirm your file type is in include_ext (go, html, css, js) and that you are editing a source file, not a generated output excluded by exclude_regex.

Go build failing

docker compose logs atcr-appview
# Air prints build errors inline and (with stop_on_error=true) holds the old
# binary until the build succeeds again. Fix the error and save.

Volume mount not working

docker compose exec atcr-appview ls -la /app
# Should show your source tree. On macOS/Windows check Docker Desktop file
# sharing for the project directory.

Asset changes not showing

CSS/JS/icon changes only take effect after the go generate pre-build runs and the binary rebuilds. If a save did not trigger a rebuild, you likely edited a generated output file (excluded from watching) instead of its source under pkg/appview/src/.