at-container-registry/docs/DEVELOPMENT.md

# Development Workflow for ATCR

## Goal

Run the ATCR services (AppView, Hold, Labeler) locally with hot reload so that
Go, template, CSS, and JS changes show up after a fast incremental rebuild
instead of a full production image rebuild.

The mechanism is **Air** (`github.com/air-verse/air`) running inside a
development container. Air watches the mounted source tree and rebuilds the
relevant binary on change. Production images are unaffected — they use the
multi-stage `Dockerfile.appview` / `Dockerfile.hold` / `Dockerfile.scanner`
builds with embedded assets.

## How It Works

All UI assets are embedded into the binary via `//go:embed` in
`pkg/appview/ui.go` (`//go:embed public` and `//go:embed templates/**/*.html`).
There is **no** filesystem-vs-embed toggle and no `ATCR_DEV_MODE` switch — the
binary always serves embedded assets. Hot reload therefore works by having Air
**rebuild the binary** whenever a watched file changes, not by reading templates
off disk at request time.

When Air rebuilds the AppView binary it runs a `pre_cmd` of
`go generate ./pkg/appview/...`. The `//go:generate` directive in
`pkg/appview/ui.go` shells out to `npm run build:appview`, which regenerates the
CSS bundle (`pkg/appview/public/css/style.css`), the JS bundle
(`pkg/appview/public/js/bundle.min.js`), and the icon sprite
(`pkg/appview/public/icons.svg`) before they are re-embedded into the new
binary.

> Do **not** run `npm run css:build` / `npm run js:build` manually. The
> `go generate` step driven by Air handles asset builds. Editing a source asset
> (CSS/JS/template) and saving triggers an Air rebuild, which regenerates and
> re-embeds the assets automatically.

### Architecture Flow

```
┌─────────────────────────────────────────────────────┐
│ Your editor                                          │
│ Edit: *.go, templates/*.html, src/css/*, src/js/*   │
└─────────────────┬───────────────────────────────────┘
                  │ (files saved to disk)
                  ▼
┌─────────────────────────────────────────────────────┐
│ Volume mount (docker-compose.yml)                    │
│   volumes:                                           │
│     - .:/app:z   (entire codebase mounted)           │
└─────────────────┬───────────────────────────────────┘
                  │ (changes appear in container)
                  ▼
┌─────────────────────────────────────────────────────┐
│ Container (mirror.gcr.io/library/golang:1.26.7)     │
│                                                      │
│  ┌──────────────────────────────────────┐           │
│  │ Air (github.com/air-verse/air)        │           │
│  │ poll = true, poll_interval = 500      │           │
│  │ Watches: *.go *.html *.css *.js       │           │
│  │                                       │           │
│  │ On change:                            │           │
│  │   1. pre_cmd: go generate (npm build) │           │
│  │   2. cmd: go build → ./tmp/atcr-*     │           │
│  │   3. restart binary (entrypoint)      │           │
│  └──────────────────────────────────────┘           │
│                 │                                    │
│                 ▼                                    │
│  ATCR AppView (serves embedded assets from           │
│  the freshly built binary)                           │
└─────────────────────────────────────────────────────┘
```

Polling (`poll = true`, `poll_interval = 500`) is **required**: inotify/fsnotify
events do not propagate reliably across Docker bind mounts, so Air polls the
mounted tree every 500ms instead.

## Files Involved

| File | Purpose |
|------|---------|
| `Dockerfile.dev` | Single dev image used by all three services. `golang:1.26.7-trixie` base with Air, Node/npm, and SQLite installed. Source comes from a volume mount, not `COPY`. Accepts an `AIR_CONFIG` build arg to select which `.air.*.toml` to run. |
| `docker-compose.yml` | The dev compose file (this *is* the primary compose file — there is no separate `docker-compose.dev.yml`). Defines `atcr-appview`, `atcr-hold`, `atcr-labeler`, and `victorialogs`, all on a fixed `172.28.0.0/24` network. |
| `.air.toml` | AppView Air config (default `AIR_CONFIG`). |
| `.air.hold.toml` | Hold Air config (selected via `AIR_CONFIG=.air.hold.toml`). |
| `.air.labeler.toml` | Labeler Air config (selected via `AIR_CONFIG=.air.labeler.toml`). |

## `Dockerfile.dev`

```dockerfile
# Development image with Air hot reload
FROM mirror.gcr.io/library/golang:1.26.7-trixie

ARG AIR_CONFIG=.air.toml

ENV DEBIAN_FRONTEND=noninteractive
ENV AIR_CONFIG=${AIR_CONFIG}

RUN apt-get update && \
    apt-get install -y --no-install-recommends sqlite3 libsqlite3-dev curl nodejs npm && \
    rm -rf /var/lib/apt/lists/* && \
    go install github.com/air-verse/air@latest

WORKDIR /app

# Copy go.mod first for layer caching
COPY go.mod go.sum ./
RUN go mod download

# For development: source mounted as volume, Air handles builds
CMD ["sh", "-c", "air -c ${AIR_CONFIG}"]
```

Note the Air install path is `github.com/air-verse/air@latest`. The old
`github.com/cosmtrek/air` module is archived and must not be used.

## `.air.toml` (AppView)

This is the real file — keep it in sync rather than copying a hand-written
version. Load-bearing settings:

```toml
root = "."
tmp_dir = "tmp"

[build]
# Use polling for Docker volume mounts (inotify doesn't work across mounts)
poll = true
poll_interval = 500
# Pre-build: generate assets if missing (each string is a shell command)
pre_cmd = ["go generate ./pkg/appview/..."]
# GO_TAGS (set by Dockerfile.dev / `make dev`) appends build tags, e.g. testmode.
cmd = "go build -tags billing${GO_TAGS:+,$GO_TAGS} -buildvcs=false -o ./tmp/atcr-appview ./cmd/appview"
entrypoint = ["./tmp/atcr-appview", "serve", "--config", "config-appview.example.yaml"]
include_ext = ["go", "html", "css", "js"]
exclude_dir = ["bin", "tmp", "vendor", "deploy", "docs", ".git", "dist", "node_modules", "scanner", "pkg/hold", "pkg/labeler"]
exclude_regex = ["_test\\.go$", "cbor_gen\\.go$", "\\.min\\.js$", "public/css/style\\.css$", "public/icons\\.svg$"]
delay = 3000
stop_on_error = true
send_interrupt = true
kill_delay = 3000
```

Key points that differ from a naive config:

- `poll = true` / `poll_interval = 500` — needed for Docker bind mounts.
- `pre_cmd` runs `go generate ./pkg/appview/...`, which regenerates and
  re-embeds CSS/JS/icons before the build.
- `cmd` builds with `-tags billing` (AppView dev runs with billing support) and
  `-buildvcs=false`. Air runs the command through `sh -c`, so `${GO_TAGS:+,$GO_TAGS}`
  appends whatever `GO_TAGS` holds. docker-compose passes `GO_TAGS: testmode` as a
  build arg to `Dockerfile.dev`, and `make dev` exports the same, so every dev
  build is a **testmode build**: `pkg/atproto/indigo_local.go` replaces
  `indigo_prod.go`, letting a did:web on an IP, `localhost`, or any port (the
  hold's `did:web:localhost%3A8080`, the appview's `did:web:127.0.0.1%3A5000`,
  the labeler's `did:web:172.28.0.4%3A5002`) resolve over plain HTTP, and letting
  the OAuth client reach a PDS on loopback. Production images never set the tag
  and cannot be configured to resolve local DIDs at runtime.
- `entrypoint` is the full argv for the built binary: it runs
  `serve --config config-appview.example.yaml`. (The example config is the dev
  base config; env vars in `docker-compose.yml` override it.)
- The `exclude_regex` deliberately ignores the *generated* asset outputs
  (`*.min.js`, `public/css/style.css`, `public/icons.svg`) so regeneration does
  not trigger an infinite rebuild loop.

`.air.hold.toml` and `.air.labeler.toml` are analogous: they build
`./cmd/hold` / `./cmd/labeler`, generate `./pkg/hold/...` (the labeler has no
generate step), and exclude the other services' packages from watching.

## Configuration via Environment Variables

`docker-compose.yml` sets a base config file per service via the Air
`entrypoint` (`config-appview.example.yaml`, `config-hold.example.yaml`,
`config-labeler.example.yaml`) and overrides specific values with environment
variables. Viper maps env var names from the YAML path, prefixed with the
service prefix and joined with `_`.

Real AppView env vars (note these are the *Viper-mapped* names, not invented
shorthand):

| Env var | Maps to |
|---------|---------|
| `ATCR_SERVER_ADDR` | `server.addr` (listen address, e.g. `:5000`) |
| `ATCR_SERVER_BASE_URL` | `server.base_url` |
| `ATCR_SERVER_MANAGED_HOLDS` | `server.managed_holds` — comma-separated DID list; **the first entry is the default blob-storage hold**. Viper splits on commas. |
| `ATCR_AUTH_CERT_PATH` | `auth.cert_path` |
| `ATCR_JETSTREAM_BACKFILL_ENABLED` | `jetstream.backfill_enabled` |
| `ATCR_LABELER_DID` | `labeler.did` |
| `ATCR_LOG_LEVEL` | `log.level` |

There is **no** `ATCR_DEV_MODE` variable anywhere in the codebase. Likewise
`ATCR_HTTP_ADDR`, `ATCR_BASE_URL`, `ATCR_DEFAULT_HOLD_DID`, `ATCR_AUTH_KEY_PATH`,
and `ATCR_BACKFILL_ENABLED` are *not* real — use the Viper-mapped names above.

Hold and Labeler use the `HOLD_` and `LABELER_` prefixes respectively
(e.g. `HOLD_SERVER_PUBLIC_URL`, `HOLD_SERVER_APPVIEW_DID`,
`LABELER_LABELER_PUBLIC_URL`). See `docker-compose.yml` for the dev values.

S3/Storj credentials and shared secrets are loaded from an external
`../atcr-secrets.env` file referenced via `env_file:` in `docker-compose.yml`.

## Usage

### Start the dev environment

`docker-compose.yml` is the dev compose file, so no `-f` flag is needed:

```bash
# Build and start everything (appview, hold, labeler, victorialogs)
docker compose up --build

# Or in the background
docker compose up -d

# Tail a single service
docker compose logs -f atcr-appview
```

Services bind to fixed ports on the host:

- AppView: http://localhost:5000
- Hold: http://localhost:8080
- Labeler: http://localhost:5002
- Victoria Logs: http://localhost:9428

On a clean start you should see Air bootstrap, run the pre-build generate step,
build, and launch the binary, e.g.:

```
atcr-appview  | watching .
atcr-appview  | !exclude tmp
atcr-appview  | running pre_cmd: go generate ./pkg/appview/...
atcr-appview  | building...
atcr-appview  | running...
atcr-appview  | <appview startup logs: server listening on :5000 ...>
```

### Daily workflow

- **Edit Go code** → save → Air rebuilds (`go build`) and restarts the binary in
  a few seconds.
- **Edit a template** (`pkg/appview/templates/**/*.html`) → save → Air rebuilds
  so the new template is re-embedded.
- **Edit CSS source** (`pkg/appview/src/css/main.css`) → save → the `pre_cmd`
  `go generate` regenerates `pkg/appview/public/css/style.css` via `npm run
  build:appview`, then the binary rebuilds.
- **Edit JS source** (`pkg/appview/src/js/main.js`) → save → `go generate`
  regenerates `pkg/appview/public/js/bundle.min.js`, then the binary rebuilds.

Important asset-source vs. generated-output distinctions:

| You edit (source) | Do NOT edit (generated) |
|-------------------|--------------------------|
| `pkg/appview/src/css/main.css` | `pkg/appview/public/css/style.css` |
| `pkg/appview/src/js/main.js` | `pkg/appview/public/js/bundle.min.js` |
| icon references in templates | `pkg/appview/public/icons.svg` |

Refresh the browser after the rebuild completes.

### Stop the dev environment

```bash
# Stop containers
docker compose down

# Stop and wipe volumes (fresh DB / PDS / labeler state)
docker compose down -v
```

## Local Development (No Docker)

For a tighter loop you can run a single service on the host. The `make dev`
target runs the AppView under Air using `.air.toml`:

```bash
make dev
```

`make dev` ensures Air is installed (`go install github.com/air-verse/air@latest`),
builds the generated assets, and runs `air -c .air.toml`.

You can also run Air directly, or skip hot reload entirely:

```bash
# Air, AppView config (GO_TAGS makes it a testmode build, as `make dev` does)
GO_TAGS=testmode air -c .air.toml

# No hot reload — build and run once
go build -tags billing,testmode -o bin/atcr-appview ./cmd/appview
./bin/atcr-appview serve --config config-appview.example.yaml
```

Leave `testmode` off only when the appview talks exclusively to public
identities (a real PDS, a hold on a public HTTPS hostname); with it off, any
did:web naming an IP, `localhost`, or a port fails to resolve, exactly as in
production. Tests need the tag too: `make test` sets it, and a bare
`go test ./...` fails fast in the tests that depend on it with a message naming
the tag.

Running on the host requires a working toolchain for the build:
Go 1.26.7 (see `go.work`), Node/npm (for the `go generate` asset step), and
SQLite headers. Override config values with the `ATCR_*` env vars listed above,
or edit your local config file.

## Production Builds (Unchanged)

Production images use the multi-stage Dockerfiles and embed all assets at
compile time:

```bash
make docker            # build appview + hold + scanner images
make docker-appview    # just the appview image
```

These do not involve Air, do not bind-mount source, and serve embedded assets
exactly as the dev binary does — the only difference is that the dev container
rebuilds on change.

## Troubleshooting

### Air not rebuilding

```bash
docker compose logs atcr-appview
# Confirm Air is running and polling. poll=true is required for bind mounts;
# without it, saved files are never detected.
docker compose restart atcr-appview
```

Confirm your file type is in `include_ext` (`go`, `html`, `css`, `js`) and that
you are editing a *source* file, not a generated output excluded by
`exclude_regex`.

### Go build failing

```bash
docker compose logs atcr-appview
# Air prints build errors inline and (with stop_on_error=true) holds the old
# binary until the build succeeds again. Fix the error and save.
```

### Volume mount not working

```bash
docker compose exec atcr-appview ls -la /app
# Should show your source tree. On macOS/Windows check Docker Desktop file
# sharing for the project directory.
```

### Asset changes not showing

CSS/JS/icon changes only take effect after the `go generate` pre-build runs and
the binary rebuilds. If a save did not trigger a rebuild, you likely edited a
generated output file (excluded from watching) instead of its source under
`pkg/appview/src/`.