at-container-registry/pkg/appview/db/migrations/README.md

Database Migrations

This directory contains database migrations for the ATCR AppView database.

Schema vs Migrations

schema.sql (in parent directory) contains the complete base schema for fresh database installations. It includes all tables, indexes, and constraints.

Migrations (this directory) handle changes to existing databases:

• CREATE TABLE statements (see below — new tables need a migration too)
• ALTER TABLE statements (add/modify/drop columns)
• UPDATE statements (data transformations)
• DELETE statements (data cleanup)
• Creating/modifying indexes on existing tables

New tables go in BOTH places

InitDB skips schema.sql entirely once schema_migrations has any rows (see hasAppliedMigrations in schema.go). A table added only to schema.sql will never be created on an existing database — it appears on fresh installs, works in every test, and is silently absent in production.

So a new table needs two changes:

1. schema.sql, for fresh installs.
2. A migration with CREATE TABLE IF NOT EXISTS, for existing databases.

TestSchemaMatchesMigrations enforces this. It builds the schema both ways and fails if they disagree, so forgetting either half is caught before it ships.

Two more rules the test cannot enforce for you

Migrations must not return rows. go-libsql rejects a row-returning statement passed to Exec with Execute returned rows. A migration that opens with a bare SELECT fails on every database that has not already recorded it. (Migration 0001 does exactly this; it survives only because every real database recorded it years ago.)

Rebuild migrations must name their columns. Column order in schema.sql is illustrative, not authoritative: migrations append with ADD COLUMN while schema.sql places the same column mid-table, so a fresh database and an upgraded one legitimately differ in column order on manifests, users, devices and repo_pages. INSERT INTO new_table SELECT * FROM old_table will therefore silently write the wrong values into the wrong columns. Always write INSERT INTO new_table (a, b, c) SELECT a, b, c FROM old_table, as migrations 0009 and 0011 do.

Migration Format

Each migration is a YAML file with the following structure:

description: Optional human-readable description of what this migration does
query: |
  SQL commands to apply the migration

Version and name are parsed from the filename, so you don't need to specify them in the YAML.

Naming Convention

Migration files must be named: {version:04d}_{migration_name}.yaml

The filename determines:

• Version: Numeric prefix (e.g., 0001 → version 1)
• Name: Everything after first underscore (e.g., add_repository_labels → "add repository labels")

Examples:

• 0001_remove_star_count_from_repository_stats.yaml → version 1, name "remove star count from repository stats"
• 0002_add_repository_labels.yaml → version 2, name "add repository labels"
• 0003_create_webhooks_table.yaml → version 3, name "create webhooks table"

Creating a New Migration

1. Choose the next version number - Look at existing migrations and increment by 1
2. Create a new YAML file with format 000N_descriptive_name.yaml
3. Add description (optional) - Explain what the migration does
4. Write your SQL in query - Use the | block scalar for clean multi-line SQL
5. Use IF EXISTS / IF NOT EXISTS where possible for idempotency

Examples

Adding a column to existing table:

Filename: 0007_add_readme_url_to_manifests.yaml

description: Add readme_url column to manifests table for storing io.atcr.readme annotation
query: |
  ALTER TABLE manifests ADD COLUMN readme_url TEXT;

IMPORTANT: After creating this migration, also add the column to schema.sql so fresh installations include it!

Data transformation migration:

Filename: 0005_normalize_hold_endpoint_to_did.yaml

description: Normalize hold_endpoint column to store DIDs instead of URLs
query: |
  -- Convert HTTPS URLs to did:web: format
  UPDATE manifests
  SET hold_endpoint = 'did:web:' || substr(hold_endpoint, 9)
  WHERE hold_endpoint LIKE 'https://%';

  -- Convert HTTP URLs to did:web: format
  UPDATE manifests
  SET hold_endpoint = 'did:web:' || substr(hold_endpoint, 8)
  WHERE hold_endpoint LIKE 'http://%';

Adding an index to existing table:

Filename: 0008_add_repository_description_index.yaml

description: Add index on manifests description field for faster searches
query: |
  CREATE INDEX IF NOT EXISTS idx_manifests_description ON manifests(description);

How Migrations Run

1. Migrations are loaded from this directory on startup
2. Sorted by version number (ascending)
3. Each migration is checked against the schema_migrations table
4. Only unapplied migrations are executed
5. After successful execution, the version is recorded in schema_migrations

Important Notes

• Never modify existing migrations - Once applied, they're immutable
• Test migrations before committing - Ensure they work on existing databases
• Version numbers must be unique - The migration system silently skips a duplicate, so the second file never runs (see TestMigrationVersionsAreUnique)
• Migrations run automatically on InitDB() - Schema first, then migrations
• CRITICAL: Update schema.sql for every structural change - Columns, tables and indexes all need both the migration AND the schema.sql entry, or fresh and existing databases diverge. TestSchemaMatchesMigrations fails the build if they do
• Migrations must not return rows - a bare SELECT fails under go-libsql with Execute returned rows
• Rebuild migrations must name columns explicitly - never INSERT INTO new SELECT * FROM old; column order differs between fresh and upgraded databases
• Drift is reported at boot - InitDB logs a warning for any difference between an existing database and schema.sql. It never fails the boot; treat the warning as a request for a corrective migration