diff --git a/.agents/skills/yatm/SKILL.md b/.agents/skills/yatm/SKILL.md index 438237a..be0d6ee 100644 --- a/.agents/skills/yatm/SKILL.md +++ b/.agents/skills/yatm/SKILL.md @@ -5,7 +5,7 @@ description: "Use yatm-cli to operate YATM: Library, live Location files, archiv # YATM -Use the `yatm-cli` executable through a shell. Use the matching installed CLI help and JSON output as the operational source of truth. A mutating command reports what it would change under `--dryrun` and writes when the flag is omitted, so resolve a large or destructive operation with a report before running it. Catalog, Job bundle and Library backup formats have independent identities and format versions. Upgrade v0.1.x installations through the documented offline migration workflow before opening their data with v1. +Use the `yatm-cli` executable through a shell. Use the matching installed CLI help and JSON output as the operational source of truth. A mutating command reports what it would change under `--dryrun` and writes when the flag is omitted, so resolve a large or destructive operation with a report before running it. Catalog, Job bundle and Library backup formats have independent identities and format versions. Upgrade v0.1.x installations through the installer and `docs/operations/migration.md` before opening their data with v1. Instant fields in CLI JSON use decimal strings with `_ns` suffixes: signed 64-bit Unix nanoseconds, not milliseconds or floating-point numbers. Keep them exact when processing or reusing output. Optional absent dates remain unknown; established zero sentinels remain zero. Restore `--before` accepts RFC3339 with an explicit timezone and preserves nanosecond precision; out-of-range dates are rejected. Durations and video offsets keep their named units. @@ -33,10 +33,12 @@ routes work; it does not duplicate the installed command inventory. | Work | CLI domain | Workflow guide | | --- | --- | --- | -| Connection and Settings | `status`, `settings` | `docs/operations/install.md` | -| Files and Locations | `ls`, `du`, `mv`, `rm`, `mkdir`, `files`, `identical`, `tag`, `location` | `docs/operations/locations.md` | -| Media | `media`, `volume`, `tape` | `docs/operations/library.md` | -| Jobs and workflows | `job`, `archive`, `restore`, `scan`, `analyze`, `preview`, `verify` | `docs/operations/library.md` | +| Connection and Settings | `status`, `settings` | `docs/operations/cli.md`, `docs/operations/install.md` | +| Files and Locations | `ls`, `du`, `mv`, `rm`, `mkdir`, `files`, `tag`, `location` | `docs/operations/locations.md` | +| Identical files | `identical` | `docs/operations/library.md` | +| Volumes | `media`, `volume` | `docs/operations/library.md` | +| Tape devices and Tape Media | `tape`, `media`, `settings devices` | `docs/operations/devices.md`, `docs/operations/read-order.md` | +| Jobs and workflows | `job`, `archive`, `restore`, `scan`, `analyze`, `preview`, `verify` | `docs/operations/jobs.md` | | Library metadata | `library` | `docs/operations/library.md` | ## Operate @@ -57,10 +59,10 @@ routes work; it does not duplicate the installed command inventory. - Legal UTF-8 filenames preserve literal backslashes, whitespace and control characters; quote operands for the shell. Non-UTF-8 names are unsupported. `ls` retains unreadable children as display-only rows with `error` and no actionable reference, drains all batches, then exits with the existing failure code and `code: incomplete`. Escaped unsupported names are display text, not paths. Directory and transport failures remain request failures. Never treat incomplete reads or failed Scan/Archive preparation as complete success. - `du` explicitly measures all matching entries and selected directory contents, independently of Search pagination. Use `--file-id` or `--location-id`/`--path`, with optional `--query`; `--recursive` searches Library roots recursively. Results stream as JSON Lines and end with a deduplicated summary. Unknown/unreadable ranges retain known bytes but return nonzero; do not interpret that subtotal as complete. Use a suitable global `--timeout` (`0` disables the deadline) for large trees. Measurement does not collect, hash or create a Job. - Before requesting Preview generation, check `preview capabilities` and the saved Preview settings. Generation requires an available native helper and explicit enablement; installing the optional helper does not enable it. Capability checks inspect the saved helper command; retry a failed read without resubmitting a Job. Known-only signature policy skips unknown content when Preview is requested; it can still read known content for decoding. Report-only Scan can generate Preview assets even though it does not publish catalog observations. Output options apply to new Jobs, while disabling generation lets running files finish and prevents later files from starting. Reading existing Preview assets remains available independently of generation enablement. -- `settings devices` reads the complete DeviceSettings group; `--device-json FILE` replaces user-owned Device settings from protobuf JSON without operating hardware; server-owned `learned_read_parameters` are retained and incoming client values for that field are ignored. Read the current group before replacing it. Write defaults attach to configured device names and apply to FORMAT. Tape `read_order` accepts `TAPE_READ_ORDER_LOGICAL` or `TAPE_READ_ORDER_AUTO`; omission/unspecified resolves to LOGICAL. The selected device's read order is frozen at each admission, independently of Job pipeline limits. Optional `capacityPollIntervalSeconds` defaults to Auto (5 seconds) for supported native counters; explicit `0` disables tar polling and custom whole intervals must be at least `5` seconds. This separate execution setting applies to both FORMAT and APPEND. Only one query is in flight; busy/unissued polls skip. Its private descriptor does not hold the primary WRITE lock, although shared drive firmware can still delay commands. Saving performs no device operation and changes neither an active attempt nor an existing Tape. Optional `passiveReadLearning` defaults On, records ordinary completed I/O observations in each Job bundle and refits learned read parameters after each Job from all existing Jobs with equal Job weight; deleting a Job removes its observations; manual read parameters take priority. `readPlanningSeconds` defaults to 300 and must be positive when supplied. Inspection may include `prepared_read` for the timestamped last prepared read; it does not identify the current cartridge or final read order. Device settings and learned read parameters are absent from Library metadata backups. -- Tape format selection belongs to an existing Archive Job's write operation. `archive write tape format` accepts `--format ltfs_v1|tar_pax_v1`; omission uses the selected device's default, falling back to LTFS. Tar accepts `--record-mode fixed|variable` and `--record-bytes BYTES` aligned to 512, with actual drive/transport validation before initialization. Omitted FORMAT record size uses the saved device size or 524288 bytes (512 KiB); APPEND and reads keep the recorded size. `--hardware-compression true|false` preserves omission and explicit On/Off; the leased attempt rechecks compression support, required decoding and cleanup. Encryption setup for tar and LTFS belongs to the configured `scripts.encrypt` adapter; there are no tar encryption/algorithm write flags. Supply the existing saved Media key through the operation's temporary 0600 file. Optional observed algorithms and key counters are diagnostics, not key presence or native setup authority. `archive write tape append` follows the registered immutable profile; overrides must agree. Tar validates its label and the real terminating filemark/EOD before writing. It checks only the tail with SPACE to EOD and long READ POSITION/LOCATE; missing long positioning, a rejected command, or a contradictory or uncertain tail refuses writing without a sequential scan. CM and cached locators do not authorize APPEND. LTFS rejects tar options. +- `settings devices` reads the complete DeviceSettings group; `--device-json FILE` replaces the user-owned Device settings without operating hardware, and server-owned `learned_read_parameters` in the input are ignored. Read the current group before replacing it and keep every entry you still want. Saving changes neither an active attempt nor an existing Tape. `docs/operations/devices.md` and `docs/operations/read-order.md` explain each field and its default. +- Tape format selection belongs to an existing Archive Job's write operation. `archive write tape format` takes `--format ltfs_v1|tar_pax_v1` and, for tar, `--record-mode`, `--record-bytes` and `--hardware-compression`; omitted values use the device defaults described in `docs/operations/devices.md`. `archive write tape append` follows the registered profile, and explicit options must agree with it. There are no encryption flags: the configured `scripts.encrypt` adapter owns encryption. Tar refuses APPEND when its label or recorded tail cannot be verified; report that refusal instead of retrying. - `restore run tape`, Tape `verify run` and `scan run` identify Tape automatically; `--media-id ID` is a manual hint restricted to the Job's frozen candidates. Tar label/initialization/profile verification remains mandatory. All-LTFS candidates, including migrated ltfs_v0, retain their configured adapter identification path. Ordinary LTFS reads do not require SG support; Auto on ltfs_v1 may obtain optional raw-device facts before mounting. LOGICAL preserves file order and permits eligible exact seek. AUTO plans the complete loaded-Media selection before reading, using one directed file/group reinsertion optimizer and complete stateful costs, including normal completion. Missing applicable geometry or timing retains LOGICAL with its reason in `job log-lines`. LTFS uses its extents and ordinary native ACP paths; unknown initial access retains one common first file. Its completion estimate assumes no Index rewrite and includes rewind/unload from the final position. This calculation changes neither mount scripts nor write protection. Read preparation prefers REOWP, then the supported HP LTO-5 CM directory, then a matching saved Media position map; CM is queried once before LTFS mounts and never per file. Missing LTFS extents, including migrated ltfs_v0 copies without historical XML, can fall back to the current mounted Index matched by path and size; existing extents and saved Job/Library records remain unchanged. Unusable geometry or Index data retains ordinary reads. REOWP/CM command support alone does not qualify the model; see the Media architecture owner for eligibility. -- `--refresh-position-map` on `restore run tape` or Tape `verify run`/`scan run` explicitly requests preparation for tar Restore or a frozen Scan VERIFY_COPIES policy. It defaults false and is not a Device preference. The server requires applicable non-ejecting HOLD/LOAD semantics, positively supported exact forms and current ready map/navigation observations; command spelling and physical validation alone do not establish eligibility. It performs one HOLD, confirms the held state, loads once and recaptures the map, then rechecks identity before the stored read controls/key and label. Active/foreign encryption blocks motion. Physical freshness stays unknown; unvalidated physical evidence does not prevent eligible explicit use, while failed validation blocks it. LTFS, Volume, inventory and other Media Scan policies reject it. Omit it for ordinary reads; AUTO never reloads automatically. After failure/cancellation, inspect any held cartridge and arrange explicit loading if needed; never automatically repeat the cycle or add recovery motion. +- `--refresh-position-map` on `restore run tape` or Tape `verify run`/`scan run` is an explicit request for tar Restore or a frozen Scan VERIFY_COPIES policy; it unloads without ejecting, reloads once and recaptures the position map. Use it only when the user asks. If it fails or is cancelled, inspect the device and report whether the cartridge needs an explicit load; never repeat the cycle automatically. - `settings job --execution-json` holds the shared Job pipeline limits that new attempts run with: `read_batch` is the manifest page, and `read_buffer_max`, `write_buffer_max`, `write_batch_size` and `flush_interval_ms` are ACP's read buffer, result queue, result batch and result flush interval; the runner's result writer only persists the batches ACP hands over. A write batch must be at least 1 and no larger than the write buffer. Change these only when a large Job needs different batching, and read the stored group before assuming how a Job persists results. - `mv`, `mkdir`, and `rm` share Library/Location organization rules and finish within the request, without creating a Job. Resolve the source and destination before mutation. Run them with `--dryrun` first when the selection is large or destructive: the report resolves the same targets and changes nothing, its entries stay `UNPROCESSED`, and a completed plan exits successfully. Planning failures, failed items and pending Library publication still exit nonzero. Results are JSON Lines with an item result and final summary; partial failure or interruption returns nonzero. Use a suitable request timeout; disconnecting cancels pending work. Inspect results and current paths before retrying, never blindly repeat completed mutations. - `rm` moves Library entries to logical Trash or Location entries to the owned `.trash` directory. It does not permanently delete content. Location removal detaches the original association while retaining Library organization and archived versions. Use `mv` to move content out; reassociation requires an explicit Scan, annotation, or Archive. Trash occupies disk space and cannot be selected for Scan, Archive, or a Restore destination. diff --git a/AGENTS.md b/AGENTS.md index 66a4ec2..c0cbc61 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,17 +11,17 @@ - Follow the supported operating model in the [architecture overview](docs/architecture/overview.md#supported-operating-model). Defensive behavior for an unsupported situation is not a user-visible capability: remove its code, tests and lower-level documentation when the supporting requirement is absent. - Write source, comments, documentation, changelogs, commit messages, and release notes in English. - Keep this public repository self-contained; do not reference or depend on private source repositories. -- Preserve configured Tape scripts wherever possible. They are user-owned adapters for LTFS distributions and hardware; keep their configuration, helpers and execution context. A necessary incompatible change needs explicit developer authorization, an actionable installer upgrade notice and automated checks of the affected boundary. Preserve the full [Tape lifecycle](docs/architecture/media-io.md#tape-lifecycle), including initialization of a new cartridge with a user-supplied barcode. The pre-stable API/data policy does not waive these requirements; the [Tape script contract](docs/architecture/media-io.md#tape-script-compatibility) owns compatibility. +- Preserve configured Tape scripts wherever possible. They are user-owned adapters for LTFS distributions and hardware; keep their configuration, helpers and execution context. A necessary incompatible change needs explicit developer authorization, an actionable installer upgrade notice and automated checks of the affected boundary. Preserve the full [Tape lifecycle](docs/architecture/tape.md#tape-lifecycle), including initialization of a new cartridge with a user-supplied barcode. The pre-stable API/data policy does not waive these requirements; the [Tape script contract](docs/architecture/tape.md#tape-script-compatibility) owns compatibility. ## Required Reading -Read [CONTEXT](CONTEXT.md), the [documentation convention](docs/README.md), and the current architecture document that owns the area you are changing before you change it. Record confirmed terminology, constraints and behavior in that owning document during the task, not only in the final response; the convention owns the document workflow and the delivery checks. +Read [CONTEXT](CONTEXT.md), the [documentation rules](docs/development/documentation.md), and the current architecture document that owns the area you are changing before you change it. Record confirmed terminology, constraints and behavior in that owning document during the task, not only in the final response; the convention owns the document workflow and the delivery checks. -Architecture and operations documentation describe current behavior. Public research belongs in `docs/research/` under the [documentation convention](docs/README.md): distinguish sourced findings, community observations, hypotheses and unresolved questions, and identify the versions studied. Research does not authorize implementation or define a product guarantee. Keep concise confirmed pending work in [TODO](TODO.md) and detailed implementation plans in an Issue or pull request; after implementation, remove the TODO item, update the current owner and delete replaced text instead of adding history, superseded-design or redirect documents. Published change notes belong to GitHub Releases, and raw verification evidence belongs to CI artifacts. Repository documentation must be self-contained: repository-relative references resolve to files committed in the same repository, while external references use stable public URLs. +Architecture and operations documentation describe current behavior. Public research belongs in `docs/research/` under the [documentation rules](docs/development/documentation.md): distinguish sourced findings, community observations, hypotheses and unresolved questions, and identify the versions studied. Research does not authorize implementation or define a product guarantee. Keep concise confirmed pending work in [TODO](TODO.md) and detailed implementation plans in an Issue or pull request; after implementation, remove the TODO item, update the current owner and delete replaced text instead of adding history, superseded-design or redirect documents. Published change notes belong to GitHub Releases, and raw verification evidence belongs to CI artifacts. Repository documentation must be self-contained: repository-relative references resolve to files committed in the same repository, while external references use stable public URLs. ## Repository Layout -Choose a file's owner before adding it. Use the following locations; the [development guide](docs/operations/testing.md#repository-layout-and-entrypoints) owns commands and output overrides. +Choose a file's owner before adding it. Use the following locations; the [development guide](docs/development/testing.md#repository-layout-and-entrypoints) owns commands and output overrides. | Location | Owns | | --- | --- | @@ -55,22 +55,19 @@ Follow the document that owns each area. The [architecture overview](docs/archit | APIs, services, generated code | [APIs and services](docs/architecture/api.md), [Interface contracts](docs/architecture/contracts.md) | | Jobs, Scan phases, publication | [Jobs and execution](docs/architecture/jobs.md), [Preview storage](docs/architecture/preview.md) | | File operations and Location behavior | [Library](docs/architecture/library.md), [APIs and services](docs/architecture/api.md) | -| Media, Tape and physical I/O safety | [Media backends and file I/O](docs/architecture/media-io.md) | -| Frontend and UI design | [User interface](docs/architecture/ui.md), [Preview storage](docs/architecture/preview.md) | +| Media, Tape and physical I/O safety | [Media backends and file I/O](docs/architecture/media-io.md), [Tape](docs/architecture/tape.md), [Tape read planning](docs/architecture/read-planning.md) | +| Frontend and UI design | [User interface](docs/architecture/ui.md), [frontend conventions](frontend/README.md) | ## Verification -Use existing verification entrypoints first and maintain recurring checks with the implementation under the [test maintenance policy](docs/operations/testing.md#test-maintenance). Choose affected checks during development/review and retain unaffected evidence under [Check Scope and Reuse](docs/operations/testing.md#check-scope-and-reuse); reserve full verification for final integration/release. Performance-sensitive changes follow [Performance Acceptance](docs/operations/testing.md#performance-acceptance). - -Before remote E2E acceptance or NAS delivery, check ignored top-level workspace directories for an `AGENTS.md` and read any applicable operator notes. Such notes provide host context only and do not authorize production actions. +Use existing verification entrypoints first and maintain recurring checks with the implementation under the [test maintenance policy](docs/development/testing.md#test-maintenance). Choose affected checks during development/review under [Check Scope and Reuse](docs/development/testing.md#check-scope-and-reuse) and retain unaffected evidence under [Evidence Reuse](docs/release/release.md#evidence-reuse); reserve full verification for final integration/release. Performance-sensitive changes follow [Performance Acceptance](docs/development/performance.md#performance-acceptance). ## Delivery and Authorization - Obtain explicit developer authorization before changing API contracts, entity definitions or database structure/storage semantics. This includes additions, removals, renames, field types, defaults, observable behavior, relationships, tables, indexes, constraints, migrations and data-format changes. - Present the affected contracts/models, proposed changes, consumer impact and compatibility/migration implications for approval before implementation or regeneration. A general feature, refactor or performance request is not authorization for these changes; approval of a plan that explicitly specifies them is sufficient within that scope. -- Until the first stable v1 release, changes to data formats, APIs, entities, schemas, Job bundles and metadata backups between unpublished or pre-stable v1 builds require no compatibility layers or version/revision bumps. Preserve the supported `v0.1.x` migration/import path. Remove this temporary rule when preparing the first stable v1 release; the authorization requirement above still applies. +- Until the first stable v1 release, follow the [pre-stable compatibility policy](docs/README.md#pre-stable-compatibility): no compatibility layers or version bumps between pre-stable builds, while the supported `v0.1.x` migration/import path is preserved. The authorization requirement above still applies. - Read-only investigation and clearly labeled Draft proposals may precede approval. Database data modifications require authorization for the specific operation; ordinary writes within an explicitly requested business operation do not authorize schema or model changes. - Keep commits single-purpose and exclude temporary files, local configuration, backups, generated test data, and obsolete implementations. -- For every release request, execute the [Release SOP](docs/operations/testing.md#release-sop), including content/consumer/documentation review and `make release-check` before candidate assembly. Report required checks that remain unrun; source checks do not replace exact-package acceptance or publication approval. -- Update the bundled [CLI Skill](.agents/skills/yatm/SKILL.md) in the same commit as any CLI command, flag, output or safety rule it describes; the [CLI document](docs/architecture/cli.md) owns that rule. -- Keep `master` aligned with `origin/main` and perform v1 development on `v1`. Push ACP integration only to its development branch until review is complete. Do not push YATM, merge ACP main, tag, or publish a release without explicit confirmation. +- For every release request, execute the [Release SOP](docs/release/release.md#release-sop), including content/consumer/documentation review and `make release-check` before candidate assembly. Report required checks that remain unrun; source checks do not replace exact-package acceptance or publication approval. +- Update the bundled [CLI Skill](.agents/skills/yatm/SKILL.md) in the same commit as any CLI command, flag, output or safety rule it describes; the [CLI coverage and Skill rule](docs/development/testing.md#cli-coverage-and-skill) owns that rule. diff --git a/CONTEXT.md b/CONTEXT.md index ad80785..30b2971 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1,6 +1,6 @@ # YATM Domain -YATM organizes ongoing files independently of their originals and archival copies. This glossary defines the domain; the [Library contract](docs/architecture/library.md) owns implemented behavior. The [documentation index](docs/README.md) owns the published Alpha compatibility policy and the exception for unpublished Draft data. +YATM organizes ongoing files independently of their originals and archival copies. This glossary defines the domain; the [Library contract](docs/architecture/library.md) owns implemented behavior. The [documentation index](docs/README.md#pre-stable-compatibility) owns the pre-stable compatibility policy. ## Language diff --git a/README.md b/README.md index 5e9e262..cf09668 100644 --- a/README.md +++ b/README.md @@ -9,14 +9,21 @@ YATM is an open-source archive manager for LTFS tapes and already-mounted disk V ## Features - Archive and Restore on Tape, HDD, and HM-SMR Media, with streaming SHA-256 verification through [ACP](https://github.com/samuelncui/acp). -- LTFS storage-order reads, append to compatible Tapes, and hardware-encryption integration through configured device scripts. +- LTFS storage-order reads and sequential tar workflows with configurable records; + both use configured encryption scripts, with public-tool recovery instructions for tar. +- Auto read order for Tape Restore and Verify: it plans the whole selection to cut tape + positioning, and keeps recorded order when that is fastest. On the tested hardware it substantially + shortened Restores of files scattered across a tar cartridge; see + [Faster Restores with Auto Read Order](docs/operations/read-order.md#faster-restores-with-auto-read-order). - A logical Library with folders, Tags, notes, paginated boolean search, and [Chonky](https://github.com/TimboKZ/Chonky)-based dual-pane browsing. - Content-addressed image/video Previews independent of archive Media. - Mounted Volume content access, automatic validated Scan publication, and metadata-only Media deletion. - Live registered directories, optional basic collection, shared Library/Location organization and query filters, and selected Archive. -- Typed gRPC services and durable Archive, Restore and Scan Jobs. Archive and Restore retain completed work between explicit Media selections; recreating a Job opens its creation form for review and manual submission. Scan supports optional hashing, content lookup, previews, inventory publication and recorded-copy verification; ordinary file operations remain request-bound. +- Durable Archive, Restore and Scan Jobs, typed gRPC services and a scriptable CLI. Scan supports optional hashing, content lookup, previews, inventory publication and recorded-copy verification. -Tape workflows require compatible LTFS hardware/software; Volume-only workflows do not require a Tape drive. See [installation requirements](docs/operations/install.md#requirements). +LTFS uses compatible hardware/software and configured adapters. Raw tar Tape uses the eligible Linux +device path; physical qualification is reported separately from protocol support. Volume-only +workflows do not require a Tape drive. See [installation requirements](docs/operations/install.md#requirements). ## Get Started @@ -24,9 +31,9 @@ The default installer selects the stable release. Install or upgrade the Alpha u ```shell curl --fail --location --output install-release.sh \ - https://raw.githubusercontent.com/samuelncui/yatm/v1.0.0-alpha.2/install-release.sh -sudo bash install-release.sh --version v1.0.0-alpha.2 --check -sudo bash install-release.sh --version v1.0.0-alpha.2 + https://raw.githubusercontent.com/samuelncui/yatm/v1.0.0-alpha.3/install-release.sh +sudo bash install-release.sh --version v1.0.0-alpha.3 --check +sudo bash install-release.sh --version v1.0.0-alpha.3 ``` Run with permission to manage the installation and systemd service. See [Installation and Configuration](docs/operations/install.md) for prerequisites and fresh-install configuration. Existing `v0.1.x` installations follow the [upgrade guide](docs/operations/migration.md); review it before proceeding. The installer displays that guide from the verified release package and asks for approval before stopping the service. @@ -37,19 +44,18 @@ For an isolated local review environment from a source checkout, run: make demo ``` -See [Local Demo](docs/operations/testing.md#local-demo) for reset behavior and safety boundaries. The Demo uses disposable data and does not exercise a physical Tape drive. +See [Local Demo](docs/development/demo.md#local-demo) for reset behavior and safety boundaries. The Demo uses disposable data and does not exercise a physical Tape drive. The release includes `yatm-cli` and an optional [agent Skill](docs/operations/install.md#optional-agent-skill). Linux amd64/systemd is the primary installation target; other platform archives are experimental. ## Documentation -- [Documentation index](docs/README.md) -- [Library and Volume workflows](docs/operations/library.md) -- [Live Locations and Scan](docs/operations/locations.md) -- [Domain glossary](CONTEXT.md) and [current architecture](docs/architecture/overview.md) -- [Development layout, checks and delivery gates](docs/operations/testing.md) +- [User Guide](docs/operations/index.md): installation, configuration, daily workflows and operation + through the CLI or an agent. +- [Developer Documentation](docs/README.md): architecture, interfaces, research, tests and contribution + rules. -Repository documentation describes only the current implementation and operating procedures. Published change history belongs to [GitHub Releases](https://github.com/samuelncui/yatm/releases); proposals and implementation plans live in Issues or pull requests until their durable result is incorporated into the current documentation. +Architecture and user documentation describe the current implementation and operating procedures; research records evidence and open questions. Published change history belongs to [GitHub Releases](https://github.com/samuelncui/yatm/releases); proposals and implementation plans live in Issues or pull requests until their durable result is incorporated into the current documentation. ## Thanks diff --git a/TODO.md b/TODO.md index b148bd2..c15ad0a 100644 --- a/TODO.md +++ b/TODO.md @@ -7,5 +7,6 @@ implemented behavior or authorize API/data changes. - [ ] Add Archive selection guidance ([#20](https://github.com/samuelncui/yatm/issues/20)). Explain **Add to list** and dragging into the selection. - [ ] Research selecting newly added files for Archive ([#12](https://github.com/samuelncui/yatm/issues/12)). Define how this relates to changed content and existing archived coverage. A path/name match alone cannot establish that current content is already archived. Preserve Recreate's review and explicit submission flow. - [ ] Fix inconsistent loading behavior in Settings. Review initial loading, switching and saving so pending and error feedback stay coherent. -- [ ] Research a tar Tape backend for LTO-4 and older drives ([#14](https://github.com/samuelncui/yatm/issues/14)). Define Archive/Restore, inventory and Media identity requirements before choosing an implementation. +- [ ] Qualify the software position estimator, read model and cartridge-memory decoding beyond the studied HP LTO-5 configuration ([#14](https://github.com/samuelncui/yatm/issues/14)). +- [ ] Revisit LTFS passive read learning after resolving its measurement ownership. Deferred: preserve native ACP reads and keep LTFS observations separate from native tar measurements. - [ ] Research indexing CD/DVD/Blu-ray archives ([#16](https://github.com/samuelncui/yatm/issues/16)). Define read-only inventory and offline browsing requirements, and assess what existing Location/Scan behavior can provide. diff --git a/build/release/build-documents.test.mjs b/build/release/build-documents.test.mjs index 6b024c7..fdfa2d6 100644 --- a/build/release/build-documents.test.mjs +++ b/build/release/build-documents.test.mjs @@ -23,7 +23,7 @@ test("packaged user entry and Tape procedures stay local while developer referen const target = fs.mkdtempSync(path.join(os.tmpdir(), "yatm-user-guides-")); t.after(() => fs.rmSync(target, { recursive: true, force: true })); const source = path.resolve(import.meta.dirname, "../.."); - const guides = ["index", "install", "migration", "library", "locations", "devices", "tar-tape-recovery"]; + const guides = ["index", "install", "migration", "locations", "library", "jobs", "devices", "tape-scripts", "read-order", "cli", "tar-tape-recovery", "offline-migration"]; fs.mkdirSync(path.join(target, "docs/operations"), { recursive: true }); for (const guide of guides) { fs.copyFileSync(path.join(source, `docs/operations/${guide}.md`), path.join(target, `docs/operations/${guide}.md`)); @@ -45,8 +45,8 @@ test("packaged user entry and Tape procedures stay local while developer referen assert.match(documents.get("index"), /\]\(tar-tape-recovery\.md\)/); assert.match(documents.get("library"), /\]\(\.\.\/\.\.\/templates\/config\.example\.yaml\)/); assert.match(documents.get("migration"), /\]\(\.\.\/\.\.\/templates\/scripts\/mount\)/); - assert.match(documents.get("migration"), /\]\(\.\.\/\.\.\/templates\/scripts\/readinfo\)/); - assert.match(documents.get("migration"), /\]\(\.\.\/\.\.\/templates\/scripts\/umount\)/); + assert.match(documents.get("tape-scripts"), /\]\(\.\.\/\.\.\/templates\/scripts\/readinfo\)/); + assert.match(documents.get("tape-scripts"), /\]\(\.\.\/\.\.\/templates\/scripts\/umount\)/); assert.match(documents.get("devices"), /\]\(tar-tape-recovery\.md\)/); assert.match(documents.get("tar-tape-recovery"), /\]\(devices\.md\)/); assert.match(documents.get("index"), /\]\(https:\/\/github\.com\/samuelncui\/yatm\/blob\/v1\.0\.0-alpha\.1\/docs\/README\.md\)/); diff --git a/build/release/build.sh b/build/release/build.sh index ae11ab0..11479e2 100755 --- a/build/release/build.sh +++ b/build/release/build.sh @@ -42,7 +42,7 @@ for script in encrypt get_device mkfs mount readinfo umount README.md; do done cp cmd/httpd/yatm-httpd.service config.example.yaml "$OUTPUT_DIRECTORY/templates/" cp LICENSE README.md CONTEXT.md install-release.sh "$OUTPUT_DIRECTORY/" -for guide in index install migration library locations devices tar-tape-recovery; do +for guide in index install migration locations library jobs devices tape-scripts read-order cli tar-tape-recovery offline-migration; do cp "docs/operations/$guide.md" "$OUTPUT_DIRECTORY/docs/operations/" done cp .agents/skills/yatm/SKILL.md "$OUTPUT_DIRECTORY/skills/yatm/" diff --git a/build/release/check-release.mjs b/build/release/check-release.mjs index 30ecc5b..84b4dc3 100644 --- a/build/release/check-release.mjs +++ b/build/release/check-release.mjs @@ -14,7 +14,7 @@ const members = readArchive(archive); const { files } = members; const roots = ['yatm-httpd', 'yatm-cli', 'yatm-export-library', 'yatm-lto-info', 'yatm-migrate', 'install-release.sh', 'VERSION', 'COMMIT', 'LICENSE', 'README.md', 'CONTEXT.md']; -const documents = ['index', 'install', 'migration', 'library', 'locations', 'devices', 'tar-tape-recovery'] +const documents = ['index', 'install', 'migration', 'locations', 'library', 'jobs', 'devices', 'tape-scripts', 'read-order', 'cli', 'tar-tape-recovery', 'offline-migration'] .map(name => `docs/operations/${name}.md`); const scripts = ['encrypt', 'get_device', 'mkfs', 'mount', 'mount.openltfs', 'readinfo', 'umount'].map(name => `templates/scripts/${name}`); const testing = ['encrypt', 'get_device', 'mkfs', 'mount', 'readinfo', 'umount', 'README.md'].map(name => `templates/testing/ltfs-file-backend/${name}`); diff --git a/build/release/check-release.test.mjs b/build/release/check-release.test.mjs index 29775ac..eab669f 100644 --- a/build/release/check-release.test.mjs +++ b/build/release/check-release.test.mjs @@ -35,6 +35,11 @@ function fixture(t, change = () => {}) { ["docs/operations/migration.md", "# Upgrade from v0.1.x to v1\n"], ["docs/operations/library.md", "# Library\n"], ["docs/operations/locations.md", "# Locations\n"], + ["docs/operations/jobs.md", "# Archive, Restore and Scan\n"], + ["docs/operations/read-order.md", "# Faster Restores with Auto Read Order\n"], + ["docs/operations/cli.md", "# Command-Line Reference\n"], + ["docs/operations/offline-migration.md", "# Offline Migration and Repair\n"], + ["docs/operations/tape-scripts.md", "# Tape Scripts\n"], ["install-release.sh", "#!/usr/bin/env bash\nexit 0\n"], ["frontend/index.html", ``], ]); diff --git a/build/release/committed-inputs.test.mjs b/build/release/committed-inputs.test.mjs index 19f6766..406d04d 100644 --- a/build/release/committed-inputs.test.mjs +++ b/build/release/committed-inputs.test.mjs @@ -31,7 +31,7 @@ function fixture(t) { write(root, name, fs.readFileSync(path.join(repository, name)), fs.statSync(path.join(repository, name)).mode); } for (const name of ['LICENSE', 'README.md', 'CONTEXT.md', 'install-release.sh', 'config.example.yaml', 'cmd/httpd/yatm-httpd.service', - 'docs/README.md', 'docs/architecture/media-io.md', 'docs/research/tar-tape.md', ...['index', 'install', 'migration', 'library', 'locations', 'devices', 'tar-tape-recovery'].map(guide => `docs/operations/${guide}.md`), '.agents/skills/yatm/SKILL.md', + 'docs/README.md', 'docs/architecture/media-io.md', 'docs/research/tar-tape.md', ...['index', 'install', 'migration', 'locations', 'library', 'jobs', 'devices', 'tape-scripts', 'read-order', 'cli', 'tar-tape-recovery', 'offline-migration'].map(guide => `docs/operations/${guide}.md`), '.agents/skills/yatm/SKILL.md', ...['encrypt', 'get_device', 'mkfs', 'mount', 'mount.openltfs', 'readinfo', 'umount'].map(name => `scripts/${name}`), ...['encrypt', 'get_device', 'mkfs', 'mount', 'readinfo', 'umount', 'README.md'].map(name => `e2e/ltfs-file-backend/${name}`)]) write(root, name, 'committed fixture'); write(root, 'e2e/ltfs-file-backend/mount.test.mjs', '// maintained local regression, not a runtime adapter'); diff --git a/dev/check-documents.mjs b/dev/check-documents.mjs index 5cecc30..6ea3c55 100644 --- a/dev/check-documents.mjs +++ b/dev/check-documents.mjs @@ -13,7 +13,7 @@ const repositoryFiles = new Set(execFileSync( const markdown = [...repositoryFiles].filter((name) => name.endsWith(".md")).sort(); const documentation = markdown.filter((name) => name.startsWith("docs/")); -const allowedDocumentation = /^(?:docs\/README\.md|docs\/(?:architecture|operations|research)\/[a-z0-9-]+\.md)$/; +const allowedDocumentation = /^(?:docs\/README\.md|docs\/(?:architecture|development|operations|release|research)\/[a-z0-9-]+\.md)$/; const retiredDocumentationReference = /docs\/(?:history|designs|decisions|releases)\/|docs\/(?:demo|e2e-test|physical-tape-e2e|platform-changes|platform-design)\.md|docs\/operations\/(?:demo|preview-build)\.md|docs\/architecture\/(?:api-ui|states)\.md/; for (const document of documentation) { diff --git a/docs/README.md b/docs/README.md index 8cd3800..271d600 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,12 +1,20 @@ -# Documentation +# Developer Documentation + +This is the implementation and contributor entry point. For installation, configuration and daily +operation, start with the separate [User Guide](operations/index.md). Architecture and operations describe the current YATM v1 development line; research records -the evidence and open questions behind further work. Start with -[installation](operations/install.md), the [v0.1.x upgrade](operations/migration.md), -[Library and Media workflows](operations/library.md), or [live Locations](operations/locations.md). +the evidence and open questions behind further work. Published versions and their change notes belong to [GitHub Releases](https://github.com/samuelncui/yatm/releases); this tree is not a release archive. +Start with the [contributor rules](../AGENTS.md), the [domain vocabulary](../CONTEXT.md) and the +[documentation rules](development/documentation.md). Component guides live beside their code: +[frontend](../frontend/README.md), [Preview helper](../previewworker/README.md) and its +[build recipes](../previewworker/build/README.md), the +[LTFS file backend adapters](../e2e/ltfs-file-backend/README.md) and the +[cartridge memory fixtures](../internal/media/tape/testdata/README.md). + ## Pre-stable Compatibility The required compatibility baseline is the supported `v0.1.x` migration/import path. Until the first @@ -18,7 +26,7 @@ identities and versions independently of software release numbers. Configured Tape scripts have a separate, long-term compatibility requirement. Upgrades preserve user adapters wherever possible; a necessary incompatible change requires explicit approval and an actionable notice before upgrade, including pre-stable releases. The -[Tape script contract](architecture/media-io.md#tape-script-compatibility) owns that boundary. +[Tape script contract](architecture/tape.md#tape-script-compatibility) owns that boundary. Known incompatible installations are rejected before writes. Their data is retained until the operator chooses backup, migration, conversion or reinstallation; YATM never clears databases or physical @@ -33,106 +41,53 @@ published stable upgrade policy. - [Library identities, organization and backups](architecture/library.md) - [Jobs and execution](architecture/jobs.md) - [Media backends and file I/O](architecture/media-io.md) +- [Tape devices, lifecycle and formats](architecture/tape.md) +- [Tape read planning](architecture/read-planning.md) - [Preview storage](architecture/preview.md) - [Interface contracts](architecture/contracts.md) - [APIs and services](architecture/api.md) - [User interface](architecture/ui.md) -- [Command-line interface](architecture/cli.md) -## Operations +## User Documentation +- [User Guide entry](operations/index.md) - [Installation and service configuration](operations/install.md) -- [Library and mounted Volume workflows](operations/library.md) +- [Upgrade from v0.1.x](operations/migration.md) +- [Offline migration and repair](operations/offline-migration.md) - [Locations](operations/locations.md) -- [v0.1.x to v1 migration](operations/migration.md) -- [Development checks and test environments](operations/testing.md) -- [Automated E2E](operations/e2e-test.md) -- [Physical Tape E2E](operations/physical-tape-e2e.md) +- [Library and mounted Volumes](operations/library.md) +- [Archive, Restore and Scan](operations/jobs.md) +- [Devices and Tape operations](operations/devices.md) +- [Tape scripts](operations/tape-scripts.md) +- [Faster Restores with Auto read order](operations/read-order.md) +- [Command-line reference](operations/cli.md) +- [Independent tar Tape recovery](operations/tar-tape-recovery.md) -## Release Packages +## Contributing -The release package includes this index and the four operator guides: installation, migration, -Library/Media workflows and Locations. Architecture, research and developer validation remain repository -documentation; release packaging uses an explicit allowlist rather than copying the whole tree. +- [Documentation rules](development/documentation.md) +- [Development checks and test environments](development/testing.md) +- [Code layout and test ownership](development/code-layout.md) +- [Automated E2E](development/e2e-test.md) +- [Performance acceptance](development/performance.md) +- [Local Demo](development/demo.md) + +## Release + +- [Release process](release/release.md) +- [Package acceptance](release/package-acceptance.md) +- [Physical Tape E2E](release/physical-tape-e2e.md) +- [Physical Tape host runbook](release/physical-tape-runbook.md) +- [Physical tar acceptance](release/physical-tar-acceptance.md) +- [Physical Tape measurements](release/physical-measurements.md) ## Research - [Tape capacity and safe completion](research/tape-capacity.md) +- [Tar on Tape](research/tar-tape.md) +- [Tape read ordering](research/tape-read-ordering.md) Research is published with the repository. Each study states its objective, source versions, confirmed findings, community observations, unresolved questions and criteria for evaluating possible approaches. A study describes the current state of knowledge; it does not establish implemented behavior or authorize experiments and production changes. - -## Documentation Ownership - -Every maintained fact has one owner. Other documents link to that owner instead of restating it. - -| Location | Owns | Does not own | -| --- | --- | --- | -| Root `README.md` | Released product introduction, quick start and documentation entry points | Development status, architecture detail or repeated runbooks | -| Root `CONTEXT.md` | Domain vocabulary and core domain invariants | Schemas, algorithms, task notes or UI layout | -| Root `AGENTS.md` | Contributor rules, safety gates, repository layout and required verification | Product or architecture contracts | -| Root `TODO.md` | Concise confirmed pending work with source Issue links | Detailed designs, implemented behavior or completed-work history | -| `docs/architecture/` | Current implemented behavior, module interfaces, invariants and durable constraints | Proposals, implementation plans, acceptance transcripts or superseded behavior | -| `docs/operations/` | Current setup, migration, use, maintenance and reproducible validation | Domain definitions or another architecture specification | -| `docs/research/` | Public technical studies, evidence, limitations and evaluation of possible approaches | Product guarantees, implementation backlogs, private incident reports or raw acceptance logs | -| `.agents/skills/yatm/` | Installed agent operating and safety instructions | Human-facing architecture or change history | - -## Decision Authority - -Within overlapping scope, broader accepted constraints bound narrower descriptions: the repository -engineering rules, the domain invariants and compatibility policy, the architecture overview, area -architecture, operations guides, and finally implementation and tests. A lower level may provide -evidence and implementation detail, but its existence cannot justify complexity or override a broader -decision. When they conflict, update the lower-level documents, code and tests together. - -The repository intentionally has no documentation archive, ADR archive or redirect stubs. -Git already retains deleted text and rationale. Root `TODO.md` records concise confirmed pending -work; detailed implementation plans belong in an Issue or pull request. Remove completed TODO -items instead of retaining a work history. Public research follows the scope above. -Published change notes belong to GitHub Releases. Raw acceptance logs, checksums, -measurements, screenshots and host-specific reports belong in CI artifacts rather than Markdown. - -## Update Workflow - -1. Establish the intended behavior from the decision authority above, then inspect `CONTEXT.md`, the - owning document and the implementation. A mismatch is something to resolve, not a reason to - promote the current implementation into a requirement. -2. Update the implementation, semantic tests, owning architecture document and affected operating - guide together. Describe the final behavior in present tense; remove the replaced text instead of - marking it historical, implemented or superseded. -3. Keep unsettled implementation designs in a PR or Issue. Public studies may evaluate alternatives - in `docs/research/`, with evidence, assumptions and remaining proof clearly identified. After - implementation, update the current architecture and link to research for supporting evidence. - Update or remove invalidated research conclusions instead of accumulating superseded studies. -4. Keep implementation detail only when callers or maintainers must know it to use the module - correctly. Function names, CSS placement, one measurement, one candidate checksum and step-by-step - development history belong in code, tests or verification output. -5. Keep repository documentation self-contained. Every repository-relative link or file reference - must resolve to a tracked file, or to a new file committed in the same change; external references - use stable public URLs. Private notes, ignored artifacts and machine-local paths are never sources - for repository documentation. -6. Before delivery, run `make check` or at minimum `node dev/check-documents.mjs` and - `git diff --check`. The document check enforces the allowed tree, complete index, valid relative - links and the absence of private/history references. - -For research, use official specifications and source code together with relevant original community -reports and maintainer responses. Record what was observed, what was actually resolved and which -explanations remain hypotheses; a forum's resolved label alone is not evidence of a verified fix. -Use stable public citations and concise summaries. Keep credentials, local paths, device identifiers, -private host details and raw logs out of public studies; include only the technical context needed -to assess the evidence. Hardware measurements establish the tested configuration's behavior, while -general guarantees require an applicable specification or other evidence with a defined scope. - -Use English Markdown, short topic-based kebab-case filenames and repository-relative links. Current -architecture filenames are version-independent. Adding a new documentation file requires a distinct -current owner that cannot be expressed clearly in an existing file; update this index in the same -change. - -## UI Prototypes - -Build UI prototypes with the existing React modules, styling and layout, and capture their rendered -pages for review. Store temporary sources and screenshots under ignored `frontend/.drafts/`, then -remove them after the accepted behavior and constraints have been recorded in the owning current -architecture document. Generated concept images are not interface specifications. diff --git a/docs/architecture/api.md b/docs/architecture/api.md index 6e693b3..d79b1f6 100644 --- a/docs/architecture/api.md +++ b/docs/architecture/api.md @@ -4,7 +4,7 @@ Status: Current v1 development architecture. The published programmatic surface comprises gRPC services, HTTP resource endpoints, and their contracts. [User interface](ui.md) owns the browser behavior built on them, -the [command-line interface](cli.md) owns the scripted surface that calls the same +the [command-line interface](../operations/cli.md) owns the scripted surface that calls the same methods, and [Interface contracts](contracts.md) owns the rules shared across surfaces. | Service | Responsibility | @@ -18,17 +18,20 @@ methods, and [Interface contracts](contracts.md) owns the rules shared across su | [FilesService](../../entity/files.proto) | Projected List, explicit Measure, basic Get, annotations, Mkdir/Move/Remove, versions, copies, single-signature duplicates, retained identical Find results with bounded rows, [Library-defined sorting](library.md#identical-file-tools) and group/member pages, Keep/Merge, inventory admission and original relocation | | [ScanJobService](../../entity/job_scan.proto) | Configurable source/content/result policies, GetCreation, ReadMedia, progress and paged entries | | [LocationService](../../entity/location.proto) | Register/list/get/update/delete, access summary and administrator discovery of unregistered paths | -| [SettingsService](../../entity/settings.proto) | Typed Get/Update for Library visibility/removal confirmation, Preview generation and Job pipeline limits | +| [SettingsService](../../entity/settings.proto) | Typed Get/Update for Library visibility/removal confirmation, Preview generation, Job pipeline limits and Device settings | -Archive/Restore use typed Tape/Volume operation targets. A kind's service is registered with its runner; Job listings include kind and optional target identity/name for routing and filtered history. New input browsing reads registered Location directories or administrator-authorized paths. Archive accepts Library/Location `selections`; Scan accepts those `selections` or a whole Media ID. A Scan whose selections all name one Location reports it as the primary target, including partial ranges; mixed Library/Location or multiple-Location selections have no single primary target. Job properties still index every Location actually involved. +Archive/Restore use typed Tape/Volume operation targets. Archive Tape FORMAT carries optional +format/write choices; APPEND uses the registered immutable profile. Read Tape targets carry an +optional Media hint. Restore resolves it against pending frozen candidates; Scan requires its +configured Media. Their existing WriteMedia, RestoreMedia and ReadMedia envelopes remain the +operation boundary, with no separate Media-selection RPC. [Interface contracts](contracts.md#tape-options-and-selection) +owns option resolution and [Jobs](jobs.md) owns attempt admission/publication. + +A kind's service is registered with its runner; Job listings include kind and optional target identity/name for routing and filtered history. New input browsing reads registered Location directories or administrator-authorized paths. Archive accepts Library/Location `selections`; Scan accepts those `selections` or a whole Media ID. A Scan whose selections all name one Location reports it as the primary target, including partial ranges; mixed Library/Location or multiple-Location selections have no single primary target. Job properties still index every Location actually involved. JobService exposes no Job retry/restart operation. Job listings filter by kind, Location and Media; -`JobFilter` has no status filter. Typed Media operations require an idle `READY` Job and follow the -[shared lifecycle](jobs.md#shared-lifecycle): pre-admission failures leave state unchanged, -admitted Scan/Index errors or operator cancellation settle as terminal `FAILED`, and only admitted -Archive/Restore Media errors or operator cancellation, including while `QUEUED`, return to pre-Media -`READY` with the prepared manifest, successful per-file/Media checkpoints and reason/timing. -Failed or cancelled Scans require a new Scan; process restart never restarts a Job or revives terminal work. +`JobFilter` has no status filter. Typed Media operations require an idle `READY` Job; the +[shared lifecycle](jobs.md#shared-lifecycle) owns their admission and settlement. Each kind's `GetCreation` reads the existing Job configuration and priority under the Job deletion guard, through a read-only bundle connection. It returns that kind's typed Create request and an @@ -41,6 +44,14 @@ Registered directory selectors, including Restore targets, use complete `Files.L [HTTP resource endpoints](../../internal/apis/files.go) serve Library JSONL metadata export/import and [Preview assets](../../internal/apis/preview_service.go). They do not serve full original or archived file bytes; obtaining saved content is a Restore workflow. There is no file or Preview Download CLI command. Loopback-only upgrade endpoints report running operations and quiesce admission for preflight. +Library export streams through HTTP/1.0, HTTP/1.1 and HTTP/2 without staging a complete snapshot. +A failure before response commitment returns an error without attachment headers; a later failure +aborts the response without appending JSON. The single HTTP recovery boundary preserves that abort. +HTTP/1.1 framing and HTTP/2 stream resets expose interrupted transfers. Close-delimited HTTP/1.0 +cannot distinguish interruption from normal EOF; download success alone does not validate a backup. +The existing JSONL `end` record identifies completed snapshot records, and import rejects a missing +end record while rolling back all changes. The CLI checks transport errors, not JSONL completeness. + `JobService.GetLog` retains its raw forward byte read for CLI and evidence consumers. `JobService.ListLogLines` serves the browser's bidirectional log view: no cursor starts at the tail for older reads or the beginning for newer reads; returned byte cursors name the @@ -55,6 +66,68 @@ the caller continues until the corresponding `has_older` or `has_newer` flag is Files.Get supplies organization, original navigation and applicable content references for explicit workflows without opening file bytes in the browser. User Ignore hides an entry from browsing, measurement and Location selection but does not prevent explicit admission. Physical mutations accept typed relative references, never shell commands or unrestricted absolute paths. +## Device Discovery + +MediaService.ListDevices returns all configured Tape entries as DeviceSummary values with the +configured device spelling, kind, busy status and detail. It reports configuration and lease state +without issuing SCSI commands. Busy entries remain visible; Job resource admission retains its +existing queued behavior. + +MediaService.InspectDevice accepts one configured device and returns transient passive observations. +Busy devices return a typed reason without hardware queries. +The optional `prepared_read` is separately timestamped last-prepared-read information from this +process, including Job/Media, requested/prepared order, parameter and position sources, applicability, +coverage and the selected learned read parameters' mean and maximum error. It can accompany a busy response; it never identifies the +currently loaded cartridge or claims the final execution route. It adds no hardware query. Optional identity/mode/limit facts +remain absent when unknown; feature support, current availability and validation are separate. +Capacity limits report optional native bytes before EW, partition, observation timestamps and error/ +unreflected-write allowances, with estimate/lower-bound quality. Existing feature fields distinguish +support, availability and measured validation. +They are transient diagnostics, not a per-file fit guarantee or persisted FORMAT preference. +Compression/decompression limits expose an optional current numeric algorithm and enable-bit +changeability. Current-enabled observations remain in TapeDeviceReport. Missing changeability +stays unknown, and an algorithm observation is not a supported-algorithm list or physical validation. +Encryption limits expose public algorithm names, device-local indexes and key sizes, plus optional +current mode, scope, parameter-control and key-instance observations. Indexes are resolved on each +reader and are not FORMAT preferences. Encryption setup belongs to the configured Tape script; +these observations do not add native key management. A key counter is not evidence that a key is present or correct; +neither keys nor key-associated data enter inspection responses. +Geometry limits expose the reported all-wrap command form and source, with count only after an +interpretable response and optional freshness. Command-form support is not proof of full coverage; +passive queries leave freshness unknown. Software ordering and non-ejecting reload retain separate +qualification and availability. +Permission, missing tools, no medium, clean unsupported forms, malformed responses and unknown +ownership have distinct reasons. No capability report is saved in the Catalog. The +[Tape device contract](tape.md#device-observation-and-raw-transport) defines physical access and cleanup. +InspectDevice can include supported baseline tar write recommendations; absent recommendations +remain unknown and never certify physical writes. The existing MediaService.Inspect retains its +cartridge-oriented LTFS/Volume behavior. Tar label validation occurs under the operation lease. +Settings Get/Update stores typed Device settings without issuing hardware commands or granting +access to an unconfigured device. Media List exposes tar profiles without the saved key; Media responses +report its presence only through `encryption_key_saved`; existing ScanEntry storage and +Library JSONL backups carry the locators. There is no separate recovery export. +Device Settings also carries the optional tar capacity-polling interval. The admitted Device +snapshot owns its effective value for both FORMAT and APPEND; the [tar capacity admission contract](tape.md#tar-capacity-admission) +owns its defaults, query-failure handling and confirmed-WRITE accounting. +The optional `TapeDeviceSettings.read_parameters` uses this same replacement and snapshot path. +It contains measured applicability conditions, nominal geometry, seven positioning terms, +READ-rate/overhead estimates, separate normal-unload `completion` timings and descriptive provenance. +Completion has optional `full_span_seconds` and `fixed_seconds`; supplied +full-span time must be positive and fixed time nonnegative. Supplied values are type/range checked; +partial measurements retain absent fields, separately from explicit zero, false and plaintext. +The longitudinal coefficient uses either seconds per absolute LPOS unit or seconds per complete +longitudinal span (`longitudinal_seconds_per_wrap`), mutually exclusive by field presence. +Explicit zero positioning terms are valid; omitted measurements stay unknown. Saving input does not establish its +applicability or accuracy; the prepared tar factory checks actual conditions and map coverage. +There is no measurement RPC, registry, cartridge-position table or new tar metadata. +The same Device group carries optional `passive_read_learning` (omitted means enabled) and +`read_planning_seconds` (omitted means 300; explicit values must be positive). Read-only +`learned_read_parameters` are server-owned: Settings Update replaces user-owned Device settings while retaining +the latest learned read parameters, including when a client submits an older snapshot. The observations behind +them have no API. Media responses and backups +include the optional derived `tape_position_map`; the [persistence owner](persistence.md#derived-tape-data) +defines its initialization binding and separate update boundary. + ## Browser Endpoint Addresses `listen` selects the bound service address. `domain` independently selects the browser's @@ -64,10 +137,9 @@ without a trailing slash. An empty or omitted value yields `/services` and `/fil the page's current origin. Preview resource URLs are always root-relative `/files/preview` URLs; CLI consumers select their own server address. -The service does not infer this address from Host or forwarded headers, enforce a canonical -hostname, or terminate TLS based on `domain`. Deployment is at the origin root: frontend -assets, navigation and resource URLs do not support an arbitrary path prefix. The -[reverse-proxy guide](../operations/install.md#reverse-proxy) shows the existing configuration. +The service does not infer this address from Host or forwarded headers. The +[reverse-proxy guide](../operations/install.md#reverse-proxy) owns operator configuration and +deployment limits. ## Listing projection diff --git a/docs/architecture/contracts.md b/docs/architecture/contracts.md index 9f8c4ce..be88da7 100644 --- a/docs/architecture/contracts.md +++ b/docs/architecture/contracts.md @@ -3,7 +3,7 @@ Status: Current v1 development architecture. These rules bind every published surface together: the [gRPC and HTTP API](api.md), the -[user interface](ui.md) and the [command-line interface](cli.md). A surface document +[user interface](ui.md) and the [command-line interface](../operations/cli.md). A surface document describes what its surface publishes; a rule that governs more than one surface belongs here, so one decision has one owner and no surface restates it. @@ -67,7 +67,44 @@ One word per concept on every surface, in the [domain vocabulary](../../CONTEXT. A Job publishes one durable state and, while an attempt is active, one live phase. `QUEUED` identifies an admitted resource wait; an idle Job reports no phase even when its runner is cached. A surface reads the durable state for the retained result and reads the live phase to identify admitted execution, including resource queueing. Media actions require an idle `READY` Job; no surface derives permission from counters, a retained manifest or a phase name. -No surface offers Job-level retry or restart. Failed or cancelled Scan execution and manifest preparation require a new Job. Only admitted Archive/Restore Media errors or operator cancellation, including while `QUEUED`, return the Job to its pre-Media `READY` state for another explicit load-Media operation, preserving successful per-file/Media checkpoints and the reason/timing. Rejections before attempt admission leave state unchanged, and process restart never restarts a Job or revives terminal `FAILED` work. The [Job lifecycle](jobs.md#shared-lifecycle) owns settlement and retained facts; transport and page-read Retry actions repeat only their reads. +No surface offers Job-level retry or restart; the [Job lifecycle](jobs.md#shared-lifecycle) owns which failures return a Job to `READY` and which are terminal. Transport and page-read Retry actions repeat only their reads. + +## Tape options and selection + +FORMAT resolves explicit fields before saved Device defaults. Unset tar record policy defaults to +variable records with a 512 KiB (524288-byte) maximum; validated recommendations can supply other +still-unset write fields. Native record and transport limits remain mandatory: an incompatible +choice fails preparation rather than silently changing the mode or size. +With no explicit or saved format, FORMAT defaults to LTFS; a recommendation never chooses the format +or overrides the record policy. Optional false remains distinct from omission. Unsupported record +choices and controls are field errors. Tar FORMAT can select hardware compression +On or Off; APPEND enforces the immutable cartridge choice after validating its label and physical +tail. Read preparation preserves write compression controls and independently requires hardware +decoding for a compressed profile. The existing `Scripts.Encrypt` adapter owns encryption setup +for tar and LTFS. FORMAT generates the existing key representation; APPEND/read uses the saved +Media key through a temporary 0600 file. Native encryption modes and any observed public algorithm +are diagnostics, not new write options or key-presence evidence. No extra persistent key copy is made. + +The UI submits its resolved explicit choices. CLI/API omissions use the same server resolution. +Job admission snapshots Device Settings before asynchronous work or a device wait; subsequent +saves affect later FORMAT and read attempts. Resolved non-secret choices are visible in diagnostics. +APPEND and reads use the immutable cartridge profile, with conflicting APPEND options rejected. +Saving Device Settings validates all entries structurally, including inactive entries, without +opening hardware; deployment configuration alone authorizes devices. Tape read order defaults to +LOGICAL. AUTO requests a bounded permutation only with an eligible provider; unknown geometry, +freshness or costs preserve the complete logical selection. Eligible LTFS uses the same planner +and retains native file paths; optional raw observations finish before its mount. + +Position-map refresh is an explicit read-preparation action for tar Restore and VERIFY_COPIES. +Known LTFS, Volume and other Scan policies reject it before physical preparation. +A missing qualified non-ejecting reload/freshness contract rejects the action before motion; AUTO +never implicitly requests it. The [position-map refresh contract](read-planning.md#position-map-refresh) owns provider +eligibility and source-boundary settlement. + +A read Media hint selects existing frozen work; it cannot supply expected identity/profile. +Restore rejects hints outside pending candidates and Scan rejects a hint different from its +configured Media before device access. Automatic identification and manual hints have the same +physical identity checks and [Job settlement](jobs.md#shared-lifecycle). ## Synchronous operation reports diff --git a/docs/architecture/jobs.md b/docs/architecture/jobs.md index 05cf1a9..f35d474 100644 --- a/docs/architecture/jobs.md +++ b/docs/architecture/jobs.md @@ -40,7 +40,13 @@ Process restart never retries or restarts a Job, revives terminal `FAILED` work, Completion is not a universal assertion that every recovered byte or checked copy is healthy. Job deletion removes its retained execution bundle, not physical copies, versions or original associations. -The durable completion checkpoint can precede the end of an active runner's cleanup. An attached runner reports `JOB_PHASE_COMPLETED` when its work and cleanup finish; a working phase still requires waiting. A completed Job without an attached runner reports `JOB_PHASE_UNSPECIFIED` and has no runner cleanup left to await. `FAILED` is an actionable result, and a `READY` Job whose runner is not working needs its operator. The [CLI waiting command](cli.md) succeeds for a completed Job even after its runner detaches. +The durable completion checkpoint can precede the end of an active attempt's cleanup. The attempt +keeps its last working phase until its result, timing, resource release and final catalog publication +settle; admission before its first runner transition reports `QUEUED`. Get, List and typed progress +use that same live phase. An idle snapshot reads its durable result after observing settlement, so +it cannot pair an earlier empty error with a newly idle phase. A completed idle Job reports +`JOB_PHASE_UNSPECIFIED`, and the [CLI waiting command](../operations/cli.md) succeeds without an attached runner. +`FAILED` is an actionable result, and an idle `READY` Job needs its operator. The Executor records the latest attempt's start and finish in the common Job bundle record. Typed progress replies expose optional `elapsed_ms`: live server-measured duration while executing, then a frozen duration after success, error or cancellation. Waiting between attempts is excluded; a new attempt starts a new measurement. Historical Jobs without timing and attempts interrupted by process loss have unknown duration rather than a value inferred from catalog timestamps. Progress polling does not write timing or catalog revisions. @@ -64,7 +70,9 @@ The typed `jobs` Settings group supplies one complete set of pipeline limits. De `read_batch=256`, `read_buffer_max=4096`, `write_buffer_max=4096`, `write_batch_size=256` and `flush_interval_ms=1000`. Read batch and write batch must be at least 1; each buffer is limited to 1–1,000,000 items; the write batch cannot exceed the -write buffer; and the flush interval must be at least 100 milliseconds. A malformed saved +write buffer; and the flush interval must be at least 100 milliseconds. The Settings UI labels +these in their actual units: manifest batch, queued reads, queued results, results per update and +update interval in milliseconds. Queue counts are item counts, not byte or memory limits. A malformed saved group is an error rather than permission to substitute defaults. Each new attempt reads the current group and keeps its own effective limits. @@ -83,18 +91,29 @@ One [Media operation](../../internal/executor/archive/media.go) proceeds in this 1. Acquire the target's exclusive attempt lease and create its typed Write Session. 2. Submit PENDING items to ACP in target order, one bounded manifest page per submission. Each item uses the current original Location's read mode; Archive does not regroup files by Location because that would change Media order and the prefix that fits on a full target. -3. Persist each successful ACP result as STAGED immediately, including the actual Media path, hash, size, mode, and times. The Archive records the bytes ACP transferred instead of running a second source-stability policy. -4. Call Write Finalize exactly once after ACP returns, including error/cancellation, using a cleanup context without the original deadline. -5. Publish only backend-verified STAGED items with `CommitMedia`. Create or reuse the selected File's version from the actual transferred facts in that same transaction; when those facts match the earlier observation, preserve its opaque signature. Raw inputs already have per-Job File associations; no later signature-based File merge or binding occurs. +3. Persist each successful ACP result as STAGED immediately, including the actual Media path, hash, size, mode, times and any target-provided provisional storage position. The Archive records the bytes ACP transferred instead of running a second source-stability policy. +4. Call Write Finalize exactly once after ACP returns, passing the copy/stop error and using a cleanup context without the original deadline. Combine any newly returned write stop with the copy cause; keep physical-finalization errors independently observable. +5. Reconcile staged positions against the completed Media result in the existing path-update pass. On normal completion or an exclusively capacity/EW stop, tar excludes members beyond its accepted complete-member cutoff and normalizes optional anchors; LTFS retains its captured Index reconciliation. Publish only backend-verified STAGED items with `CommitMedia`. Create or reuse the selected File's version from the actual transferred facts in that same transaction; when those facts match the earlier observation, preserve its opaque signature. Raw inputs already have per-Job File associations; no later signature-based File merge or binding occurs. 6. Mark published items SUBMITTED with their Media ID; complete the Job when nothing remains pending. Return copy/finalization/publication errors after retaining eligible successes. -The [result stream](../../internal/executor/archive/stream.go) validates completed copy facts and hands them to the shared [result writer](../../internal/executor/result_writer.go): the results callback only starts a write of the outcomes a durable row can express, which ACP hands over in batches of up to `write_batch_size` items, and the writer stages them outside the feed. A staging failure is the writer's terminal error and fails the phase; a result no durable row can express, such as a target that refused its file, is a per-item outcome the attempt reports while it keeps the item PENDING. STAGED is a candidate result, not durable archive success. Initialization resets leftover unsubmitted STAGED items and attempt report data to PENDING; resetting also clears Media path and result. Only submitted files count toward durable progress, historical speed, ETA, and reports; verified zero-byte files count too. +The [result stream](../../internal/executor/archive/stream.go) validates completed copy facts and hands them to the shared [result writer](../../internal/executor/result_writer.go): the results callback only starts a write of the outcomes a durable row can express, which ACP hands over in batches of up to `write_batch_size` items, and the writer stages them outside the feed. A staging failure is the writer's terminal error and fails the phase; a result no durable row can express, such as a target that refused its file, is a per-item outcome the attempt reports while it keeps the item PENDING. STAGED is a candidate result, not durable archive success. Initialization resets leftover unsubmitted STAGED items to PENDING; resetting also clears Media path and result. Only submitted files count toward durable progress, historical speed, ETA, and reports; verified zero-byte files count too. ACP target no-space normalization retains the original error chain, including joined write, close and temporary-output cleanup failures. The failed item stays PENDING without a staged copy result. An Archive Media operation ending in an error or operator cancellation settles the runner and returns the Job to its pre-Media `READY` state with its reason and timing. The prepared manifest and successfully submitted items remain; another explicit load-Media operation processes the remaining items. +Tar keeps each provisional locator in the existing item result, with no separate member report. +A successful member Finish alone cannot authorize publication while bytes remain buffered. On normal completion or an exclusively capacity/EW stop, the accepted-member cutoff and mandatory +physical gates determine which staged rows survive. Cancellation or an in-progress member failure +returns no publishable tar result; cleanup does not seal or salvage that Archive. +A source-open failure can leave a gap before later complete files. Incomplete members remain +pending. A successfully finalized zero-member tar FORMAT registers the initialized Media, its +Job property and zero-file report while retaining the stop reason and remaining work in READY. +An EW first observed during the final buffered WRITE remains a no-space result. Unaccepted staged +members return to pending under that known stop; missing evidence on normal completion remains an error. +It advances no content progress. Failed physical completion or cleanup publishes no usable result. + [Media finalization](media-io.md) defines which candidates survive normal completion, cancellation, target-full conditions, or validation failure. Prefix directories have no batch identity or cleanup ownership. ## Restore @@ -103,7 +122,7 @@ An Archive Media operation ending in an error or operator cancellation settles t No saved version, only newer archive dates, and unknown archive dates are unmatched selection results, separate from a chosen version with no usable copy. Unmatched selections block preparation by default. `skip_unmatched_versions` requires a cutoff and explicit consent; it skips only automatic unmatched selections, never missing-copy errors. An all-skipped selection cannot become an executable Job. Inspection reports unmatched/skipped counts and resolved version/date details only for bounded explicit regular-file roots or explicit versions, not an entire directory manifest. Directory membership and logical output paths use the Library at preparation, not historical directory snapshots. -The `files` table freezes one row per selected version, including content facts, logical name/parent, the reconnect decision and output state. `copies` contains only persisted Media-candidate facts. Execution explicitly assembles a private runtime candidate from one Copy row and the matching File row; it never hydrates execution fields into the persisted Copy type. The latest selected version per logical File is the only reconnect candidate. Once the manifest is frozen, the remaining Media work uses its persisted output paths without selecting newer versions from the Library. +The `files` table freezes one row per selected version, including content facts, logical name/parent, the reconnect decision and output state. `copies` contains only persisted Media-candidate facts, including nullable frozen storage metadata. Profile and locator messages are copied when captured and when assembled for use; reopening the bundle uses the same facts. Native and migrated v0.1.x candidates can have absent locators. Execution explicitly assembles a private runtime candidate from one Copy row and the matching File row; it never hydrates execution fields into the persisted Copy type. The latest selected version per logical File is the only reconnect candidate. Once the manifest is frozen, the remaining Media work uses its persisted output paths without selecting newer versions from the Library. A Restore Media operation ending in an error or operator cancellation returns the Job to its pre-Media `READY` state with its reason and timing. It preserves the prepared manifest and successful per-file/Media checkpoints; another explicit load-Media operation processes the remaining work without restarting the Job. @@ -111,7 +130,31 @@ A Restore Media operation ending in an error or operator cancellation returns th [Native name reservation](../../internal/executor/restore/output_names.go) uses temporary directory-only namespaces under each actual existing output parent, including parent aliases and nested mounts. Filesystem lookup handles case and Unicode equivalence; no SQL collation guesses those rules. A target whose new directories do not inherit its observed name rules is rejected, as are selected file/directory conflicts. Reservations are rebuilt from bounded Job pages when an attempt starts. Probe directories contain no file content, are excluded dynamically from original readers and are removed after indexing; interrupted leftovers contain only directories and cannot become indexed ordinary files. -The [Restore runner](../../internal/executor/restore/media.go) opens a typed Read Session and owns the ACP items of its candidates. The [results callback](../../internal/executor/restore/buffer.go) only starts a write of each batch ACP hands over, and the shared [result writer](../../internal/executor/result_writer.go) completes and stages those batches of up to `write_batch_size` items outside the feed; a failed persistence is the writer's terminal error and stops the run exactly like a caller cancellation, while the drain still persists every accepted item. One target that wrote nothing is an item outcome, not a pipeline failure: its copy keeps its PENDING status and the attempt reports that error while the other candidates of the Media are still persisted and published. Sequential-read Media follows backend storage order; random-read Media uses ordinary path order. [Copy completion](../../internal/executor/restore/stream.go) compares ACP's real hash and size with the expected content and restores the version's own mode/mtime. The output retains read-source and ready facts, but Library association and item/alternative completion wait for successful Session Finalize. Failed physical finalization retains complete bytes without claiming success; a later attempt cannot adopt its own unfinalized output as an unrelated preexisting file. Completion is per item/version, not every version of its File. Metadata restoration may invalidate disposable ACP caches; completed targets are not reread to rebuild caches. Restore does not share Archive's STAGED/submit state machine. +Name probes are recorded in [temporary allocations](persistence.md#catalog-connections), which own +their cleanup. Closing a duplicate runner only releases that instance's handles and cannot delete +another instance's active name probes. Durable chosen paths and results remain in the Job database. + +The [Restore runner](../../internal/executor/restore/media.go) opens a typed Read Session and owns the ACP items of its candidates. The [results callback](../../internal/executor/restore/buffer.go) only starts a write of each batch ACP hands over, and the shared [result writer](../../internal/executor/result_writer.go) completes and stages those batches of up to `write_batch_size` items outside the feed; a failed persistence is the writer's terminal error and stops the run exactly like a caller cancellation, while the drain still persists every accepted item. After draining, the attempt preserves both transfer and persistence failures; an earlier item failure cannot hide a later checkpoint error. One target that wrote nothing is an item outcome, not a pipeline failure: its copy keeps its PENDING status and the attempt reports that error while the other candidates of the Media are still persisted and published. + +Sequential-read Media uses backend storage order as its baseline. An eligible provider plans the +complete chosen-Media read selection once under the [read-order contract](read-planning.md#software-read-ordering). +Verified local outputs are settled before planning; they never contribute phantom Tape movement. +The derived execution index is disposable and leaves frozen physical locators unchanged. Bounded +Copy/File joins consume its ranks, independently of ACP handoff size and result batches. No page +waits for source closure to choose its next order. LOGICAL keeps the existing manifest keyset stream; +random-read Media uses ordinary path order. + +[Copy completion](../../internal/executor/restore/stream.go) compares ACP's real hash and size with the expected content and restores the version's own mode/mtime. The output retains read-source and ready facts, but Library association and item/alternative completion wait for successful Session Finalize. Failed physical finalization retains complete bytes without claiming success; a later attempt cannot adopt its own unfinalized output as an unrelated preexisting file. Completion is per item/version, not every version of its File. Metadata restoration may invalidate disposable ACP caches; completed targets are not reread to rebuild caches. Restore does not share Archive's STAGED/submit state machine. +Finalize receives the read/stop cause so native tar can distinguish normal unload from failed or +canceled cleanup. The original read error and physical-release error remain independently visible. + +Read preparation selects exclusively from pending frozen candidates. An explicit Tape Media hint +outside those candidates is rejected before device access; expected identity/profile always come +from the Job. Automatic tar selection reads the cartridge memory barcode under the attempt lease and returns a +trusted frozen candidate; a Tape without a cartridge memory barcode fails identification. +Explicit tar hints bypass LTFS barcode scripts. All-LTFS selection retains its configured adapter. +Sequential tar reads use logical Tape file/header order. [Media/I/O](media-io.md) owns label validation +and physical lifetime; [interface contracts](contracts.md#tape-options-and-selection) owns defaults. Candidate listing reports missing catalog Media explicitly, including its ID; it never silently removes a frozen candidate from the response. Catalog metadata must be restored before that candidate can be used. @@ -145,9 +188,9 @@ Default Restore candidates are healthy or unchecked and are still validated duri [ScanJobService](../../entity/job_scan.proto) owns one SCAN kind, [runner](../../internal/executor/scan/runner.go), manifest and [pipeline](../../internal/executor/scan/pipeline.go) for Location/Library `selections` or a whole Volume/Tape Media. One `LocationSelection` with an empty path selects an entire Location; selected paths and overlapping roots use the same bounded expansion. Only selections all naming one Location give the Job a single Location target, while properties retain all involved Location IDs. Content acquisition, comparison, Preview, validation and publication are stages, not delegated Jobs. Source adapters supply access, enumeration, order, leases and identity checks. Ordinary browsing and Refresh remain request-bound reads. 1. Freeze source bindings and old content baselines into the Job bundle. Media keeps identity, profile, expected Positions and storage order; Location selections keep the registered root and bounded selected scopes. -2. Enumerate real entries in ordered, bounded pages into the existing `entries` manifest. Logical selections expand once across all Locations; physical and logical overlaps share the `(location_id, path)` key. All selected sources finish preparation before content processing or Library publication begins. Location selections prune Ignore, including an explicitly named path; administrator authorization is never bypassed. Media inventory requires fresh physical enumeration, including the current trusted LTFS index for Tape; old Positions cannot stand in for it. Enumeration reports discovered files and bytes but keeps the total explicitly unknown until its manifest is frozen. +2. Enumerate real entries in ordered, bounded pages into the existing `entries` manifest. Logical selections expand once across all Locations; physical and logical overlaps share the `(location_id, path)` key. All selected sources finish preparation before content processing or Library publication begins. Location selections prune Ignore, including an explicitly named path; administrator authorization is never bypassed. Media inventory requires fresh physical enumeration, using the current trusted LTFS index or complete tar members as appropriate; old Positions cannot stand in for it. Enumeration reports discovered files and bytes but keeps the total explicitly unknown until its manifest is frozen. 3. Acquire content facts under the signature policy below, optionally look up matching Library content, and optionally generate Preview assets in this same pipeline. -4. Finalize the physical Media session where applicable. Location publication uses the observations already produced by enumeration and content reads; it does not run a second filesystem pass to prove that the source stayed unchanged. +4. Finalize the physical Media session where applicable, forwarding the read/stop cause. Location publication uses the observations already produced by enumeration and content reads; it does not run a second filesystem pass to prove that the source stayed unchanged. 5. Publish the selected result policy through metadata-only Library transactions, then checkpoint the Job independently. Keep detailed results in the paged `entries` table; range descriptions come from the specification and current attempt. A failed or cancelled Scan is terminal at every stage, including an admitted Media operation. @@ -172,7 +215,7 @@ scope; paths, content and tracking rows stay in their existing paged stores. Valid cached SHA-256/size facts do not replace opaque YATM signatures. Matching prior content retains its existing signature. Reuse requires applicable metadata and binding/native evidence; metadata-preserving changes require FORCE_READ. Cache read failures never silently turn KNOWN_ONLY into hashing. A Scan records what its reads return under the supported single-operator content-stability model; it does not implement snapshot or drift-detection semantics. -Indexing resolves and stores each selected source path once; the content phase reads that stored path without re-resolving it or comparing before/after metadata. The phase submits caller-owned items to ACP one `read_batch`-sized manifest page at a time, and its results callback only starts a write: it performs no I/O and never waits for a database write, so the one place it waits is a busy writer. ACP owns the result queue and the batch, and the shared [result writer](../../internal/executor/result_writer.go) persists each batch the engine hands over through at most eight concurrent write goroutines and records the first write error once; that error stops the feed and fails the phase, and the drain before the phase returns persists everything the callback accepted. Verification persists a failed item as a per-file finding and continues, while every other result policy reports the cause and stops its scope. `read_batch` is the manifest page, and `read_buffer_max`, `write_buffer_max`, `write_batch_size` and `flush_interval_ms` are ACP's read buffer, result queue, result batch and result flush interval, so the engine owns the queue and the delivery cadence while the runner owns persistence. +Indexing resolves and stores each selected native source path once; the content phase reads that stored path without re-resolving it or comparing before/after metadata. A Media content endpoint has no native path or filesystem signature cache: its transient reference binds the recorded Media path and storage facts when the page is submitted. Sequential Media still cannot supply native Preview inputs. The phase submits caller-owned items to ACP one `read_batch`-sized manifest page at a time, and its results callback only starts a write: it performs no I/O and never waits for a database write, so the one place it waits is a busy writer. ACP owns the result queue and the batch, and the shared [result writer](../../internal/executor/result_writer.go) persists each batch the engine hands over through at most eight concurrent write goroutines and records the first write error once; that error stops the feed and fails the phase, and the drain before the phase returns persists everything the callback accepted. Verification persists per-file findings and continues; device errors stop the scope, as do content errors under every other result policy. `read_batch` is the manifest page, and `read_buffer_max`, `write_buffer_max`, `write_batch_size` and `flush_interval_ms` are ACP's read buffer, result queue, result batch and result flush interval, so the engine owns the queue and the delivery cadence while the runner owns persistence. | Result policy | Publication boundary | | --- | --- | @@ -181,7 +224,56 @@ Indexing resolves and stores each selected source path once; the content phase r | PUBLISH_INVENTORY | Automatically replace inventory only after complete trustworthy Media observation and identity validation; no Apply step | | VERIFY_COPIES | Require real uncached reads against frozen old Position baselines; publish guarded historical check findings, not replacement content | -Inventory publication permits unsigned content, does not manufacture health checks, and preserves applicable prior bad observations. A genuinely empty observed range can remove original references or inventory Positions, never File organization or saved history. Location publication covers the manifest observations directly; it does not omit entries because a later `stat` differs. An unrecoverable read failure stops later Locations, while earlier published Locations remain; the failed Scan is terminal, and a new Scan covers the full selection again, including the Locations that already published. Inventory failures preserve the previous inventory. Library and Job commits remain separate, with errors reported directly. +Inventory publication permits unsigned content, does not manufacture health checks, and preserves +applicable prior bad observations. A genuinely empty observed range can remove original references +or inventory Positions, never File organization or saved history. Location publication covers the +manifest observations directly; it does not omit entries because a later `stat` differs. An +unrecoverable read failure stops later Locations, while earlier published Locations remain; the +failed Scan is terminal, and a new Scan covers the full selection again, including the Locations +that already published. + +Tar inventory can omit the incomplete last member of a filemark-terminated capacity/EW archive +while retaining complete later Tape files. This framing support is not an interrupted-write recovery workflow. A +complete inventory can therefore remove its old Position. Read, cancellation or Session finalization +failure withholds replacement inventory for the entire selected scope, preserving the previous +inventory. Verify instead checks frozen expected Positions, retains individual findings and +continues when possible. Unread candidates receive no invented finding; raw command uncertainty +or lost access ownership stops Verify without changing per-file findings. Whole-Media +unavailability is not per-file corruption. Library and Job commits remain separate, with errors +reported directly. + +Content endpoints retain whether their Open was actually entered, independently of read ordering. +An ACP stop cause propagated to an unopened endpoint is attempt unavailability, not that member's +missing/unreadable finding. Its check timestamp and pending content work remain unchanged. +Restore uses the same rule for unopened copy candidates, preserving PENDING without manufacturing +copy-health observations. + +Optional ordering in VERIFY_COPIES plans the complete selected-Media manifest once, using the +same disposable [execution index](read-planning.md#software-read-ordering) as Restore. Submission pages +only deliver the fixed ranks; they neither replan nor wait for per-page source acknowledgements. +Inventory keeps sequential discovery, and absent providers retain the ordinary keyset stream. +Cancellation and ACP's fatal-stop notification stop submission; the existing Close/Wait path +drains accepted work before Session finalization. + +Scan uses the same append-only `jobs//job.log` and public log reader as Archive and Restore. Logs are retained until Job deletion without disk rotation or a size limit; bounded log reads do not bound retained disk usage. Logs record phase and source-range boundaries, read outcomes, Preview starts/results and summary counts with duration; high-frequency progress events remain counters rather than log lines. Per-file completion does not imply that the source range has been published. + +The Location walker reads at most 256 directory entries per call and rejects traversal beyond 256 directory levels; its recursive implementation can therefore retain at most one open directory handle per allowed level. Manifest traversal, matching and publication use 256-row ordered pages. Public Scan result queries return at most 1,000 rows per request. These are structural bounds, not a fixed RSS guarantee: the per-Job SQLite manifest and its journal still scale with the number of observed entries, while callers must replace result windows rather than retain the whole manifest. + +Verification findings distinguish match, mismatch, missing, unreadable and unavailable baseline. A per-file issue permits continued checking; device errors/cancellation retain completed observations and leave the rest unexamined. Session finalization failure prevents health publication. Expected hashes are never replaced by damaged bytes. Mounted Volume scans execute automatically; Tape scans wait for an explicit matching device through ReadMedia. Waiting holds no Media lease. Sequential sources do not support Preview or implicit Restore/staging. + +Location analysis is the original-publication configuration of Scan, and integrity checking is its +VERIFY_COPIES result policy; [copy health](library.md#copy-health) remains independent of inventory +and accessibility. Neither is a separate runner or Job kind. + +Preview is optional and off by default. Its policy selects missing-only generation or forced regeneration of all identified content. KNOWN_ONLY without a usable identity skips generation and counts it. Generator failures are per-item findings, not a rollback of otherwise valid scanning. Asset addressing and validity belong to [Preview storage](preview.md). + +## Original Matching + +[Matching](../../internal/executor/observation/matching.go), shared with live Archive preparation, runs ordered rounds over the current complete Location selection: path → available signature → native identity. One shared observation-store implementation applies matching to the common identity/evidence columns of disposable Archive observations and durable Scan entries. Each caller supplies its own scoped table; consumer tests verify that assignments cannot cross the admitted Location. Matching evidence lives in the owning row. Old File IDs and new paths are ordered; later rounds never override earlier matches. Matching does not hash. Unmatched entries become independent Files; occupied paths preserve their File without retaining invalid content facts. + +Cross-Location reassociation requires a successfully removed old binding and valid same-Executor tracking evidence. Unavailable Locations and unobserved ranges cannot lose bindings to another analysis. Native tracking keys are derived from existing stat observations and published with originals; matching never reads or writes tracking xattrs. Matching is organization continuity, not proof of move versus copy-and-delete. + +## Progress and Estimates Progress is stage-local rather than a false whole-Job percentage. One decision in the shared [stage helper](../../internal/executor/stage_progress.go) answers what a phase may display: phases `PROCESSING_CONTENT`, `VERIFYING_MEDIA`, `GENERATING_PREVIEWS`, `COPYING_TO_MEDIA` and `COPYING_FROM_MEDIA` may show a percentage and an estimate, `COMPARING_CONTENT` and `COMPLETED` may show a percentage, and every other phase reports counters alone. Content acquisition opens its window once per attempt from the durable manifest — the active scope's denominator and its already completed work — and ACP work events advance it from there; comparison uses manifest outcomes for its processed/total window, Preview uses its attempt's own completions over the manifest's denominator because a successful Preview is never stored per entry, and validation, waiting and publication name their phase without inventing a denominator. A new Location Scan Job builds fresh observations and progress; it does not resume a failed or cancelled Scan. Completion reports the non-removed manifest total. @@ -189,32 +281,10 @@ All typed Job progress replies include the shared `StageProgress` contract: a wi The bounded [stage helper](../../internal/executor/stage_progress.go) uses wall-clock throughput, minimum five-second sampling windows and EWMA smoothing (0.25). Three productive windows are required; Preview also requires three actual decoder-start/completion observations. Samples come from work rather than from polling: Scan read, Archive write and Restore read sample the one-second ACP progress event that already advances their counters, and Preview samples decoder completions, while a progress request only reads the recorded window without recording a sample. Cache hits and joined content-addressed work never become generation samples. Preview measures decoder items: its work unit is never the bytes of the content whose previews settled. Preview first streams ordered content groups and reads existing manifests to classify its unique generation workload without hashing or invoking a decoder; only one group is retained, and SQL does not grow per row. Runtime disabling makes its workload unknown. Estimates exclude later stages and remain `ESTIMATING` until their own denominator and samples are meaningful. A gap longer than the greater of 30 seconds or three observed average completion intervals invalidates the rate; three productive windows restore it, and because the work path observes that gap, a stalled stage is still detected while no card is visible. A merely slower-than-average file no longer repeatedly removes the ETA. Comparison, waiting, validation, publication and finalization expose `NOT_APPLICABLE`, never a fabricated remaining time. -Job cards use one stable metric layout and the server-provided stage percentage/estimate for every kind. One shared module owns the frame, the counter wording and the waiting/active/terminal classification behind the status label and the action buttons; only active work animates a bar of unknown length. Polling keeps the last coherent snapshot through refresh/error/offscreen transitions and rejects obsolete requests, and it never changes an estimate, because every sample is recorded on a work path. Current and session-average byte speeds remain distinct from persisted business results; elapsed time stays the common latest-attempt measurement. - -Scan uses the same append-only `jobs//job.log` and public log reader as Archive and Restore. Logs are retained until Job deletion without disk rotation or a size limit; bounded log reads do not bound retained disk usage. Logs record phase and source-range boundaries, read outcomes, Preview starts/results and summary counts with duration; high-frequency progress events remain counters rather than log lines. Per-file completion does not imply that the source range has been published. - -The Location walker reads at most 256 directory entries per call and rejects traversal beyond 256 directory levels; its recursive implementation can therefore retain at most one open directory handle per allowed level. Manifest traversal, matching and publication use 256-row ordered pages. Public Scan result queries return at most 1,000 rows per request. These are structural bounds, not a fixed RSS guarantee: the per-Job SQLite manifest and its journal still scale with the number of observed entries, while callers must replace result windows rather than retain the whole manifest. +## Manifest Paging Every Job manifest listing shares one page request: a bounded `limit`, an order key cursor, an `order` direction, an `offset`, and an optional total. The listing defines one order key and applies the same clauses in every case; a cursor restricts the sequence to rows beyond that key in the requested direction, an offset addresses a position in the filtered sequence, and the two combine without a separate jump mode. A page reads one sentinel row to report `has_more`, and continuous paging uses the cursor so a page never re-walks the rows it skips; `offset` is only how a client anchors a jump to a position it has not loaded. `include_total` additionally reports the filtered result-set size, which is the height a client needs to present the manifest as one list; a request without it issues no count. Scan orders by entry ID, Archive by the frozen logical target path, and Restore Media by Media ID; Restore Files orders by the composite `(media_id, id)` and carries an index for it, so one sequence spans every required Media. -Verification findings distinguish match, mismatch, missing, unreadable and unavailable baseline. A per-file issue permits continued checking; device errors/cancellation retain completed observations and leave the rest unexamined. Session finalization failure prevents health publication. Expected hashes are never replaced by damaged bytes. Mounted Volume scans execute automatically; Tape scans wait for an explicit matching device through ReadMedia. Waiting holds no Media lease. Sequential sources do not support Preview or implicit Restore/staging. - -Preview is optional and off by default. Its policy selects missing-only generation or forced regeneration of all identified content. KNOWN_ONLY without a usable identity skips generation and counts it. Generator failures are per-item findings, not a rollback of otherwise valid scanning. Asset addressing and validity belong to [Preview storage](preview.md). - -## Analyze - -Location analysis is the original-publication configuration of [Scan](#scan), not a separate runner or Job kind. - -## Verify - -Integrity checking is Scan's VERIFY_COPIES result policy; [copy health](media-io.md#copy-health) remains independent of inventory and accessibility. - -[Matching](../../internal/executor/observation/matching.go), shared with live Archive preparation, runs ordered rounds over the current complete Location selection: path → available signature → native identity. One shared observation-store implementation applies matching to the common identity/evidence columns of disposable Archive observations and durable Scan entries. Each caller supplies its own scoped table; consumer tests verify that assignments cannot cross the admitted Location. Matching evidence lives in the owning row. Old File IDs and new paths are ordered; later rounds never override earlier matches. Matching does not hash. Unmatched entries become independent Files; occupied paths preserve their File without retaining invalid content facts. - -Cross-Location reassociation requires a successfully removed old binding and valid same-Executor tracking evidence. Unavailable Locations and unobserved ranges cannot lose bindings to another analysis. Native tracking keys are derived from existing stat observations and published with originals; matching never reads or writes tracking xattrs. Matching is organization continuity, not proof of move versus copy-and-delete. - ## Location File Operations Ordinary mutations use the shared [organization engine](library.md#shared-organization-interface), not Jobs. Library edits, browsing, annotation and manual original relocation also remain request-bound. Initial whole-Location collection uses Scan with KNOWN_ONLY and PUBLISH_ORIGINALS. - -An active attempt retains its last working phase until result publication, timing and resource settlement complete. Job hydration observes this phase before reading the durable result. Copy-page failures remain visible while accepted ACP work is drained. diff --git a/docs/architecture/library.md b/docs/architecture/library.md index f17b5fa..61023c6 100644 --- a/docs/architecture/library.md +++ b/docs/architecture/library.md @@ -28,7 +28,20 @@ Signature remains opaque bytes in VARBINARY(256), with empty values normalized t [Position](../../internal/library/position.go) stores Media ID, actual path, derived parent_path, directory flag, signature/hash/size, physical mode/times, storage order and typed storage metadata. It has no File or FileVersion owner column. Media/path remains unique; indexed parents support direct-child browsing. Derived directory Positions are not ownership or deletion units. -Verified Archive publication writes Positions and creates/reuses the selected File's version in the same metadata transaction. [Volume Scan publication](../../internal/library/scan.go) reconciles physical inventory without creating logical File identities. Explicit [inventory admission](../../internal/library/catalog.go) creates independent Files under Unforged using known copy facts, with unknown archive times; discovering content is not a new archive event. Only new logical names receive collision suffixes. +Verified Archive publication writes Positions and creates/reuses the selected File's version in the same metadata transaction. [Media Scan publication](../../internal/library/scan.go) reconciles physical inventory without creating logical File identities. Explicit [inventory admission](../../internal/library/catalog.go) creates independent Files under Unforged using known copy facts, with unknown archive times; discovering content is not a new archive event. Only new logical names receive collision suffixes. + +Tar Tape profiles retain the initialization UUID and immutable fixed/variable record policy. Each +regular Position, including an empty member, retains a tar locator and the 16-byte big-endian +`(tape_file, header_offset_bytes)` order key. Its path is +`archives/<16-digit lowercase hexadecimal tape_file>/`; label file zero is +excluded. Library publication and import validate format/profile/locator agreement, namespace, +512-byte alignment and paired optional physical anchors in partition zero. The physical header +offset cannot exceed the tar header offset or the declared record bound, and the exclusive end +must follow the header. Derived directory Positions have no +member locator. JSONL backups preserve these existing profile/storage BLOB payloads; import +allows Position records before their Media records and validates their joined storage facts in bounded +pages after all records arrive, before committing replacement. Inconsistent combinations roll back. [Media/I/O](media-io.md) owns the physical +record and member-completion rules. Exact signature lookup pages matching Positions and independently organized duplicate Files. One Position can serve several Files' versions. Coverage does not imply present readability or a fresh integrity check. @@ -38,7 +51,7 @@ The version keeps this original's mode/mtime and content facts, not another File ## Registered Originals -[Location](../../internal/library/location.go) stores name, Executor/canonical root, preferred Restore destination flag, one JSON configuration for Ignore text and mmap content reads, revision and nanosecond observation/configuration times. Values queried separately, including Restore preference, remain columns. Mmap defaults off. Every Location supports live originals; preference is not write authorization. A Location's entries have exactly one shape per layer: the recorded association is one [FileLocation](../../internal/library/file_location.go), a listed entry is one [entity.LocationEntry](../../entity/location.proto), and what a read observed is one [library.ObservedEntry](../../internal/library/observed_entry.go) - the manifest a Scan publishes and the before/after facts of its result, never a stored entity. Readers return the association itself and read retained evidence separately. Duplicate roots on an Executor conflict. Registration is immediately browsable and usable, including after an import. Name changes do not reorganize Files. [Configuration migration and access](../operations/install.md#location-configuration-and-migration) own administrator boundaries and explicit installer conversion. +[Location](../../internal/library/location.go) stores name, Executor/canonical root, preferred Restore destination flag, one JSON configuration for Ignore text and mmap content reads, revision and nanosecond observation/configuration times. Values queried separately, including Restore preference, remain columns. Mmap defaults off. Every Location supports live originals; preference is not write authorization. A Location's entries have exactly one shape per layer: the recorded association is one [FileLocation](../../internal/library/file_location.go), a listed entry is one [entity.LocationEntry](../../entity/location.proto), and what a read observed is one [library.ObservedEntry](../../internal/library/observed_entry.go) - the manifest a Scan publishes and the before/after facts of its result, never a stored entity. Readers return the association itself and read retained evidence separately. Duplicate roots on an Executor conflict. Registration is immediately browsable and usable, including after an import. Name changes do not reorganize Files. [The upgrade guide](../operations/migration.md#what-changes) owns explicit installer conversion. GORM hooks advance revisions for configuration, attempts and association publication. Search continuations bind directory facts and query; complete directory List has no cursor. Accessibility is a separate timed runtime observation. @@ -62,11 +75,11 @@ Each FileLocation records the relative path and observed facts of its original. Scan creation normalizes Location selections to `ALL` regardless of the requested Library visibility scope. An empty Location path therefore covers the whole Location under its existing Ignore and access rules. Library selections retain their visibility scope and never establish whole-Location coverage. -[Admission](../../internal/executor/location_admission.go) shares path → available valid signature → native identity matching across explicit and automatic collection. Rounds finish within the observed batch before unmatched entries become independent Files under Unforged/location-name. Existing paths win; later evidence cannot steal a still-present or unavailable original. [Scan](jobs.md#scan) matches bounded Job entries and publishes complete selected Location ranges through metadata transactions. Failed or unobserved ranges never imply disappearance. Content changes preserve organization and invalidate stale content facts; versions follow the archive-evidence rule above. +[Admission](../../internal/executor/location_admission.go) shares path → available valid signature → native identity matching across explicit and automatic collection. Rounds finish within the observed batch before unmatched entries become independent Files under Unforged/location-name. Existing paths win; later evidence cannot steal a still-present or unavailable original. Admission and selection share one optional ACP-cache applicability rule: cached size/mtime must match their already-observed facts; cache errors supply no evidence and trigger no content read. [Scan](jobs.md#scan) matches bounded Job entries and publishes complete selected Location ranges through metadata transactions. Failed or unobserved ranges never imply disappearance. Content changes preserve organization and invalidate stale content facts; versions follow the archive-evidence rule above. [Private tracking keys](../../internal/library/file_tracking.go) use primary (file_id, kind) and nonunique (kind, scope, key_value), with typed validation details, observation time and Location provenance. Candidates require trustworthy same-Executor evidence. Scoped native identities are evidence, not File IDs. An available native identity records the Executor and device in its scope and the inode in its key value, without writing a marker into the file. Birth time and generation constrain matching only when their recorded values are nonzero: Darwin records both platform values, while Linux records neither. Available signature or native evidence can preserve continuity under the matching order above. Matching does not prove that a move or copy occurred or provide a platform-independent guarantee against inode reuse. Positively missing originals may retain keys; rebindings, import, explicit deletion and unregistration invalidate untrusted evidence. A root or Ignore change keeps it: evidence confirms a later observation only when that observation agrees with it, so evidence recorded under an older configuration leaves a File unconfirmed rather than claimed, and dropping it would silently unconfirm every File of the Location. Tracking never opens the observed file or reads/writes xattrs. Retired tracking UUID markers are left untouched. Catalogs containing obsolete UUID evidence are rejected before writes; importing such JSONL evidence rolls back the entire import. -Whether a File is backed up is not decided by tracking evidence: archive coverage is the recorded content signature matching published Positions, read from the catalog alone. Tracking evidence decides whether the content currently on disk is the recorded one. A listing reports what it observed - including `Local file present` for an original whose facts it read - and states `Current content not checked` where the evidence was not confirmed, rather than implying coverage; operations that act on bytes confirm identity themselves. Explicit manual relocation replaces only the original association, refusing a path owned by another File; it preserves organization and cannot move physical bytes. +Whether a File has an archived copy is not decided by tracking evidence: archive coverage is the recorded content signature matching published Positions, read from the catalog alone. Tracking evidence decides whether the content currently on disk is the recorded one. A listing reports what it observed - including `Local file present` for an original whose facts it read - and states `Current content not checked` where the evidence was not confirmed, rather than implying coverage; operations that act on bytes confirm identity themselves. Explicit manual relocation replaces only the original association, refusing a path owned by another File; it preserves organization and cannot move physical bytes. Location operations use the [supported single-operator model](overview.md#supported-operating-model): the operator does not issue conflicting mutations or replace selected files while an operation uses them. Requests therefore do not add filesystem locks, repeated observations or publication-time filesystem checks to manufacture multi-user semantics. Metadata transactions still keep one requested catalog mutation atomic. Catalog import is performed with no active Job, and a later revision may require stopping the service first. [Live access](../../internal/executor/location_live.go) owns administrator/path validation; [file operation publication](../../internal/library/file_operation.go) updates original paths after moves and removes bindings after deletion, never logically reorganizing existing Files or transferring annotations to copies. @@ -113,40 +126,6 @@ and member observation. They are separate from the flat `has:duplicates` search Observed disappearance within a completely validated Location selection, explicit physical deletion or unregistration can remove recorded originals, never Files, versions, archive bytes or Preview. Mere failed access cannot remove associations. Media deletion removes inventory metadata only. Trash remains logical organization. -### File and Version Display States - -The primary color combines current local presence with known default Restore candidates. It does not measure health or archive recency. - -| Local original | Known selectable archive copy | Color | Short meaning | -| --- | --- | --- | --- | -| Present | Yes | Green | Local and archived | -| Present | No | Yellow | Local only | -| Missing or unlinked | Yes | Blue | Archive only | -| Missing or unlinked | No | Red | No available content source | -| Either axis unknown | Any | Gray | Identify the unconfirmed axis | - -Present requires a current observation; FileLocation existence alone is insufficient. Missing means the associated path was observed absent. Unlinked means no association, not proof that matching bytes do not exist elsewhere. Inaccessible and unobservable originals remain unknown. With unknown current content and no selectable historical copy, archive coverage is unknown rather than definitely absent. A saved version without a selectable Position does not count as an archived copy. Unchecked copies can be candidates; known damaged, missing or unreadable copies are excluded by default. Candidate existence never promises immediate Media accessibility. - -| Additional scenario | Primary color and supplement | -| --- | --- | -| Present; current content has a candidate | Green | -| Present; current content uncovered but history has a candidate | Green + unbacked-change marker | -| Present; current content unknown but history has a candidate | Green + unknown-content marker | -| Present; current content unknown and no historical candidate | Gray: archive unconfirmed | -| Missing/unlinked; latest saved version has a candidate | Blue; distinguish missing from unlinked in the tooltip | -| Missing/unlinked; only an older version has a candidate | Blue + latest-version-unavailable warning | -| Missing/unlinked; every saved version lacks candidates | Red | -| Present; known saved copies are all unusable | Yellow + copy-problem marker when current coverage is known; gray when it remains unknown | -| Original inaccessible; history has a candidate | Gray: local inaccessible, archive known | -| Original unchecked; history has a candidate | Gray: local unconfirmed, archive known | -| Candidate remains, but another recorded copy is abnormal | Green or blue by local presence + copy-problem marker | - -Lists retain at most one supplement: copy/latest-version problem, then unbacked changes, then unknown content. Inspector and tooltips retain all facts. Shape and accessible text accompany color. Directories reserve the same status-column width but do not invent recursive coverage. - -[Copy health](media-io.md#copy-health) records historical actual-read observations and time. It has no automatic expiry and does not control the primary color. An unloaded Tape is not damaged; successful archive writing is not a read-back check. A new matching-content read on the expected Media and successful Session finalization are required to replace a bad finding. File/FileVersion display states are derived projections, not persisted state machines; [Restore outcomes](jobs.md#restore-outcomes) remain separate. - -Explicit [Trim](../../internal/library/library.go) removes dangling inventory and, when requested, regular Files having neither an original nor saved history. References are rechecked inside the deletion transaction. Unavailable indexed originals still protect Files. Version history is deliberately retained even after its last Position disappears; signature equality never establishes a physical-delete cascade. - ## Shared Organization Interface [FilesService](../../entity/files.proto) owns List, Search, Get, UpdateMetadata, Mkdir, Move and Remove, plus paged versions/copies/duplicates and explicit association operations. List returns one complete directory snapshot as streamed batches, with identity, breadcrumbs and total in the first batch and no cursor or limit. Search remains paged and query-bound. Both return reference/name/kind/path; include groups add attributes, status, operations or navigation. Basic reads bypass full-File hydration; attributes/status use batch queries and status uses existence rather than exact copy counts. Both sources share query grammar and apply it before returning rows. Location reads never use index fallback. Get supplies organization and original navigation independently of Preview/history. @@ -200,9 +179,13 @@ attributes for the retained old path. Another explicit Find supplies the current Frozen Restore manifests retain their selected IDs and content facts. Publication accepts that immutable version snapshot after catalog removal or reassignment and creates an independent restored File when reconnecting the source is no longer valid. Restore outcomes stay in the Job bundle; the Library stores the resulting organization and associations. +## Trim + +Explicit [Trim](../../internal/library/library.go) removes dangling inventory and, when requested, regular Files having neither an original nor saved history. References are rechecked inside the deletion transaction. Unavailable indexed originals still protect Files. Version history is deliberately retained even after its last Position disappears; signature equality never establishes a physical-delete cascade. + ## Backup and Legacy Compatibility -[Library metadata backups](../../internal/library/jsonl.go) export Files/versions, Media/Positions, Locations/originals and tracking evidence using the [version 1 JSONL format](persistence.md#published-data-formats). Each Location record includes its complete JSON configuration. The header declares only its format, version and included Library entities. Settings and operation receipts are outside this boundary, and import always preserves the target installation's Settings. Location data requires its complete File/Location/FileLocation group. Imports validate live relationships and roll back replacement on invalid input, including cross-batch conflicts. Media-only replacements preserve the immutable identity, kind and profile of retained Position owners; reused numeric IDs cannot redirect archived copies. This also applies to legacy Tape-only imports. +[Library metadata backups](../../internal/library/jsonl.go) export Files/versions, Media/Positions, Locations/originals and tracking evidence using the [version 1 JSONL format](persistence.md#published-data-formats). Each Location record includes its complete JSON configuration. Tape Media records include their saved encryption keys, so a backup must be protected like those keys. The header declares only its format, version and included Library entities. Settings and operation receipts are outside this boundary, and import always preserves the target installation's Settings. Location data requires its complete File/Location/FileLocation group. Imports validate live relationships and roll back replacement on invalid input, including cross-batch conflicts. Media-only replacements preserve the immutable identity, kind and profile of retained Position owners; reused numeric IDs cannot redirect archived copies. This also applies to legacy Tape-only imports. Import rebuilds derived Position directories, so their IDs may change, and issues fresh Location revisions. Physical Position IDs come from the imported records. Compare derived directories by Media and path and exclude Location revisions when verifying a metadata roundtrip. Location configuration, including empty or whitespace-only Ignore text, is preserved; default rules apply only to new registrations. @@ -219,4 +202,20 @@ stand in for admitted content, and repeated admission reuses an already resolved [Legacy whole-object import](../../internal/library/json_legacy.go) uses bounded temporary staging. [Offline legacy migration](../operations/migration.md) preserves IDs/organization and copies signatures without rehashing, deriving versions only from observed archive evidence. Isolated File content facts do not create invented history. -Legacy whole-object JSON and the version 1 metadata format have explicit readers. Draft JSONL versions and unsupported schemas are rejected before replacement; data is retained for inspection. The [persistence contract](persistence.md#published-data-formats) owns artifact identities and version handling. +Legacy whole-object JSON and the version 1 metadata format have explicit readers. Unpublished pre-stable JSONL versions and unsupported schemas are rejected before replacement; data is retained for inspection. The [persistence contract](persistence.md#published-data-formats) owns artifact identities and version handling. + +## Copy Health + +Position stores the latest health observation, check time and local checking Job ID. Detailed findings belong to Verify or Restore. Verification reads against the Job's frozen expected content, and [publication](../../internal/library/position_health.go) records the completed check after successful Session Finalize on the Media validated before access. + +| Health | Evidence | Restore and subsequent changes | +| --- | --- | --- | +| Unknown | No usable completed verification baseline | Candidate by default, but actual transfer must verify; never counted as healthy | +| Healthy | Actual hash/size matched and Session finalized successfully | Candidate; a newer complete check can supersede the observation | +| Damaged | Complete read mismatched frozen expected content | Excluded by default; explicit salvage may retain complete mismatched output | +| Missing | A specific recorded path was absent during valid Media access | Not a candidate; expected content and version history remain | +| Unreadable | A specific path could not be read | Excluded by default; explicit salvage permits another read attempt | + +Successful Archive writing does not manufacture a readback check or check timestamp. Inventory publication preserves health only for unchanged content and never accepts damaged bytes as a replacement baseline. Only actual successful reads on the expected Media after successful finalization can clear bad observations. Checks have no automatic expiry and do not claim continuous health. Whole-Media unavailability and canceled/unattempted entries do not mark individual copies bad. Export/import preserves historical observations tied to the same expected content; installation-specific Job links do not survive import. + +Media access (unchecked/reachable/unavailable), lease state (free/owned/uncertain), and Position health are independent. Inspection observes Media identity/capacity, not bytes. Preparation acquires a lease; the attempt releases it after physical finalization and settlement. Uncertain device cleanup preserves the existing lifetime-level unavailable guard; it is not a Position corruption finding. No health transition deletes physical bytes or saved history. diff --git a/docs/architecture/media-io.md b/docs/architecture/media-io.md index 78920cf..3c46982 100644 --- a/docs/architecture/media-io.md +++ b/docs/architecture/media-io.md @@ -15,18 +15,31 @@ and transfer/integrity checks remain separate required boundaries. The [Backend](../../internal/media/backend.go) is a stateless factory for Read/Write Sessions and accepts only the typed physical target. It never receives a Job database. Sessions prepare physical -access, resolve paths, report capabilities and finalize; Archive, Restore and Scan retain manifest +access, resolve content references, report capabilities and finalize; Archive, Restore and Scan retain manifest paging, ACP orchestration, item transitions and Library publication. ReadSession uses a neutral ReadMediaTarget with expected identity/profile; it never queries Restore `copies` or requires fake -Restore tables for Scan. Tape expectations are checked before encryption/mount; Volume expectations -precede path exposure. +Restore tables for Scan. LTFS expectations precede encryption/mount; tar validates its physical label +after establishing the required record mode and before content access. Volume expectations precede +path exposure. + +Read Finalize also receives the caller's read/stop error. A successful raw tar read uses normal +physical release; failed or canceled work only restores owned controls and closes the descriptor. +LTFS still delegates release to its configured unmount adapter; Volume has no Tape release action. `Media` is a zero-I/O boundary descriptor on a prepared Session, not a Library persistence row. -Write target resolution receives the Archive row's frozen path and size, so Volume capacity planning -does not query that row. Finalize receives only whether the copy stopped at a target-full boundary and -returns a `WriteResult`: the selected Media descriptor, physical path prefix and, for Tape, a bounded -post-unmount inventory source. The Archive runner reconciles that result with its own `items` table in -one transaction and then asks Library to publish it. No Media implementation imports an Archive row, +Read resolution receives the frozen Media path and available storage facts. A source or target +reference contains exactly one native path or ACP content endpoint; constructing it performs no +content I/O. LTFS and Volume retain native paths. Endpoint identifiers come from the caller's Media +path and are never treated as filesystem paths. Write target resolution receives the Archive row's +frozen path and size, so Volume capacity planning does not query that row. Finalize receives the +copy/stop error and returns a `WriteResult`: the selected Media descriptor, physical path prefix and, +for Tape, bounded completed path/size/storage facts after physical finalization. LTFS supplies its +captured Index after unmount; tar validates positions already retained in Archive item results. LTFS derives its +target-full behavior from the copy error. The returned finalization error describes physical completion; +the result separately retains a stop first discovered while settling buffered data or the trailer. +The runner combines that stop with the copy error before reconciliation, preserving a usable +physical result and its actual termination reason. The Archive runner reconciles that result with its +own `items` table in one transaction and then asks Library to publish it. No Media implementation imports an Archive row, changes a Job table or owns a Library transaction. Capabilities derive from immutable [Media profiles](../../entity/media.proto), not duplicate database flags: @@ -39,127 +52,6 @@ Capabilities derive from immutable [Media profiles](../../entity/media.proto), n Concurrent random access uses configured ACP concurrency, random access uses a single device thread, and sequential access uses ACP's linear behavior. Read access determines storage-order requirements. An [attempt lease](../../internal/executor/device.go) exclusively reserves a Tape device or Volume UUID; these locks also prevent conflicting YATM operations, not just removable-media swaps. A conflicting Job waits for the holder in arrival order and reports `QUEUED`, so a device in use never fails a Job; an unconfigured Tape device remains an explicit configuration error. Restore acquires the drive before probing its cartridge identity, and Session setup reuses that attempt's lease. The attempt retains the lease through finalization and settlement. -## Tape Script Compatibility - -Configured scripts are a long-lived user interface. Preserve previously supported adapters wherever -possible, including their helpers, input variables, output format, working directory and exit-status -meaning. LTFS distribution and hardware differences remain inside those adapters. A necessary -incompatible change requires explicit developer approval, an actionable installer notice before -upgrade and regression checks of the changed boundary. Installation preserves customized adapters -and their execution context instead of replacing them automatically. This contract applies -independently of the pre-stable API/data policy and preserves the [Tape lifecycle](#tape-lifecycle). - -Script changes require automated checks of the published interface and representative unchanged -historical adapters. Physical acceptance checks the selected host adapter separately; it does not -replace the historical compatibility checks. - -Archive requires a completed per-Job captured Index; historical adapters without that output need -manual adaptation before Tape operations. This required incompatibility is described in the -[upgrade notice](../operations/migration.md#tape-script-adaptation) displayed by the installer. - -## Tape - -Configured scripts are the anti-corruption boundary for environment-specific executable paths, device mapping, vendor options and helper logic. The service supplies operation inputs and validates identity, content and final Index output; it does not absorb those local decisions into configuration switches. The [migration guide](../operations/migration.md#tape-script-adaptation) owns input/output contracts and upgrade examples. Installer preflight never invokes these scripts. - -Operators may edit configured scripts and their helpers to match local software versions, -platforms and hardware. Bundled commands are starting points; local changes retain the -published inputs, outputs, exit-status meaning and physical finalization requirements. - -The format script passes the resolved device, barcode and complete Tape name as separate literal -arguments to `mkltfs`; spaces and shell pattern characters in the name do not split or expand it. -Regression checks execute the shipped script with a stub LTFS command, while physical acceptance -uses the packaged script with only the host executable path and test placement rule adapted. - -The Linux `get_device` adapter resolves the Tape node through the kernel's sysfs mapping to its -SCSI generic node. This lookup does not open the drive, so it still works while LTFS owns the -device and during unmount; an absent or ambiguous mapping fails explicitly. - -Bundled mount scripts resolve the Job's Tape artifact directory to an absolute path before passing it to LTFS. LTFS can change its working directory when it runs in the background; captured indexes must still land in the supplied Job directory when `paths.work` is relative. - -The official LTFS file-backend adapters select a supported capture option at setup: an absolute -directory value, or bare `capture_index` with an absolute work directory for older builds. They wait -for their mount's LTFS process to exit after FUSE detaches. FUSE detachment alone can precede the final Index -write; the test adapter therefore supplies the same completed-finalization boundary as physical -device release. - -The bundled encryption script loads the drive and retries `stenc` at most 60 times. If -every attempt fails, it exits nonzero so the Tape Session cannot proceed to formatting or -mounting. Its `stenc` arguments remain part of the environment-specific script contract. -The script includes a commented stenc 2.0.0 command, which reads the barcode descriptor -from the key file's second line instead of accepting `-kd`; operators select the form -supported by their local tool. - -[Tape Sessions](../../internal/executor/media_tape.go) identify the cartridge, configure encryption, format when requested, mount LTFS, and unmount/finalize. FORMAT accepts a successful probe reporting an empty electronic barcode and uses the explicit user-supplied six-character barcode; the format script passes that identity to `mkltfs` to write it during initialization. A non-empty device barcode must match the requested identity, and probe failures stop the Session before encryption or formatting. An empty barcode does not establish that the cartridge contains no data: FORMAT remains an explicit destructive operation. FORMAT rejects a barcode already in the Library; explicit metadata deletion is required before creating a new Media identity. - -APPEND and Restore require a non-empty valid device barcode before encryption or mounting; APPEND also requires an exact match with the requested identity. Device inspection may use its documented read-only manual identity fallback when automatic MAM lookup is unavailable, but that fallback does not authorize a write or Restore. APPEND requires compatible `ltfs_v1`, preserves Media ID/key/Positions, and chooses a short base36 timestamp prefix only to avoid path collisions. - -### Tape Lifecycle - -1. For an uninitialized cartridge, the operator chooses its six-character barcode and explicitly - submits FORMAT. The electronic barcode may be empty; the formatting script writes the supplied - identity and name while initializing LTFS. An empty barcode does not establish empty content. -2. Archive writes files, normally unmounts and waits for device release, then validates the final - captured Index and publishes Media and Positions. The Library records the Tape's identity and key. -3. On later loads, APPEND verifies the existing barcode and compatible Library Media, reuses its key - and adds files without replacing earlier Positions. Restore verifies the loaded Tape against its - recorded identity, uses its key and reads the selected saved content. -4. Metadata deletion leaves the cartridge's bytes intact. Reinitialization requires explicit FORMAT - after deleting the registered Media; formatting erases the old physical contents. - -Script and caller changes must keep every step available, including FORMAT before an electronic -barcode exists. Per-operation identity and finalization rules remain defined above and below. - -The final captured LTFS Index is the physical fact source. [LTFS processing](../../internal/media) retains backend extents in the typed storage-metadata envelope and stores opaque binary order from partition/block/offset. `ltfs_v0` is path-order compatibility; `ltfs_v1` has validated LTFS order/extents and supports append. Versioning belongs to the storage format, not a second independent version field. - -After physical Finalize, Archive matches staged files against the returned Index by actual Media path, -size and valid extents. On normal completion every staged result must match. On target-full it submits -only the continuous verified prefix and resets the later suffix to PENDING. A successful normal unmount -includes waiting up to ten minutes for the LTFS process to release the SG device and for `mt status` to -report `DR_OPEN`; the Job attempt, device lease and Job bundle remain owned throughout this wait. An -unmount timeout or invalid/missing final Index publishes no unverified result. An uncertain device -remains unavailable for that Executor lifetime. Force and lazy unmount are manual stopped-service -recovery tools whose write result must be discarded. Ordinary LTFS files stay readable without a -YATM-specific segmented format. - -Read Sessions validate the expected Tape before mounting and keep the attempt's device lease -through normal unmount. Finalize does not run `readinfo` again: that script loads the cartridge, -which must not occur while LTFS owns it. Successful reads become publishable only after normal -unmount succeeds; a failed unmount still withholds publication and keeps the device unavailable. - -### Tape Capacity - -Capacity management must preserve continuous data supply during normal Archive. It must not -introduce recurring pause/drain/synchronize/resume cycles per file or batch; normal finalization -when ending use of the current Tape remains necessary. The -[capacity study](../research/tape-capacity.md#streaming-and-mechanical-wear) explains the mechanical -wear concern and the limits of the available evidence. - -ACP checks a cached available-space estimate immediately before starting each positive-size -linear target, debiting that target's logical size. One independent worker per filesystem starts -an observation immediately and refreshes every five seconds, with at most one query in flight. -Copying does not wait for a query: unknown initial capacity permits a file, while a completed -query error is returned to subsequent targets until a successful refresh. A refresh deducts -admissions that overlapped its query; it does not reserve the outstanding bytes of files already -in progress. Tape target-path resolution can run ahead of copying and does not perform a second -capacity decision. The estimate can refuse a file before output creation, but cannot guarantee -that an admitted file will finish or interrupt it when a later observation falls. ACP does not -credit failed-output cleanup as reclaimed Tape space. Stream shutdown joins outstanding queries. - -Capacity observations have three distinct boundaries: bytes accepted by LTFS, bytes accepted by -the drive, and bytes physically written. A successful file Close does not establish that the -drive's remaining-capacity counter includes every accepted byte. Reading `statfs` does not itself -flush either buffer. LTFS can also adjust the reported capacity for its Index and warning state; -raw drive capacity is not interchangeable with filesystem available space. - -Actual write, Close and finalization errors remain authoritative. On exhaustion, retain the -complete error chain and publish only the verified prefix described above. Removing an LTFS -temporary file is a namespace cleanup, not proof that its written Tape capacity was reclaimed. -YATM does not automatically roll back LTFS history, shorten cartridges, configure hardware early -warning thresholds, or force recurring Index synchronization for capacity management. The public -[Tape capacity study](../research/tape-capacity.md) owns the supporting hardware/source evidence, -community reports and unresolved prevention/recovery questions. Authorized experiments follow the -[physical acceptance procedure](../operations/physical-tape-e2e.md#capacity-measurement). - ## Volume [Volume Sessions](../../internal/executor/media_volume.go) manage only already-mounted filesystems. [Volume discovery](../../internal/media/volume.go) accepts configured roots or direct children, resolves the immutable `.yatm.json` UUID/profile, and rejects duplicate mounted identities, profile changes, symlink path components, and path escapes. An absent marker is skipped; a present but unreadable/invalid marker fails discovery, including checks separating originals from archive storage. YATM does not mount, unmount, or eject Volumes; current discovery validates the directory/marker rather than requiring an OS mount-point record. @@ -179,12 +71,38 @@ advance while the preceding file's Close and metadata work finish. Each result s own hash, source/target closure, cache and publication dependencies. Conflicting output paths retain their required publication order; independent paths can finish concurrently. Terminal target errors stop new source work once observed while draining and closing bounded prefetched work. This -application ordering does not prescribe physical LTFS extent placement; the final Index remains +application ordering does not prescribe physical LTFS extent placement; its final Index remains authoritative. +ACP's optional content endpoints use the same transfer engine and require its linear mode only in +the endpoint's direction; the native side retains its configured options. An opened source is a +standard `fs.File`. ACP observes its Stat and optional Xattrs once in the ordered lane, then copies +portable metadata before creating any target. Tar supplies its actual parsed header and raw PAX +attributes; ACP owns ordinary-xattr selection, managed-cache exclusion and snapshot limits. External +Info.Sys does not establish native ownership. Failed or panicking metadata observation fails that +item before output creation and still closes its acquired source. The codec retains its bounded +decoding and PAX-specific representation checks; ACP retains native size probes before allocation. Each provider bounds its +own allocations before ACP receives its snapshot. + +Source Close must succeed before native output publication and before the next content source opens; +target Finish runs once after writes, source closure and hashing settle, retaining their causes, +before the next content target opens. A source returned with an Open error still closes once. +Restore/Scan tracking and read-window wrappers forward Stat/Xattrs and preserve observation errors; +source-boundary settlement does not replace the terminal item result. Native linear completion +keeps its existing overlap. Endpoint identifiers never enter native stat, cache, capacity or cleanup +operations. Tar uses these endpoints for Archive and Restore/Scan; native LTFS/Volume paths retain +their applicable metadata/cache/output behavior. + +ACP's SourceFromFS binds a real `fs.FS` and relative name without inventing a filesystem tree for +tar. It opens once and observes that file, avoiding fs.Stat's possible extra Open/Close. Injected +filesystem paths follow io/fs rules and do not acquire native mmap or ownership guarantees. Native +paths retain their existing absolute/arbitrary-byte names and platform behavior. Context-aware tar +operations keep their command cancellation; wrapping an arbitrary fs.FS does not make blocking +Open/Read calls interruptible. + Location configuration selects ACP's source read mode for original content in Archive preparation/copying and Scan content reads; buffered is the default. Archive applies the setting of each item's current Location without reordering the Media write. Scan applies it per Location scope, including indexed original content reads. Media sources and Restore output verification keep their own read behavior. On Linux buffered ACP reads first attempt `O_NOATIME` and fall back to an ordinary open when the filesystem or permissions refuse it; mapped reads use a different open path. Avoiding access-time updates is therefore best effort, not an operation guarantee. -Callers own their items: the runner pages the Job manifest into ACP one bounded page per submission, and every accepted item is reported exactly once through one results callback, called from one goroutine. A result that carries any error reaches that callback immediately, while results without an error wait in the engine's result queue until a delivery holds the result batch, the result flush interval elapses, or the run closes. The caller's callback only starts a write of the batch it received, so ACP never waits for a database write; the caller's writer is where a persistence failure is recorded, reported back as the callback's cancellation and finally reported by the drain. A completion reports one outcome per requested target in request order, so an item whose every target failed is still a completion, and target errors keep their identity for classification. A missing or unreadable source remains an observable item failure rather than a silent omission. `Wait` returns the run's terminal error, and `Close` still flushes what its buffer holds. The `af05f05c` surface is a compatibility shell over the same engine and the only path that publishes ACP's `Job` report rows; YATM's runners call the push engine directly. +Callers own their items: the runner pages the Job manifest into ACP one bounded page per submission, and every accepted item is reported exactly once through one results callback, called from one goroutine. A result that carries any error reaches that callback immediately, while results without an error wait in the engine's result queue until a delivery holds the result batch, the result flush interval elapses, or the run closes. The caller's callback only starts a write of the batch it received, so ACP never waits for a database write; the caller's writer is where a persistence failure is recorded, reported back as the callback's cancellation and finally reported by the drain. A completion reports one outcome per requested target in request order, so an item whose every target failed is still a completion, and target errors keep their identity for classification. A missing or unreadable source remains an observable item failure rather than a silent omission. `Wait` returns the run's terminal error, and `Close` still flushes what its buffer holds. Page/Submit, Wait and Close failures are retained independently; a single failure is returned unchanged and multiple failures use the existing multierror aggregation, without introducing another error classification. ACP's older Job-based API is a compatibility shell over the same engine and the only path that publishes ACP's `Job` report rows; YATM's runners call the push engine directly. Archive and Restore always hash real source bytes in ACP. Completed targets are not reread for hashing. Ordinary targets, including LTFS files, are written to an exclusively created `.tmp_*` file beside the @@ -198,37 +116,26 @@ reserves distinct native output names during preparation, including case and Uni Failed copies clean up only their owned temporary output and preserve existing destinations; cleanup failures remain alongside the copy error without losing no-space error identities. Linear targets skip preallocation and per-file Sync; -successful LTFS unmount and the final Index still govern Tape publication. Staging adds no filesystem +successful LTFS unmount and the final Index still govern LTFS publication. Tar uses its physical +complete-member gate above. Staging adds no filesystem transaction or power-loss durability guarantee. ACP's managed signature xattr is a disposable cache keyed by size and nanosecond mtime, never an immutable identity or transfer-integrity proof. Scan's KNOWN_ONLY uses the cache-only read without hash fallback; FILL_MISSING hashes misses, and FORCE_READ reads every selected file. Verification always reads uncached content against old expected facts. The managed key is excluded from ordinary -xattr copying. Cache lookup uses the content source's descriptor and indexed facts. Source Close can +xattr copying. Native cache lookup uses the source's descriptor and indexed facts. Source Close can finish before hashing; cache refresh then uses the source path and finishes before result delivery. This follows the supported operating model in which selected files remain unchanged during an operation. A failed cache write is counted, never a Job failure. Read-only/unsupported xattrs do not -fail copying. +fail copying. Custom content sources have no native cache reuse/refresh capability; their optional +Xattrs supplies ordinary metadata only. Tar stores neither this cache nor a checksum manifest. +Restore hashes actual bytes and recreates a supported native output's cache under its selected policy. +Applying the selected FileVersion's metadata can subsequently invalidate this disposable cache; +the [Restore contract](jobs.md#restore) owns that behavior. The Archive runner normalizes ACP and operating-system no-space errors before Finalize, preserving the original copy error independently. Finalize performs physical cleanup and identity validation; Archive then applies the returned facts, publishes verified successes and returns the original/normalized error. -[Jobs](jobs.md) defines these publication checkpoints; [E2E](../operations/e2e-test.md) defines physical +[Jobs](jobs.md) defines these publication checkpoints; [E2E](../development/e2e-test.md) defines physical validation coverage. - -## Copy Health - -Position stores the latest health observation, check time and local checking Job ID. Detailed findings belong to Verify or Restore. Verification reads against the Job's frozen expected content, and [publication](../../internal/library/position_health.go) records the completed check after successful Session Finalize on the Media validated before access. - -| Health | Evidence | Restore and subsequent changes | -| --- | --- | --- | -| Unknown | No usable completed verification baseline | Candidate by default, but actual transfer must verify; never counted as healthy | -| Healthy | Actual hash/size matched and Session finalized successfully | Candidate; a newer complete check can supersede the observation | -| Damaged | Complete read mismatched frozen expected content | Excluded by default; explicit salvage may retain complete mismatched output | -| Missing | A specific recorded path was absent during valid Media access | Not a candidate; expected content and version history remain | -| Unreadable | A specific path could not be read | Excluded by default; explicit salvage permits another read attempt | - -Successful Archive writing does not manufacture a readback check or check timestamp. Inventory publication preserves health only for unchanged content and never accepts damaged bytes as a replacement baseline. Only actual successful reads on the expected Media after successful finalization can clear bad observations. Checks have no automatic expiry and do not claim continuous health. Whole-Media unavailability and canceled/unattempted entries do not mark individual copies bad. Export/import preserves historical observations tied to the same expected content; installation-specific Job links do not survive import. - -Media access (unchecked/reachable/unavailable), lease state (free/owned/uncertain), and Position health are independent. Inspection observes Media identity/capacity, not bytes. Preparation acquires a lease; the attempt releases it after physical finalization and settlement. Uncertain device cleanup preserves the existing lifetime-level unavailable guard; it is not a Position corruption finding. No health transition deletes physical bytes or saved history. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 1085bdd..28e984a 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -10,7 +10,7 @@ one original, saved FileVersions, and signature-linked independent Tape/Volume c | Module | Responsibility | Interface | | --- | --- | --- | | Library | Files, annotations, versions, Media/Positions and health, Locations, originals and tracking | Metadata transactions; no physical copy orchestration | -| Settings | Typed Library, Preview and Job preferences | Complete independently stored groups; not part of Library metadata backup | +| Settings | Typed Library, Preview, Job and Device settings | Complete independently stored groups; not part of Library metadata backup | | Executor | Job catalog projections, runner lifetime, cancellation, Job locking, accessible paths and device allocation | One execution owner per Job and attempt-scoped Media leases | | Job runner | Typed specification, durable manifest, live phase and progress | Archive, Restore and Scan own their operation-specific execution | | Media Backend/Session | Physical preparation, safe paths, capabilities and finalization | Stateless factory and short-lived read/write sessions | @@ -21,7 +21,7 @@ Application entry and configuration are in the [HTTP service](../../cmd/httpd/ma [configuration](../../internal/config/config.go), and [example configuration](../../config.example.yaml). [Executor construction](../../internal/executor/executor.go), [Job registration](../../internal/executor/job_type.go), and [API wiring](../../internal/apis/api.go) connect the modules. Source ownership and build entrypoints -follow the [repository layout](../operations/testing.md#repository-layout-and-entrypoints). +follow the [repository layout](../../AGENTS.md#repository-layout). ## Supported operating model @@ -64,7 +64,9 @@ concurrent mutations. Keep and Merge validate their current connected component physical guarantees; [Library](library.md) owns organization, versions and inventory; [Persistence](persistence.md) owns durable stores and recovery. Accessibility, copy health, Job state, live phase and UI summaries are separate facts. Library organization never renames or removes physical -Media files, whose ordinary LTFS/filesystem paths remain recoverable without a YATM-specific format. +Media files. LTFS/filesystem files and tar members use ordinary public formats; +[Media/I/O](media-io.md) and the [recovery guide](../operations/tar-tape-recovery.md) own their +physical recovery requirements. ## Source Modules and Dependency Direction @@ -78,7 +80,8 @@ The source tree therefore follows the business modules above: | `internal/settings/` | Typed Settings groups, validation and persistence | `entity` and its Settings store | | `internal/executor/` | Job catalog, execution admission, resources and composition of runners | Library, Settings, Media contracts and operation modules | | `internal/executor/archive/`, `restore/`, `scan/` | One Job kind's bundle rows, runtime pipeline and results | Executor capabilities plus the Library and Media operations that Job kind uses | -| `internal/media/` | Backend and session contracts for physical Media behavior | `entity`; never a Library row or Job database | +| `internal/media/` | Backend and session contracts, Tape records and tar encoding for physical Media behavior | `entity`, ACP content endpoints; never a Library row or Job database | +| `internal/media/readplan/` | Software read ordering of one frozen selection in a disposable index | `entity`, temporary resources; never `internal/media`, Tape I/O or locators | | `internal/apis/`, `cmd/` | Transport and process composition | Public module interfaces; no duplicate domain state | Dependencies point toward contracts and owning modules. A physical Media session receives the facts @@ -88,35 +91,3 @@ interface: Archive observation rows and Scan entry rows each map their own schem Extract shared behavior when it materially removes duplication between current callers or establishes one independently testable ownership or invariant boundary. Similar structs, a hypothetical future caller, or a generic `models`, `services` or `repositories` layer are not sufficient. - -## Type Roles and File Layout - -Every durable type remains in the package that owns its invariant. When one package needs several -representations of the same subject, names make their lifecycle explicit: - -- a private `...Row` is the exact persisted table shape and carries only persistence behavior; -- a runtime aggregate or snapshot carries execution state but has no ORM tags or hooks; -- a projection is a read result assembled for a particular query and is never saved as a row; -- an adapter translates between an owning representation and a real cross-module interface; -- an `entity` type is a wire or stored-blob contract, not the default internal model. - -Files follow the same roles. A cohesive type may keep its ordinary domain filename; when roles would -otherwise mix, use focused names such as `job_row.go`, `job.go`, `file_projection.go` or -`match_store.go`. Focused responsibility files live inside their existing owner; a catch-all -`models.go` file is not an ownership boundary. -A new subpackage is introduced only when it owns behavior and invariants behind a useful interface; -moving passive structs into a separate package only adds navigation and dependency cost. - -Persistence hooks never assemble presentation state. Queries explicitly load the projection or -snapshot they promise, so production behavior does not depend on callers remembering to disable an -ORM hook. The process composition root creates the Settings module, and attempt admission freezes the -validated effective settings used by a runner. - -## Test Ownership - -Tests live beside the behavior they protect. Row tests own table names, columns and indexes; module -tests own transactions and state changes; interface contract tests run against each real adapter; API, -CLI and end-to-end tests retain only behavior visible at those boundaries. A lower-level test is not -repeated through test-only compatibility methods, and tests for removed commands or superseded wire -shapes are deleted rather than retained as historical documentation. Stable wire numbers, published -formats and supported migration inputs remain positive compatibility tests. diff --git a/docs/architecture/persistence.md b/docs/architecture/persistence.md index 7d33afa..dacf6d1 100644 --- a/docs/architecture/persistence.md +++ b/docs/architecture/persistence.md @@ -6,9 +6,9 @@ | --- | --- | --- | | Executor catalog | `jobs`, `job_properties` | Job identity, state/name projections, Location/Media search values, timestamps and revisions | | Library catalog | `catalog_metadata`, `files`, `file_tags`, `file_versions`, `file_version_archives`, `media`, `positions`, `locations`, `file_locations`, `file_tracking_keys` | Format identity, organization, saved content/dates, copy health, originals and tracking | -| Settings store | `settings` | Independently stored Library, Preview and Job preference groups | +| Settings store | `settings` | Independently stored Library, Preview, Job and Device settings groups | | Archive Job database | `job`, `config`, `items` | Specification, frozen inputs, copy results and submission checkpoints | -| Restore Job database | `job`, `config`, `files`, `copies` | One selected version/output per File row; separate Media candidates | +| Restore Job database | `job`, `config`, `files`, `copies` | One selected version/output per File row; separate Media candidates with frozen storage locators | | Scan Job database | `job`, `config`, `entries` | Specification and one manifest of observations, matching evidence, content/check/Preview results | These are three ownership categories. Executor, Library and Settings use the configured Catalog connections; Job bundles have separate SQLite databases. Each table row is declared in its owning module and new databases use those GORM row definitions, not independently maintained SQL schemas or a shared models layer. Runtime snapshots and query projections are assembled explicitly and are never schema inputs. @@ -25,11 +25,40 @@ SQLite sidecars belong to the database. Backups must use the supported metadata Ordinary SQLite handles used by Jobs, temporary work and retained Identical results also default to a five-second busy wait on every connection, including replacements. Independent handles to the same file wait for short write transactions to finish instead of immediately failing concurrent progress reads or status writes. -Archive selection preparation uses a disposable SQLite workset under `work/.selection-*`, removed when preparation returns. Its `candidate_ids` table retains only File IDs for matching; current original facts are read from the Library, not copied into another persistent table. Scan selection ranges are derived from its specification and attempt state; matching evidence is embedded in `entries`. Neither workflow retains separate observation/original/scope tables. +Archive selection preparation uses a disposable SQLite workset under `work/temporary/`, removed when preparation returns. Its `candidate_ids` table retains only File IDs for matching; current original facts are read from the Library, not copied into another persistent table. Scan selection ranges are derived from its specification and attempt state; matching evidence is embedded in `entries`. Neither workflow retains separate observation/original/scope tables. Request-bound file operations use a temporary SQLite `items` manifest for bounded traversal and outcomes. It is protected from Location access and removed when the request ends; it is neither a retained Job bundle nor another persistent database category. -[Temporary database ownership](../../internal/resource/temporary.go) closes the database before removing its uniquely allocated directory, including failed construction. File operations, naming/import worksets and Keep staging share this cleanup; each caller still owns its schema and semantics. Durable Jobs and retained Identical results use their own lifetimes. +[Temporary database ownership](../../internal/resource/temporary.go) closes the database before removing its uniquely allocated directory, including failed construction. Disposable runtime worksets, execution indexes, request results, script output and transient key files share `work/temporary/`. Each allocation has a UTC creation-time prefix and a unique suffix. Normal completion deletes it immediately; startup and hourly cleanup scan bounded directory batches and delete abandoned allocations older than seven days, and every unheld transient key directory regardless of age. Process-local ownership protects live operations without leases or heartbeat files. Failed deletion retains the path for retry and produces a warning; it does not prevent other leftovers from being removed. Each caller still owns its schema and semantics. Restore name-probe allocations also retain exact +external directory paths and native parent identities. Cleanup removes those probes before their +records; unavailable targets retain the records without blocking other allocations. These disposable +records are independent of Job deletion and do not change Library or Job schemas. +The HTTP service and Demo preparation configure Library's Work directory before admitting operations. + +Durable Job bundles, recovery material and retained verification evidence are outside this expiry rule. +Same-filesystem atomic staging retains its publishing owner's lifetime. LTFS mount points are removed +only after unmount; expiry never recursively follows them or external Restore output paths. + +## Derived Tape Data + +The nullable `media.tape_position_map` stores geometry separately from the immutable Tape profile. +It records initialization binding, declared partition coverage, wrap endpoints, direction, source and +spatial uncertainty. Actual EOD limits a partial final wrap independently of its interpolation scale. +The Library validates and replaces this field using the observed initialization and previous map; +ordinary Media publication cannot overwrite it. Library JSON and JSONL backups retain the field. +Reads validate the physical tar label before using retained geometry and bind its lifetime to the +current prepared Session. A failed map save logs a warning without failing content access. + +Device settings retain learned read parameters independently of user-owned values. Client +Save ignores supplied learning fields and preserves the latest server-owned learned read parameters. Their publication +compares the prior value for one exact device/configuration and leaves user-owned values and other learned read parameters +unchanged. Manual read parameters take precedence; admitted settings and parameters remain frozen +through the current operation. Learned read parameters are bounded to 8 MiB of protobuf-encoded values per +device. Publishing retires that device's oldest automatic results when necessary, retaining the +newly fitted result and every manual preference; other devices are unaffected. + +The observations behind learned read parameters are not a Settings table; each Job keeps its own in its +[bundle](#job-bundle), so deleting the Job removes them from later refits. ## Timestamps @@ -54,10 +83,9 @@ do not change the millisecond unit of the public elapsed duration. ## Published Data Formats The current v1 implementation uses revision 1 for the Catalog and Job bundle. The Library JSONL -contract uses version 1. Until the first stable v1 release, changes to these formats between -unpublished or pre-stable v1 builds require no compatibility readers or migrations and no -version/revision bumps; the same marker does not promise a compatible layout. The supported -`v0.1.x` migration/import path remains. Software release numbers and data-format revisions are +contract uses version 1. The [pre-stable compatibility policy](../README.md#pre-stable-compatibility) +governs changes between unpublished or pre-stable v1 builds; the same marker does not promise a +compatible layout. Software release numbers and data-format revisions are independent. | Artifact | Identity | Current revision | @@ -74,27 +102,35 @@ Job bundles share one format definition across runners and the migrator. An older Alpha Catalog with the same revision but obsolete Settings, file-operation tables, tracking UUID evidence or pre-nanosecond timestamp columns is rejected before writes. Bundle, -backup and Volume marker readers likewise require their current timestamp shape. Existing -pre-stable v1 layouts require an explicit conversion or reinstall. +backup and Volume marker readers likewise require their current timestamp shape. Restore bundles +without the current nullable `copies.storage_metadata` column are rejected on opening, without +adding it or changing the bundle. Migrated v0.1.x candidates have that column with absent locators. +Existing pre-stable v1 layouts require an explicit conversion or reinstall. -Unmarked nonempty catalogs, prototype JSONL headers, unmarked Draft bundles and +Unmarked nonempty catalogs, prototype JSONL headers, unmarked pre-stable bundles and unknown format revisions are rejected with their data retained. Prototype identifiers do not reserve revisions in the supported families. Disposable development fixtures require an explicit reset. Legacy whole-object JSON remains a separate supported import. Its compressed protobuf header, encryption-key prefix and opaque signature bytes retain their original -encoding. LTFS profiles and Volume markers identify physical storage contracts, +encoding. LTFS/tar profiles and Volume markers identify physical storage contracts, not Catalog revisions or backup versions. Existing wire names such as the compressed protobuf header, key/signature headers and `ltfs_v1` identify actual encodings; they are independent of the `v0.1.x` and v1 software release names. ## Job Bundle -Each Job owns `work/jobs//`: `job.json` describes the bundle, `state.db` stores execution state, and `job.log` records events. Tape-specific reports, LTFS logs, and captured indexes live below `tapes//`. The complete append-only log remains with its Job until deletion; it is not rotated or truncated. A bundle is the retention and cleanup unit; detailed Job state is never a Library blob. +Each Job owns `work/jobs//`: `job.json` describes the bundle, `state.db` stores execution state, and `job.log` records events. Tape-specific reports and LTFS logs/indexes live below `tapes//`; the existing Media/Job profiles retain their keys. Native tar reads with [passive read learning](read-planning.md#software-read-ordering) append `tapes//read-observations.jsonl`, one JSON object per completed Tape action with raw facts only: `device`, `scope` (protobuf JSON of the learned model's applicability), `job_id`, `barcode`, optional `map_source`, `map_observed_at_ns`, `min_lpos` and `max_lpos`, `opcode`, `action` (`LOCATE`, `REWIND`, `READ` segment, `FIRST_READ`, `FILEMARK_READ` or `UNLOAD`), optional `previous_action`, `started_at_ns`, optional `idle_seconds`, `from`/`to` endpoints with optional `partition`, `object`, `file`, `wrap`, `offset` and `span`, and `commands`, `bytes`, `device_seconds`, `host_seconds`, `max_host_gap_seconds` and `discard`. Instants are decimal strings. Each attempt appends after native I/O settles; a refit skips a truncated tail. Temporary script key files do not become additional durable recovery records. The complete append-only log remains with its Job until deletion; it is not rotated or truncated. A bundle is the retention and cleanup unit; detailed Job state is never a Library blob. [Bundle creation](../../internal/executor/db.go) finishes before Create returns. [Job records](../../internal/executor/job.go) keep authoritative kind, durable status and priority in the Job database's single `job` row. The private `jobs` catalog row contains only identity, navigation, revision and query projections; runtime `Job` snapshots combine that row with the authoritative Job record and the attached runner's live phase. The Executor's `catalog_kind` is an indexed query projection, published with a new revision when the durable state changes. A Job's state and live phase are read from its own record on every hydration, so no catalog column carries either. Append-only `job_properties(job_id, key, value)` stores `location_id` and `media_id` values for catalog filters, without source/destination roles. Values come from frozen selections or actual operations. +Tar profile and locator variants occupy the existing Media profile and Position storage BLOBs; +Restore copies freeze a nullable storage metadata BLOB alongside their existing path/order/profile. +Scan uses its existing entries and expected-position configuration. There are no separate tar +inventory, locator-map or per-file xattr tables. [Library](library.md#archive-inventory) owns validation +and [Jobs](jobs.md#restore) owns frozen candidate use. + ## Mutation and Pagination Rules - Application reads and writes use GORM so hooks, nullable signature handling, and timestamp semantics remain active. Job catalog timestamps and soft deletion follow the shared nanosecond storage contract. @@ -103,8 +139,9 @@ Each Job owns `work/jobs//`: `job.json` describes the bundle, `state.db` - Explicit transactions are avoided by default: an existing operation boundary or a single statement carries ordinary writes. A transaction is introduced only for a concrete consistency requirement that nothing else satisfies; the invariant that requires it is documented, its scope stays minimal, and it never substitutes for operation admission, validation or recovery. Catalog transactions never span physical I/O or long-running work. - Separately queried or indexed values are columns. Location's unindexed Ignore and mmap settings share one JSON `config` column; its separately queried Restore preference remains a column. Typed protobuf blobs hold cohesive specification/results or backend metadata; Scanner/Valuer implementations handle their encoding. - Location revisions advance through GORM save hooks for configuration, attempted Job and successful publication changes. A live Search cursor binds the Location, path, query and lexical continuation, not a directory snapshot. Request references name the Location and relative path; an operation resolves the current object once for the action it is performing. -- Global configuration uses `settings(key, value, created_at_ns, updated_at_ns)`. The independent typed groups in the [Settings module](../../internal/settings/settings.go) bind `library`, `preview` and `jobs` to protobuf JSON values, defaults and validation. Reads never insert defaults, and each save replaces one complete group. A malformed stored group is reported instead of becoming defaults. Unbacked visibility and Trash-removal confirmation default on and remain independent; preference changes never collect files. Queries resolve visibility before filtering/paging, and jobs freeze the resolved scope. Library metadata backups do not contain Settings. Incompatible pre-stable automatic-collection/physical-delete schemas are rejected before AutoMigrate; data is preserved for an explicitly selected conversion or reinstall. +- Global configuration uses `settings(key, value, created_at_ns, updated_at_ns)`. The independent typed groups in the [Settings module](../../internal/settings/settings.go) bind `library`, `preview`, `jobs` and `devices` to protobuf JSON values, defaults and validation. The DEVICE group occupies one `devices` row; its configured-device entries are preferences, not hardware authorization or persisted capability reports. Reads never insert defaults, and each save replaces one complete group. A malformed stored group is reported instead of becoming defaults. Fields unknown to the running build are ignored on read and omitted by the next save, so Settings stay readable across builds. Unbacked visibility and Trash-removal confirmation default on and remain independent; preference changes never collect files. Queries resolve visibility before filtering/paging, and jobs freeze the resolved scope. Library metadata backups do not contain Settings. Incompatible pre-stable automatic-collection/physical-delete schemas are rejected before AutoMigrate; data is preserved for an explicitly selected conversion or reinstall. - Large manifests and Positions use ordered, indexed pages or streams. `parent_path` indexes direct physical children. Query aggregates from authoritative rows instead of adding duplicate counters when the query is bounded and indexed. +- Restore's optional complete read plan is a disposable execution index under `work/temporary/`, not part of the Job bundle or Library. It maps computed ranks to existing Copy IDs, retains physical locators unchanged and is rebuilt for a new explicit attempt. The [read-order owner](read-planning.md#software-read-ordering) defines its bounded planning and cleanup; no retained schema migration or automatic resume is involved. - File, Tag, duplicate-group and duplicate-member cursors have distinct centrally declared operation kinds as well as query bindings. A cursor from another operation cannot be reused as a directory continuation. - Keep Library transactions metadata-only and limited to their publication work; perform source reads, ACP, mount operations, and validation before opening them. @@ -112,7 +149,7 @@ Maintenance constraints require new catalog-visible mutation paths to maintain r ## Recovery Boundary -The durable Job states are `PREPARING`, `READY`, `COMPLETED` and `FAILED`. A runner reports a live phase only while an attempt is active; execution is not another durable state. Preparation and Scan failure or cancellation are terminal. Only Archive/Restore Media failure or cancellation returns to pre-Media `READY` for another explicit Media choice. A retained manifest cannot admit a `FAILED` Job. Normal shutdown lets an active operation finish or clean up; process restart does not resume an interrupted live phase. [Jobs](jobs.md#shared-lifecycle) owns recreation through a new Job. +The [Job lifecycle](jobs.md#shared-lifecycle) owns durable states, live phases and which failures return to `READY`. Normal shutdown lets an active operation finish or clean up; process restart does not resume an interrupted live phase. Library and Job commits are separate. A successful Library commit remains authoritative even if the following Job checkpoint fails; there is no distributed transaction, mid-operation attempt journal, or extra SIGKILL/power-loss recovery protocol. Archive initialization resets leftover STAGED rows. Unexpected Volume files can be examined through explicit Scan. @@ -121,5 +158,3 @@ Restore publishes Library associations before its Job checkpoint. If that checkp File operations update their original associations and return the request outcome without a persistent result table or automatic metadata retry. Their temporary manifest is removed at request end. [Physical-operation outcomes](library.md#physical-file-operations) define partial failure and explicit follow-up; no filesystem rollback is attempted. [Recovery](../../internal/executor/recovery.go) removes only literally empty incomplete Job directories. A missing bundle preserves its catalog entry and reports an error. Missing metadata/state files, invalid manifests/configuration, symlinks and nonempty orphan bundles are reported and preserved for inspection, not recursively deleted as unfinished creations. [Catalog updates](../../internal/executor/job.go) and [kind-specific runners](jobs.md) define the implemented checkpoints. [Offline migration](../operations/migration.md) owns temporary staging/backup tables; those tables are not part of the runtime schema. - -Disposable files use the shared Work temporary directory with timestamped names, explicit active ownership and seven-day expiry. Cleanup retains failed removals for retry; Restore name probes remain on their target filesystem and are tracked by that owner. diff --git a/docs/architecture/preview.md b/docs/architecture/preview.md index ff12e86..0d40196 100644 --- a/docs/architecture/preview.md +++ b/docs/architecture/preview.md @@ -18,16 +18,7 @@ The installer explicitly converts legacy YAML generator options and removes that Output dimensions, sampling, assets, manifest, and output bytes are bounded. Video defaults use 320×180 timeline tiles, a 10-second sampling interval and at most 30 frames. Video timeline tiles retain their configured dimensions, including odd sizes, so encoded sprites and WebVTT coordinates agree regardless of the source pixel format. The native timeline slider occupies the full height of its controls row for pointer interaction. Pointer positions select the nearest sampled time; arrow keys move between actual sampled frames, including irregularly spaced frames. -Video position and duration use the same fields from `days:hours:minutes:seconds`, omitting -leading days and hours only when they are zero in the total duration. The current position -retains those fields even at zero. The first displayed component has no leading zero; -following components have two digits. For example, a 65-second clip starts at `0:00 / 1:05`, -a two-hour clip at `0:00:00 / 2:00:00`, and a one-day clip at `0:00:00:00 / 1:00:00:00`. -Fractional seconds are rounded down; the display retains minutes and seconds for short clips. - -The settings editor uses a searchable Type selector and one card at a time, without repeating the selected type as a card heading. Image/video types use descriptive names; multiple routes of the same type are distinguished by extensions. File extensions use a searchable multi-select: selection enables a route, deselection retains it disabled, and custom extensions remain editable. Per-kind helper capabilities supply suggestions; unconfirmed extensions stay visible rather than being discarded. Switching cards preserves unsaved edits, and Save returns to any card with invalid parameters. The Supported formats section shows helper capabilities and dependency versions before Save and Cancel; failed or pending rechecks retain the prior information and draft. The master switch defaults off; configured kinds and extensions default on beneath it. Disabling a kind or extension retains its parameters. The helper path, input pixel limit and timeout are read after a file acquires an execution slot; changes do not alter the Job's frozen output dimensions, format or quality. `PreviewService.GetCapabilities` and `yatm-cli preview capabilities` probe the installed helper separately from enablement and asset reads. Explicit Scan or Archive generation requests fail before Job creation when the master switch is off or the helper is unavailable. A later failure creating Archive's companion does not invalidate the prepared Archive. - -Unchecking an extension retains its disabled route. Deleting its selected chip removes that route, allowing corrections or reassignment to another configured generator without discarding the other draft settings. +The [user interface](ui.md#preview) owns the timeline display and the Settings editor layout. ### Native Helper @@ -37,15 +28,17 @@ The command defaults to a helper beside the main executable, then `PATH`; it nev Relative source paths and `paths.work` resolve from the service working directory. An absolute `preview.root` is used directly; a relative root resolves under `paths.work`, with `previews` as the default. The shared caller passes absolute source and output paths to the helper; generation does not rewrite deployment configuration. -Image generation uses LibRaw for DNG, CR2, CR3, NEF, NRW, ARW, RAF, ORF, RW2, PEF and SRW. It selects the smallest embedded preview large enough for the requested output, falling back to actual RAW development when previews are absent, damaged or undersized. Development uses camera white balance with LibRaw's missing-metadata fallback, sRGB output and half-size decoding when sufficient. Orientation is applied once before final resizing. These extensions are routing capabilities, not a guarantee that every camera or compression variant is supported. The installed helper reports its LibRaw version and per-kind input formats; RAW shares Image output settings and the existing asset contract. +Image generation uses LibRaw for DNG, CR2, CR3, NEF, NRW, ARW, RAF, ORF, RW2, PEF and SRW. It selects the smallest embedded preview large enough for the requested output, falling back to actual RAW development when previews are absent, damaged or undersized. Development uses camera white balance with LibRaw's missing-metadata fallback, sRGB output and half-size decoding when sufficient. Orientation is applied once, after target-size resampling. These extensions are routing capabilities, not a guarantee that every camera or compression variant is supported. The installed helper reports its LibRaw version and per-kind input formats; RAW shares Image output settings and the existing asset contract. ### Resource Limits -One manager bounds active files across all Jobs (default two, configurable 1–16), shares in-flight generation for identical content and admits at most 64 distinct pending content keys. Queue saturation is an explicit item failure. The per-file timeout defaults to 600 seconds and starts after acquiring a slot, not while waiting. Each decoder uses two slice threads. Input dimensions are checked before decoding against the configured limit (default and maximum 64 million pixels). Native allocation requests, bytes read per seek, packets inspected, output dimensions, asset count and bundle bytes are bounded independently. These limits are not a fixed process-RSS ceiling: a 100-million-pixel sprite alone needs roughly 400 MB of RGBA storage, in addition to decoder and encoder buffers. Capacity acceptance must measure the media and concurrent workload; service cgroup memory can include filesystem cache, not just process RSS. +One manager bounds active files across all Jobs (default two, configurable 1–16), shares in-flight generation for identical content and admits at most 64 distinct pending content keys. Queue saturation is an explicit item failure. The per-file timeout defaults to 600 seconds and starts after acquiring a slot, not while waiting. Input dimensions are checked before decoding against the configured limit (default and maximum 64 million pixels). Native allocation requests, bytes read per seek, packets inspected, output dimensions, asset count and bundle bytes are bounded independently. These limits are not a fixed process-RSS ceiling: a 100-million-pixel sprite alone needs roughly 400 MB of RGBA storage, in addition to decoder and encoder buffers. Capacity acceptance must measure the media and concurrent workload; service cgroup memory can include filesystem cache, not just process RSS. -RAW uses LibRaw's decoder-allocation guard, scaled with the configured input-pixel limit and capped at 512 MiB, and rejects processed buffers over 256 MiB. The decoder guard does not include all development, resize and encoder allocations. Large developed outputs can therefore exceed 512 MiB of process RSS; concurrency and output dimensions remain separate resource controls. - -Video timelines open the input and decoder once, issue ascending sparse backward seeks and decode keyframes only. The poster and timeline share sampled frames. Repeated keyframes are deduplicated; WebVTT carries actual normalized presentation timestamps rather than claiming an exact frame at each requested time. Sampling does not fall back to sequential decoding or per-frame subprocesses. The worker bounds demux reads to 64 MiB for opening and each seek and inspects at most 4,096 packets per sample. Containers lacking sufficient header metadata, unsupported codecs, unavailable keyframes and corrupt frames produce explicit failures instead of an unbounded probe. Temporary frames remain in bounded memory rather than per-frame files; publication failure or cancellation removes temporary output assets. +The [helper's sampling and resource bounds](../../previewworker/README.md#sampling-and-resource-bounds) +own the native per-file limits: demux and packet budgets, keyframe-only sparse sampling, output +dimensions and the LibRaw allocation guard. Sampling never falls back to sequential decoding or +per-frame subprocesses, unsupported or under-described inputs fail explicitly, and publication +failure or cancellation removes temporary output assets. ## Addressing and Update Policy @@ -53,7 +46,7 @@ The current producer uses the Library v1 signature from SHA-256 plus size. The [ Missing-only generates assets for known content without an existing bundle. Regenerate-all replaces existing bundles even when generator settings are unchanged, independently of the signature policy; it also permits replacing damaged derivatives. Identical content shares assets across independently organized Files and versions and is generated once per Scan. Archive's companion Scan uses the same policy. Job deletion does not remove bundles. -[PreviewService.Get](../../entity/preview.proto) reads metadata independently of Files.Get/List and version pagination. Its request is the content signature itself, because Preview storage is content-addressed: a caller resolves a File, a saved version or a recorded Position to that signature before asking, and content the Library cannot name is never requested. Results distinguish READY from NOT_GENERATED. Storage errors stay within the Preview region. Server-generated [resource URLs](../../internal/apis/preview_service.go) bind the content identity and the manifest-owned role; asset requests revalidate that binding. A current original's Preview is read only under the identity its own observation published - `FilesDetail.content_signature`, present while that observation applies - so unknown or changed current originals never borrow saved-version assets. Reads neither admit, hash, observe nor generate content. [Generator tests](../../internal/preview/generator_test.go) and [Preview E2E](../operations/e2e-test.md#preview-update-policy) cover generation separately. +[PreviewService.Get](../../entity/preview.proto) reads metadata independently of Files.Get/List and version pagination. Its request is the content signature itself, because Preview storage is content-addressed: a caller resolves a File, a saved version or a recorded Position to that signature before asking, and content the Library cannot name is never requested. Results distinguish READY from NOT_GENERATED. Storage errors stay within the Preview region. Server-generated [resource URLs](../../internal/apis/preview_service.go) bind the content identity and the manifest-owned role; asset requests revalidate that binding. A current original's Preview is read only under the identity its own observation published - `FilesDetail.content_signature`, present while that observation applies - so unknown or changed current originals never borrow saved-version assets. Reads neither admit, hash, observe nor generate content. [Generator tests](../../internal/preview/generator_test.go) and [Preview E2E](../development/e2e-test.md#preview-update-policy) cover generation separately. ## Asset State diff --git a/docs/architecture/read-planning.md b/docs/architecture/read-planning.md new file mode 100644 index 0000000..5caf428 --- /dev/null +++ b/docs/architecture/read-planning.md @@ -0,0 +1,243 @@ +# Tape Read Planning + +How Tape reads choose an order: Logical and Auto read order, the cost model and its parameter +sources, passive read learning and position maps. The +[Tape contract](tape.md) owns cartridge identity, formats and Sessions. + +## Software Read Ordering + +The Device's LOGICAL/AUTO preference and parameters are frozen at admission. LOGICAL retains the +manifest order, eligible member seek and the same per-member cost decisions as AUTO. AUTO plans +all selected pending reads on the loaded Media after identification and Session preparation, before +feeding ACP. Verified local Restore outputs are excluded first. Other cartridges and explicit Media +attempts get their own plan; Library edits do not retarget a Job's frozen content or locators. +Scan with `VERIFY_COPIES` plans its complete frozen baseline without a fresh Inventory traversal. +Other Scan policies retain their enumeration/read flow. `read_batch` controls delivery to ACP, +not optimization scope. Native source paths remain native ACP inputs. +Native paths can still supply Media extents for planning; endpoint close tracking does not determine +whether those extents are usable. + +LTFS prefers the extents already frozen with the selection. When they are missing, including a +migrated `ltfs_v0` Tape without an old XML capture, the same planning database can fill only the +missing ranges from this mount's captured Index. The parser verifies the mounted volume UUID and +Index generation, scans once, and supplies path/size/extent entries. Bounded batch joins retain +only selected matches; duplicate paths, size differences and incomplete parses keep the complete +LOGICAL route. Existing valid ranges are not replaced. No Job or Library locator is rewritten, +and ACP still receives the original native paths. The whole preparation shares the planning +budget and disposable index; a fully located selection does not open the captured XML. +Extents must fit the declared file length; sparse holes remain valid. Exceeding the bounded Index +parser's limits retains complete LOGICAL reading and does not restrict ordinary inventory parsing. +Native Scan sources retain their Media path and size for this lookup independently of ACP paths. + +The [planner](../../internal/media/readplan/plan.go) orders one complete selection in a disposable +SQLite index under `Work/temporary`; Job and Library locators are never rewritten and ACP receives the +original sources. Providers must quote every member transition, a certified incoming-independent exit +state, spatial neighbourhood coordinates and the terminal action. The search starts from the complete +LOGICAL route and adopts only a lower complete-route estimate that covers every member exactly once; +it is a deterministic heuristic, not a proof of optimality. A candidate move whose cost is unknown is +skipped; LOGICAL is kept with a recorded reason when a member's range, exit state or spatial +coordinate is unknown, when the current route cannot be priced, or when no lower route is found. + +The total includes the prepared initial position to the first file, complete target reads, each +later transition and the provider's terminal action from the forecast exit. Tar preparation reads +the identity label and establishes a qualified host-next address; its terminal action is +synchronous unload, priced as fixed work plus the physical longitudinal fraction times full-span +rewind time. Missing completion parameters keep LOGICAL. LTFS delegates completion to the configured +unmount adapter and prices rewind/unload from the final forecast position, assuming completion does +not rewrite the Index; an adapter-caused Index rewrite lies outside this estimate. Scores are +route-comparison estimates, not elapsed-time guarantees. A provider with an unknown initial position +fixes the first LOGICAL file and compares only the remainder; otherwise an unknown origin keeps LOGICAL. + +One planning budget covers indexing and search: absent `read_planning_seconds` allows 300 seconds; +explicit values are positive. Expiry keeps the last completely verified route or the LOGICAL +manifest, never a partial selection. Cancellation, index corruption and temporary-disk failure remain +errors. Closing the index removes it; abandoned directories follow the shared seven-day cleanup +policy. After planning or its fallback, the Session records the effective order (AUTO only when a +lower route was adopted) and its reason in the Device read-optimization status, and the attempt's +read-order log records `planning_ns`. + +The [cost model](../../internal/media/read_cost.go) retains all seven CERN factors: intercept, +wrap, band, landing zone, direction, same-wrap step-back and longitudinal distance. Signed terms +are regression coefficients; a negative term is not an independent mechanical benefit. Nonfinite, +negative or overflowing total predictions cannot quote elapsed time. Absolute LPOS interpolation +uses the actual interval midpoint `min + (max-min)/2`. Alternatively, +`longitudinal_seconds_per_wrap` prices one whole longitudinal tape span using normalized coordinates +0–1 and midpoint 0.5; it is mutually exclusive with seconds per LPOS. Reverse wraps reflect that +coordinate. Its estimates are not guaranteed latency bounds. + +Parameter priority is manual read parameters, learned read parameters matching +all observed conditions, then source-labelled built-in initial values. Generation/density sources +supply documented layout and native-rate references. The generic seven-term timing seed comes from +the recorded HP LTO-5 experiment; it is an unmeasured borrowed prior for other readers, not an LTO +standard or portable validation result. Missing serial, firmware, density, record or written control +facts remain unknown. Partial manual input is never silently filled from learning. Hardware-compressed +written content retains ordinary reading and exact seek but is excluded from linear smart ordering. +Changing current drive compression does not change the cartridge's written policy. +The optional `completion` parameters keep full-span rewind and fixed unload work separate. Built-in +completion values borrow provisional HP references; they are not generation-wide mechanical guarantees. + +Passive read learning defaults on and adds no measurement command. The native transport emits completed +scalar observations into a bounded 256-entry queue; overflow drops observations and increments a +counter without waiting. The consumer only buffers records in memory; nothing is persisted or fitted +while the record lane is active. Sequence gaps and failed commands withdraw the inferred logical +cursor, idle time and preceding action, so only actions whose endpoints were known are recorded. +LOCATE(10)/LOCATE(16), REWIND and synchronous UNLOAD each form one record. A first READ after LOCATE +and a READ that crosses a filemark together with its following data READ are single records. Other +data READs form a READ segment: the contiguous READs between any other commands, split only where +the discard label changes, never by time. Successful nonmoving restoration preserves the cursor but +ends READ continuity. SPACE is not decoded, so it withdraws nothing and records nothing; command +observations do not expose RECOVERED ERROR sense, so records carry no recovered-error count. + +Each record keeps raw facts only: the logical address of both endpoints (partition, logical object +and file number when known), their wrap and longitudinal offset within the wrap's interpolation span +as resolved by the Session's position map, that map's source, observation time and LPOS bounds, the +SCSI opcode, action, preceding action, absolute start, host idle time before the action, and for +READ records the command count, bytes, device seconds, host wait, largest host gap and discard label. +Each record also names its device and applicability. Band, direction and access features are derived +at fit time from the wrap and the generation layout through the same feature function the cost model +uses. After native I/O settles and the drive is released, the attempt's records are appended to its +Job bundle's [observation file](persistence.md#job-bundle). A background refit, serialized per device +and bounded by the process lifetime, then streams the observation files of every Job in the catalog, +keeps those of the same device and applicability, and refits every action: + +- LOCATE uses the seven-term model in the base parameters' longitudinal unit. +- READ uses seconds = bytes / rate, plus the first-READ or filemark overhead for those records. +- UNLOAD uses fixed work plus the physical fraction of one full-span rewind; it never relabels LOCATE + coefficients as rewind time. + +The fit minimizes weighted absolute deviations. Each record weighs one over its Job's record count for +that action, so every Job carries equal weight and one abnormal Job among N moves the result by about +1/N. The solve uses iteratively reweighted least squares with a fixed iteration bound and a +deterministic stopping rule; the same observations always give the same parameters. A coefficient is updated +only when its weighted feature column is independent under the scaled/reorthogonalized rank check, +and only when the result is a valid parameter (positive rate and full span, nonnegative overheads, +fixed work and longitudinal slope); otherwise it keeps the base value. The published model records +the observation count and the unweighted mean and maximum absolute residual over those observations. +Isolated Device Settings writes publish it for subsequent tasks, replacing the stored model read at +the start of that refit. Deleting a Job removes its observations from later refits; there is no other +retention. A write or refit failure only logs a warning; manual preferences remain unchanged. Turning +learning off records and publishes nothing; existing observations remain with their Jobs. + +## LTFS Reads + +LTFS uses the same planner with its existing file extents and native ACP paths. Optional device +facts are acquired under the existing owner before mount; that raw descriptor closes before LTFS +owns physical I/O. AUTO preparation can read each physical partition's VOL1/filemark/XML Label/ +filemark prefix using qualified CP LOCATE and variable READ, without changing controls or the +adapter-installed key. Two complete, agreeing Labels establish the symbol-to-physical mapping; +the bounded reader does not scan file contents. Its two partition-start seeks have separate +preparation cost, recorded in the Job log. The XML cap is 512 KiB per Label, not a format limit. +Commands use a twenty-minute native timeout and retain ownership until settlement. Unavailable +host buffers, unsupported forms or unusable complete Labels withdraw this optional source; +cancellation, native I/O faults and uncertain completion remain errors. Cleanup must finish before mount. + +Mounted LTFS supplies the written block size, compression policy, volume UUID and partition roles, +which must agree with the pre-mount Labels. Its current index generation binds retained geometry; +Labels contain no index generation. Mount can reposition, so its initial cursor remains unknown. +The read-only `indexLocation` must place the current Index on the Index partition for this normal +completion comparison. A missing/invalid pointer, unknown matching data-partition EOD or incomplete +completion parameters withdraw optimization. The old Index start is never substituted for EOD. +`ltfs.sync` is not a probe: reading it can write an Index and is not used. +When the raw source is unavailable, unambiguous mounted MAM +selectors remain usable. Contradictory sources never substitute for one another. Partition letters +and capacity ratios alone do not establish addresses. Missing or conflicting facts retain LOGICAL +without changing native reads. No raw command is issued after LTFS mounts. + +The LTFS evaluator visits a file's extents in file-offset order, retaining sparse gaps and partial +block uncertainty. It prices the demonstrated last-block cache, adjacent reads and ordinary seeks; +it grants no unimplemented gap-discard benefit. The mounted initial cursor is unknown, so every +route keeps the same complete first file. This terminal model only prices data-partition content; +other or cross-partition selections retain LOGICAL. Matching native learned read parameters can supply mechanical priors, but do not validate LTFS +whole-file latency. LTFS has no whole-file passive read learning and no ACP source-timing +extension. Finalization +invalidates the Session's model before another attempt can use it. + +## Position Maps + +Position maps belong to Media and contain spatial facts, source, binding and uncertainty. A live +complete REOWP report takes priority. Read preparation tries cartridge memory once on the owned +HP LTO-5 descriptor if REOWP has no complete directory; ordinary device Inspect does not read CM. +The READ BUFFER descriptor bounds the image to 8 KiB and each transfer to 4 KiB. Unsupported, +malformed or inconsistent CM leaves matching retained geometry available; cancellation, uncertain +completion and medium changes remain preparation errors. No failure triggers LOAD or another motion. +LTFS obtains the same facts before mounting and closes the raw owner before native reads. + +The [CM decoder](../../internal/media/tape/cm_directory.go) accepts the studied version-4 directory +and version-2 EOD pages for one or two partitions. Checksums protect the tables, directory header, +each entry and each EOD page. The device's confirmed partition count selects active EOD pages; +an unused second page can remain after single-partition FORMAT and does not create a partition. +Tar reuses its configured actual count; pre-mount CM fallback otherwise reads the bounded MAM +partition list once. Unknown partition identity withdraws CM. Complete valid runs must each match exactly one completed EOD's +terminal dataset/write pass and separate record/filemark totals. Partition starts come from these +runs, never cartridge identity or nominal capacity. Unexplained extended counts and other formats +withdraw this source. The [research](../research/tape-read-ordering.md#direct-physical-coordinates) +states its empirical scope and remaining limits. CM cumulative object counts use exclusive bounds. +The paired HP Ultrium 5-SCSI/Z6ED density-0x58 REOWP response uses the same qualified count convention; +other readers retain their existing reported-object interpretation. Earlier inclusive REOWP maps +from that HP5 configuration require a new observation before reuse. Source selection never changes a saved +file's logical locator. Accepted derived geometry uses the existing Media map and session lifetime; +no raw CM image or per-file wrap is persisted. A new read after APPEND observes the current directory. + +When timing parameters omit nominal layout, +a self-described tar map remains usable only if current full boundaries agree and current EOD covers +its saved prefix; confirmation never extends retained coverage. Tar maps bind to initialization ID; LTFS maps +bind to volume UUID and index generation. Each map owns its explicit endpoint convention. Missing +or ambiguous coverage refuses cost estimates without changing exact logical seek addresses. Session +invalidation withdraws live estimates. The four-band, alternating-wrap interpolation assumption uses +a sourced nominal wrap count; complete intervals use their own boundaries. A partial final interval +uses the greater of its observed length and typical complete interval length for scale, while actual +EOD still bounds readable data. Locations whose wrap classification is ambiguous remain unknown. + +Tar WRITE geometry inference uses existing start/final position checks and completed WRITE facts, +with no new command. The consumer bins native time by cumulative confirmed bytes, separately retains +host-supply gaps, checks first-half frequency, recurrence, phase and harmonics, and rejects starved, +constant-slow or continuously backhitching signals. Fixed bin storage is bounded; raw observations +are discarded after deriving the map. A successful APPEND preserves the complete prefix and replaces +only the affected tail; insufficient evidence retains old coverage without claiming the new tail. +Rejected inputs allocate no byte-bin storage. +Unknown acceptance never becomes inferred written bytes. Map publication follows physical cleanup, +and failure to save optional geometry cannot fail content publication. The [research evidence](../research/tar-tape.md) +separates the same-cartridge blind replay from prospective hardware qualification. + +Tar planning and execution share one full-range evaluator for retained-record reuse, continued READ +with gap discard/filemarks, and LOCATE plus target reads. Headers, padding, cross-record ends and +actual host-next/cache state are included. Missing seek costs never justify reading an avoidable +gap; immediately adjacent or retained records can still be reused. Actual reads validate header, +path, content length and padding. A safe state mismatch withdraws further estimates; physical I/O +failure retains the existing Tape error/cleanup boundary. A planned sequence remains a fixed estimate, +not a physical cursor observation. The optimizer is a deterministic heuristic, not a global optimum. + +Passive REOWP inspection scopes the A3/1F/45 maintenance CDB family to IBM and HP/HPE Ultrium +devices, using independently documented IBM and HPE response framing. Standard RSOC reports retain +exact service-action and field-mask checks. A complete vendor-specific support-5 report for HP/HPE +Ultrium permits one bounded report-all query on loaded, ready media; rejection never triggers a +second form. Other vendor/product families remain unknown without a vendor query. A successful +query supplies OBSERVED evidence, with physical validation UNVALIDATED. Typed observations expose the reported +long command form and source, with count only after an interpretable response; these do not prove +full-wrap coverage. Successful querying +alone leaves freshness unknown. The live reported observation preserves gaps, pair identities and +endpoint semantics, and becomes unusable after its owner writes, initializes, loses certainty or +closes. Direction, band, full-wrap normalization and empirical costs require separate applicable +evidence. + +## Position-Map Refresh + +Explicit position-map refresh precedes read settings/key application and label validation under the +attempt lease. Applicable documented HOLD/LOAD semantics, positively reported exact synchronous +forms, a ready cartridge, usable all-wrap reporting and logical navigation establish eligibility; +physical validation remains separate. The provider requires encryption/decryption off without +foreign control before motion. It invalidates old geometry/navigation, performs one non-ejecting +HOLD, confirms the held/unthreaded state and performs one LOAD. It then reobserves drive identity, +readiness, navigation and the bounded map on the same descriptor. The factory cross-checks the +cartridge identity again before applying the frozen read policy/key and validating the label. +Only that post-HOLD readiness check accepts the exact current held-state response; it does not +close shared command admission. Ordinary readiness loss, uncertain completion and cancellation +still prevent further motion. +Original non-secret controls are captured before motion and retained through ordinary cleanup. +Cancellation or failure never schedules a compensating LOAD, retry or eject; failed restoration +keeps the existing device-unavailability boundary. + +The replacement map is a newly acquired reported observation. The cycle does not establish map +reconstruction, full physical coverage or `Fresh=true`, and supplies no timing model or AUTO +recommendation. AUTO never adds a reload. Restore preserves the explicit action through frozen +selection; Scan accepts it only for frozen tar VERIFY_COPIES. diff --git a/docs/architecture/tape.md b/docs/architecture/tape.md new file mode 100644 index 0000000..cf99cc1 --- /dev/null +++ b/docs/architecture/tape.md @@ -0,0 +1,472 @@ +# Tape + +Tape device observation, the configured script contract, cartridge lifecycle, the sequential tar +format and capacity admission. [Media backends](media-io.md) own the shared Session, Volume and +integrity boundaries; [read planning](read-planning.md) owns read order. + +## Device Observation and Raw Transport + +[Device discovery](../../internal/executor/device_inspect.go) accepts only configured Tape entries. +The configured spelling remains the request and script input; passive Linux sysfs resolution unifies +st, nst, SG and configured symlink aliases under one device resource. A captured lease key survives +loss of discovery and uncertain cleanup. A new observed identity may replace a free resource mapping; +it cannot bypass an existing hold, waiter or retained unavailable guard. Failure to resolve raw access +does not prevent the existing LTFS adapters from using their configured entry. + +ListDevices reports every configured entry with its busy status and reason, using configuration, +sysfs and Executor lease state without SCSI commands. InspectDevice takes a non-waiting lease and +checks ownership before opening raw access. A busy device returns that reason without probing it. +Raw access requires visible host process descriptor tables and a clean negative holder check; +incomplete visibility is unknown ownership, not evidence that the drive is free. This observation +uses the supported same-host namespace and does not coordinate another operator's device access. +Inspection reports identity, readiness, block limits, current mode facts, non-secret encryption state +and available READ POSITION/LOCATE form observations. It sends no positioning, write, mode-setting, key-loading +or measurement command. +Cartridge Inspect remains the distinct LTFS adapter action and can load the cartridge. + +Feature reports separate support, current availability and validation, with typed reasons, limits +and evidence. Missing or unreadable facts remain absent or explicitly unknown. A clean unsupported +optional query does not invalidate other observations. Passive queries do not qualify write, seek, +recovery or cleanup behavior; features whose applicable operation is not established remain +unvalidated. Reports are transient observations, not Library facts or saved configuration. +Archive displays optional-query diagnostics without treating unsupported, malformed or unimplemented +optional observations as a blanket write veto. Known busy, missing-medium, write-protection and +readiness/ownership failures still block its form; leased Executor preparation owns final admission. +The Tape transport owns command status, sense and residual classification, including clean optional +rejection and qualified WRITE warnings. Format backends consume that evidence and retain their +own stop, framing and publication decisions; an incompatible joined error cannot become a safe warning. + +A zero reported maximum block length leaves the maximum absent; it is not a zero-byte limit. +The SG descriptor's reserved buffer can grow for a selected record, so its current allocation is +not reported as a transport maximum. Preparation requests the selected size and checks the actual +readback; an insufficient allocation fails before record I/O. An immutable transport limit can be +reported, while an unknown adapter maximum remains absent. +Position overflow/unknown flags prevent treating a returned object address as exact. Navigation +selects a reported READ POSITION/LOCATE pair before motion, checks its address width and partition, +and never retries another LOCATE form after a real command failure. These follow +the [SSC-3 Revision 01c](https://www.13thmonkey.org/documentation/SCSI/ssc3r01c.pdf) block-limit contract +and the applicable [IBM 2026 reference](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference%20GA32-0928-08%20%28EXTERNAL%29.pdf) +position forms. Holder visibility follows the +[psmisc 23.7 fuser contract](https://gitlab.com/psmisc/psmisc/-/blob/v23.7/doc/fuser.1); an empty +fuser result alone cannot establish permission to observe all holders. + +The [raw Tape provider](../../internal/media/tape) uses Linux SG v3 and a serialized data/control lane. +Recurring capacity observations alone use a private descriptor under the same attempt lease; +ordinary passive inspection remains separate. +Read/write access submits each command header once and receives its matching response separately; +restarting an interrupted receive cannot submit the CDB again. Submission errors and short headers +are never retried. Passive inspection retains read-only SG_IO access and its idempotent query +allowlist. It does not require write permission. The split exchange avoids SG_IO's post-submission +`ERESTARTSYS` behavior under signal handlers with `SA_RESTART`. + +Cancellation before admission issues no command. Once issued, the owner retains the lane and waits +for the native response and any failure settlement before returning or closing its descriptor. +The header timeout remains the kernel request timeout, including its error handling; it is not an +independent wall-clock cancellation deadline. Read/write transports own reusable anonymous memory +for command, sense and record bytes. The kernel retains those stable addresses instead of caller +Go buffers; only a correlated response with valid counts copies returned bytes to the caller. +Normal close clears and releases the allocation. Unknown settlement retains both it and the fd. +These private CLOEXEC descriptors are never forwarded or used across credential changes. +Host/driver failures, invalid accepted-count evidence and unknown completion prevent later command +grants. With concurrent capacity observation, the sole WRITE granted before a fatal query result +may still reach native submission afterward. It must settle, and the attempt publishes nothing. +The Executor retains the affected resource as unavailable until it stops; these failures do not +create Position health findings. Closing a raw transport sends no unload, reset or filemark. + +Record commands distinguish fixed-record counts from variable-record byte counts. Current/deferred +sense, residuals and warning scope govern acceptance; an unexplained residual cannot prove a complete +write. EW handling uses an applicable provider contract, not an LTO-generation assumption. Filemark +completion uses non-immediate commands: count one synchronizes prior data and records the terminating +mark; count zero synchronizes without adding a mark. A qualified current completion warning must prove +the entire requested count completed and remains observable as EW. Other completion errors withhold +publication. These primitives and their synthetic tests do not establish physical qualification. +LTFS and Volume retain their native I/O paths; tar uses the raw provider under the same attempt lease. + +## Tape Script Compatibility + +Configured scripts are a long-lived user interface. Preserve previously supported adapters wherever +possible, including their helpers, input variables, output format, working directory and exit-status +meaning. LTFS distribution and hardware differences remain inside those adapters. A necessary +incompatible change requires explicit developer approval, an actionable installer notice before +upgrade and regression checks of the changed boundary. Installation preserves customized adapters +and their execution context instead of replacing them automatically. This contract applies +independently of the pre-stable API/data policy and preserves the [Tape lifecycle](#tape-lifecycle). + +Script changes require automated checks of the published interface and representative unchanged +historical adapters. Physical acceptance checks the selected host adapter separately; it does not +replace the historical compatibility checks. + +LTFS Archive requires a completed per-Job captured Index; historical adapters without that output +need manual adaptation for Archive. This required incompatibility is described in the +[upgrade notice](../operations/migration.md#tape-script-adaptation) displayed by the installer. + +## Tape + +The immutable Tape profile selects LTFS or `tar_pax_v1`. FORMAT resolves explicit choices and the +selected device's frozen defaults; APPEND and reads use the stored profile. Unset format defaults to +LTFS. Tar defaults to variable records with a 512 KiB (524288-byte) maximum, overridden by saved +Device settings and then explicit FORMAT values; device/transport limits remain mandatory. +APPEND and reads retain the recorded policy. Tar options cannot be supplied to LTFS, and conflicting +APPEND options fail before access. +Compression omission survives admission so leased preparation can apply a qualified recommendation +under the [shared precedence](contracts.md#tape-options-and-selection); without one its final +profile defaults off. [Encryption](#encryption) has no per-device or per-write switch. + +The Device capacity-polling interval is an execution preference for tar FORMAT and APPEND, +independent of the cartridge profile. Its admitted snapshot resolves omission to five seconds, +explicit zero to Off, and other values to an interval of at least five seconds. Saving it performs +no hardware operation and cannot change an active attempt. + +Configured Tape scripts are the anti-corruption boundary for environment-specific executable paths, device mapping, vendor options and helper logic. The service supplies operation inputs and validates identity, content and final Index output; it does not absorb those local decisions into configuration switches. The [Tape scripts guide](../operations/tape-scripts.md) owns their inputs, outputs and completion requirements; the [upgrade guide](../operations/migration.md#tape-script-adaptation) owns v0.1.x adaptation. Installer preflight never invokes these scripts. + +Operators may edit configured scripts and their helpers to match local software versions, +platforms and hardware. Bundled commands are starting points; local changes retain the +published inputs, outputs, exit-status meaning and physical finalization requirements. + +The format script passes the resolved device, barcode and complete Tape name as separate literal +arguments to `mkltfs`; spaces and shell pattern characters in the name do not split or expand it. +Regression checks execute the shipped script with a stub LTFS command, while physical acceptance +uses the packaged script with only the host executable path and test placement rule adapted. + +The Linux `get_device` adapter resolves the Tape node through the kernel's sysfs mapping to its +SCSI generic node. This lookup does not open the drive, so it still works while LTFS owns the +device and during unmount; an absent or ambiguous mapping fails explicitly. + +The [Tape scripts guide](../operations/tape-scripts.md#bundled-templates) describes the bundled templates and +the file-backend test adapters. + +[LTFS Sessions](../../internal/executor/media_tape.go) identify the cartridge, configure encryption, format when requested, mount LTFS, and unmount/finalize. FORMAT accepts a successful probe reporting an empty cartridge memory barcode and uses the explicit user-supplied six-character barcode; the format script passes that identity to `mkltfs` to write it during initialization. A non-empty device barcode must match the requested identity, and probe failures stop the Session before encryption or formatting. An empty barcode does not establish that the cartridge contains no data: FORMAT remains an explicit destructive operation. FORMAT rejects a barcode already in the Library; explicit metadata deletion is required before creating a new Media identity. + +LTFS APPEND and Restore require a non-empty valid device barcode before encryption or mounting; APPEND also requires an exact match with the requested identity. Device inspection may use its documented read-only manual identity fallback when automatic MAM lookup is unavailable, but that fallback does not authorize a write or Restore. APPEND requires compatible `ltfs_v1`, preserves Media ID/key/Positions, and chooses a short base36 timestamp prefix only to avoid path collisions. + +### Tape Lifecycle + +The configured LTFS adapter lifecycle remains: + +1. For an uninitialized cartridge, the operator chooses its six-character barcode and explicitly + submits FORMAT. The cartridge memory barcode may be empty; the formatting script writes the supplied + identity and name while initializing LTFS. An empty barcode does not establish empty content. +2. Archive writes files, normally unmounts and waits for device release, then validates the final + captured Index and publishes Media and Positions. The Library records the Tape's identity and key. +3. On later loads, APPEND verifies the existing barcode and compatible Library Media, reuses its key + and adds files without replacing earlier Positions. Restore verifies the loaded Tape against its + recorded identity, uses its key and reads the selected saved content. +4. Metadata deletion leaves the cartridge's bytes intact. Reinitialization requires explicit FORMAT + after deleting the registered Media; formatting erases the old physical contents. + +Script and caller changes must keep every step available, including FORMAT before a cartridge memory +barcode exists. Tar FORMAT follows the same first step natively, writing the supplied barcode into +the cartridge memory as described in [Sequential tar Tape](#sequential-tar-tape). Per-operation +identity and finalization rules remain defined above and below. + +The final captured LTFS Index is the physical fact source. [LTFS processing](../../internal/media) retains backend extents in the typed storage-metadata envelope and stores opaque binary order from partition/block/offset. `ltfs_v0` is path-order compatibility; `ltfs_v1` has validated LTFS order/extents and supports append. Versioning belongs to the storage format, not a second independent version field. + +After physical Finalize, Archive matches staged files against the returned Index by actual Media path, +size and valid extents. On normal completion every staged result must match. On target-full it submits +only the continuous verified prefix and resets the later suffix to PENDING. A successful normal unmount +includes waiting up to ten minutes for the LTFS process to release the SG device and for `mt status` to +report `DR_OPEN`; the Job attempt, device lease and Job bundle remain owned throughout this wait. An +unmount timeout or invalid/missing final Index publishes no unverified result. An uncertain device +remains unavailable for that Executor lifetime. Force and lazy unmount are manual stopped-service +recovery tools whose write result must be discarded. Ordinary LTFS files stay readable without a +YATM-specific segmented format. + +LTFS Read Sessions validate the expected Tape before mounting and keep the attempt's device lease +through normal unmount. Finalize does not run `readinfo` again: that script loads the cartridge, +which must not occur while LTFS owns it. Successful reads become publishable only after normal +unmount succeeds; a failed unmount still withholds publication and keeps the device unavailable. + +### Sequential tar Tape + +[Tar Sessions](../../internal/executor/media_tar.go) use a known single data partition and fixed or +variable records. Physical Tape file zero is an ordinary PAX tar containing `YATM/identity.json` +and `YATM/README.txt`; it records the barcode, name, random initialization ID and immutable record +profile. Each later physical Tape file is an independent PAX archive for one Archive attempt. +Members use `archives/<16-digit hexadecimal tape-file-number>/`, preserving +separate saved versions during independent extraction. The label is identity/recovery data, not a +content index. The [recovery guide](../operations/tar-tape-recovery.md) owns the public-tool procedure. + +Every tar FORMAT, APPEND and read runs the configured encryption adapter; there is no plaintext tar +path. Writes establish the selected hardware compression policy; the immutable profile records that choice. +Writes also require known rewritable, unprotected media and applicable buffering/EOD generation; +the baseline does not offer WORM writes. +Known-absent features need no unsupported control command. Reads require the record/decoding policy +without writable-media or write-compression prerequisites. Compressed reads independently establish +hardware decompression. New tar FORMAT generates a key and invokes the configured Encrypt script; +APPEND and reads restore the stored key, and a tar Media without one is refused before device access. +Software read ordering uses the qualified +provider boundary below and otherwise retains logical order. +Optional cached capacity admission +does not replace EW/error handling; unknown capacity does not prevent writing. Record choices must fit observed block limits +and the selected transport allocation. Generic applicable SSC forms can establish eligibility without +an exact-model allowlist, but unknown required semantics or failed safety evidence refuses access. +Passive observations remain UNVALIDATED and supply no physically validated write recommendations. + +FORMAT remains explicit and destructive and rejects a registered barcode. Its record profile and +key use the existing Media/Job publication path; failure does not create a separate recovery copy. As in LTFS, the +cartridge memory barcode is the Tape identity. FORMAT accepts an absent barcode, which does not imply a blank cartridge, and a +reported conflicting barcode fails. After initialization, FORMAT writes an absent barcode as the +requested identity with WRITE ATTRIBUTE, as `mkltfs -s` does: attribute 0806h in ASCII format, a +32-byte value left-aligned and space-padded. It reads the attribute back and requires equality +before the Encrypt script and the label; a present matching barcode is not rewritten. A write or +readback failure stops FORMAT before any data write, without retry or fallback. APPEND and reads +require a matching cartridge memory barcode. The label verifies the selected Media and never substitutes for that identity. +Actual partition-list evidence confirms committed layout; mode-page counts may remain unchanged until +FORMAT. A required transition from multipartition media or a nonzero active partition needs an +applicable supported form before mutation. Initialization establishes partition zero/BOT and a known single partition; +the new label's synchronized terminating filemark establishes the fresh recorded boundary. This +replaces the recorded archive layout rather than claiming to erase every physical byte. A proven EW +on the label's filemark closes content admission while permitting zero-member Media settlement, +without another data WRITE or filemark. + +APPEND checks the registered label/initialization/profile before proving the physical tail. It +requires the long READ POSITION/LOCATE pair, issues SPACE to EOD (mandatory in SSC) without a +support query, obtains the actual object/filemark counters at EOD, locates the preceding object, +and verifies a zero-byte filemark followed by zero-byte EOD. It restores and checks the proven +post-filemark position before enabling write controls. This needs two boundary READs regardless +of the written prefix. A missing long position form, a rejected command, an EOD position without +object or file counters, failed motion or a contradictory tail fails APPEND without retry or a +record/filemark walk from BOT. The proof never uses CM or catalog counts as write authority or +rewrites the previous tar terminator. A capacity/EW-terminated archive ending at a proven filemark can precede another archive. Unterminated trailing records, unreadable required +boundary data and unresolved final position refuse APPEND. Existing Media and Positions are preserved. + +The [codec](../../internal/media/tar_write.go) packs members across records without per-member device +flush/sync. It uses standard Go PAX framing, regular files, permissions including special mode bits, +subsecond mtime and local `SCHILY.xattr` records with raw empty/binary values. ACP excludes its managed +cache keys before supplying ordinary metadata; the codec owns PAX representation and encoded-header +limits. Metadata/header allocation is bounded; oversized or unsupported metadata fails before +member emission. Reference construction may run ahead of writing; dynamic admission belongs to +the ordered target Open lane, without reading mutable writer state from the feeder. A prepared +label warning can reject references immediately. Global PAX state, sparse extensions and links are unsupported. Sequential locators +contain the actual Tape file ordinal and header-chain byte offset. Optional paired header/exclusive-end +anchors use partition zero, a logical object and an offset within its actual record. Their containing +record's tar offset cannot underflow. The Tape provider owns eligibility for the positively reported +standard READ POSITION/LOCATE pairing, partition-zero scope and address width. Its logical-object +contract counts records and filemarks independently of compression or record byte lengths. Missing +pairing or applicable failed evidence withdraws optional navigation. Physical validation remains +UNVALIDATED until its own acceptance; it is not another protocol-eligibility switch. + +With an eligible navigator, the writer captures one fresh start position before content emission, +counts accepted complete records and checks the expected next object after its synchronized +terminating filemark. It also checks the file ordinal when the device reports it exactly. Member +boundaries are retained on existing target references and included in Archive's ordinary staged item +results; there is no second member inventory, whole-Tape record map or per-member position query. A clean unsupported observation or valid unknown address omits both +anchors while retaining otherwise proven sequential content. Contradictory or uncertain position +evidence makes finalization unusable. Logical counters and command-width bounds are checked. +Exhausting an optional address width omits the attempt's anchors while preserving sequential +completion; it does not wrap a logical object or reject otherwise valid content. An observed +early-warning position closes content admission under the same physical completion rules. + +Successful member Finish proves complete header/payload/zero padding, separately from physical record acceptance. +Archive's existing item results retain provisional positioning after ACP and member framing succeed. +Two optional scalar header offsets track the last successful member still buffered and the last successful +member covered by fully accepted records. Byte zero remains valid; an active unfinished member cannot +advance confirmation. Finalize requires synchronized count-one filemark completion and verified cleanup +before its frozen validator admits staged positions on normal completion or an exclusively +capacity/EW stop. Archive applies it in the existing path-update pass, +including optional-anchor withdrawal and final short-record normalization. Staged rows cannot authorize +restart publication; a new attempt resets them. +Successful normal completion or an exclusively capacity/EW stop also requires synchronous unload +before publication. An independent failure or cancellation uses control cleanup without unload; +the runner still retains its original content/stop result. +Archive joins those path/size/storage facts with successful STAGED content results. Complete members +can survive source-open gaps; they need not form a continuous manifest prefix. Successful physical +finalization makes a newly initialized Media publishable even when no content member survives. + +An exclusively capacity/EW stop closes admission and discards the unsubmitted suffix without +adding payload or a tar trailer. Its completed filemark and unload can admit only fully accepted, +complete members. Cancellation, a source failure after a member starts, or another independent +failure stops the attempt, runs ordinary cleanup and returns no usable result from that Archive. +It does not add a filemark to salvage a prefix, repair a tail or resume the interrupted stream. +Completed command observations remain diagnostic facts; they do not authorize publication. Unknown +or partial acceptance, deferred/hardware errors and failed completion or cleanup also withhold publication. Qualified non-immediate count-zero synchronization or +count-one filemark EW can establish completion from current NO SENSE/EOM command semantics; +absent sense information does not contradict that success, while a valid remainder must be zero. +Data WRITE acceptance still requires qualified valid transferred-unit evidence. Contradictory native +status/residuals or mixed errors remain failures. The runner retains the copy/stop cause independently +when physical finalization succeeds. An EW first reported by the completing filemark also reaches +the runner as the stop reason without discarding completed members. Normal completion writes two zero tar blocks, then a short final +variable record or legal fixed-record padding. + +#### Tar Capacity Admission + +Capacity admission uses a transient cached native-space allowance. Preparation can observe the +current partition's recorded-EOD-to-EW estimate from applicable page 0C parameters 4 and 5; current +head position, nominal capacity and logical object counts cannot substitute. Missing/invalid counters +retain the sequential baseline. Clean optional query failures remain feature-local; observed medium +loss/not-ready state or uncertain ownership/command settlement invalidates the affected attempt. +The existing current-mode active partition can associate a passive sample without logical-position +support; conflicting partition observations leave capacity unknown. + +Ordered target Open reserves the actual source metadata size, encoded PAX/header chain and 512-byte +payload padding before the first header. Reference creation performs no reservation or query. +Cumulative native cost counts shared records once; failed members do not reclaim Tape space. +A separate reserve covers tar end blocks, fixed-record final padding and the terminating filemark. +A cached refusal admits no bytes of the next member and closes a normal tar from that reserve; +EW during WRITE overrides the budget and stops further data/trailer writes. + +The optional observer owns at most one query. On Linux, its first recurring observation lazily +opens one private SG descriptor with independent native buffers and response matching. Only the +exact current-value page 0C query is allowed there, with kernel queue-tail placement. Explicit Off +never opens it. The writer does not wait for this query's I/O or an I/O-spanning software lock. +The kernel, HBA and drive remain shared, so this does not promise physical zero delay. +Its issue callback captures the cumulative cost of WRITEs already known to have returned full +acceptance. Atomic publication deducts all reservations beyond that prefix, completion reserve and +known error/unreflected allowances from an absolute sample, saturating at zero. An overlapping +WRITE remains charged even if its before-submission callback has run; a conservative double debit +is possible. No admission lock spans I/O; finalization joins the observer before physical +completion, control/key cleanup or descriptor release. +Live accounting requires an explicit transport issue-callback contract for both queries and WRITE; +an Execute-only transport retains ordinary records but cannot promise that accounting boundary. +Both descriptors share terminal failure admission. A clean unsupported query, or a clean optional +attachment rejection before any command, disables recurrence. Preparation, passive Inspect and +recurring observations use the LTFS LTO LOG SENSE default native timeout of sixty seconds, without +an additional sample deadline or failure-count cutoff. The timeout is not a wall-clock cleanup +bound through kernel error handling; cancellation never releases an issued request early. +Inspection admits its finite query sequence under the caller's context and each native timeout; +it adds no shorter whole-inspection deadline that could promote an optional miss to device failure. +A matched, settled capacity-only timeout or busy response, undecodable counters within a GOOD +native response, or an unknown counter value leaves cached debits and EW handling intact. +Clean local descriptor/memory exhaustion while opening the private query attachment also leaves +WRITE available and retries attachment on the next poll; no query was submitted. +Capacity polling continues at the configured cadence and can recover after any number of local misses. +One worker issues its first observation immediately and then follows a ticker. An outstanding query +prevents another from starting; missed ticks are dropped, with at most one due tick consumed immediately +after a slow query. There is no retry backlog, separate retry loop or exponential backoff. +The observer logs failed queries and missing values at warning level with the consecutive failure +count, and logs recovery at info level. Busy admission and ordinary shutdown cancellation are quiet. +Logging uses the existing Job logger outside capacity-accounting and native WRITE locks. +Unknown native outcome, reset, deferred write errors, medium loss/not-ready and other unclassified +query/attachment failures still invalidate the attempt. A query-local error cannot hide a joined +failure. Finalization joins the issued request before completion and cleanup. + +Only fresh lower-bound observations with known allowances and bounded native costs establish bounded +admission. Estimates cannot promise that a member fits or increase an applicable bounded budget. +The current provider supplies estimates with unknown error, buffer lag and physical overhead. +Production retains the preparation observation and cached debits. Unless the optional query is +confirmed unsupported, the admitted polling interval enables background capacity polling even when +preparation could not obtain a value. Explicit Off retains only the preparation/debit path. +Unsupported queries retain EW/error handling without a polling loop. +This protocol-based policy does not certify device performance or physical costs. Unknown freshness +cannot establish a lower bound; the [capacity study](../research/tape-capacity.md) explains the +separate measurement limits. +The worker and data path remain independent; per-file admission consumes cached facts only. +Polling remains advisory and does not establish a bound on native counter lag or physical costs. + +[Reads](../../internal/media/tar_read.go) without eligible anchors rewind/space to the containing Tape file and reuse +the forward cursor for selected members. They validate the stored header boundary/path, stream actual +payload and require observed legal padding at Close; incomplete/failed Close invalidates the cursor. +Eligible selected reads locate the containing header record, check its actual length, parse the +PAX/header chain and compare the observed exclusive member end at Close. The session retains one +record for adjacent or earlier headers in those same bytes; record lengths need not equal the +configured maximum. Failed motion or record reads stop the source lane, and later unstarted reads +issue no retry or fallback commands. Session-wide unavailability does not create per-file corruption +findings; it does not by itself imply an uncertain hardware outcome. Read cleanup remains a separate +publication gate for previously successful reads. +Fresh inventory traverses every content Tape file and excludes only a readable incomplete tail at a +clean filemark. Actual I/O faults or an unterminated recorded tail fail enumeration; complete inventory +and successful read Finalize are both required before replacement publication. Reads emit no filemark. +Automatic tar selection reads the cartridge memory barcode under the lease and resolves it against frozen +candidates; a Tape without one fails identification. Known LTFS selections retain +their configured adapter flow without requiring SG permissions. + +Preparation captures touched non-secret session controls before the first mutation, uses MODE SELECT +with SP clear and verifies readback. Both session and partition selection preserve buffering and speed. +Cleanup restores captured record/compression controls and uncommitted pending partition controls after +commands settle; it never saves defaults or resets. Normal tar Archive, Restore and Scan completion +then issues one synchronous UNLOAD, which owns rewind/unthread/eject; no extra REWIND is added. +Failed or canceled work and transport Close do not issue that motion. Restoration failure prevents +unload. Required release failure or preserved command uncertainty retains device unavailability. +Synthetic record/SG tests and GNU tar disk interoperability establish software behavior. Physical +acceptance separately covers initialization, EW/count/completion, reload-stable seek, cleanup and +Linux public-reader recovery for each tested record policy and device path. Discovery, a local +commit or disk extraction does not establish those gates. + +### Hardware Compression + +Hardware compression uses the SSC data-compression page's DCC capability and DCE write setting, +independently of its DDE read setting. The legacy device-configuration compression selector is +resolved consistently with DCE for writes; APPEND first validates the label and recorded tail under +read controls, then applies the cartridge's write policy. READ preserves write DCE and does not +require writable media or a changeable write-compression setting. A compressed profile requires +enabled hardware decoding with supported exception handling before reading its label. Plain reads +remain usable on devices without that optional capability. + +The inspection report exposes the observed numeric algorithm identifier and optional enable-bit +changeability through `TapeCompressionLimits`, reusing the existing current-enabled observations. +An observed algorithm is neither a supported-algorithm list nor a compatibility guarantee for another +cartridge. Algorithms are preserved rather than selected by name or LTO generation. Required +controls need known changeability when changed and exact readback. Cleanup restores only touched +controls and preserves actual partition state and any last-read decompression algorithm. It never +saves power-on defaults. Unknown active write settings fail preparation; a known absent compressor +with explicit Off needs no unsupported compression control. No software compression is introduced, +and the capacity estimate grants no expected compression savings. Physical compression, cleanup +and compatible Linux-reader recovery qualification remains pending. +Job diagnostics distinguish the immutable write choice from effective write compression and read +decompression in FORMAT, READ and both APPEND preparation phases. Algorithm observations are +refreshed from existing current-mode queries; missing values remain unknown. Unexpected write-algorithm +drift cannot count as verified cleanup, while normal last-read decoder algorithm changes are retained. + +### Encryption + +The configured `Scripts.Encrypt` adapter owns encryption setup for both LTFS and tar, using the +existing `DEVICE`, `KEY_FILE`, `TAPE_BARCODE` and `TAPE_NAME` inputs and the service working directory. +FORMAT generates the existing `v1:` plus 64-hex-digit key; successful publication retains it in +Media and the normal Job profile. APPEND and reads restore that saved key for the adapter. The +Executor supplies an owner-only (0600) temporary key file and removes it through ordinary cleanup. +There is no additional persistent recovery descriptor, key copy or failed-FORMAT recovery store. + +The native tar provider owns records, partition/buffering and compression controls, not secret +keys or encryption commands. Read-only encryption diagnostics report available modes, algorithm +and counters; a counter is not key presence. A present encryption protocol with an unnamed +algorithm remains partially observed, rather than unsupported. Malformed optional algorithm +reports retain their diagnostic reason without preventing script-prepared record access. A script-prepared session is not subjected to a second +native key setup/verification/clearing state machine. Optional `TarMediaProfile.encryption_algorithm` +is an observed public fact, not a user control; a saved key does not require this observation, and +an unknown algorithm leaves read-parameter applicability unknown. + +The adapter runs under the attempt lease with raw descriptors closed. FORMAT first finishes +its partition initialization and barcode assignment on the raw descriptor, closes it, invokes Encrypt, +then opens a fresh descriptor, requires the matching barcode and reapplies record controls before +writing the label. APPEND/read first opens the raw device and cross-checks its +barcode (after any explicit position-map refresh), closes it, runs Encrypt with the saved key, then +reopens a fresh raw session and cross-checks the barcode again. Script failure stops preparation. The cartridge memory +barcode selects one saved Media and therefore one key; selection never tries candidate keys. The label contains no key. +Normal independent recovery uses the existing protected Library export and a compatible reader, +as described in the [public-tool guide](../operations/tar-tape-recovery.md). Media deletion removes +its saved profile; Job deletion removes its attachments, including old key files if present. Those +are existing deletion operations, not a new key-retention or recovery workflow. + +### Tape Capacity + +Capacity management must preserve continuous data supply during normal Archive. It must not +introduce recurring pause/drain/synchronize/resume cycles per file or batch; normal finalization +when ending use of the current Tape remains necessary. The +[capacity study](../research/tape-capacity.md#streaming-and-mechanical-wear) explains the mechanical +wear concern and the limits of the available evidence. + +For LTFS, ACP checks a cached available-space estimate immediately before starting each positive-size +linear target, debiting that target's logical size. One independent worker per filesystem starts +an observation immediately and refreshes every five seconds, with at most one query in flight. +Copying does not wait for a query: unknown initial capacity permits a file, while a completed +query error is returned to subsequent targets until a successful refresh. A refresh deducts +admissions that overlapped its query; it does not reserve the outstanding bytes of files already +in progress. Tape target-path resolution can run ahead of copying and does not perform a second +capacity decision. The estimate can refuse a file before output creation, but cannot guarantee +that an admitted file will finish or interrupt it when a later observation falls. ACP does not +credit failed-output cleanup as reclaimed Tape space. Stream shutdown joins outstanding queries. + +Capacity observations have three distinct boundaries: bytes accepted by LTFS, bytes accepted by +the drive, and bytes physically written. A successful file Close does not establish that the +drive's remaining-capacity counter includes every accepted byte. Reading `statfs` does not itself +flush either buffer. LTFS can also adjust the reported capacity for its Index and warning state; +raw drive capacity is not interchangeable with filesystem available space. + +Actual write, Close and finalization errors remain authoritative. On exhaustion, retain the +complete error chain and publish only the verified prefix described above. Removing an LTFS +temporary file is a namespace cleanup, not proof that its written Tape capacity was reclaimed. +YATM does not automatically roll back LTFS history, shorten cartridges, configure hardware early +warning thresholds, or force recurring Index synchronization for capacity management. The public +[Tape capacity study](../research/tape-capacity.md) owns the supporting hardware/source evidence, +community reports and unresolved prevention/recovery questions. Authorized experiments follow the +[physical acceptance procedure](../release/physical-measurements.md#capacity-measurement). diff --git a/docs/architecture/ui.md b/docs/architecture/ui.md index 028f75e..ef07f72 100644 --- a/docs/architecture/ui.md +++ b/docs/architecture/ui.md @@ -4,7 +4,7 @@ Status: Current v1 development architecture. The browser application: routes, browsers, dialogs, selection models and presentation. [APIs and services](api.md) owns the endpoints it calls, the -[command-line interface](cli.md) owns the scripted surface over the same operations, and +[command-line interface](../operations/cli.md) owns the scripted surface over the same operations, and [Interface contracts](contracts.md) owns the rules shared across them, which this document does not restate. @@ -32,88 +32,10 @@ describes only what has loaded. This is the contract for every surface that list whether the rows come from the catalog or from a live filesystem directory. A query is not a directory listing and keeps its own paged form. -## State and Styling - -Each fact has one owner. Redux stores shared application state; it does not replace the router, -controlled forms or request lifetimes. - -| State | Owner and lifetime | -| --- | --- | -| Archive/Restore selections and Restore version policy | Application Redux store; hydrate browser-session preferences once, then persist changes. Successful submission clears only the submitted revision, preserving later edits. | -| Committed Library/Preview/Job Settings | Application Redux store; only a current Get or successful Save publishes values. Older reads cannot replace a Save or a newer read, and an initial failure leaves values unknown until explicit Retry. Editors keep local drafts. | -| Configured Locations | Application Redux store; fetch the complete catalog once, share choices with Settings, and publish successful configuration edits. Live availability and latest analysis use detail reads. | -| File list sorting and folders-first | Application Redux store, persisted to local browser storage by list identity, independently of the current path. | -| Jobs return position, loaded rows, revisions and tombstones | In-memory Redux snapshot per route filter; request lifetimes remain local and late responses cannot publish after leaving. | -| Navigation and active route filters | Router. | -| Panes, sparse Identical pages, log/result windows, dialog drafts and pending actions | The owning view or shared feature hook; discard or invalidate them when their request or source changes. | - -The [application state module](../../frontend/src/state/store.ts) uses serializable protobuf JSON -and memoized decoded selectors. Controllers, timers and in-flight promises remain outside Redux. -Committed Settings and Job snapshots are not persisted. Browser storage is an optional preference -boundary: failed reads use the established default, and failed writes cannot turn a successful -server operation into a failure. A preference retains its own commit rule, such as a confirmed -Restore destination or successful Scan submission; merely editing a draft does not persist it. - -Instants follow the [shared time contract](contracts.md#time-values). Decoded values remain -`bigint`; serializable state uses decimal strings. [Shared date helpers](../../frontend/src/tools/time.ts) -convert to `Date` only for display and date controls, with range validation and correct rounding -before the Unix epoch. An unchanged Restore cutoff keeps its original nanoseconds rather than -round-tripping through the control's lower precision. -File rows keep their original nanoseconds as the shared browser date-sort key; saved-version -ordering and Preview selection follow the Library's [latest saved version order](library.md#organization-and-content) -with exact nanosecond comparisons. A first-date display fallback does not replace an unknown last archive date in that order. - -Shared read hooks keep requests local to each consumer. Preview and saved-version reads share their -feature hooks; the [Location-choice hook](../../frontend/src/components/use-location-choices.ts) owns -the root menu, Library filters, Restore, Locate original and Identical scope's choices from the -same configured Location catalog as Settings. The application preloads its complete paged catalog; -opening a chooser and filtering its names use that shared cache. An initial read shows loading, -and a failed read exposes Retry. Settings Refresh explicitly reloads the catalog; successful -configuration changes update it. A successful Library import refreshes the catalog and supersedes any earlier read. Late reads cannot overwrite newer edits. Live filesystem state -and latest analysis remain separate detail reads. Selection, Restore eligibility checks and save -actions remain with their components. Library filters retain their selected Location and label -when canceled and reopened. Media search keeps its successful current query/page when reopened; -changed queries/pages and explicit Retry perform a new read. Scan Location search and the Location choices -in Jobs filters use the same configured catalog; changing a Job filter retains loaded Media choices. The RPC transport does not log request or -response bodies; request owners present failures, and obsolete cancellation is not an application error. - -Styling has four owners: - -| Concern | Implementation | -| --- | --- | -| Application colors, type, shape and framework-wide defaults | The application theme. | -| Shared component appearance and interactive states | The owning React component, using MUI `styled`, `sx`, theme overrides and supported slots. Reuse Feedback, ActionRow, Settings shells, search popup shells and navigation instead of copying their appearance. | -| Page layout and native application-owned elements | Co-located Less/CSS; shared components keep these styles together with their markup, and no stylesheet targets another component's private DOM or state classes. | -| Measured row positions and dimensions | Dynamic styles owned by the virtual list or resizer. | - -`styled` and `sx` use MUI's Emotion styling system. Less styles page layout and application-owned -HTML; it does not reskin framework components. A shared component's native markup may retain -co-located Less, while MUI controls use the theme and component APIs above. -Components own text alignment; root layout containers leave it unset. -Settings editors keep one shared page shell across lazy code loading and the first data read, -with one progress indicator in that shell. Revisiting committed Settings uses the shared cache; -a failed read remains an actionable error until Retry. The router owns the selected tab, -including detail and trailing-slash routes. -The [entrypoint](../../frontend/src/main.tsx) configures MUI utility names before importing component -modules, including production chunks. Use exported utility classes or slots instead of hardcoded -`app-Mui*` selectors. The application theme wraps application controls, including controls placed -inside Chonky slots; Chonky owns its theme and browser appearance. Do not override all Chonky -buttons from an application stylesheet. The [application FileBrowser](../../frontend/src/components/file-browser.tsx) -owns its layout element and supplies Chonky's public theme props, so pages need no `.chonky-*` -stylesheet selectors. Chonky owns the native scroll container for ordinary, grouped and sparse -rows. Rows fill its client width; the browser allocates scrollbar space. Fixed content insets -must not stand in for a scrollbar, including when the list fits without scrolling. -Component states must work under lazy route loading, -portals, narrow layouts and nested browsers without depending on import order. - -`pnpm styles:check` and ESLint reject known boundary violations. Direct configured-Location -list reads and Settings Get/Update calls are restricted to their shared owners, as are Job result -page reads. Visual acceptance still checks -alignment, wrapping, focus, disabled/loading states and both browser panes; a static guard cannot -prove rendered appearance. - ## Browsing and Interaction +### Files Browsing + The shared Files controller records a loaded view only after publishing its successful result. Request replacement before that point, including when Settings resolves the Library scope, remains initial loading. Superseded responses cannot settle the replacement request; each page boundary rejects obsolete work before fetching another page, and obsolete failures do not surface against the new view. A read publishes batch by batch, so a read failure has already shown part of the directory: those rows are discarded rather than presented as the directory, and a failed Library refresh falls back to the listing it last read completely. Navigation or a read failure that clears rows also clears their loaded-view identity; only successful empty responses display the empty state. A list without rows states which of the three it is - reading, empty, or failed - through one [shared placeholder](../../frontend/src/components/list-placeholder.tsx); the loading state never doubles as the empty state, and no placeholder outlives the read that owned it. Navigation publishes the target directory's breadcrumb together with the loading list, before its rows: a Location target is derived from its path, and an opened Library folder from the row or ancestor entry that opened it. A Library directory without a known entry, such as a remembered position or a located parent, keeps its breadcrumb until the response. Every YATM file browser uses the list view through the shared application FileBrowser. Grid and view-switching actions are unavailable in menus, including selection browsers and waitlists. Browser sorting, folders-first, and Select all apply only when all rows for the current directory or query are loaded. Lists that still stream or have another page hide these actions. When sorting is available, size and date start descending, name starts ascending, and choosing the active sort reverses its direction. Each list remembers its sort, direction and folders-first setting across paths, page visits and reloads. The shared FileBrowser initializes row order and menu indicators from the same preference, including after initial loading. Media hides its hidden-file toggle because its rows have no hidden state. @@ -148,32 +70,96 @@ Authorized path browsing skips explicit policy/resource exclusions; filesystem o Retain the product logo, build-derived version, contextual Library/Location root labels, compact row height, full-width separators, inherited surfaces, and usable empty spacing. Tape drawers close independently and physical-file panel height stays stable when expanded/collapsed. Visible dot-prefixed directories remain navigable unless a backend explicitly excludes them. Shared browser interaction fixes belong in Chonky, not per-page CSS/event workarounds. -[MUI class-name initialization](../../frontend/src/mui-classnames.ts) must run before component imports in both the application entry and browser test setup. The entry dynamically loads [application bootstrap](../../frontend/src/bootstrap.tsx) after initialization, preserving that order across production shared chunks. Cached internal selectors and rendered classes must use the same prefix; otherwise default select overflow and outlined-label styles can fail. Verify long selected values and floating labels in real rendering. +### Locations and Source Navigation -The [application theme](../../frontend/src/theme.ts) owns MUI primitive appearance; `sx` and `styled` use the same Emotion engine. Shared components own repeated structure and placement, including tabs, section cards, action rows, confirmations, feedback, result rows and saved-version choices. Less owns page layout and scoped third-party adaptations; it does not restyle those shared components. Bootstrap loads the reset before importing the application and its global styles. Component styles stay scoped, so opening a lazy route cannot change another surface. Chonky retains its own Emotion styles and notifications retain their distributed CSS. - -The [Locations page](../../frontend/src/pages/locations.tsx) lives in **Settings → Locations**. Its compact table leads to routed add/detail pages; **Add location** and Refresh sit in that order at the left below the list instead of competing with the page title, and unrelated Library tools are not repeated here. Details separate configuration, live files and related Jobs, with unsaved-change protection. The detail header shows the Location's name and availability; the Configuration form owns the root path along with the other values. Its bottom action row places Save changes, Scan, Cancel and Remove location together in that order. The Directory field offers **Browse directories** before registration or while editing. Its read-only chooser uses the existing authorized path service, retains exact server paths and pages directories without collecting files; Choose changes only the draft, Cancel leaves it unchanged, and Save registers or updates the Location. Failed pages retain their retry context and closed or superseded navigation discards late replies. Browsing registered contents remains the Files tab's job; the tab fills the height the page leaves it rather than a fixed pane. **Restore destination** is a recommendation, not permission. Ignore is verbatim gitignore text; matching entries disappear from the live file list and from Restore destination choices. An explicitly named destination still writes normally, and its output stays unlinked. New and imported Locations are immediately usable. Registration does not create a collection Job. Access errors and analysis results remain separate. Shared containers unify gutters and panel gaps. +The [Locations page](../../frontend/src/pages/locations.tsx) lives in **Settings → Locations**. Its compact table leads to routed add/detail pages; **Add location** and Refresh sit in that order at the left below the list instead of competing with the page title, and unrelated Library tools are not repeated here. Details separate configuration, live files and related Jobs, with unsaved-change protection. The detail header shows the Location's name and availability; the Configuration form owns the root path along with the other values. Its bottom action row places Save changes, Scan, Cancel and Remove location together in that order. The Directory field offers **Browse directories** before registration or while editing. Its read-only chooser uses the existing authorized path service, retains exact server paths and pages directories without collecting files; Choose changes only the draft, Cancel leaves it unchanged, and Save registers or updates the Location. Failed pages retain their retry context and closed or superseded navigation discards late replies. Browsing registered contents remains the Files tab's job; the tab fills the height the page leaves it rather than a fixed pane. **Restore destination** is a recommendation, not permission. Ignore is verbatim gitignore text. New blank input uses the server's default YATM entry rules; nonempty custom text replaces them. Updates retain intentionally empty rules. Matching entries disappear from ordinary live browsing, measurement and Scan/Archive selections. Only root/Ignore edits show the scope warning: existing history remains, future reads use current configuration and a frozen Job cannot be redirected. An explicitly named destination still writes normally, and its output stays unlinked. New and imported Locations are immediately usable. Registration does not create a collection Job. Access errors and analysis results remain separate. Shared containers unify gutters and panel gaps. Each Chonky pane has a keyboard-accessible **Library / Locations** root menu. Files' dual panes share one source: both Library or both the same Location. Either menu changes the shared source, including the hidden pane in Inspector mode; source changes discard stale rows and selections. Directory positions, search, pagination and scrolling remain independent, with each pane remembering its directory per source. Selection browsers retain their separate waitlists. Library actions change logical organization; Location cut/paste, rename/move, mkdir and delete change actual disk entries. Neither offers user-level Copy. Shared focused-pane hotkeys and drag actions enforce same-source operation boundaries. Cross Library/Location drags never imply a filesystem transfer. Unadmitted regular files support explicit annotation/admission and Archive; unknown content has no invented version or Preview. Library visibility uses [server-side saved-version scope](library.md#visibility-and-selection), independent of Location and duplicate browsing. Root composition and interaction extensions belong to Chonky, not page overlays. -Clicking the root name returns to that source's root; its separate arrow switches sources and restores remembered navigation. Media roots also remain clickable. The compact single-line navbar uses measured available width to fold intermediate ancestors into an accessible menu, retaining root and current directory without wrapping height. Copy path writes a source-qualified display path, not an executable object reference. **Include unbacked files in Library** belongs to **Settings → Library**, separately from import/export; it is not a Files toolbar control. Settings separates its surfaces the same way the preference groups are stored: **Library** holds visibility and Trash confirmation, **Preview** holds Generation, Runtime, Output and reported formats, and **Jobs** holds the pipeline limits an attempt runs with. Each editor loads one complete typed group, keeps a local draft, and replaces that group on Save; Cancel restores the last loaded value. Save/Cancel is one page-level action row below the editable sections, never an action embedded in one section. A page states what its values apply to. All four surfaces render through one [Settings page shell](../../frontend/src/components/settings-page.tsx), one [page heading](../../frontend/src/components/page-heading.tsx), uniform section cards, one shared action row and one [field](../../frontend/src/components/settings-field.tsx) that carries a control's label, control and explanation. The framework's own components own the frame, widths and spacing, so a title, group or explanation cannot take a different size or place on one surface than on another. +Clicking the root name returns to that source's root; its separate arrow switches sources and restores remembered navigation. Media roots also remain clickable. The compact single-line navbar uses measured available width to fold intermediate ancestors into an accessible menu, retaining root and current directory without wrapping height. Copy path writes a source-qualified display path, not an executable object reference. **Include unbacked files in Library** belongs to **Settings → Library**, separately from import/export; it is not a Files toolbar control. Settings separates its surfaces the same way the preference groups are stored: **Library** holds visibility and Trash confirmation, **Preview** leads with generation/type/output and places execution, resource limits, extensions and sampling in Advanced, and **Jobs** holds the pipeline limits an attempt runs with. **Devices** holds per-device FORMAT preferences and Tape read order; saving them performs no hardware operation. Each editor loads one complete typed group, keeps a local draft, and replaces that group on Save; Cancel restores the last loaded value. Save/Cancel is one page-level action row below the editable sections, never an action embedded in one section. A page states what its values apply to. All Settings surfaces render through one [Settings page shell](../../frontend/src/components/settings-page.tsx), one [page heading](../../frontend/src/components/page-heading.tsx), uniform section cards, one shared action row and one [field](../../frontend/src/components/settings-field.tsx) that carries a control's label, control and explanation. The framework's own components own the frame, widths and spacing, so a title, group or explanation cannot take a different size or place on one surface than on another. + +### Settings Surfaces + +The [Device editor](../../frontend/src/pages/settings-devices.tsx) uses the existing typed DEVICE +Settings Get/Update group and shared committed cache/editor. Configured devices and saved inactive +entries remain distinct; inactive Device settings grant no access. FORMAT defaults are resolved for the +selected drive in an existing Archive Job's **Choose archive storage** dialog after selecting +**Storage type → Tape**, following manifest preparation. The absent +format means LTFS; explicit operation edits override saved defaults and remain when the drive changes. +APPEND and reads use the cartridge's immutable stored profile. **Read order** is an ordinary field beside FORMAT defaults and edits the selected device's +LOGICAL/AUTO preference; unspecified resolves to LOGICAL and remains unspecified +through unrelated edits. Saving validates the enum without hardware access and retains inactive +entries. Each admitted read freezes that preference. LOGICAL preserves manifest order while keeping +eligible exact seek. AUTO plans the complete selected-Media sequence when a usable position map +and applicable read parameters are available. The Media backend records the effective order and +fallback reason in the existing Job log. Device inspection is an explicit action +separate from Save. Unknown observations stay unknown; omitted tar record options resolve as the +[Tape option contract](contracts.md#tape-options-and-selection) defines. Persistence checks record shape; +actual admission checks current device/driver bounds without clamping. The +[Device guide](../operations/devices.md) owns operator instructions. + +The separate collapsed **Manual read parameters** section opens a JSON dialog copied from the page draft. +Apply validates it and updates only that draft; the page Save persists it. Cancel leaves the page +unchanged, using the existing discard confirmation for edits. Starting from device details uses +only the displayed report and performs no new query; it creates fresh applicable identity fields +instead of relabeling previous timings. Clearing manual read parameters is also a draft edit and +returns selection to automatic parameters only after Save. Partial manual read parameters retain priority +and may leave insufficient costs for AUTO; absence stays unknown rather than zero. Variable-record +maximum size and written compression/encryption conditions cannot be inferred from current controls. + +Advanced also owns passive read learning (omitted means enabled) and the complete-selection +planning budget (omitted means 300 seconds; explicit values are positive). Learned +models are server-owned Device Settings facts; a form Save cannot erase or replace them. +Manual read parameters take precedence over matching learned read parameters and source-labelled initial +values. Device details show the last prepared read's identity, parameter/position sources, +coverage gaps and the learned read parameters' mean and maximum error when available. This historical snapshot does not certify +the currently loaded cartridge or report the final read order; Inspect does not start +measurement or extra planning queries. + +Job pipeline tuning, Preview runtime tuning and Device tar options use SettingsSection.collapsible +in initially collapsed **Advanced** sections. Hidden fields stay mounted and retain values; validation +expands the affected section and shows its error without discarding the draft. Each page keeps its +normal Save/Cancel actions. Devices places Save, Cancel and **Refresh devices** in one row below +the fields. Refresh devices reloads availability only and retains the settings draft; refreshing +saved Settings through the editor is a separate action that replaces the page draft after a +successful read, without a discard prompt. A failed reload retains the draft and reports the error. +Read-only Device details are collapsed after the editable sections. Raw tar hardware compression offers On/Off with leased validation. +Encryption setup belongs to the configured adapter for both tar and LTFS; there are no native +key, encryption-On or algorithm-choice controls. +Settings retains unavailable saved Device settings for explicit correction. Compression diagnostics show the current numeric +algorithm and optional control changeability without inferring an algorithm list or physical +validation. Encryption diagnostics expose public algorithm/index/key-size and optional non-secret +mode/scope/counter facts; a counter is never labeled key presence. No key input or download surface +is added. An optional observed algorithm does not establish whether a key is present or the +cartridge is plaintext. Every stored Tape shows one **Encryption** value in Media details, the +APPEND summary and the read chooser: the observed algorithm, **Configured · algorithm not recorded** +when a key is saved without one, or **Off** when no key is saved. Responses report only key presence. +Geometry diagnostics distinguish the reported +all-wrap command form from qualified coverage and preserve unknown freshness and reported-wrap count +until their observations exist. Stored Media details show format and non-secret profile facts and link to +the Device editor and [independent recovery guide](../operations/tar-tape-recovery.md); there is no +recovery RPC or download action. + +### Shared Interaction Patterns Actions for one page or entity belong in one left-aligned inline row after its content, ordered from the main action at the left to less central actions at the right. Editable forms place Save first in that row and Cancel to its right; related actions, including Scan, import, export and removal, share the same row. List-level actions sit below their list; dialogs keep their actions in the footer. Navigation tabs, status indicators, field controls and contextual browser tools stay with the content they control. The [shared action row](../../frontend/src/components/action-row.tsx) owns alignment, spacing and horizontal scrolling on narrow screens, keeping actions together without shrinking their buttons. Settings pages and their forms use the available content width without a separate narrow maximum. Settings groups use one restrained card treatment: a clear section heading, a separated body with consistent field spacing, and explanations next to the setting they describe. Location's Advanced settings use the same section component and are initially collapsed in one bordered section; expanding it reveals default-off mmap content reads without a nested card. Double-clicking a file in ordinary Library/Location and Archive/Restore source browsing opens details, never a content tab or an implicit waitlist addition. Location entries with a File association use the same Library details and saved versions; unadmitted entries show live properties without forcing admission. Double-clicking a directory navigates into it. Original details link to the live containing folder and reveal the file; they do not add a separate Open button. Archived copies show health and availability with Restore guidance, not an HTTP content link. A dedicated file picker may use double-click to choose its entry; Library search retains its locate-result behavior. -Library, Jobs and Preview Settings share the same loading, retry and save feedback. Every group reuses its committed Redux value on revisit. Before the first value arrives, the editor shows shared loading feedback without inventing field values; an explicit reload retains its existing form while the read runs. Location removal uses the shared confirmation dialog, retaining an error in that dialog for correction and retry. +Library, Jobs, Preview and Device Settings share the same loading, retry and save feedback. Every group reuses its committed Redux value on revisit. Before the first value arrives, the editor shows shared loading feedback without inventing field values; an explicit reload retains its existing form while the read runs. Location removal uses the shared confirmation dialog, retaining an error in that dialog for correction and retry. Composed browsers connect pagination to `FileList.onScroll`; `FileBrowser` is the state container and does not forward that event to an explicitly supplied child. Integration tests render the installed list component as well as testing the paging hook, so a mocked wrapper cannot conceal a disconnected native scroll handler. The [shared query bar](../../frontend/src/components/library-filters.tsx) keeps optional structured controls out of the workspace. Find identical files opens **Tools → Identical files**. Ordinary queries, including `has:duplicates`, remain flat file results. Duplicate semantics belong to [Library search](library.md#annotation-search-and-cleanup). Inspector is the default unless Dual pane was saved; both keep the complete Chonky toolbar and dot-directory navigation. -Rows use the [Library state matrix](library.md#file-and-version-display-states): present/archive green, present/no-archive yellow, absent/archive blue, absent/no-archive red and uncertainty gray. Current changes and historical anomalies use a supplementary mark, never an implicit health expiry. Files.List computes requested status using batched existence facts and bounded current-original observations, without hashing or reading Media. No exact copy counts, versions or signature bytes accompany ordinary list rows. +Rows use the [Library state matrix](#file-and-version-display-states): present/archive green, present/no-archive yellow, absent/archive blue, absent/no-archive red and uncertainty gray. Current changes and historical anomalies use a supplementary mark, never an implicit health expiry. Files.List computes requested status using batched existence facts and bounded current-original observations, without hashing or reading Media. No exact copy counts, versions or signature bytes accompany ordinary list rows. The [controlled detail flow](../../frontend/src/pages/file-detail.tsx) is keyed by a `FileOperationRef` and uses `FilesService.Get` for both Library and live Location entries. The [shared detail surface](../../frontend/src/components/detail-surface.tsx) owns the same compact header, optional status, fixed tab row, Preview region, and scroll boundary in the Files Inspector, Files dialog, and Media/Position Inspector; each caller supplies its own facts and operations. The Files dialog uses the same content as the Inspector and becomes full-screen on small viewports. Basic detail loading does not wait for Preview or history; versions load on first opening Saved versions, and that version's copies load with its detail. Each region owns errors, retry and stale-response rejection. A refresh retains the last coherent snapshot with one unobtrusive indicator; late responses for another entry cannot clear or replace the current selection. Selecting another entry clears the previous snapshot and holds its layout as a skeleton until the new detail arrives, while retaining the selected Overview or Saved versions tab. The [File content](../../frontend/src/components/file-content.tsx) keeps a fixed Overview/Saved versions tab row while Preview stays above the selected tab's content. The image or video display area is a responsive 16:9 frame; video controls sit below it, and media fit without stretching. Overview keeps the original, inline Tags/Note editing with Save/Cancel, and collapsed technical details together. Overview shows **Add to Archive list** only when the entry publishes the Archive operation and a regular Library File has an original association. Directories offer their root without checking descendant original associations. Saved versions use a paginated dated list with the newest loaded archive time first. File details and the Restore picker use one [saved-version choice](../../frontend/src/components/saved-version-choice.tsx) for selected, disabled and keyboard-focus presentation; each caller retains its own content and selection. A version shows its recorded last archive date, then its first archive date; without either it reads **Date not recorded**. Matching copy write dates do not establish when that FileVersion was archived. **Add to Archive list** and **Add to Restore list** lead the bottom action rows of Overview and the selected saved version, respectively. The selected saved version can be added to the Restore waitlist by its exact `FileVersion` identity. Both additions retain the Inspector or dialog, report whether the item was added or already present, and expose **View list**; neither chooses Media or a destination, estimates a Job, or creates one. Copies are immediately visible as individually collapsible MUI cards with a right-side icon arrow; signature, hash and permissions stay in collapsed technical details. A current original's Preview is never replaced by saved assets. An entry with no current original previews its newest saved version instead, under a label naming that version, and shows no Preview when it has no saved version either. Archive-only Files use the same eligible-history summary as the list. Saved content remains available through [Library publication](library.md#archive-inventory); Restore execution remains the explicit waitlist review and Job creation workflow. Duplicate queries never merge organization. Originals are not archive copies. Persistent copy is limited to names, paths, facts and actions. Location names are user-defined directory labels, not product modes. Overview labels the association **Location** and shows its name and physical relative path as inline links separated by a slash. The name opens the live Location root; the path opens its containing folder and reveals the file, including later pages. An original hidden by user Ignore opens that folder without appearing in it, because explicit access does not make the entry browsable. It has no redundant location heading or missing-location paragraph. Archive-only entries use the neutral **Saved versions only** label. Location pages and Scan dialogs omit generic introductions and repeated operation descriptions. The selected File header does not repeat the domain model; version lists do not repeat archive rules, and copy lists do not repeat their section context. Empty/unavailable/error states retain concise next steps. The selected version's Preview uses that version's endpoint; switching versions clears the previous image, including when a prior response arrives late. Successful File refreshes update original facts, loaded version/copy pages and selected-version Preview while retaining the tab and selection. A current Preview is read under the content signature the detail published, which the Library only publishes while the live original still agrees with its record. Path review, Ignore semantics, destructive-action scope and Restore destination/non-in-place warnings remain at their relevant operations. +### Job Creation + **Jobs → New → Scan** orders its controls as source, target, signature policy, result handling and Preview. Location and Media use required single-choice search fields; free text is not a selection. Location choices reuse the shared configured catalog. Media searches use bounded server ID-cursor pages. Search matches registered names and directory paths or Media identities, treating wildcard characters literally. Loading, empty results and retry stay in the chooser. Stale responses cannot overwrite a newer search or selection. Inventory update and recorded-copy verification are mutually exclusive policies. Verification forces real reads; sequential Media disallow Preview. The shared Preview select offers **Don’t generate / Generate missing / Regenerate all** in Scan and Archive. [Preview](preview.md) owns execution semantics. All modes use the same SCAN runner and result types. The browser remembers successfully submitted Scan choices in local storage. Returning reloads the selected target and checks eligibility/binding before enabling submission; incoming source/selection links take precedence. Invalid storage or unavailable targets cannot silently select another resource. Signature, result and Preview constraints still apply to remembered choices. @@ -192,9 +178,15 @@ Generic workflow explanations belong in documentation, not persistent Job cards, When an original's content signature becomes unknown, its Preview clears the preceding error and offers no retry for that unknown content. -All Scan entry points open the same [controlled form](../../frontend/src/components/scan-form.tsx) at `/scan`: source/range, Signatures, Results, Find matching Library content, Previews, then **Start scan**. Location and Media use required server-search selectors. Location supports the whole root or selected entries within that Location; Files supports Library/Location roots accumulated across directories; Media remains whole-resource. The read-only Chonky range dialog edits a temporary selection: Choose commits it, Cancel discards it. Roots retain their source, path and scope; an empty Location path selects its whole root, and directories expand only on the server. Browsing does not collect files, organize entries or start Jobs. +All Scan entry points open the same [controlled form](../../frontend/src/components/scan-form.tsx) at `/scan`: source/range, Results and Previews, then **Start scan**. Advanced holds signature acquisition and archived-copy counting. A compact summary keeps the effective reading/publication choices visible, including remembered non-default choices. Location and Media use required server-search selectors. Location supports the whole root or selected entries within that Location; Files supports Library/Location roots accumulated across directories; Media remains whole-resource. The read-only Chonky range dialog edits a temporary selection: Choose commits it, Cancel discards it. Roots retain their source, path and scope; an empty Location path selects its whole root, and directories expand only on the server. Browsing does not collect files, organize entries or start Jobs. -The route initializes defaults, applicable last-used settings, then explicit [prefill](../../frontend/src/pages/scan-input.ts) once per navigation. Files context actions supply editable selections; both file and directory Add to Library supply known-only/original-publication/no-comparison/no-Preview defaults. Location/Media URL shortcuts and `collect=1` resolve into the same prefill, not separate form modes. Only actual capabilities constrain options: copy verification forces real reads and Tape disables Preview. Target changes discard incompatible ranges; late lookups cannot overwrite a newer choice. Invalid or stale targets require correction rather than falling back to a whole Location. Successful creation alone saves final options and valid Location/Media identity; individual file selections are never stored as long-term preferences. All submissions use the existing ScanJobSpec/Create contract. +The route initializes defaults, applicable last-used settings, then explicit [prefill](../../frontend/src/pages/scan-input.ts) once per navigation. Files context actions supply editable selections; both file and directory Add to Library supply known-only/original-publication/no-comparison/no-Preview defaults. Location/Media URL shortcuts and `collect=1` resolve into the same prefill, not separate form modes. Only actual capabilities constrain options: copy verification forces real reads and Tape disables Preview. Target changes discard incompatible ranges; late lookups cannot overwrite a newer choice. Invalid or stale targets require correction rather than falling back to a whole Location. Successful creation alone saves final options and valid Location/Media identity; individual file selections are never stored as long-term preferences. All submissions use the existing ScanJobSpec/Create contract. Known-only with Preview skips files +whose content signature is unknown and offers a choice to read missing facts; known signatures can +still require reads for decoding. Report-only suppresses catalog publication, not content I/O or +Preview generation. Optional comparison counts recorded archived copies and displays those counts +in results; an unrequested or unknown count is not shown as zero. A visible Scan Job loads its +creation settings in any state, and rows add no comparison text until they arrive. Capability errors expose Retry +without leaving the draft. Copy-verification remains an explicit real-read policy. Confirmations and name prompts use the application-owned [action dialog](../../frontend/src/components/action-dialog.tsx), not browser `alert`, `confirm` or `prompt`; lint enforces this boundary. Dialogs name the operation and target, retain destructive-action scope, and accept operation-specific content/options. The dialog paper is the form, keeping its scrolling content and fixed footer in one flex layout. Cancel/Escape does not execute the action; submission locks dismiss and repeat-submit controls, and failures remain in the dialog. A successful mutation followed by a refresh failure reports the refresh separately instead of inviting another mutation. Inline errors and notifications remain appropriate for non-blocking feedback; browser unload protection is a separate platform capability. @@ -202,23 +194,62 @@ Inline operation errors and Alerts with actions use the shared [Feedback](../../ File Overview offers **Locate original** for regular Library Files, including associated entries viewed through a Location. It reviews the selected Location and observed file before replacing the association through the existing relocation operation. Choosing another Location or file clears the prior selection's error. Successful linking closes its dialog before refreshing details; a refresh failure reports separately and cannot repeat the association change. +### Job Review and Cards + Job creation must have a review surface before execution: Archive/Restore creation dialogs, Scan configuration, and Media read/write dialogs provide it through their target/options and final action button. Creation pages use their top tabs as labels, without redundant New headings or a Locations shortcut. Check integrity opens the prefilled Scan form; stop and delete require confirmation. Navigation to a creation page, Properties, results or logs does not require confirmation or start a Job. **Recreate** opens the normal Archive, Restore or Scan form with the retained creation inputs from the [read-only creation API](api.md). It never changes or submits the old Job. Archive/Restore ask before replacing a nonempty waitlist, and concurrent draft edits require a fresh confirmation. Missing roots, versions or destinations stay visible and require correction; a failed lookup never widens a selection or chooses another destination. Original options, priority and an unchanged Restore cutoff are preserved exactly. Submission remains an explicit action through the normal estimates and pre-submit checks. Imported Jobs without retained creation inputs explain why the form cannot be restored. -Job cards offer no Job-level retry or restart. Media choices require an idle `READY` Job. Admitted Archive and Restore Media errors or operator cancellation, including while `QUEUED`, return to pre-Media `READY`, retaining the prepared manifest, successful per-file/Media checkpoints and reason/timing so the operator can load Media for the remaining work. Failed or cancelled Scan execution and manifest preparation offer no start action; a new Job is required. `total_known`, retained counters and an `UNSPECIFIED` phase never enable a `FAILED` Job. Active and queued attempts suppress another Media choice, and process restart never restarts a Job or revives terminal `FAILED` work. Storage dialogs reject responses from a cancelled opening, and a successful start closes the dialog before refreshing Jobs; a failed refresh reports separately without reopening the Start action. Successful Volume initialization or registration likewise closes its dialog before refreshing Media. Page and transport Retry actions repeat reads only. +Job cards use one stable metric layout and the server-provided stage percentage/estimate for every kind. One shared module owns the frame, the counter wording and the waiting/active/terminal classification behind the status label and the action buttons; only active work animates a bar of unknown length. Polling keeps the last coherent snapshot through refresh/error/offscreen transitions and rejects obsolete requests, and it never changes an estimate, because every sample is recorded on a work path. Current and session-average byte speeds remain distinct from persisted business results; elapsed time stays the common latest-attempt measurement. + +Job cards offer no Job-level retry or restart, and offer a Media choice only for an idle `READY` Job as the [Job lifecycle](jobs.md#shared-lifecycle) defines. `total_known`, retained counters and an `UNSPECIFIED` phase never enable a `FAILED` Job, and active or queued attempts suppress another Media choice. Storage dialogs reject responses from a cancelled opening, and a successful start closes the dialog before refreshing Jobs; a failed refresh reports separately without reopening the Start action. Successful Volume initialization or registration likewise closes its dialog before refreshing Media. Page and transport Retry actions repeat reads only. + +### Tape Dialogs + +Tape Archive uses format-aware preflight. Raw tar FORMAT inspects the configured device passively, +checks exact Library barcode registration, resolves supported record values and requires explicit +barcode confirmation before destructive submission. Resetting recommendations reuses the loaded +report and saved defaults; it does not perform another inspection. Changing FORMAT/APPEND retains +the selected device and barcode-registration result while clearing only action-specific overrides. +Tar uses the encryption adapter without the LTFS formatting/mounting/readinfo workflow. +The [read chooser](../../frontend/src/components/read-media-dialog.tsx) keeps **Refresh position map +before reading** in Advanced and offers it only for known tar Restore or Scan VERIFY_COPIES. +An enabled action retains a visible consequence summary even when Advanced is collapsed. Scan reuses GetCreation to load +its frozen result policy once per chooser admission; unknown or failed policy loading hides this +optional action without blocking ordinary reads. Native LTFS/Volume does not fetch an unused policy. +The selected device's qualified non-ejecting reload support controls availability. A supported, +READY observation enables the action unless validation failed; UNVALIDATED alone does not disable it. +An explicit request is revalidated under the lease and refused +before motion without that lifecycle; AUTO alone never reloads. LTFS, Volume and inventory/other +Scan policies omit the action. Late responses cannot restore a closed chooser's policy. + +Raw APPEND requires an explicit action choice and displays its registered immutable profile; the +Session verifies the label/profile and tail under the attempt lease. Failed submissions retain the +form and error. LTFS retains cartridge inspection and **Check Tape**, together with its +initialization, key, mount and normal unmount workflow. + +When Tape inspection cannot identify the cartridge, the Archive storage dialog requests its barcode with an **ABC123** example and helper text specifying six letters or digits without the LTO suffix. Input is trimmed and uppercased for validation and submission; **Check Tape** remains required before starting the Archive. + +### Archive and Restore Details Presentation cleanup includes tracing response fields and server calculations to their consumers. Remove presentation-only data with no remaining business or client use through the protobuf generation path; retain fields used for validation, selection binding, content identity, pagination and actionable diagnostics. The selection-review API follows the [Library aggregate contract](library.md#visibility-and-selection), not a hidden output-path preview. +Archive's **Recheck source hashes for previews** remains an advanced preparation option. It +changes hashing before the companion Preview Scan, not the integrity check performed while +copying Archive content. Disabling Preview keeps its draft value without making it active. + Archive and Restore waitlists use Chonky's optional `footer` slot for their creation-dialog entry action. One browser frame owns the outline, divider and outer corners; there is no separately rounded list inside another card. The list scrolls independently of its footer. Configuration scrolls inside the modal, with Cancel and final preparation in its action bar. The [shared component integration](../../frontend/README.md) documents the published dependency and consumer regression coverage. Rows containing path or version details use content-measured heights in the shared virtual list, including after details change. Ordinary single-line rows remain compact. Waitlists omit a secondary path when it is identical to the displayed name. -Restore shows a compact destination path control. Its compact save-location dialog puts a preferred-Location selector above the live directory browser, compact breadcrumbs and **Choose / New folder / Cancel** actions. The dialog follows content height; only the directory list scrolls, without a fixed-height sidebar or empty space below its actions. Double-click enters a directory; Choose confirms the current directory. Re-selecting the current breadcrumb, including the root, is a no-op rather than a reload. New folder performs an immediate checked creation; Cancel does not undo it. Navigation and cancellation do not collect files or change Library. There is no preferred/all toggle or fallback; an empty preferred list links to Settings. Failed navigation clears old rows; Retry reloads the complete directory. Advanced damaged-copy recovery remains off by default. [Job indexing](jobs.md) freezes contents, logical targets, destination bindings and collision-safe outputs. +Restore shows a compact destination path control. Its compact save-location dialog puts a preferred-Location selector above the live directory browser, compact breadcrumbs and **Choose / New folder / Cancel** actions. The dialog follows content height; only the directory list scrolls, without a fixed-height sidebar or empty space below its actions. Double-click enters a directory; Choose confirms the current directory. Re-selecting the current breadcrumb, including the root, is a no-op rather than a reload. New folder performs an immediate checked creation; Cancel does not undo it. Navigation and cancellation do not collect files or change Library. There is no preferred/all toggle or fallback; an empty preferred list links to Settings. Failed navigation clears old rows; Retry reloads the complete directory. Advanced damaged-copy recovery remains off by default; enabling it shows that complete mismatched +output is retained without a verified original association. [Job indexing](jobs.md) freezes contents, logical targets, destination bindings and collision-safe outputs. The target picker filters preferred Locations on the server before pagination; additional-page requests cannot overlap or append duplicate IDs. Closing the modal invalidates its pending responses. Only a confirmed choice is saved in browser-local storage, including Location ID, root path and relative directory. Returning to Restore reloads that Location and reuses the choice only if it remains preferred with the same root path. Deleted, moved or ineligible targets require selection again; a late lookup cannot override a new choice. Cancel leaves the prior choice intact. This is a browser preference, not a change to CLI destinations or server write authorization. -All creation results, Jobs rows and related-object histories link to the canonical `/jobs/:id` detail. The detail names the Job and when it was created through the shared page heading, and the navigation strip's **All** tab is what returns to the list, so the page carries no second back link; **Back to Location** remains as its page action. Returning to Jobs preserves filters, loaded pages and list position. The same detail component owns progress, results, logs, cancellation and explicit Media actions. **View Log** opens at the newest lines, pages backward or forward by cursor in a bounded virtual list, and follows new lines while the operator is at the bottom or the filtered view is empty. It keeps raw lines copyable, colors their severity, highlights timestamps, and filters by level or text across the retained log. An oversized line fragment whose match cannot be confirmed is labeled as such, and the filtered search continues. A filter or closed dialog invalidates pending reads; a search with no confirmed match in one byte page continues through the log and can be stopped. Settled Jobs do not poll. Read dialogs offer only the kinds represented by required Media: a single-kind Job opens directly on that kind, and Verify displays its fixed Media identity rather than offering another backend. Restore with both kinds defaults to Volume and allows switching to Tape; Tape inspection must match a required identity before submission. Missing catalog Media is an error, not a fallback choice. Job creation is never presented as a finished Archive or Restore. +### Jobs Pages + +All creation results, Jobs rows and related-object histories link to the canonical `/jobs/:id` detail. The detail names the Job and when it was created through the shared page heading, and the navigation strip's **All** tab is what returns to the list, so the page carries no second back link; **Back to Location** remains as its page action. Returning to Jobs preserves filters, loaded pages and list position. The same detail component owns progress, results, logs, cancellation and explicit Media actions. **View Log** opens at the newest lines, pages backward or forward by cursor in a bounded virtual list, and follows new lines while the operator is at the bottom or the filtered view is empty. It keeps raw lines copyable, colors their severity, highlights timestamps, and filters by level or text across the retained log. An oversized line fragment whose match cannot be confirmed is labeled as such, and the filtered search continues. A filter or closed dialog invalidates pending reads; a search with no confirmed match in one byte page continues through the log and can be stopped. Settled Jobs do not poll. Restore offers its Media chooser only after candidate IDs are known; retained snapshots remain usable during progress refresh, while a known empty list reports no pending Media. Read dialogs offer only the kinds represented by required Media: a single-kind Job opens directly on that kind, and Verify displays its fixed Media identity rather than offering another backend. Restore with both kinds defaults to Volume and allows switching to Tape; All-LTFS candidates, including migrated `ltfs_v0`, and an explicit LTFS Media hint use the existing cartridge adapter and require its matching identity before submission. Raw tar or mixed-format automatic reads show passive details and offer an explicit Media hint from the Job's candidates; the server proves identity/profile under the lease. Hint-required failures retain the form for reviewed resubmission, and missing raw prerequisites never force all-LTFS reads through SG inspection. Missing catalog Media is an error, not a fallback choice. Job creation is never presented as a finished Archive or Restore. The top **All** tab and sidebar **Jobs** preserve an originating `/jobs` list URL and its filters; other origins and direct detail links go to `/jobs`. **Back to Location** returns @@ -233,12 +264,72 @@ All Job kinds use the same card title/status/body/footer spacing and one shared Job cards keep their bodies mounted while offscreen; visibility gates only a shared serialized progress hook. One shared progress card owns the frame, its counter wording and the waiting/active/terminal classification that the status label and the action buttons read, so the three kinds render the same frame for the same phase and counter set. `QUEUED` displays **Queued — waiting for Media**, keeps **Cancel** available and suppresses another start and **Delete Job**; active attempts also suppress deletion. Live working phases take precedence over `COMPLETED` and `FAILED` while cleanup is active. An idle `UNSPECIFIED` stage displays **--** as its current phase. Counter wording names the phase's own work: discovery reports **Found**, Preview reports **Generated**, comparison reports **Compared**, content, verification and publication report **Processed**, and copy reports **Copied**. An idle Scan keeps **Processed** for its retained counters; Archive and Restore keep **Copied**. A phase measured in items alone reports no byte counter, because the bytes beside a decoder count describe the source rather than the stage; the Job's own total rows stay visible beside it. The bar animates only while the Job is active on a stage of unknown length. An idle completed Job shows 100%, including before its counters load or when the workload is empty; active cleanup still uses its live stage progress. Other waiting or settled Jobs show a still bar at the percentage they have. The failure alert keeps its own space under the title. A settled attempt that carries a reason is shown as a failure whatever settled it, including the operator's own cancellation. Leaving a Job or hiding its card aborts its current read and prevents late publication, while completed snapshots remain visible. The Jobs list owns the bounded flex scroll region and pagination observer. Chonky synchronizes controlled rows before paint so loading completion cannot briefly show an empty folder. -Media names stay separate from secondary capability/availability details so prompts and breadcrumbs identify only the target. A Media Scan with **Update inventory** offers **Add to Library** beside **Start scan**: it counts the records that admission would add, shows the count with its folder/unsigned breakdown, and writes only after confirmation. **Don't show this next time** records that answer in browser-local storage so later clicks import directly without the count. The dialog admits the Media's published records, which are not the entries this Scan is about to publish. Mounted Volumes start scanning directly; Tape requires its explicit device/read step and fresh LTFS inventory. **Add to Library** accepts signed regular archive entries or inventory directories; directory descendants expand in bounded Media-path pages and only ordinary signed content becomes independent Library Files. Empty or unsigned-only selections return an explicit result. Position copies are labeled **Archived copies**, inventory deletion is **Remove from Library**, and **Clean up Library** explains its retention scope before execution. Single-Media operations reject multiple eligible selections with an actionable notice rather than silently doing nothing or choosing one. Context actions use Chonky's filtered action selection. +### Media -The [command-line interface](cli.md) exposes the same named methods and projections to scripts and agents; [CLI E2E](../operations/e2e-test.md) records its executable acceptance and the [local Demo](../operations/testing.md#local-demo) remains a review surface. +Media names stay separate from secondary capability/availability details so prompts and breadcrumbs identify only the target. A Media Scan exposes **Add existing inventory to Library** separately from starting the configured Scan. The default action reviews the dry-run count, including already-present and unsigned records, before writing. The existing browser opt-out may skip that confirmation after a successful admission, but **Review what will be added** always runs the dry run independently and lets the operator restore confirmation. Its label states that the preference applies across Media in this browser. This action admits already published inventory, not the entries a new Scan might later publish. Zero-result feedback distinguishes empty inventory, already-present Files and records skipped because signatures are missing. Mounted Volumes start scanning directly; Tape requires its explicit device/read step and fresh inventory through the selected format. **Add to Library** accepts signed regular archive entries or inventory directories; directory descendants expand in bounded Media-path pages and only ordinary signed content becomes independent Library Files. Empty or unsigned-only selections return an explicit result. Position copies are labeled **Archived copies**, inventory deletion is **Remove from Library**, and **Clean up Library** explains its retention scope before execution. Single-Media operations reject multiple eligible selections with an actionable notice rather than silently doing nothing or choosing one. Context actions use Chonky's filtered action selection. + +The [command-line interface](../operations/cli.md) exposes the same named methods and projections to scripts and agents; [CLI E2E](../development/e2e-test.md) records its executable acceptance and the [local Demo](../development/demo.md#local-demo) remains a review surface. The Media Properties dialog uses the shared detail surface, retaining Tape drive selection and physical inspection results in its body. +Archive, Restore/Scan and Media Properties share the Tape device chooser. It lists all configured +drives with busy reasons, disables busy operation choices and selects a drive automatically only +when exactly one available entry remains. Pending or failed device discovery is distinct from a +confirmed empty list. **Refresh devices** refreshes list facts; **Device details** and **Refresh +device details** explicitly request passive diagnostics, including for a busy entry. Reports keep +unknown facts visible and separate support, availability, validation and evidence. These actions +do not replace cartridge inspection or authorize an operation. In Restore/Scan, changing Tape Media +preserves a pending device-list refresh, which clears its loading state when it settles. Selection +changes invalidate earlier cartridge inspection and device diagnostics; closing or reopening a +dialog also invalidates its device-list reads. Device details +are not polled automatically; [Media/I/O](tape.md#device-observation-and-raw-transport) owns their +physical access rules. + +### Preview + +Video position and duration use the same fields from `days:hours:minutes:seconds`, omitting +leading days and hours only when they are zero in the total duration. The current position +retains those fields even at zero. The first displayed component has no leading zero; +following components have two digits. For example, a 65-second clip starts at `0:00 / 1:05`, +a two-hour clip at `0:00:00 / 2:00:00`, and a one-day clip at `0:00:00:00 / 1:00:00:00`. +Fractional seconds are rounded down; the display retains minutes and seconds for short clips. + +The settings editor uses a searchable Type selector and one card at a time, without repeating the selected type as a card heading. Image/video types use descriptive names; multiple routes of the same type are distinguished by extensions. File extensions use a searchable multi-select: selection enables a route, deselection retains it disabled, and custom extensions remain editable. Per-kind helper capabilities supply suggestions; unconfirmed extensions stay visible rather than being discarded. Switching cards preserves unsaved edits, and Save returns to any card with invalid parameters. Basic fields cover generation enablement, type and output dimensions/format. Advanced holds helper execution, resource limits, extensions and video sampling. PNG quality is inapplicable and disabled without clearing its stored value. Video controls are **Minimum sampling interval** and **Maximum timeline frames**: long videos spread samples across the duration, and keyframe reuse can produce fewer frames. The Supported formats section reports the saved helper, not an unsaved command draft. Check and Retry use that saved helper; a successful settings Save refreshes its report. Failed or pending reads retain prior information and the draft. The master switch defaults off; configured kinds and extensions default on beneath it. Disabling a kind or extension retains its parameters. The helper path, input pixel limit and timeout are read after a file acquires an execution slot; changes do not alter the Job's frozen output dimensions, format or quality. `PreviewService.GetCapabilities` and `yatm-cli preview capabilities` probe the installed helper separately from enablement and asset reads. Explicit Scan or Archive generation requests fail before Job creation when the master switch is off or the helper is unavailable. A later failure creating Archive's companion does not invalidate the prepared Archive. + +Unchecking an extension retains its disabled route. Deleting its selected chip removes that route, allowing corrections or reassignment to another configured generator without discarding the other draft settings. + +## File and Version Display States + +The primary color combines current local presence with known default Restore candidates. It does not measure health or archive recency. + +| Local original | Known selectable archive copy | Color | Short meaning | +| --- | --- | --- | --- | +| Present | Yes | Green | Local and archived | +| Present | No | Yellow | Local only | +| Missing or unlinked | Yes | Blue | Archive only | +| Missing or unlinked | No | Red | No available content source | +| Either axis unknown | Any | Gray | Identify the unconfirmed axis | + +Present requires a current observation; FileLocation existence alone is insufficient. Missing means the associated path was observed absent. Unlinked means no association, not proof that matching bytes do not exist elsewhere. Inaccessible and unobservable originals remain unknown. With unknown current content and no selectable historical copy, archive coverage is unknown rather than definitely absent. A saved version without a selectable Position does not count as an archived copy. Unchecked copies can be candidates; known damaged, missing or unreadable copies are excluded by default. Candidate existence never promises immediate Media accessibility. + +| Additional scenario | Primary color and supplement | +| --- | --- | +| Present; current content has a candidate | Green | +| Present; current content uncovered but history has a candidate | Green + unbacked-change marker | +| Present; current content unknown but history has a candidate | Green + unknown-content marker | +| Present; current content unknown and no historical candidate | Gray: archive unconfirmed | +| Missing/unlinked; latest saved version has a candidate | Blue; distinguish missing from unlinked in the tooltip | +| Missing/unlinked; only an older version has a candidate | Blue + latest-version-unavailable warning | +| Missing/unlinked; every saved version lacks candidates | Red | +| Present; known saved copies are all unusable | Yellow + copy-problem marker when current coverage is known; gray when it remains unknown | +| Original inaccessible; history has a candidate | Gray: local inaccessible, archive known | +| Original unchecked; history has a candidate | Gray: local unconfirmed, archive known | +| Candidate remains, but another recorded copy is abnormal | Green or blue by local presence + copy-problem marker | + +Lists retain at most one supplement: copy/latest-version problem, then unbacked changes, then unknown content. Inspector and tooltips retain all facts. Shape and accessible text accompany color. Directories reserve the same status-column width but do not invent recursive coverage. + +[Copy health](library.md#copy-health) records historical actual-read observations and time. It has no automatic expiry and does not control the primary color. An unloaded Tape is not damaged; successful archive writing is not a read-back check. A new matching-content read on the expected Media and successful Session finalization are required to replace a bad finding. File/FileVersion display states are derived projections, not persisted state machines; [Restore outcomes](jobs.md#restore-outcomes) remain separate. + ## Identical Files [The tool](../../frontend/src/pages/identical.tsx) separates Library connected histories from scoped Location originals. One continuous sparse Chonky list spans all groups with a global scrollbar; unloaded positions retain their row geometry with passive placeholders and visible pages load automatically. One list-level indicator reports pending reads; an error leaves quiet gaps and an explicit Retry rather than rows that claim to be loading forever. Group headers are not file identities. Group selection and actions remain scoped, and the right Inspector follows the last operated member while actions consume the full selection. @@ -247,4 +338,4 @@ Selected members remain pinned while pages are replaced, including selection cha Opening the tool queries nothing: an explicit **Find** starts a global search and repeats it on demand, and a source or scope change clears results and selection. The Locations source has a searchable multi-select whose chips are the selected whole Locations. The retained result supplies bounded rows for arbitrary scroll positions; missing pages retry without discarding loaded pages. The hidden-file toggle switches between two positions computed by the same Find, keeping the previous projection visible until replacement rows and selected positions are ready. A transient read failure keeps that projection; a shorter result clamps the viewport to its new end. The list exposes only **Sort by size**, using the same control as ordinary lists. Groups start largest first; selecting it reverses and remembers the direction. Members stay in ascending path order. The retained indexes supply both directions without another Find. Sorting keeps the current rows visible until the replacement viewport, selected rows and collapsed/removed group positions are ready together. Delete, Keep and Merge wait while a sort remaps row positions; sorting also waits for active file operations. Date sorting, view switching, folders-first and Select all are unavailable on this sparse list. A result lost through expiration or restart asks for another Find rather than silently searching again. Version removal remains available through the shared Saved versions detail. -Independent Remove requests may overlap, including different files in one Location. The action is labeled **Delete**, as in ordinary file lists; its confirmation explains that the selected files move to Trash. Identical Delete and Keep confirmations close after admission; per-file progress and failure remain in the list, allowing another file to be deleted while the first request is pending. A physically removed row disappears when its result arrives; a publication-pending row also disappears with an explicit Library warning. Failed and unprocessed rows retain diagnostics. Changed groups show their remaining file count but are not regrouped until Find; Keep and Merge are unavailable on them. A successful Merge removes its group; partial Keep updates the affected members, including members that were not loaded. Neither these actions nor Inspector or metadata updates automatically start a new Find. [Library](library.md#identical-file-tools) owns matching, merge, Trash and historical Job semantics. +Independent Remove requests may overlap, including different files in one Location. The action is labeled **Delete**, as in ordinary file lists; its confirmation explains that the selected files move to Trash. Identical Delete and Keep confirmations close after admission; per-file progress and failure remain in the list, allowing another file to be deleted while the first request is pending. A physically removed row disappears when its result arrives; a publication-pending row also disappears with an explicit Library warning. Failed and unprocessed rows retain diagnostics. Changed groups show their remaining file count but are not regrouped until Find; Keep and Merge are unavailable on them. Merge confirmation states that the chosen File retains its original while the others move to Library Trash and lose original links without deleting their disk bytes. A successful Merge removes its group; partial Keep updates the affected members, including members that were not loaded. Neither these actions nor Inspector or metadata updates automatically start a new Find. [Library](library.md#identical-file-tools) owns matching, merge, Trash and historical Job semantics. diff --git a/docs/development/code-layout.md b/docs/development/code-layout.md new file mode 100644 index 0000000..5b7e2f3 --- /dev/null +++ b/docs/development/code-layout.md @@ -0,0 +1,37 @@ +# Code Layout and Test Ownership + +Where new Go types, files and tests belong. [AGENTS.md](../../AGENTS.md#repository-layout) owns the +repository layout, and the [architecture overview](../architecture/overview.md) owns module ownership +and dependency direction. + +## Type Roles and File Layout + +Every durable type remains in the package that owns its invariant. When one package needs several +representations of the same subject, names make their lifecycle explicit: + +- a private `...Row` is the exact persisted table shape and carries only persistence behavior; +- a runtime aggregate or snapshot carries execution state but has no ORM tags or hooks; +- a projection is a read result assembled for a particular query and is never saved as a row; +- an adapter translates between an owning representation and a real cross-module interface; +- an `entity` type is a wire or stored-blob contract, not the default internal model. + +Files follow the same roles. A cohesive type may keep its ordinary domain filename; when roles would +otherwise mix, use focused names such as `job_row.go`, `job.go`, `file_projection.go` or +`match_store.go`. Focused responsibility files live inside their existing owner; a catch-all +`models.go` file is not an ownership boundary. +A new subpackage is introduced only when it owns behavior and invariants behind a useful interface; +moving passive structs into a separate package only adds navigation and dependency cost. + +Persistence hooks never assemble presentation state. Queries explicitly load the projection or +snapshot they promise, so production behavior does not depend on callers remembering to disable an +ORM hook. The process composition root creates the Settings module, and attempt admission freezes the +validated effective settings used by a runner. + +## Test Ownership + +Tests live beside the behavior they protect. Row tests own table names, columns and indexes; module +tests own transactions and state changes; interface contract tests run against each real adapter; API, +CLI and end-to-end tests retain only behavior visible at those boundaries. A lower-level test is not +repeated through test-only compatibility methods, and tests for removed commands or superseded wire +shapes are deleted rather than retained as historical documentation. Stable wire numbers, published +formats and supported migration inputs remain positive compatibility tests. diff --git a/docs/development/demo.md b/docs/development/demo.md new file mode 100644 index 0000000..2dd7e3c --- /dev/null +++ b/docs/development/demo.md @@ -0,0 +1,95 @@ +# Local Demo + +`make demo-dev` prepares the same Demo fixture and builds the backend incrementally, then serves +Vite hot reload at `http://localhost:5173`, proxying the selected backend listen address. +`YATM_DEMO_FRONTEND_PORT` overrides the strict frontend port. A port conflict fails startup; +both children stop on signals or either child's exit. Fixture reset remains explicit through +`YATM_DEMO_RESET=1`. `make demo` retains the production frontend build for packaged-browser checks. + +`make demo` builds and serves the disposable review fixture at `http://127.0.0.1:18080`, normally +under the system temporary directory. Restarting preserves reviewer changes. Select an isolated +temporary root and reset it explicitly after fixture or pre-stable format changes. Stop its server +before resetting that root; concurrent reviews use distinct roots: + +```shell +YATM_DEMO_ROOT=/private/tmp/yatm-demo-review \ +YATM_DEMO_LISTEN=127.0.0.1:18090 \ +YATM_DEMO_RESET=1 make demo +``` + +Reset replaces only a validated Demo root whose basename begins with `yatm-demo` and which is below +a temporary directory. Never place valuable files there. The fixture uses normal Library tables, +Job runners, Volume markers and ACP transfers; it has no production mock branch and must never be +used for a physical Tape operation. + +The default fixture keeps five Locations and thirteen Jobs. Setup retains only Jobs needed by current +Location links or distinct review results; saved versions and Preview assets remain. A complete +Documents Scan supplies its LastSync link while later partial Scans supply current results. +Archive preparation collects Incoming's sources directly, and one Scan publishes their content +observations and generates Previews. The retained examples cover these review paths: + +| Fixture | Review coverage | +| --- | --- | +| Library | Annotations, mixed-field search, 101 invoices spanning two 100-result pages, logical organization and Trash, saved and unknown legacy archive dates | +| Documents | Original-only, unknown and covered content under the `archive-review` tag, unarchived edits, three text/image saved versions and date cutoffs, Ignore and dot directories, unadmitted arrivals, empty folders, failed Scan with a reconnected source | +| Shared files | One 201-member group with three dot files, three additional pairs, three/four-member groups, arbitrary scrolling, collapse, hidden rows and independent Delete; a.txt/b.txt connect transitively to unavailable c.txt with different current sizes | +| Unavailable originals | Retained originals under an inaccessible root, cross-Location duplicates and saved-version evidence | +| Incoming | Four Archive sources across two directories, a pending Archive and a completed Scan with image/video Previews | +| Restored files | Preferred Restore destination, pending multi-file Restore and completed adoption of identical existing output | +| Media | Random-write HDD, sequential-write HM-SMR, offline Volume, uninitialized disk, metadata-only LTFS Tape probe and a tar Tape whose profile and member locators await Add to Library; inventory additions/changes/removals, healthy/mismatched/missing Verify findings and a Tape-bound pending check | + +For large browser reviews, opt in with `YATM_DEMO_IDENTICAL_FILES`, forwarded to `cmd/demo` as +`-identical-files`. The integer is the **total number of members across Shared files' Large group +and Many groups together**, including the default 207 members; it excludes the three/four-member +groups, transitive-history example and other Demo files. Values below 207 are rejected before +reset or writes. Any explicit count, including 207, requires reset when reusing an existing root; +omit the option on later restarts to preserve the fixture and reviewer changes. + +```shell +YATM_DEMO_ROOT=/private/tmp/yatm-demo-identical-10k \ +YATM_DEMO_IDENTICAL_FILES=10000 YATM_DEMO_RESET=1 make demo + +YATM_DEMO_ROOT=/private/tmp/yatm-demo-identical-100k \ +YATM_DEMO_IDENTICAL_FILES=100000 YATM_DEMO_RESET=1 make demo +``` + +Run one server at a time or set distinct listen addresses. At 10,000 members there is one +5,000-member group and 2,500 pairs; at 100,000 there are 50,000 members in that group and 25,000 +pairs. Smaller totals retain at least the original 201-member group, with rounding assigned to +that group so every other group is a complete pair. Every eightieth member of the large group +has a dot name. Existing feature examples and the five Locations/thirteen Jobs remain intact. +Setup writes added files in bounded batches, then uses one normal complete Scan of Shared files +to publish signatures and replace its setup Job. Cancellation stops pending batches and drains +active Jobs before closing the catalog. Large setup duration and disk use depend on the requested +count and host. + +For each size, use explicit Find with Library and with Locations selecting Shared files. Review +far scrolling within the large group and across the pairs, collapse/expand, size sorting in +both directions under the [Library sorting contract](../architecture/library.md#identical-file-tools), +hidden files, and independent Delete/Keep in the disposable Location. Repeat Find +after mutations. The small saved-history example still exercises transitive Library grouping. + +Use an explicit Find for either identical source; Locations uses current records rather than +saved-version links. `Photos/preview-video.mp4` is a licensed, silent five-minute Big Buck Bunny +excerpt at 640×360 with an actual native-worker poster, thirty distinct 320×180 timeline frames +in a three-row sprite and VTT spanning 00:00–05:00. The timeline uses the production defaults of +320×180 tiles, a 10-second interval and at most 30 frames. These bundled +assets pass through normal content-addressed Preview publication and require no installed helper. +Its File Note displays the credit and license; [asset attribution and reproduction](../../licenses/demo-video.LICENSE) +records the exact source, tools, commands and checksums. Bundled assets do not establish acceptance +of a current native-helper release. + +`YATM_DEMO_VIDEO=/path/to/clip.mp4 YATM_DEMO_RESET=1 make demo` preserves the optional MP4 override. +It generates derivatives with `yatm-preview` on `PATH` or beside the executable and requires that +native helper. To reproduce the bundled output and check override creation and reuse: + +```shell +YATM_TEST_PREVIEW_HELPER=/path/to/yatm-preview go test -count=1 ./internal/demo \ + -run 'TestBundledVideoAssetsReproduceWithNativeWorker|TestPrepareVideoOverrideUsesNativeWorker' +``` + +Frontend-visible changes update `internal/demo` and its semantic tests. Keep fixtures deterministic, +bounded and representative of actionable, unavailable, empty and error states. Verify with +`go test ./internal/demo`, `bash -n dev/demo.sh`, an explicit reset and a browser smoke test. Demo +review supplements semantic and E2E tests; the exact visible behavior belongs to the UI architecture +and automated tests rather than a fixture transcript. diff --git a/docs/development/documentation.md b/docs/development/documentation.md new file mode 100644 index 0000000..9213158 --- /dev/null +++ b/docs/development/documentation.md @@ -0,0 +1,75 @@ +# Documentation Rules + +How repository documentation is owned, decided and updated. Contributors follow these rules for +every change to a document. + +## Documentation Ownership + +Every maintained fact has one owner. Other documents link to that owner instead of restating it. + +| Location | Owns | Does not own | +| --- | --- | --- | +| Root `README.md` | Released product introduction, quick start and documentation entry points | Development status, architecture detail or repeated runbooks | +| Root `CONTEXT.md` | Domain vocabulary and core domain invariants | Schemas, algorithms, task notes or UI layout | +| Root `AGENTS.md` | Contributor rules, safety gates, repository layout and required verification | Product or architecture contracts | +| Root `TODO.md` | Concise confirmed pending work with source Issue links | Detailed designs, implemented behavior or completed-work history | +| `docs/README.md` | Developer entry, complete documentation index and the pre-stable compatibility policy | Daily-use instructions or documentation rules | +| `docs/operations/index.md` | User entry for installation, configuration and operation | Architecture, contributor rules or test procedures | +| `docs/architecture/` | Current implemented behavior, module interfaces, invariants and durable constraints | Proposals, implementation plans, acceptance transcripts or superseded behavior | +| `docs/operations/` | Packaged user guides: current setup, migration, configuration, use and maintenance | Domain definitions, architecture specifications, test procedures or measurement results | +| `docs/development/` | Contributor procedures: documentation rules, code layout, checks, E2E, performance and the Demo | Release gates, user procedures or product contracts | +| `docs/release/` | Maintainer release procedures: SOP, builds, package acceptance, the physical Tape gate and physical measurements | Contributor checks, user procedures or measurement research | +| `docs/research/` | Public technical studies, evidence, limitations and evaluation of possible approaches | Product guarantees, implementation backlogs, private incident reports or raw acceptance logs | +| `.agents/skills/yatm/` | Installed agent operating and safety instructions | Human-facing architecture or change history | +| Component `README.md` files | Commands and usage local to that component, such as `frontend/` conventions | Product behavior owned by `docs/` | + +## Decision Authority + +Within overlapping scope, broader accepted constraints bound narrower descriptions: the repository +engineering rules, the domain invariants and compatibility policy, the architecture overview, area +architecture, operations guides, and finally implementation and tests. A lower level may provide +evidence and implementation detail, but its existence cannot justify complexity or override a broader +decision. When they conflict, update the lower-level documents, code and tests together. + +The repository intentionally has no documentation archive, ADR archive or redirect stubs. +Git already retains deleted text and rationale. Root `TODO.md` records concise confirmed pending +work; detailed implementation plans belong in an Issue or pull request. Remove completed TODO +items instead of retaining a work history. Public research follows the scope above. +Published change notes belong to GitHub Releases. Raw acceptance logs, checksums, +measurements, screenshots and host-specific reports belong in CI artifacts rather than Markdown. + +## Update Workflow + +1. Establish the intended behavior from the decision authority above, then inspect `CONTEXT.md`, the + owning document and the implementation. A mismatch is something to resolve, not a reason to + promote the current implementation into a requirement. +2. Update the implementation, semantic tests, owning architecture document and affected operating + guide together. Describe the final behavior in present tense; remove the replaced text instead of + marking it historical, implemented or superseded. +3. Keep unsettled implementation designs in a PR or Issue. Public studies may evaluate alternatives + in `docs/research/`, with evidence, assumptions and remaining proof clearly identified. After + implementation, update the current architecture and link to research for supporting evidence. + Update or remove invalidated research conclusions instead of accumulating superseded studies. +4. Keep implementation detail only when callers or maintainers must know it to use the module + correctly. Function names, CSS placement, one measurement, one candidate checksum and step-by-step + development history belong in code, tests or verification output. +5. Keep repository documentation self-contained. Every repository-relative link or file reference + must resolve to a tracked file, or to a new file committed in the same change; external references + use stable public URLs. Private notes, ignored artifacts and machine-local paths are never sources + for repository documentation. +6. Before delivery, run `make check` or at minimum `node dev/check-documents.mjs` and + `git diff --check`. The document check enforces the allowed tree, complete index, valid relative + links and the absence of private/history references. + +For research, use official specifications and source code together with relevant original community +reports and maintainer responses. Record what was observed, what was actually resolved and which +explanations remain hypotheses; a forum's resolved label alone is not evidence of a verified fix. +Use stable public citations and concise summaries. Keep credentials, local paths, device identifiers, +private host details and raw logs out of public studies; include only the technical context needed +to assess the evidence. Hardware measurements establish the tested configuration's behavior, while +general guarantees require an applicable specification or other evidence with a defined scope. + +Use English Markdown, short topic-based kebab-case filenames and repository-relative links. Current +architecture filenames are version-independent. Adding a new documentation file requires a distinct +current owner that cannot be expressed clearly in an existing file; update this index in the same +change. diff --git a/docs/operations/e2e-test.md b/docs/development/e2e-test.md similarity index 58% rename from docs/operations/e2e-test.md rename to docs/development/e2e-test.md index d659012..1990114 100644 --- a/docs/operations/e2e-test.md +++ b/docs/development/e2e-test.md @@ -1,10 +1,10 @@ # E2E Testing -Status: Maintained validation for the current v1 development line. See [test environment safety](testing.md) before selecting a host or fixture. Actual release results belong to the release workflow, not this procedure. +Status: Maintained validation for the current v1 development line. See the [development checks](testing.md) before selecting a host or fixture. Actual release results belong to the release workflow, not this procedure. Primary business acceptance uses actual `yatm-cli` subprocesses and production HTTP/gRPC-Web transport. The [complete installation workflows](../../e2e/workflows_test.go) and [indexing recovery workflows](../../e2e/indexing_cli_test.go) also start the actual `yatm-httpd` binary. Development runs build both binaries from the checkout; release acceptance sets `YATM_E2E_BIN_DIR` to the extracted candidate directory and uses those programs unchanged. Each test owns isolated configuration, databases, source files, output directories and loopback listeners. No production installation or Tape device is used by local tests. -Local/CI Mounted Volume, Preview-policy and LTFS fixtures use the same CLI process adapter, with an in-process server only where generator configuration, finalization faults or detailed checkpoint assertions require it. The adapter has explicit command mappings and **no direct-RPC fallback**. Direct filesystem writes represent external changes, not hidden catalog setup. Public registration, scanning, selection expansion, archive, restore, annotation, import/export and Job mutations go through CLI commands. Remote package acceptance instead uses only packaged application binaries and [local SSH orchestration](#remote-packaged-binary-acceptance). +Local/CI Mounted Volume, Preview-policy and LTFS fixtures use the same CLI process adapter, with an in-process server only where generator configuration, finalization faults or detailed checkpoint assertions require it. The adapter has explicit command mappings and **no direct-RPC fallback**. Direct filesystem writes represent external changes, not hidden catalog setup. Public registration, scanning, selection expansion, archive, restore, annotation, import/export and Job mutations go through CLI commands. Remote package acceptance instead uses only packaged application binaries and [local SSH orchestration](../release/package-acceptance.md#remote-packaged-binary-acceptance). New primary capabilities require a CLI command, a repeatable CLI E2E scenario and an entry in the matrix below. Move existing CLI-expressible business steps to subprocesses instead of keeping a parallel private-service workflow. Command availability and RPC coverage tests do not prove end-to-end acceptance. Retain unit/integration tests for bounded traversal, failed Library/Job commits and supported recovery boundaries; CLI smoke tests cannot replace those checks. @@ -39,9 +39,9 @@ The scenario names below are executable tests, not a release claim. Record actua | Export/import | `library export/import` | Complete-installation tests; LTFS archive test | Version 1 metadata without Settings despite view filtering, tags/Note, imported roots immediately usable, legacy whole-object import | macOS/Linux; LTFS Linux | | Job navigation data and lifecycle | `job list/changes/get/log/wait/delete` | Complete-installation tests and Volume test | Location/Media resource history, kind and target filters, machine-readable results, independent Scan Jobs, explicit deletion | macOS/Linux | | LTFS append/finalization/capacity/integrity | `archive write tape format/append`, `restore run tape`, `verify create/run/entries`, `media inspect/delete` | `TestLTFSArchiveRestore`, `TestLTFSFullTapeSpansMediaAndRestores` | Final-Index/unmount boundaries, continuous prefix, two-Media restore, storage order, actual Tape reads and healthy observations | Linux, official file backend only | -| Physical Tape release gates | Same CLI commands, separately documented explicit fault probes | [Physical Tape E2E](physical-tape-e2e.md) | Hardware identity, append, restart restore, end-of-Media evidence | Explicitly assigned scratch hardware only | +| Physical Tape release gates | Same CLI commands, separately documented explicit fault probes | [Physical Tape E2E](../release/physical-tape-e2e.md) | Hardware identity, append, restart restore, end-of-Media evidence | Explicitly assigned scratch hardware only | -Preview assets and Library JSONL backup use HTTP; full original or archived file bytes do not. Business setup remains CLI-driven. Physical barcode-mismatch acceptance checks the public CLI refusal; backend Session tests separately check rejection before encryption, formatting or mounting. The [physical suite](physical-tape-e2e.md#pt-04-reject-a-different-requested-barcode) records that coverage boundary. +Preview assets and Library JSONL backup use HTTP; full original or archived file bytes do not. Business setup remains CLI-driven. Physical barcode-mismatch acceptance checks the public CLI refusal; backend Session tests separately check rejection before encryption, formatting or mounting. The [physical suite](../release/physical-tape-e2e.md#pt-04-reject-a-different-requested-barcode) records that coverage boundary. Run the full suite with Tape opt-ins unset for ordinary local acceptance: @@ -65,159 +65,6 @@ File-operation transport tests additionally cover partial-result exit codes, abs The [integrity CLI tests](../../e2e/integrity_cli_test.go) supplement semantic tests for Media finalization failure, missing baselines, cancellation, unreadable entries, guarded health publication and explicit Restore publication failures. Verify and Restore must not fabricate successful catalog observations when only physical bytes were produced. Resource-filter departures are covered by catalog change-feed tests; UI tests cover restoring filters and navigation. -## Release Candidates and Upgrades - -Follow the [Release SOP](testing.md#release-sop) for source preflight, candidate assembly, acceptance and publication. The [release build guide](testing.md#release-backend-builds) owns toolchains, the required archive set and the separate accepted-candidate upload procedure. Hosted CI skips opt-in LTFS and physical Tape tests. Remote release acceptance replays the public CLI workflows with the official LTFS file backend, using packaged application binaries. Physical Tape acceptance follows the affected source and nonphysical checks and may use a reviewed local build while CI proceeds; [Check Scope and Reuse](testing.md#check-scope-and-reuse) owns final-artifact checks and evidence reuse. - -For local/CI harness runs, set `YATM_E2E_BIN_DIR` to an extracted candidate and run the full suite without a `-run` filter. Set `YATM_E2E_LTFS=1` only in a Linux harness environment with the official file backend. Physical Tape uses the separate [package controller](physical-tape-e2e.md#automated-stages) and explicitly assigned scratch hardware. Do not move this harness or source tree to the remote package-acceptance host. Record every pass, failure and skipped test with its reason. Use a temporary filesystem supporting user xattrs; the full-capacity LTFS fixture needs at least 12 GiB of free space for its source, virtual Media and restored bytes. - -A missing or non-executable candidate fails explicitly. Package checks reject altered checksums, missing programs, mixed identities, active configuration, development dependencies and hidden filesystem metadata. Packaged guides retain local links; source references resolve at the matching public tag. - -### Remote Packaged-Binary Acceptance - -Use repeatable `--case NAME` to select exact existing case names. The controller automatically -runs their required cases in normal order and always allocates a fresh isolated environment. -Omit `--case` for the complete selected profile. The report's `case_scope` distinguishes requested -cases, prerequisites, executed cases and skipped cases with reasons; a passing subset does not -establish full acceptance. The legacy profile supports the same selection: upgrade is required -before historical repair or JSONL roundtrip, while review/decline/abort checks do not run those actions. - -Reports use monotonic elapsed times for preparation, transfer, installation, workflows and cleanup, -plus individual commands, cases and the whole run. Independent read-only host, tool and identity -checks run in batches of at most four; each retains its result and exit status even if another fails. -Nested or concurrent durations overlap and must not be added to claim total elapsed time. - -Run the maintained controller over SSH or directly on the explicitly selected test host with `--host local`. Verify and extract the reviewed release archive into an isolated attempt directory; invoke only its shipped YATM programs. Read their offline `--version` output and compare the commit/version and archive checksum before starting. For host-local execution, transfer the controller and its package validators with the artifacts and operating instructions under [Remote Candidate Acceptance](testing.md#remote-candidate-acceptance). Do not deploy an application checkout or separately built harness. - -The [package controller](../../e2e/package_acceptance.py) requires Python 3.9 or newer and Node.js for -the existing package validators; SSH mode also requires OpenSSH. The explicitly selected -Linux amd64 test session must permit an isolated systemd installation and provide the installer's -dependencies plus `timeout`, `ss`, `setfattr` and `getfattr`. It does not install host dependencies. - -```bash -python3 e2e/package_acceptance.py \ - --host isolated-test-host --test-parent /srv/yatm-tests \ - --archive "$MAIN_ARCHIVE" --preview-archive "$PREVIEW_ARCHIVE" \ - --version "$VERSION" --commit "$COMMIT" --out "$NEW_PRIVATE_REPORT_DIRECTORY" --ltfs -``` - -Checksum companions must be beside each archive. Preview validation also requires its matching -corresponding-source archive beside the controller's Preview package; it is validated without being -installed. The controller checks package identities before preparing the host, allocates a unique -root and service name, uses real systemd and the package installer, and records CLI exits, JSON and -output-byte assertions in its evidence directory. -Its workflows cover literal UTF-8 and invalid-byte filenames, exact signed file times, Scan's -all-source preparation barrier and overlapping multi-Location selections, read-only Job creation -inputs, explicit new submission, recorded/live Search, dry run and recoverable Delete, Volume -Archive/Restore/Scan, explicit zero/pre-epoch Restore cutoffs, JSONL export/import with rejected -timestamp shapes preserving existing data, and Verify findings -for healthy, damaged, missing and explicitly repaired copies, including result paging and health. -Volume inventory checks added/changed/removed files, a stat-preserving edit under force-read, -explicit and repeated inventory import, trim, metadata-only deletion, marker registration and -reconstruction through another Scan. -With a Preview archive, it also generates an image with a literal backslash in its name and the -licensed five-minute Demo video, checks HTTP asset bytes and thirty 320x180 timeline frames, and -compares missing-only preservation with explicit regeneration. `--ltfs` adds official file-backend -format/append, finalized-prefix/pending-suffix behavior at full capacity, explicit second-Media -selection, Restore and Verify. It requires the optional LTFS dependencies, `findmnt`, `truncate` -and 16 GiB of free space including the installation backups (4 GiB without LTFS). Its cartridge paths -are newly allocated beneath the attempt root; -physical device paths are not accepted. Omitting this flag records the missing acceptance explicitly. -SQLite acceptance enables WAL on the stopped test service, writes through the CLI, restarts with -WAL disabled and compares committed Catalog data. Two same-version replacements check decline/EOF, -whole-directory backup contents, hidden files, -links, permissions, user attributes, customization and removal of obsolete managed files. A -same-root recovery retains the displaced tree and restores Catalog/Job identities without overlaying -new-only files. These cases do not simulate every installer failure stage or a cross-version upgrade. -Browser form interaction, legacy migration and physical Tape retain their separate acceptance -requirements below. Helper-only RAW/codec acceptance remains separate. - -Cleanup verifies service ownership and stops only this attempt's service. Failed runs retain their -root and transcripts; successful runs remove the root unless `--keep-root` is selected. The controller -stops the test service and unmounts any remaining owned FUSE paths before removing successful -fixtures. Unexpected mounts or unverifiable ownership retain the root and fail cleanup. -`make test-package-tools`, also included in `make check`, checks argument quoting, fixture encoding, -exit/timeout handling and cleanup without connecting to a host. -The report covers its named scenarios; it is not an overall release approval. - -`YATM_E2E_BIN_DIR` remains a local/CI selector. Some detailed Go fixtures start the checkout server -in-process even when their CLI comes from that directory. Those tests retain their source-level -coverage and do not substitute for this remote whole-package boundary. - -Select an already-mounted, xattr-capable filesystem with sufficient free space, then create one owned test root for the extracted package, fixtures, isolated installation and results. Use the archive's `install-release.sh` for remote installation acceptance; the README's exact-tag download remains the normal user bootstrap. Check free space for the complete fixture, including archive and Restore outputs, rather than relying on the system temporary directory's capacity. - -Use the same public CLI steps and observable assertions as the coverage matrix: registration, live Files operations, Scan policies, archive/restore, versions, copy verification, import/export and Job lifecycle. Compare exit codes, JSON, Job terminal results and actual output bytes from the local controller. Exercise real systemd installation/update and the official LTFS file backend. Retain invocation transcripts and artifact hashes privately. - -Low-level fault injection requiring a custom server remains local/CI coverage. Record any remote scenario that cannot be expressed using the shipped tools as unrun, with its exact reason; previous source-based or separately compiled harness runs do not satisfy this gate. Physical Tape remains a distinct explicitly authorized test. - -### Packaged browser acceptance - -The maintained Playwright suite in `frontend/e2e/` operates on disposable Demo data -and deletes two files. Create a fresh Demo root locally from the candidate source, -with `go run ./cmd/demo -root /tmp/yatm-demo-browser-UNIQUE -reset -listen 127.0.0.1:18080 --identical-files 1001`. Then start the extracted candidate's `yatm-httpd` from its -package directory with `-config /tmp/yatm-demo-browser-UNIQUE/config.yaml`; this loads -the packaged frontend and server while all data stays inside the Demo root. The Demo -generator prepares fixtures only. Never use a regular installation or physical Tape. - -Run `YATM_BROWSER_URL=http://127.0.0.1:18080 YATM_BROWSER_COMMIT=FULL_COMMIT -pnpm --dir frontend test:e2e` after installing Chromium as described in the -[frontend guide](../../frontend/README.md#packaged-browser-acceptance). Record the main -archive checksum alongside the output. Stop the owned server after the run. CI runs -this same suite on the extracted Linux package. Browser acceptance covers actual -rendering and interaction; backend large-data tests own million-entry completion. - -### Installer acceptance - -Installer semantic tests cover read-only checks, version ordering, cancellation, file ownership and failure boundaries. Actual systemd acceptance uses unique `--install-dir` and `--service` values on an isolated Linux host with candidate `--archive` and `--checksum` inputs. The cases below are required acceptance, not claims that the candidate has passed them: - -| Workflow | Public command/entry point | Required assertions | -| --- | --- | --- | -| Fresh installation | Exact-version `install-release.sh`, `--config` | Shipped binaries, real systemd/API/frontend readiness, Volume-only configuration, template ownership and optional Skill. | -| Read-only review | Installer `--check`; `yatm-migrate -phase preflight` | No persistent attempt, database write, quiesce or service interruption; pass/manual/blocker distinctions; Tape scripts never run. | -| Pre-stop rejection | Installer confirmation/package/version checks | Cancel, EOF, unknown version, missing packaged migration guide, wrong checksum and Busy preserve service PID/state and active file hashes. | -| Legacy activation | Installer; `yatm-migrate` Prepare/Commit/validate/cleanup | Guide text comes from the verified archive; two explicit approvals; retained report on Prepare failure/cancel; item-by-item Library/Job/log validation before cleanup. | -| Configuration review | `yatm-migrate` config-plan/config-check/config-apply; installer | Unified YAML diff and Settings summary before stop; stale inputs rejected; explicit access preserved, legacy authorization materialized; saved/deleted registrations and Preview preferences preserved; idempotent conversion. | -| Repeated upgrade | Two successive installer updates, same-version replacement and unchanged rerun | Whole-installation tar includes hidden files, hard/symbolic links, permissions and supported xattrs; only `.backup/` excluded; archive/source verified; old checksums unchanged; successful scratch removed; unchanged rerun offers Skill without another archive. | -| Customization | Installer update | Reviewed config changes only; scripts/helpers, unit, permissions, original working-directory path meanings unchanged; complete replacement removes obsolete managed resources. | -| Failure and rollback | Installer faults plus documented in-root recovery | Backup/Prepare/Commit/config-apply/cleanup/replacement/readiness failure retains reports; uncertain mutation stays stopped; failed tree saved before whole-archive restore; `.backup/` retained. | -| Historical repair | `yatm-migrate -phase repair-job -backup-root ...` | Operates after active legacy tables are removed; every repaired item/log matches backup evidence; already frozen Restore destination survives config changes. | -| Old installation entry | `main` branch's legacy `install-release.sh` | Full parameter validation and cross-major rejection; `v0.1.x`-to-`v0.1.x` update remains available. | - -Compare configuration/script/helper/unit hashes, Catalog contents and original service state at each boundary. Inject faults only in the isolated test's candidate/environment; remove only its own service registration afterward. Persist exact candidate commit/archive hashes, stdout/stderr and each pass/fail/skip reason outside the public repository. - -Skill acceptance uses an isolated ordinary-user home and the pinned public Skills CLI from [installation](install.md#optional-agent-skill). Test the real interactive picker and existing-target cancel/replace, repeat through sudo from a root working directory, and exercise missing dependencies, non-TTY input and source-read failure. Compare installed copy hashes and ownership; cancellation must preserve existing content and optional failure must leave YATM installed. - -Real legacy archive acceptance uses an approved timestamped copy, input checksums, independent Library/Job expectations and a full CLI import/export roundtrip. Run Prepare → Abort → Prepare → Commit → validate → cleanup, repeat idempotent phases, and repair a historical Job from the unchanged backup. Compare every historical item, frozen Restore path and archived log, rather than only Catalog counts. Preserve reports privately and record absent fixture categories explicitly. Production updating follows the public guide only after candidate acceptance; it is not a migration test fixture. - -The same package controller automates this flow for a copied `v0.1.x` SQLite metadata fixture: - -```bash -python3 e2e/package_acceptance.py \ - --host isolated-test-host --test-parent /srv/yatm-tests \ - --archive "$MAIN_ARCHIVE" --version "$VERSION" --commit "$COMMIT" \ - --legacy-package "$PUBLISHED_LEGACY_LINUX_ARCHIVE" \ - --legacy-fixture "$APPROVED_METADATA_ARCHIVE" --out "$NEW_PRIVATE_REPORT_DIRECTORY" -``` - -The metadata tar contains root-level `tapes.db`, its retained WAL if present, and installation-relative -`captured_indices`, `job-logs`, `write-reports` or transitional `jobs` evidence as applicable. -The controller copies only those data members. It rejects links and escaping paths, excludes saved -configuration and scripts, and supplies an isolated configuration with no Tape devices or live -Locations. The old server never starts. The exact candidate installer exercises declined Prepare, -Abort, activation, validation and cleanup; the candidate CLI performs a complete JSONL roundtrip. -The stopped installation then exercises repeated cleanup and historical Job repair. - -[Independent metadata checks](../../e2e/legacy_metadata.py) compare copied legacy facts with the -migrated SQLite rows and Job files, including nanosecond instants and archived logs. This checker -supports completed Archive history with submitted inline sources and uniquely matched logged Media. -It rejects other retained Job kinds/states and transitional Job databases; the report records missing -Restore and other-state coverage. Compressed metadata needs `zstd` on the local `PATH`. -Run Python without optimization because these checks use assertions. A missing fixture category is -not acceptance. Real metadata, extracted copies and detailed transcripts remain in the private -report directory and must not be uploaded as public CI artifacts. Unit tests generate synthetic -inputs. The original copied archive is read-only and its checksum is rechecked after acceptance. - ## Mounted Volume The Volume test uses an ordinary temporary directory as an already-mounted offline disk. It runs the real ACP copy flow and verifies: @@ -307,7 +154,7 @@ YATM_E2E_LTFS=1 YATM_TEST_PREVIEW_HELPER="$PREVIEW_HELPER" \ go test -tags=e2e -v ./e2e -run 'TestLTFS(ArchiveRestore|FullTapeSpansMediaAndRestores)' ``` -The command above is not the remote package-acceptance procedure. On the isolated host, replay its public CLI scenarios using [packaged binaries and local orchestration](#remote-packaged-binary-acceptance); retain internal checkpoint/fault assertions in the local/CI tests. +The command above is not the remote package-acceptance procedure. On the isolated host, replay its public CLI scenarios using [packaged binaries and local orchestration](../release/package-acceptance.md#remote-packaged-binary-acceptance); retain internal checkpoint/fault assertions in the local/CI tests. Regular `go test ./...` runs do not include the tagged E2E suite. Physical-media behavior remains covered by the separate physical Tape E2E suite before release. diff --git a/docs/development/performance.md b/docs/development/performance.md new file mode 100644 index 0000000..991b0f8 --- /dev/null +++ b/docs/development/performance.md @@ -0,0 +1,242 @@ +# Performance Acceptance + +Use fixed, representative benchmarks to record current interface performance and detect +regressions. Compare a repair with its preserved pre-repair source, a pull request with its exact +base, and a release with its recorded accepted source baseline. Do not silently replace that +baseline or require comparison with the first Alpha release. + +Run each standard Go benchmark **once per source**, with `-count=1 -benchmem`. Go calibrates the +iterations within that run; there is no minimum number of separate samples or confidence-interval +gate. Use the same reviewed harness, fixture sizes, Go patch version, SQLite mode, CPU setting +and filesystem for both sources. Measure baseline and candidate serially, with other builds, +tests and agents stopped during timing. Record source identities, commands and raw results; +a skipped case, missing metric or incomplete run is not evidence. + +The [suite manifest](../../dev/performance/suites.json) selects the appropriate scope: + +| Tier | Coverage | Use | +| --- | --- | --- | +| `fast` | 10,000-entry List, Ignore, Search and Scan; 1,000/10,000-entry Archive result pages | Related pull requests and local changes | +| `critical` | List/Search/Scan and Archive result pages with 100,000 entries | Traversal, SQL, projection and buffering changes | +| `release` | Critical List/Search/Scan, Archive pages at 1,000/10,000/100,000/1,000,000 entries, and candidate-only million-entry correctness checks | Release acceptance | +| `migration` | 10,000 Files and 20,000 physical copies on disk SQLite | Legacy archived-content reconciliation changes | +| `read-fast` | Representative tar/LTFS planning benchmarks and all five development seeds | Restore planning development and affected pull requests | +| `read` | The same planning benchmarks, all five development and five acceptance seeds, and fixed references | Complete Restore route-quality acceptance | +| `read-resources` | Streamed 100,000/1,000,000-item budget and cancellation cases, plus production tar evaluator cases | Restore planning resource and cleanup acceptance | + +For List/Scan choose one of `fast`, `critical` or `release`; do not repeat the smaller tiers inside +a release run. Use `migration` separately when its reconciliation changes. Large-data correctness +checks remain separate from timing and run once. Frontend changes also use the relevant list +interaction and virtualization tests in the frontend suite, and the +[frontend benchmark](../../frontend/README.md#frontend-benchmarks) uses Vitest's native comparison +with one invocation per source. Browser tests cover actual rendering and interaction. Shared loading or virtual-list changes +include a cold route, deliberately delayed code/data responses, an error and explicit Retry, +and forward/reverse/distant scrolling with long rows. Verify request counts, selected menus +and real scroll geometry: a mocked virtualizer cannot establish those layout properties. + +List/Search fixtures associate half the entries with catalog Files and request attributes, +status, operations and navigation. They cover random/reverse names, 16 directory levels, +102 Ignore rules with exclusion and negation, and the first and next 100-row Search pages. +List records first-batch and complete-response latency with a receiver retaining all rows; +Search records each page's response latency. Both report allocations. Setup is outside the timer +and Go's benchmark loop reuses the fixture during calibration. The million-entry List check +verifies completeness, ordering and bounded batches with a receiver retaining only counters and +the previous name. Go `B/op` excludes native SQLite memory; these checks do not claim a peak RSS bound. + +Scan covers one selection across one, three and nine Locations, a small physical selection in a +large recorded Location, and renamed originals across three Locations. Each case reports Create, +completion, first-result-page and total latency, plus allocations. Scan intentionally uses +`-benchtime=1x -count=1`: it modifies catalog state, so a second operation on that same catalog +would measure a different workload. Catalog setup and Job deletion remain outside the timer. +The million-entry Scan check verifies completion and all result pages once on the candidate. + +Result browsing uses `go test ./internal/executor/scan -run '^$' -bench '^BenchmarkScanListEntries$' +-benchmem -count=1`: a retained 100,000-entry manifest, first-page counts, forward/reverse +cursors and far offsets, with warm or reopened runners and Preview presence checks. Fixture +creation is outside timing. This isolates the public handler from Scan execution, RPC transport +and browser rendering. Keep its small fixture correctness check in the regular Go suite. + +Archive result paging uses `go test ./internal/executor/archive -run '^$' +-bench '^BenchmarkArchiveManifestPages$' -benchmem -count=1`. It queries retained on-disk +manifests of 1,000, 10,000, 100,000 and 1,000,000 synthetic files in 200-row pages, including an initial +total, forward/reverse cursors and a far offset. Fixture construction and correctness +checks are outside timing. This measures manifest queries and result projection, not +Archive preparation, RPC transport, browser rendering or physical Tape I/O. + +Legacy migration changes use the same collector with `--tier migration`; it runs +`BenchmarkPrepareArchivedVersions` once on each source with `-benchtime=1x`, records identities +and feeds the same comparison command. Each operation consumes fresh on-disk staging tables, +including duplicate copies across pages; setup and result-count checks remain outside the timer. +This focused check measures archived-content reconciliation, while complete migration acceptance +uses the packaged installer and a copied legacy backup through the +[package controller](../release/package-acceptance.md#remote-packaged-binary-acceptance). + +Restore planning separates deterministic route quality from representative execution timing. +The common harness runs against both the frozen old source and the candidate with identical +inputs and models. `read-fast` uses the development seed set; `read` also includes the separate +acceptance seed set. Each report retains the complete case inventory, generator version, +seed, selection and population sizes, input/model hashes, full inputs, LOGICAL and retained +routes, best working cost, adoption result and termination reason. Small exact references +enumerate every complete file permutation under the actual initial-state and completion semantics. +Nominal admission makes their eligible and unconstrained minima coincide; sensitivity results remain +separate from search. Fixed references supply signed cost gaps for both implementations; +falling short of a reference does not stop measurement or discard a complete route. +The fixed mixed308 case retains its measured LTFS layout and the fixed 1.5 TB population retains +its original mixed sizes and 20% selection. Synthetic boundary cases may span more logical data +than a physical LTO-5 cartridge: their purpose is to force distinct continuous runs and exercise +31/32/33-run search boundaries, not to predict that cartridge's capacity or elapsed time. +The quality and representative timing harnesses use the same 300-second budget on both sources, +matching the Device default; the fixed 1.5 TB workload must not receive a shortened acceptance +budget. Explicit resource deadline/cancellation tests retain their separate budgets. Reports retain +bounded per-pass search diagnostics so exhausted budgets can be explained without rerunning or +resampling the workload. +Both read tiers check that the scenario budget reaches the instrumented production evaluator; +quality and timing consumers cannot independently override it. +The two fixed 1.5 TB cases retain their LOGICAL, generated and selected costs, including a tied +LOGICAL fallback. Complete the selected experiments before comparing route quality, planning +cost and resource use; no minimum improvement or per-case no-regression rule selects the winner. + +The [route comparison](../../dev/performance/read-quality.mjs) reports wins, ties, losses and +exclusions overall and by backend/family, with median, P90 and worst absolute/relative cost +changes and exact-reference gaps. These percentiles describe the fixed input population, +not timing confidence. Every case retains its input and routes, including the identified worst +case. LOGICAL-to-old, LOGICAL-to-new, old-to-new, best-working-to-retained and model-plus-planning +costs remain separate. A complete route retained at budget expiry remains comparable and carries +an explicit termination label; missing models, skipped cases and failed runs retain their status +and cannot provide an invented cost. Every nominal increase, fixed-reference gap and parameter +tradeoff stays visible. Policy labels are descriptive metadata, not a validator allowlist or an +acceptance rule. The comparison completes even when a route is slower than LOGICAL, an old +implementation or a reference. Member coverage, replay consistency and evidence integrity remain +correctness checks. Synthetic one-factor rescoring of locate, longitudinal and read coefficients +at 0.5/1/2 is sensitivity evidence, not physical-device evidence. + +Representative planning benchmarks use one fresh operation (`-benchtime=1x`) per source and +report total time/allocations, indexing, search/scoring, standalone full retained-route scoring, +nominal provider evaluator calls (`evaluations/op`), optional all-model planner diagnostic counts +(`total-evaluations/op`), temporary disk bytes, modeled route cost and model-plus-planning cost. +Selections of at most 200 members also report `lower-bound-model-s/op`: a Held-Karp bound on +the same modeled costs, below every complete route, so its distance to the retained cost bounds the +remaining search gap. It is a reported quantity, not a pass threshold; larger selections omit it. +The all-model count includes nominal and variant scoring calls; an older source without that +diagnostic reports it as unavailable, and the pair comparison does not compare it. Process +high-water RSS includes native SQLite allocations and is cumulative across the test binary; +it is not an isolated per-case peak. The explicit `read-resources` tier streams large inputs, +checks complete consumption or authoritative-manifest fallback, exercises cancellation during +ingestion and after optimization state allocation, and verifies temporary-directory cleanup. +On cancellation, `plan-ns/op` includes cleanup performed before `PlanReadSequence` returns; +`consume-cleanup-ns/op` then measures post-return settling and checks. For successful plans, +the latter also includes execution-index traversal and `Close`. +Its disk sampling includes SQLite journals and may miss peaks between samples. Resource and +CPU/allocation increases above 10% remain investigation warnings rather than automatic failures; +route costs and parameter sensitivity are likewise reported for review without automatic rejection. + +Use repeatable `--package ./internal/apis`, `--package ./internal/executor/scan` or +`--package ./internal/executor/archive` to compare only that package's suites within the +selected tier. Package names must match the manifest; +structural checks and common harness files follow the selected packages. `pair.json` records +requested, selected, executed and skipped packages and structural scope. A completed subset is +not a full release comparison. Baseline and candidate use independent mutable temporary fixtures. +The manifest and shared harness are captured once and applied identically to both sources. + +Fixture-only changes can measure preparation separately with +`go test ./internal/apis -run '^$' -bench '^BenchmarkFilesFixtureSetup$' -benchtime=1x -benchmem -count=1`. +It measures fresh file creation and catalog seeding at 10,000 and 100,000 entries; deferred cleanup +is outside the timer. It is not a List/Search runtime comparison. Reuse their existing measurements +when their implementation and equivalent input distribution remain unchanged. + +The collector snapshots the supplied sources and installs the same reviewed harness in both. +It records commits and actual source hashes, including dirty changes, then compiles both suites +before measuring. Each suite executes its standard Go test binary once per source. This is the +same benchmark mechanism as `go test -run '^$' -bench PATTERN -benchmem -benchtime=1s -count=1`; +precompilation keeps compilation out of the measurement window. It adds no custom sampling loop. + +```shell +node --test dev/check-performance.test.mjs dev/performance/*.test.mjs + +PERF_OUTPUT="$(mktemp -d)/pair" +node dev/performance/collect.mjs \ + --baseline "$PERF_BASELINE" --candidate "$PERF_CANDIDATE" \ + --harness "$PERF_HARNESS" --out "$PERF_OUTPUT" \ + --tier fast --environment "$PERF_ENVIRONMENT" --idle +node dev/check-performance.mjs "$PERF_OUTPUT" +``` + +Set the checkout paths and environment label explicitly. Output must be a new directory outside +all source checkouts. Defaults are `--cpu 2`, `--benchtime 1s` and `--cgo 1`; use `--cgo 0` for +pure-Go SQLite. The harness's `go.mod` selects the exact Go patch toolchain. Only ordinary source +files are supported; symlinks/submodules and incompatible harnesses fail instead of silently +changing the inputs. Sources are measured on the same host, not compared with unrelated old +machine timings. Ambient Go configuration and workspace/module edits are disabled. + +The [comparison](../../dev/check-performance.mjs) reports each measured time/allocation value and +percentage change. An increase above 10% flags the case for review; it does not prove a code +regression or fail a completed comparison. CI highlights these flags as warnings. Missing, invalid +or failed measurements still fail the command. Review the flagged implementation and measurement +conditions before accepting the change; if necessary, rerun only that case on both sources. Keep +the initial result and explain the finding. Finish the comparison before selecting an implementation; +weigh observed route quality, planning cost, resource use and applicability together, including +losses and tradeoffs. Do not repeat the whole suite until it passes or automatically replace the +baseline. The comparison reports measurements, not release approval. + +Keep `pair.json`, raw `.bench` files, structural logs and `comparison.json` as verification +artifacts outside maintained documentation. Read tiers additionally seal paired full-case JSON +reports and logs. Failed quality runs retain full inputs and any partial routes for diagnosis; +failed or unexecuted cases cannot complete acceptance. An interrupted collection remains incomplete. + +The [Performance workflow](../../.github/workflows/performance.yml) measures a PR's exact head +and event base with the base revision's reviewed harness. It also runs the frontend's native +directory and 1,000/10,000/100,000/1,000,000-row Job cache comparison serially after the Go +measurements. Both collectors retain source/harness identities and raw results; missing, +failed or unmatched cases cannot complete a comparison. Manual critical/release runs use the +requested full candidate commit and `PERFORMANCE_ACCEPTED_BASELINE`, an explicitly accepted full +commit SHA. Both sources run once on that runner. The workflow is read-only, retains evidence on +failure, and never updates the baseline or publishes a release. Missing tools/harness require a +reviewed change; a missing or invalid baseline fails the hosted comparison. +Affected pull requests also run `read-fast` when Media/Restore planning sources, the common +performance harness, its workflow or the canonical Restore fixture change. Explicit `read` and +`read-resources` dispatches select full quality and large-resource evidence separately. + +Release preparation requires one completed, reviewed `release` comparison of the final source +against the recorded accepted source baseline, collected locally or through the hosted workflow. +Both sides must use the same host, tools and reviewed harness under the conditions above. Evidence +must identify the exact baseline, candidate and harness commits and content hashes, including any +dirty changes, with the commands, raw results and flag resolutions. Complete the structural checks +and relevant frontend checks as well. Review the completed results and record the implementation +choice and tradeoffs before candidate or source-push approval. An incomplete comparison remains +open work. A completed local comparison +satisfies this requirement without uploading its baseline or repeating it on a hosted runner. +Candidate assembly runs its source and package gates without automatically invoking Performance. + +After measurement, review the diff against each measured operation. Retain its evidence when +the relevant runtime source, dependencies, harness, fixtures and measurement settings are +unchanged, recording the measured and final commits. Documentation or unrelated runtime/tooling +changes do not require repeating that operation. Rerun only affected benchmarks and resolve their +flags; do not repeat the whole release tier for a change to one component. The final source +preflight and acceptance of the exact release archives remain required. + +## Large Identical Results + +Use the native benchmark for before/after Find and retained-page costs with the same 10,000- +and 100,000-File fixtures. Fixture creation is outside the measured operations: + +```shell +go test ./internal/library -run '^$' -bench '^BenchmarkIdenticalRetainedResult$' -benchmem -count=1 +``` + +The opt-in performance fixture builds isolated 10,000- and 100,000-File catalogs with many pairs, +a large group, a long transitive chain, saved versions, hidden names and unknown sizes. It measures +one complete Find and retained 200-row reads, then checks every sort direction, random offsets and +File ID position lookups against an independent result. Page checks verify that catalog collection +and component construction do not run again. + +```shell +YATM_IDENTICAL_PERF=1 go test -count=1 ./internal/library \ + -run '^TestIdenticalLargeCatalogPerformance$' -v -timeout 30m +CGO_ENABLED=0 YATM_IDENTICAL_PERF=1 go test -count=1 ./internal/library \ + -run '^TestIdenticalLargeCatalogPerformance$' -v -timeout 30m +``` + +These measurements cover query staging and retained SQL reads, not API member observation or +browser rendering. Go allocation measurements exclude fixture construction and native SQLite +memory; use an external process measurement when peak memory matters. The ordinary unit suite +skips this fixture, and the default Demo remains small enough for interactive review. diff --git a/docs/development/testing.md b/docs/development/testing.md new file mode 100644 index 0000000..04f9e27 --- /dev/null +++ b/docs/development/testing.md @@ -0,0 +1,116 @@ +# Development Checks and Test Environments + +Run checks in proportion to changed behavior. Commands below run from the repository root; Go tests use isolated fixtures and do not authorize access to production data or hardware. + +## Test Maintenance + +Use the [existing entrypoints](#repository-layout-and-entrypoints) first for tests, regressions +and performance work: `go test`, maintained scripts, and the frontend's native Vitest and Node +test runners. Replace recurring manual agent sequences with reproducible checked-in harnesses +and fixtures in the owning test area. Extend an existing runner or script before adding another. + +Keep reusable release, test and benchmark scripts and synthetic or public fixtures in this +repository. Supply machine-specific hosts, paths and private inputs through arguments or +environment variables. Keep credentials, real backups, deployment configuration and sensitive +results outside tracked source and public artifacts. + +Maintain tests, harnesses and fixtures with the implementation. Update or remove obsolete +assertions, unsupported scenarios and unused tooling in the same change. Cover observable +behavior, invariants and failure boundaries; avoid duplicate tests at the same boundary and +assertions that merely repeat the implementation. Distinct unit, adapter and E2E boundaries +retain their purpose. Keep verification proportional; a small edit need not create new automation. + +Record source identities, exact commands, passes, failures, skips and unrun checks with their +reasons. [Performance Acceptance](performance.md#performance-acceptance) owns measurement rules. Existing +physical Media, device, migration and visual acceptance remains required where applicable: +automate repeatable setup and checks where useful, and record each manual step's result and +why it needs an operator or visual judgment. Tar physical acceptance runs only through the +installed package: the Python controller's tar stages drive the installed service and `yatm-cli`, +like LTFS PT-01..PT-08, and the independent reader checks public-tool recovery. There are no Go +hardware tests; [physical Tape E2E](../release/physical-tar-acceptance.md#tar-acceptance) owns the stages. Follow [Media and Migration Safety](../release/release.md#media-and-migration-safety); +automation does not expand access or publication authorization. + +### Check Scope and Reuse + +During development and review, test the changed behavior and its affected callers. Choose the +scope before running commands; include unchanged code when a shared dependency, contract or +configuration change can affect it. A new commit by itself does not invalidate every earlier +check. Batch related fixes before expensive integration checks. + +| Change | Checks during development and review | +| --- | --- | +| Documentation | Document links/examples and `git diff --check` | +| Go implementation | Changed packages and affected callers; add race checks for concurrency/shared state and both drivers for SQLite behavior | +| Frontend | Affected Vitest tests and type checking; style/lint checks for touched presentation, browser interactions for affected visible behavior | +| API, persistence or shared infrastructure | Both sides of the changed boundary, generated output where relevant, and the workflows that consume it | +| Installer, packaging or test tooling | The owning script tests and the affected installation/package workflow | +| Performance-sensitive implementation | The affected native benchmarks, once per source; larger fixtures when traversal or scale behavior changes | + +Use runner selection (`go test PACKAGE -run PATTERN`, Vitest file filters and explicit script +tests) rather than creating another test framework. After fixing a failure, rerun its reproducer +and affected suite/callers. Do not restart unrelated passed suites to get another all-green run. +A failed or interrupted full run remains incomplete; record the completed checks separately. + +For slow verification, report the whole workflow's elapsed time, including setup, transfers, +queueing, retries and diagnosis, separately from test execution. Use existing runner/CI timings +to identify the costly stages; individual step timings do not explain the whole wait. + +Final source/package checks and the reuse of release and hardware evidence follow +[Evidence Reuse](../release/release.md#evidence-reuse). + +## Repository Layout and Entrypoints + +File ownership and placement rules are maintained in [Repository Layout](../../AGENTS.md#repository-layout). Follow those rules when adding or moving files; this section owns the development commands and output overrides. + +The root [Makefile](../../Makefile) is the common development/CI entrypoint. Its default target displays help; implementation stays with the owning project or script. Frontend compilation remains in package.json. Use `make build`, `make backend`, `make backend-linux`, `make frontend`, `make release`, `make preview`, `make generate`, or `make demo` / `make demo-dev`. `make check` runs documentation and IDL naming checks, Go unit tests/vet, compilation and vet of the E2E harness, frontend checks, tool-script tests and shell syntax checks; `make test-race` and `make test-e2e` are explicit additional checks. Set `YATM_E2E_BIN_DIR` to the extracted candidate directory for `make test-e2e`; the [E2E guide](e2e-test.md) also documents source-based harness runs. Native Preview module validation remains separate from the root Go test suite. + +Scripts locate the checkout independently of the caller's working directory. `RELEASE_DIRECTORY` overrides the final archive directory; relative values resolve from the caller's working directory before scripts change directories. Build tools are source files, not build output. The root `install-release.sh` remains the user installation entrypoint, and installation package paths are independent of source layout. + +Packages under `internal/` are not an external Go API; `entity` and the public gRPC contracts are. + +## Local Checks + +Select checks using [Check Scope and Reuse](#check-scope-and-reuse). For example, a legacy +migration repair checks its implementation and installer caller: + +```shell +go test ./internal/migrate/legacy ./cmd/migrate +go vet ./internal/migrate/legacy ./cmd/migrate +go test -race ./internal/migrate/legacy ./cmd/migrate +CGO_ENABLED=0 go test ./internal/migrate/legacy ./cmd/migrate +git diff --check +``` + +The coverage below is an inventory for choosing affected checks, not a requirement to run +every listed suite after each change. The [Release SOP](../release/release.md#release-sop) owns full verification. + +For content-model changes, cover opaque nullable signatures, unique File/version pairs, independent same-content Files, one original per File, repeated-content version reuse, shared-copy lookup, cross-batch import rollback, and evidence-based legacy migration with source preservation. + +Restore association checks include existing original associations, ignored outputs, multi-version candidate freezing, exact-content adoption, namespace collisions, Media finalization failure and explicit Library/Job publication failure. Scan verification checks include real uncached reads, per-item versus device errors, absent baselines and content-guarded health publication. Run both SQLite drivers and race checks for `./internal/executor/restore`, `./internal/executor/scan` and `./internal/library`; imported restored Files and original associations must remain intact, and imported bad-copy observations must not become healthy or unchecked accidentally. + +For Files and Location changes, inspect callers in `./internal/executor/scan`, `./internal/executor/observation`, `./internal/executor/fileops`, `./internal/treeops`, `./internal/ignore`, `./internal/demo`, and `./cmd/yatm-cli`; run the affected unit/race checks and select the affected [live-file, Volume, or Preview CLI E2E](e2e-test.md). Cover optional File identity, pure browsing versus explicit collection, complete directory List, paged Search and changefeed departures, Ignore versus authorization, guarded references, physical-operation results without persistent receipts and ordered continuity where changed. Shared organization changes run the contract against its affected Library, filesystem and object-storage-semantic adapters: directory merges, retained target associations, duplicate selections, conflicts, cancellation, bounded traversal and partial publication. Do not add user Copy acceptance to the supported move/mkdir/delete contract. + +Scan tests exercise one pipeline with known-only, fill-missing and force-read policies, optional comparison/Preview and each publication policy. Known-only cache misses must not hash; verification must neither trust cache nor replace its saved baseline. Preview failures must preserve otherwise valid observations, while source-read and physical-identity failures prevent publication of the current scope; a new Scan reads its complete selection. `analyze`, `preview create` and `verify` CLI presets must resolve to this same Scan service, not separate runners. + +Capacity checks cross multiple filesystem and manifest pages, cover the 256-level traversal boundary, reject an over-depth tree, enforce the 1,000-row public result window and cancel while a recursive directory stack is open. On platforms exposing `/dev/fd`, the cancellation test records descriptor counts before traversal and after unwinding and requires no retained descriptor. The test suite does not claim a platform-independent peak RSS or fixed Job-database size; any such release evidence must be measured against the exact packaged candidate and stated with its fixture, filesystem and SQLite mode. + +Canonical CLI subprocess tests cover minimal/projected `ls`, `files get/metadata`, `mv/rm/mkdir`, independent `preview get`, typed Archive/Restore estimates and `scan create/run/results`; protocol tests cover Preview URLs and reject full-file byte routes. Read tests assert request counts, SQL cost, omitted computation and absence of collection. The [E2E coverage matrix](e2e-test.md) records complete server/CLI workflows; subprocess transport tests alone do not establish full business acceptance. Ignore tests compare against Git in an isolated temporary repository when Git is available; production matching does not invoke Git. Check the SQLite paths with both the default CGO driver and `CGO_ENABLED=0 go test ./internal/library ./entity ./internal/executor/... ./internal/apis ./internal/migrate/legacy ./internal/demo ./internal/preview/... ./internal/media/... ./internal/resource`; this includes the external-memory read planner and temporary-resource tests themselves. Regenerate IDL with `bash dev/generate.sh` and verify a second generation produces no changes. + +Replacing a workflow starts from an inventory of the baseline implementation's UI/CLI actions, results and edge cases; each maps to its replacement and regression coverage or to an explicitly approved removal. Acceptance verifies end-to-end behavior, not only the new interface shape, and stale mocks are updated with their real consumers; passing tests without capability coverage do not prove feature parity. + +For frontend changes, run the scripts in [package.json](../../frontend/package.json), regenerate protobuf Go/TypeScript outputs when IDL changes, and keep the lockfile reproducible. Vitest owns tests under `frontend/src`; Node tooling tests under `frontend/scripts` run through their explicit `node --test` command in the frontend checks. A test belongs to one runner. Use active-LTS Node and supported stable frontend dependencies, and keep the initial bundle bounded. Check shell syntax for changed scripts and run builds/generation checks when those paths change. Documentation changes run `node dev/check-documents.mjs`; the check uses Node and Git only. + +## CLI Coverage and Skill + +[Command registration](../../cmd/yatm-cli/main.go) attaches one command to each public +service method, so every operation has a CLI spelling. [CLI coverage +tests](../../cmd/yatm-cli/coverage_test.go) hold the two together by resolving every +public RPC to one real parser leaf and rejecting a binding whose RPC no longer exists, so +a new public method without its command fails that check instead of leaving the surface +unreachable. [Local checks](#local-checks) run it. + +The CLI surface and its bundled [Skill](../../.agents/skills/yatm/SKILL.md) change +together: a command, flag, output or safety rule it describes must update the Skill in the +same commit. The Skill supplies operational guidance alongside the matching installed +CLI help and JSON output. Installed copies refresh through the +[optional Skill step](../operations/install.md#optional-agent-skill). diff --git a/docs/architecture/cli.md b/docs/operations/cli.md similarity index 64% rename from docs/architecture/cli.md rename to docs/operations/cli.md index 21062b3..e37e07b 100644 --- a/docs/architecture/cli.md +++ b/docs/operations/cli.md @@ -1,25 +1,15 @@ -# Command-line interface - -Status: Current v1 development architecture. +# Command-Line Reference The `yatm-cli` executable is the scripted and agent-facing surface. It calls the same -[gRPC and HTTP API](api.md) the browser does, so a CLI request carries no separate -server semantics; [Interface contracts](contracts.md) owns the rules shared with those +[gRPC and HTTP API](../architecture/api.md) the browser does, so a CLI request carries no separate +server semantics; [Interface contracts](../architecture/contracts.md) owns the rules shared with those surfaces. -## Shape - -[Command registration](../../cmd/yatm-cli/main.go) attaches one command to each public -service method, so every operation has a CLI spelling. [CLI coverage -tests](../../cmd/yatm-cli/coverage_test.go) hold the two together by resolving every -public RPC to one real parser leaf and rejecting a binding whose RPC no longer exists, so -a new public method without its command fails that check instead of leaving the surface -unreachable. The [development guide](../operations/testing.md) owns the commands that run -it. +## Agents and Connection The [bundled Skill](../../.agents/skills/yatm/SKILL.md) routes agent work to command domains and defines operating rules. It ships inside the release package beside the -[operations guides](../operations/library.md) that own the workflows. Agents read the +[operations guides](library.md) that own the workflows. Agents read the matching installed `--help` down to the selected command for its inventory, arguments and flags, and interpret its JSON output as the operational source of truth. The Skill's domain table does not duplicate the installed command inventory; this document owns the @@ -53,19 +43,16 @@ command; full archived content is obtained through Restore. The CLI has no comma operation the API does not expose. Instant fields in JSON use decimal strings and names ending in `_ns`, following the -[shared time contract](contracts.md#time-values). Preserve them as signed 64-bit integers when +[shared time contract](../architecture/contracts.md#time-values). Preserve them as signed 64-bit integers when processing output. Restore `--before` accepts a timestamp with its stated timezone and retains its nanosecond precision; dates outside the supported range are usage errors. `job wait` succeeds for a durable `COMPLETED` Job whose phase is `COMPLETED` or `UNSPECIFIED`, including after a service restart. It keeps waiting while an attached runner reports `QUEUED` for an admitted attempt waiting for shared Media resources, or a finalizing -phase. `FAILED` is terminal: no Job kind supports Job-level retry. An idle `READY` Job -reports `UNSPECIFIED` and requires an explicit Media choice. Only an Archive or Restore -Media operation failure or operator cancellation, including while `QUEUED`, returns -the Job to pre-Media `READY`, retaining its prepared manifest and error or cancel reason -for another explicit Media operation. Initial preparation and Scan failure or cancellation -remain terminal. Waiting returns nonzero with `action_required` for `FAILED` or idle +phase. An idle `READY` Job reports `UNSPECIFIED` and requires an explicit Media choice; the +[Job lifecycle](../architecture/jobs.md#shared-lifecycle) defines which failures return a Job to +`READY` and which are terminal. Waiting returns nonzero with `action_required` for `FAILED` or idle `READY`, including the persisted error when present. `archive creation ID`, `restore creation ID` and `scan creation ID` call their kind's @@ -87,7 +74,7 @@ rows; group and member CLI pages default to 20. `identical rows` and `identical accept the same `--sort-key file-id|name|size` and `--order asc|desc`. The default is File ID ascending; name starts ascending and size starts descending when selected. Row pages and position lookups use the same requested order under the -[Library sorting contract](library.md#identical-file-tools). These reads never scan or hash physical files. +[Library sorting contract](../architecture/library.md#identical-file-tools). These reads never scan or hash physical files. `job log` reads a raw bounded byte page and returns JSON with text in `logs` and the next byte offset as a decimal string in `offset`. Follow that returned offset, not @@ -103,6 +90,27 @@ An empty filtered page can still advance its cursor, so use the returned `has_ol Continuation fragments can appear as context when their line start lies outside the bounded read, even if that fragment does not establish a filter match. +`library export` retains streamed delivery. Reported HTTP or transport failures return nonzero; +named-file output is published only after the response is received and the file is closed. The CLI +does not validate the JSONL end record. In particular, close-delimited HTTP/1.0 can make an +interrupted download appear complete; import still rejects the missing end record. Stdout may +contain a partial stream on failure, so its exit status must be checked. + +The separate [standalone export tool](../../cmd/export-library/main.go) opens the configured local +Catalog and delegates to the same Library snapshot writer. Its default selection is +`file,media,position`; related version groups follow Library selection rules. This selected export +omits Location/original groups and Settings, so it is not a complete installation backup. +Unknown or empty comma-separated type names fail before opening configuration, Catalog or output. +Named regular-file output is staged in the same directory and replaces the resolved target only +after export, resource cleanup, file synchronization and close succeed; failures retain the previous +backup and remove the owned staging file. Existing permission bits and final symlink targets are +preserved; replacement creates a new inode, so other hard links retain the old file. New files keep +the normal creation mode and process umask. Stdout and explicitly named device/pipe outputs remain +streaming and can contain partial data on failure. Interrupt and termination signals cancel the +operation cooperatively. Signal interception is released after the first cancellation, so a later +signal can terminate an operation blocked in an OS call. Cooperative cancellation runs normal +cleanup; forced termination cannot guarantee it. + ## Directory paths `ls` accepts one optional positional directory operand. A bare `ls` selects the Library @@ -147,29 +155,43 @@ drains the remaining rows, then returns the existing failure exit code with `cod directory or transport failures retain their own errors. Supported filenames are legal UTF-8, including literal backslashes, whitespace and control characters. Non-UTF-8 names are escaped for display only and are not usable path operands. Quote shell operands so the shell preserves their bytes. -The [Library guide](../operations/library.md#browse-from-the-cli) shows usage. +The [Library guide](library.md#browse-from-the-cli) shows usage. ## Reports and mutations -The [shared rule](contracts.md#synchronous-operation-reports) owns which commands report +The [shared rule](../architecture/contracts.md#synchronous-operation-reports) owns which commands report under `--dryrun`, what a report contains and which operations are excluded. This surface adds only where the CLI differs: `archive write tape format` keeps `--confirm-format`, because confirming the Tape barcode authorizes a write to a device rather than a dry-run decision, and a report needs no follow-up command — the same command without `--dryrun` performs the resolved work. -Tape FORMAT first inspects the device and Library registration. A successful inspection with -an empty electronic barcode accepts the user-supplied `--barcode`, normalized to six uppercase -alphanumeric characters; a nonempty inspected barcode must match it. `--confirm-format` must -exactly match that resolved barcode. Probe errors and registered Media stop before the write. -APPEND retains its requirement for a matching inspected identity and compatible registered Media. +## Tape Commands -## Skill +- `settings devices` reads the typed DEVICE group, and `--device-json FILE` replaces the user-owned + Device settings without device access. Server-owned `learned_read_parameters` are preserved and + incoming values for them are ignored. [Device settings](devices.md#save-device-settings) and + [Auto read order](read-order.md) explain the fields and give JSON examples. +- `archive write tape format` resolves the selected device's defaults at this Media operation. + `--format ltfs_v1|tar_pax_v1`, `--record-mode`, `--record-bytes` and + `--hardware-compression true|false` override them as [Tar write options](devices.md#tar-write-options) + describe; LTFS rejects tar options. `--barcode` is normalized to six uppercase letters or digits + and requires an exact `--confirm-format`. Registered Media and identity conflicts stop before + writing, and FORMAT erases the cartridge even when its cartridge memory barcode is empty. +- `archive write tape append` uses the registered profile and rejects conflicting explicit options. +- `restore run tape`, Tape `verify run` and `scan run` take `--device`, an optional `--media-id` hint + and `--refresh-position-map` (default false). Without a hint, the server identifies the Tape by its + cartridge memory barcode; a hint never bypasses physical verification. + [Load Tape for Restore or Scan](devices.md#load-tape-for-restore-or-scan) describes both options. +- No command sets encryption: the configured `encrypt` script owns it, and Media output carries + `encryption_key_saved`, never the key. The Device's read order applies to each read attempt, and + the Job log records the requested and effective order with any fallback reason. -The CLI surface and its bundled [Skill](../../.agents/skills/yatm/SKILL.md) change -together: a command, flag, output or safety rule it describes must update the Skill in the -same commit. The Skill supplies operational guidance alongside the matching installed -CLI help and JSON output. Installed copies refresh through the -[optional Skill step](../operations/install.md#optional-agent-skill). +## Tape Device Diagnostics -The standalone Library exporter defaults to `file,media,position`, rejects unknown types, and replaces a named output only after a complete successful export. Stdout remains streaming. HTTP Library exports abort failed partial responses; clients must validate the JSONL end record when transport framing cannot identify truncation. +`tape device list` prints every configured Tape entry with its busy status and detail; listing +sends no SCSI commands. `tape device inspect --device DEVICE` explicitly requests passive diagnostics +as protobuf JSON. Absent facts stay absent, and inspection never loads, positions or modifies the +cartridge. `media inspect tape` is the separate cartridge identification operation; it uses the +configured LTFS adapter and can load the cartridge. [Inspect a Device](devices.md#inspect-a-device) +explains the reported fields. diff --git a/docs/operations/devices.md b/docs/operations/devices.md new file mode 100644 index 0000000..174adbf --- /dev/null +++ b/docs/operations/devices.md @@ -0,0 +1,229 @@ +# Devices and Tape Operations + +Configure Tape entries under `tape_devices` in the service configuration described in the +[installation guide](install.md). Device names refer to entries on the server, not paths on the +browser or CLI client. LTFS operations retain the configured scripts and their local helpers; +[upgrades](migration.md#tape-script-adaptation) preserve those adapters. + +## Device Requirements + +Device diagnostics and raw tar need Linux SG_IO +access to the configured entry's resolved generic device, read/write permission for the service +account, and `fuser` from psmisc with visibility of host process holders; **Device details** +reports permission, missing-tool, busy or unknown-ownership reasons when that access is missing. +Listing devices sends no hardware commands. + +Raw tar additionally needs a standard sequential-access drive (SPC-3 with SSC-2/3 command +behavior), supported record operations and an observed partition layout. Reads and APPEND require +the initialized single data partition; FORMAT also requires supported destructive initialization, +including any transition from an LTFS layout. A missing or uncertain required fact fails admission +with a reason, and a model name alone establishes no qualification. LTFS workflows instead rely on +the configured [Tape scripts](tape-scripts.md) and compatible LTFS software. + +## Inspect a Device + +Open **Settings → Devices**, select a device, and explicitly request its **Device details**. Refreshing +the device list reports configured entries and busy reasons without sending SCSI commands. +Device details report observed identity, readiness, record limits, current compression/encryption +state when known, and feature support, availability and validation separately. Missing values +remain unknown. A passive report neither positions the Tape nor proves that writing, seeking or +independent recovery works on that hardware. + +Capacity diagnostics, when available, show native bytes estimated to remain before early warning, +the observed partition and observation start/end times. Unknown error/buffer-lag allowances remain unknown. +An estimate is not a guarantee that a file fits. Counter support and measured performance remain +separate observations; enabling polling does not mark a device as physically validated. + +```shell +yatm-cli tape device list +yatm-cli tape device inspect --device /dev/nst0 +``` + +Raw Tape access requires Linux, the configured device's SCSI generic mapping, suitable permissions, +and visibility of host device holders. A busy or unknown ownership result must be resolved before +raw access. `media inspect tape` is a separate LTFS cartridge action that can load the cartridge; +passive diagnostics do not replace it for LTFS. + +## Save Device Settings + +In **Settings → Devices**, choose **Default Tape format** and **Read order**. Expand **Advanced** +for tar records, hardware compression, capacity polling, planning budget and Tar passive read learning. +**Manual read parameters** has its own collapsed section. Save, Cancel and **Refresh devices** +are below the fields; Refresh devices updates device availability without changing the saved +Device settings or draft. **Device details** follows in a separate collapsed section. Saving changes +only persisted Device settings; it performs no hardware operation. Entries removed from service configuration remain editable as inactive Device settings and do not authorize +access to those devices. Validation errors retain the edited values and reveal the affected fields. + +Read order, passive read learning, the planning budget and manual read parameters are explained in +[Faster Restores with Auto read order](read-order.md). + +The CLI reads or replaces the complete DeviceSettings group using protobuf JSON. Read the current +group before editing; replacement must retain every entry you want to keep: + +```shell +yatm-cli settings devices +yatm-cli settings devices --device-json device-settings.json +``` + +For example, a saved tar preference for one configured entry is: + +```json +{"devices":[{"device":"/dev/nst0","tape":{"defaultFormat":"tar_pax_v1","tarWriteOptions":{"recordMode":"TAPE_RECORD_MODE_VARIABLE","recordBytes":262144,"hardwareCompression":false},"readOrder":"TAPE_READ_ORDER_LOGICAL"}}]} +``` + +The example size is an explicit preference, not a hardware recommendation. Omitted options remain +absent; explicit `false` remains an explicit choice. `readOrder` accepts `TAPE_READ_ORDER_LOGICAL` +or `TAPE_READ_ORDER_AUTO`; omission/unspecified uses Logical when the attempt is admitted. +The optional `capacityPollIntervalSeconds` uses Auto when omitted, preserves explicit `0` as Off, +and accepts custom unsigned 32-bit values of at least `5` seconds. + +## Tar Write Options + +The available formats are `ltfs_v1` and `tar_pax_v1`. Without a saved format, FORMAT uses LTFS. +Tar accepts fixed or variable records. FORMAT uses 512 KiB (524288 bytes) unless a saved device size +or explicit FORMAT override is supplied; APPEND and reads keep the cartridge's recorded size. +Record bytes must be positive, fit an unsigned 32-bit integer, +and be aligned to 512 bytes. Saving checks this shape; the operation checks the current device, +driver and transfer limits before allocating buffers or writing. An incompatible size fails rather +than being silently reduced. For variable records, record bytes is the maximum physical record size; +for fixed records, it is the size of every physical record. + +Tar hardware compression can be explicitly On or Off. The leased operation checks current hardware +support, required control changes and readback; a saved preference alone cannot establish support. +APPEND uses the cartridge's saved choice after verifying its label and recorded tail. Reads require +compatible hardware decoding for compressed content and preserve the drive's write-compression +controls. Diagnostics show observed algorithm identifiers and known or unknown control changeability, +separately from physical validation. No physically qualified compression recommendation is supplied +by passive inspection, and expected compression savings are not a space guarantee. + +## Encryption + +Encryption for tar and LTFS follows the configured `scripts.encrypt` adapter. There is no separate +FORMAT encryption switch or algorithm selector. FORMAT generates the existing key representation; +APPEND/read uses the key saved in the Media profile. A temporary 0600 file passes it to the adapter +and is removed during cleanup. Changing formats does not replace a customized encryption script. + +Published keys remain in the existing Media/Job profiles and Library exports. No extra persistent +JSON or `.key` copy is created. If independent recovery is needed, retain a protected Library export +and use the [public-tool guide](tar-tape-recovery.md). Media deletion removes its profile and key; +Job deletion removes the Job's attachments. Diagnostics expose only non-secret facts; an unknown +algorithm is not evidence of plaintext and a key counter is not key presence. + +## Capacity Polling + +Tar capacity polling defaults to **Auto (5 seconds)** unless the device reports that the +required query is unsupported. Leave the interval empty for Auto, enter `0` for Off, or enter a whole +number of at least `5` seconds for a custom cadence. FORMAT and APPEND freeze this preference +when admitted; it is independent of the stored record/compression profile. File admission uses cached results, and writing +does not wait for the query to return. +Unsupported counters use EW/error handling and any available preparation estimate. A missing counter value +remains unknown, and neither periodic estimates nor the selected interval guarantee that a file fits. +Writing deducts space from the cached estimate; periodic capacity samples only correct it. A settled +query timeout, busy response, unreadable payload or missing value keeps the cached debits and +ordinary polling cadence. Repeated misses do not stop Archive or disable later capacity polling. +Failed capacity samples appear as warnings in the Job log, including the error and consecutive failure +count; the first successful value after failures logs recovery. Investigate repeated warnings using +the device and host logs. Skipped busy polls and ordinary shutdown do not generate failure warnings. +Actual write errors, medium loss, device resets or unknown native command outcomes stop the +attempt and withhold its Archive results. A write already admitted before a terminal failure may +finish; a failed optional capacity sample alone does not establish such a failure. +LTFS and Volume retain their existing capacity paths. + +## Load Tape for Archive + +Create and prepare the Archive Job with the normal file selection. Format selection belongs to +**Choose archive storage → Storage type → Tape**. Select a device and **FORMAT**, review its saved defaults, +and override them for this operation when needed. Explicit operation values take precedence over +saved Device settings and the [defaults](#tar-write-options). Validated +passive recommendations can supply other remaining choices. Review the reported limits and any +admission error. + +FORMAT is destructive even when the cartridge memory barcode is empty. Enter the six-character +alphanumeric barcode and Tape name, then confirm the exact barcode before submission. A reported +different barcode or an existing Library registration stops FORMAT. To reinitialize registered +Media, explicitly delete its metadata first; metadata deletion alone never erases Tape contents. +Raw tar FORMAT uses passive diagnostics and does not require the LTFS cartridge inspection script. +Like `mkltfs`, it writes the confirmed barcode into an empty cartridge memory barcode and verifies it +before writing data. A failed write or readback stops FORMAT; it is not retried. +LTFS retains its cartridge checks, initialization, mount and normal unmount flow. + +```shell +yatm-cli archive write tape format JOB_ID --device /dev/nst0 \ + --barcode ABC123 --name 'Archive Tape' --confirm-format ABC123 \ + --format tar_pax_v1 --record-mode variable --record-bytes 262144 +``` + +Omit `--format` to use the selected device's saved format, or supply `--format ltfs_v1` for an +explicit LTFS override. Tar-only record options are invalid for LTFS. An edited operation choice +is retained while reviewing the form; selecting another device does not erase explicit overrides. + +Choose **APPEND** explicitly for an existing registered Tape. Its stored format and tar record +profile are immutable: current Device write-profile defaults do not apply. The separate capacity +polling preference still applies to the new attempt. APPEND verifies the loaded identity; +tar additionally verifies its label/profile and recorded tail before adding an independent archive. +A mismatching override fails instead of rewriting the profile. + +```shell +yatm-cli archive write tape append JOB_ID --device /dev/nst0 --barcode ABC123 +``` + +Successful raw Archive completion writes a standard tar terminator and a confirmed Tape filemark +before publishing completed copies. Normal tar Archive, Restore and Scan completion restores owned +controls and unloads the cartridge, including its rewind and eject. Reinsert the Tape for another +operation. Failed or canceled raw operations use their existing cleanup path instead. LTFS follows +the configured unmount adapter; the bundled adapter ejects. Early warning is handled as a normal capacity boundary only +when the provider supplies validated accepted-count evidence. Completion errors remain visible; +accepted bytes alone do not establish a completed copy. + +Tar can use a cached preparation estimate to refuse the next complete file before its header and +close a normal archive. File admission does not wait for a space query or pause to synchronize each +file. Unknown capacity permits baseline writing; actual EW still stops data writing and may leave +an incomplete final member. That member stays pending for another explicitly selected cartridge. + +## Load Tape for Restore or Scan + +Restore and Media Scan offer automatic identification and a manual Media selection from the Job's +prepared candidates. Start with automatic identification. If the server requires a hint, retain the +form, select the expected Media, review it, and submit again. A manual selection supplies expected +identity and the stored profile; the server still verifies the physical cartridge. It does not +permit reading an unrelated Tape or bypass a label mismatch. + +```shell +yatm-cli restore run tape JOB_ID --device /dev/nst0 +yatm-cli restore run tape JOB_ID --device /dev/nst0 --media-id MEDIA_ID +yatm-cli verify run JOB_ID --device /dev/nst0 --media-id MEDIA_ID +``` + +Tar reads use the stored record policy and sequential physical filemarks; they do not use a +current FORMAT default or require LTFS `readinfo`. LTFS keeps its configured adapter, identity, +key, mount and normal unmount path. Exact member seek requires stored physical anchors and an +applicable standard READ POSITION/LOCATE pairing in the prepared record/partition envelope. +Missing anchors or optional command support retain sequential reads. Device details reports +physical validation separately from protocol support. A real +positioning or record-read failure stops that attempt rather than silently retrying by another path. +For tar Restore and recorded-copy verification, **Refresh position map before reading** is an +explicit preparation action, not a saved Device setting. Select it in the read dialog, or pass +`--refresh-position-map` to `restore run tape`, Tape `verify run` or `scan run` for a frozen +VERIFY_COPIES Job. The drive unloads without ejecting, loads once and captures a new position map +before reading; **Device details** shows whether the drive is eligible. The +[refresh contract](../architecture/read-planning.md#position-map-refresh) owns eligibility, identity +checks and encryption handling. A refreshed map does not qualify Auto read order. + +Ordinary reads remain available with the action omitted. A failed or canceled cycle can leave the +cartridge unthreaded: inspect the device and explicitly load it before another read if required. +YATM does not retry the cycle or perform recovery motion automatically. +Auto read order never triggers a reload. LTFS, Volume, inventory and other Media Scan policies do not +accept this action. A missing Scan policy hides only refresh; it does not block normal reading. + +Use `yatm-cli job log-lines JOB_ID` or the Job's log to inspect requested/effective read order and its +reason. Geometry diagnostics retain unknown freshness; a reported all-wrap command form is not +proof of a fresh complete map or an ordering recommendation. A reported-wrap count requires a +decoded response; command-form support alone leaves that count unknown. + +LTFS Auto preparation can read both partition Labels through SG before mounting. This adds two +partition-start seeks; the Job log records that preparation time separately from planning. +YATM checks the Labels against mounted volume facts and lets LTFS own all subsequent reads. +Missing partition, geometry or timing evidence retains the normal logical order. + +Media details display the stored format and record profile. Retain the independent +[tar recovery parameters and guide](tar-tape-recovery.md) before relying on a cartridge. diff --git a/docs/operations/index.md b/docs/operations/index.md new file mode 100644 index 0000000..086f4a6 --- /dev/null +++ b/docs/operations/index.md @@ -0,0 +1,52 @@ +# User Guide + +Start here to install, configure and operate YATM. The [glossary](../../CONTEXT.md) defines the +terms these guides use. These guides describe the current software; +implementation details, contributor instructions and research have a separate +[developer documentation entry](../README.md). + +## Install and Configure + +- [Installation and service configuration](install.md): supported environments, installation, + updates, service settings and the optional operating Skill for agents. +- [Upgrade from v0.1.x](migration.md): data import, Tape-script adaptation and recovery from an + interrupted upgrade. +- [Offline migration and repair](offline-migration.md): run migration phases outside the installer + when the upgrade guide or a maintainer directs it, and repair historical Jobs. +- [Locations](locations.md): register original directories and Restore destinations, browse live + files, configure Ignore rules and operate on original files. +- [Devices and Tape operations](devices.md): inspect configured devices, save Device settings and + load LTFS or tar Tape for a Job. +- [Tape scripts](tape-scripts.md): adapt the configured scripts to the local LTFS distribution, + drive and helpers. + +## Daily Use + +Use one YATM service for one operator. Avoid conflicting changes to the same files, Library items +or paths, and keep an operation's selected content, referenced configuration and mounted Media +unchanged until it finishes. Independent Jobs and internal copying may still run concurrently. + +- [Library and mounted Volumes](library.md): organize and search files, register mounted Volumes, + back up Library metadata and find identical files. +- [Archive, Restore and Scan](jobs.md): copy files to Media, restore saved versions and publish + content facts, inventory and integrity results. +- [Faster Restores with Auto read order](read-order.md): shorten Restores of files scattered across a + Tape, and tune the read parameters it uses. +- [Command-line reference](cli.md): run the same operations from scripts and agents. +- [Independent tar Tape recovery](tar-tape-recovery.md): retain cartridge parameters and recover + ordinary files with public tools, including the required physical reader qualification. + +LTFS and raw tar use the configured encryption adapter and the existing saved Media key. Raw tar +requires Linux; writes establish the selected hardware compression policy and reads retain the +recorded decoding policy. Independent encrypted recovery uses an explicitly retained protected +Library export and a compatible public unlock procedure. The +[recovery guide](tar-tape-recovery.md#tested-reader-and-required-parameters) lists physically checked +reader combinations and remaining limits. Consult the +[release notes](https://github.com/samuelncui/yatm/releases) for the installed version. + +## Operating with an Agent + +Install the optional YATM operating Skill using the +[installation instructions](install.md#optional-agent-skill). It guides an agent through the same +CLI operations and safety boundaries used by an operator. Repository `AGENTS.md`, architecture +documents and test procedures are contributor guidance, not prerequisites for daily use. diff --git a/docs/operations/install.md b/docs/operations/install.md index f6913db..0ce4011 100644 --- a/docs/operations/install.md +++ b/docs/operations/install.md @@ -1,32 +1,37 @@ # Installation and Service Configuration -This guide describes the current installer. The examples select `v1.0.0-alpha.2.1`; the same installer accepts an explicitly supplied, checksum-verified local candidate. Existing installations follow the [migration guide](migration.md). +This guide describes the current installer. The examples select `v1.0.0-alpha.3`; the same installer accepts an explicitly supplied, checksum-verified local candidate. Existing installations follow the [migration guide](migration.md). ## Requirements -- Tape workflows require an LTFS-capable LTO drive (LTO-5 or later), its SCSI generic mapping, and compatible LTFS software. Volume-only workflows do not require a Tape drive. +- LTFS Tape workflows require an LTFS-capable LTO drive (LTO-5 or later), its SCSI generic mapping, compatible LTFS software and the configured adapters. Raw tar Tape uses the Linux protocol requirements below and does not require LTFS. Volume-only workflows do not require a Tape drive. - Linux amd64 is the primary deployment target; the existing deployment scripts have been exercised on Debian 11/12. Other release architectures are experimental, and Windows is unsupported by the Unix-oriented mount workflow. - Historical CPUs such as Pentium III are outside the maintained runtime targets. Source ports to these systems are user-maintained; a Linux 386 experimental archive does not establish support for every 32-bit x86 CPU. -- Place required tools in PATH or adapt the configured scripts: LTFS formatting/mounting, `mt`, device inspection, and [Stenc](https://github.com/scsitape/stenc) for hardware encryption. The repository's Tape scripts target HPE LTFS conventions; other LTFS implementations may require script changes. +- Place the required tools in PATH or adapt the configured scripts: LTFS formatting/mounting, `mt`, `fuser` from psmisc and device inspection where applicable. Both tar and LTFS use `scripts.encrypt` for encryption setup; the bundled adapter uses [Stenc](https://github.com/scsitape/stenc). The bundled unmount adapter uses `fuser` to verify device release. The repository's Tape scripts target HPE LTFS conventions; other LTFS implementations may require script changes. - The optional Image/Video Preview helper ships its own private native libraries; a system ffmpeg/ffprobe installation is not required. The main service and existing Preview reads work without this helper. See [native build targets](../../previewworker/build/README.md) for the helper's separate platform requirements. +Tape device access, raw tar protocol requirements and device diagnostics are described in +[Device requirements](devices.md#device-requirements). Installation never runs raw preparation or +hardware qualification, and public-tool recovery needs its own +[reader qualification](tar-tape-recovery.md). + ## Install or Update -The [release installer](../../install-release.sh) supports Linux amd64 with systemd and requires Bash 4.4 or newer. Its `--help` output is the authoritative list of required command-line tools. Run it with an account authorized to write the installation and manage the service. The default `/opt/yatm` installation normally requires `sudo`; a dedicated writable installation and user-owned unit may use an unprivileged account. The default selects the latest stable release; Alpha requires an explicit version. Obtain that revision's installer, check, then install: +The [release installer](../../install-release.sh) supports Linux amd64 with systemd and requires Bash 4.4 or newer. Its `--help` output is the authoritative list of required command-line tools. Run it with an account authorized both to write the installation directory and to manage the system-level systemd unit. The default `/opt/yatm` installation normally requires `sudo`; a writable custom directory alone does not grant system-service permissions. The installer does not manage `systemctl --user` units. The default selects the latest stable release; Alpha requires an explicit version. Obtain that revision's installer, check, then install: ```shell curl --fail --location --output install-release.sh \ - https://raw.githubusercontent.com/samuelncui/yatm/v1.0.0-alpha.2.1/install-release.sh -sudo bash install-release.sh --version v1.0.0-alpha.2.1 --check -sudo bash install-release.sh --version v1.0.0-alpha.2.1 + https://raw.githubusercontent.com/samuelncui/yatm/v1.0.0-alpha.3/install-release.sh +sudo bash install-release.sh --version v1.0.0-alpha.3 --check +sudo bash install-release.sh --version v1.0.0-alpha.3 ``` For an extracted local candidate, supply its archive and companion checksum; it follows the same installation path: ```shell -sudo bash install-release.sh --version v1.0.0-alpha.2.1 \ - --archive ./yatm-linux-amd64-v1.0.0-alpha.2.1.tar.gz \ - --checksum ./yatm-linux-amd64-v1.0.0-alpha.2.1.tar.gz.sha256 --check +sudo bash install-release.sh --version v1.0.0-alpha.3 \ + --archive ./yatm-linux-amd64-v1.0.0-alpha.3.tar.gz \ + --checksum ./yatm-linux-amd64-v1.0.0-alpha.3.tar.gz.sha256 --check ``` `--check` validates the candidate and performs read-only installation checks. Remove it to proceed to the confirmation prompts. `--install-dir` and `--service` select a dedicated installation and unit; `--config` supplies a reviewed configuration for a fresh installation. Otherwise fresh installation opens the configuration template in `$EDITOR` (default `vi`). Review the listener, database and access roots. Volume-only users leave `tape_devices: []`; no fictitious Tape device or script execution is needed to install. @@ -42,6 +47,7 @@ After systemd starts, bind acceptance to the direct-loopback API process identit For a fresh manual installation from a downloaded archive, verify its companion checksum before extraction. Then install the package and bind acceptance to the versioned programs and the systemd-owned process: ```shell +RELEASE_VERSION=v1.0.0-alpha.3 sha256sum --check "yatm-linux-amd64-${RELEASE_VERSION}.tar.gz.sha256" sudo mkdir -p /opt/yatm sudo tar -xzf "yatm-linux-amd64-${RELEASE_VERSION}.tar.gz" -C /opt/yatm @@ -58,29 +64,51 @@ sudo systemctl show yatm-httpd.service --property MainPID,WorkingDirectory,Fragm /opt/yatm/yatm-cli --server http://127.0.0.1:8080 --timeout 3s status ``` +The installer keeps programs inside the chosen installation directory. For the bare `yatm-cli` +commands in the daily guides, add that directory to the current shell's PATH: + +```shell +export PATH="/opt/yatm:$PATH" +yatm-cli --version +``` + +Use the directory selected by `--install-dir` when different, or invoke its full `yatm-cli` path. +Persist that PATH setting in your shell profile if desired. + Compare each program's reported version/commit with the package `VERSION` and `COMMIT`, and confirm that the readiness endpoint belongs to the unit's `MainPID`; the automatic installer performs these checks. Review paths, database settings, listener/domain, optional Tape devices/scripts, and Preview settings before starting. Manual commands above are for a new installation only. If installing elsewhere, update the service and scripts for that location. The configuration supports SQLite and an untested MySQL option; per-Job state uses SQLite bundles. -Automatic upgrade covers standard SQLite layouts contained in the installation backup. External databases, Job storage, scripts or captured indexes require the manual procedure with coordinated backup of those resources. Other platform archives use manual installation and carry explicit runtime-validation or experimental labels. +Automatic upgrade and the documented offline migration cover standard SQLite layouts contained in the installation backup. External databases, Job storage, scripts or captured indexes need a layout-specific migration procedure and coordinated backup before Prepare or Commit; the contained-layout command sequence does not cover them. See the [offline migration scope](offline-migration.md#offline-migration-and-repair). Other platform archives use manual installation and carry explicit runtime-validation or experimental labels. `database.sqlite_wal: true` optionally permits Catalog reads during a write; it defaults to false. Keep the SQLite database on a local disk even when the service runs on a NAS. Do not place it on an -SMB/NFS share or copy only a live main database file: WAL sidecars may contain committed data. -Use metadata export or stop the service before backing up the complete installation. Turning the +SMB/NFS share; [Backups](#backups) explains how to copy it safely. Turning the option off restores DELETE journal mode on startup; an unsuccessful switch prevents startup. -`paths.access` defines administrator-approved directory roots and optional ordered gitignore exclusions. Daily original/Restore directory choices live in **Settings → Locations**, not YAML. `paths.work` owns execution artifacts and is relative to the service working directory when not absolute. A relative `preview.root` resolves under it, defaulting to `previews`; an absolute Preview root is used directly. `paths.volumes` contains already-mounted archive Volume discovery roots. Restore destinations are not archive Media. See [Location configuration and migration](#location-configuration-and-migration). +`paths.access` defines administrator-approved directory roots and optional ordered gitignore exclusions. Daily original/Restore directory choices live in **Settings → Locations**, not YAML. `paths.work` owns execution artifacts and is relative to the service working directory when not absolute. A relative `preview.root` resolves under it, defaulting to `previews`; an absolute Preview root is used directly. `paths.volumes` contains already-mounted archive Volume discovery roots. Restore destinations are not archive Media. See [Location configuration and migration](#locations). -Scripts isolate environment-specific LTFS and device behavior from the service. Operators may edit configured scripts and helpers for local tool versions, platforms and hardware; bundled commands are starting points. Review the [script contract and adaptation sequence](migration.md#tape-script-adaptation) before Tape use. The default unmount script ejects the cartridge. Once Archive has finished and the final Index is captured, use physical write protection when retaining a cartridge offline. Retain unexpected LTFS/index diagnostics for investigation. +For LTFS, scripts isolate environment-specific LTFS and device behavior from the service. Operators may edit configured scripts and helpers for local tool versions, platforms and hardware; bundled commands are starting points. Review the [Tape script contract](tape-scripts.md) before Tape use. The default unmount script ejects the cartridge. Once Archive has finished and the final Index is captured, use physical write protection when retaining a cartridge offline. Retain unexpected LTFS/index diagnostics for investigation. -## Location Configuration and Migration +## Locations -Every Location supports live browsing and authorized physical operations. **Restore destination** includes it in the UI's preferred-target picker; this preference is not a write permission. CLI/API targets still pass the same authorization checks. Scan and Restore publish observations without whole-directory readiness. Ignore scopes ordinary Location reads and new associations, but never controls authorization and never rewrites existing facts. Settings provides path browsing, registration and inline configuration. Startup, registration, browsing and preference updates do not collect files or create Scan Jobs; use explicit Scan to publish originals. +Register original directories and Restore destinations after installation in the +[Location guide](locations.md). An upgrade from v0.1.x converts legacy `paths.source` and +`paths.target`, as listed in [What Changes](migration.md#what-changes). -The installer previews and applies legacy `paths.source` and `paths.target` conversion to Locations, then removes those YAML fields. The old target is recommended for Restore; existing registrations remain authoritative. Relative paths resolve from the original service working directory; equal normalized roots share one registration. Conversion does not scan roots or move Library nodes. Startup does not import these fields, and no migration-history table or Locations-page migration notice is retained. +## Backups -The installer materializes effective legacy authorization as explicit `paths.access` before removing source/target. Existing explicit access rules remain unchanged; an explicit empty list permits no directories. YAML never overrides later Location edits. Paths outside allowed roots, symlink descendants, runtime files (including `.backup/`) and overlapping archive storage remain denied. Ignore rules cannot reopen paths outside an allowed root, and an ignored parent requires reopening before a child exception can apply. Revoking access invalidates a frozen Job rather than redirecting it. +Two things are called a backup: -A registered or imported path is immediately usable. Original access additionally needs a current per-file observation; Restore validates the selected path and its existing parent components, and writes can still fail if permissions or space change. Root/Ignore edits do not retarget existing records: a frozen Restore destination still requires its registration to keep the same root, and later reads resolve recorded paths under the current root. [Legacy migration](migration.md) retains frozen legacy Job inputs and configured Restore roots. Unpublished Draft data is distinct from the [published format families](../architecture/persistence.md#published-data-formats). +| Backup | Contains | Owner | +| --- | --- | --- | +| Installation backup | The complete installation directory, taken by the installer before an upgrade | [Backup layout](migration.md#backup-layout-and-upgrade-phases) | +| Library metadata backup | Library, Locations, original links and saved Media keys, as JSONL; no file contents or global Settings | [Metadata backup](library.md#metadata-backup) | + +File contents on Tape or Volume Media are archived copies, not backups; see +[Archive, Restore and Scan](jobs.md). + +To back up a running installation, export Library metadata or stop the service and copy the +complete installation. Copying only a live main database file is not a backup: WAL sidecars may +contain committed data. ## Optional Agent Skill @@ -96,13 +124,9 @@ DISABLE_TELEMETRY=1 DO_NOT_TRACK=1 \ Run this as the user of the selected agent. `npx --yes` allows fetching the pinned tool; Skills still asks which agents to use and confirms installation. The bundled Skill belongs to the installed YATM release and documents its supported CLI commands. -## Preview Preferences +## Preview Helper -The installer offers the optional helper before changing the service. `--with-preview` selects it explicitly; `--without-preview` skips it. Noninteractive installation skips it unless explicitly selected. The helper package must match the main package's version and commit and pass its checksum and capability probe before installation. For an unpublished local candidate, add `--preview-archive /path/to/yatm-preview-linux-amd64-VERSION.tar.gz --preview-checksum /path/to/archive.sha256` to the ordinary install command. A same-version rerun can add the helper later. No helper source or system-wide native libraries are installed. - -Installing the helper does not enable generation. In **Settings → Preview**, enable generation, choose Image or Video and edit the corresponding extensions and output options. Shared controls set concurrent files, timeout, input-pixel limit and an optional helper command override. Disabled controls preserve their values. Existing Preview assets remain accessible when generation is disabled or the helper is absent. Check installation independently with `yatm-cli preview capabilities`. - -The installer previews legacy generator import when no saved preferences exist, preserves saved Settings, and removes `preview.generators` from YAML. Startup does not import these options; manual upgrades must run the explicit configuration conversion. The asset storage root remains deployment configuration. See the [Preview contract](../architecture/preview.md#generation) for task snapshots, enablement and resource limits. +The installer offers the optional helper before changing the service. `--with-preview` selects it explicitly; `--without-preview` skips it. Noninteractive installation skips it unless explicitly selected. The helper package must match the main package's version and commit and pass its checksum and capability probe before installation. For an unpublished local candidate, add `--preview-archive /path/to/yatm-preview-linux-amd64-VERSION.tar.gz --preview-checksum /path/to/archive.sha256` to the ordinary install command. A same-version rerun can add the helper later. No helper source or system-wide native libraries are installed. To enable generation afterwards, see [Scan signatures and Previews](jobs.md#scan-signatures-and-previews). ## Reverse Proxy @@ -120,7 +144,7 @@ requests follow the page's current origin. CLI clients select their address inde with `--server`. `domain` does not configure TLS, bind a socket or restrict incoming Host values. YATM is served at the origin root; adding a path such as `/yatm` to `domain` does not provide subpath deployment support. The [API address contract](../architecture/api.md#browser-endpoint-addresses) -owns these rules. +describes how the browser derives its request URLs. YATM serves its control API and frontend through the configured HTTP endpoint with gRPC support. The existing Nginx deployment example below uses TLS and basic authentication; adapt its certificates, authentication file, host, and upstream to the installation: @@ -152,4 +176,4 @@ server { } ``` -See [Library/Volume workflows](library.md) for normal operations and [testing](testing.md) for isolated validation requirements. +See [Library/Volume workflows](library.md) for normal operations. diff --git a/docs/operations/jobs.md b/docs/operations/jobs.md new file mode 100644 index 0000000..2fb67ab --- /dev/null +++ b/docs/operations/jobs.md @@ -0,0 +1,83 @@ +# Archive, Restore and Scan + +Run the three kinds of Job: copy files to Media, restore saved versions to a Location, and observe +files or Media to publish content facts, inventory and integrity results. Loading a Tape for these +Jobs is described in [Devices and Tape operations](devices.md). + +## Archive and Restore + +Archive and Scan with Preview accept Library selections or live Location entries without prior content analysis. Accumulate Archive files/directories across folders in the Chonky waitlist. A Library File without an original association has no Add to Archive action; a directory can be added without checking its descendants. For Library selections, the estimate counts only Files with original associations and reports skipped Files separately; live Location selections count their observed ordinary files. Preparation skips those same Files and requires at least one eligible regular File. Expansion is bounded, overlap is deduplicated, and unknown sizes are reported separately. Archive preparation establishes File identities and actual content and freezes Library-relative archive targets. Physical selection does not imply mirrored archive paths. + + yatm-cli archive create --location 2:photos --location 3:reports + yatm-cli archive create --file-id 42 --file-id 43 + yatm-cli files versions 42 + yatm-cli restore create 17 18 --target-location 3 --directory recovered + +Preparation alone is not Archive completion. Choose Media on the resulting Job to copy and verify. An associated original that cannot be read is an error, never a substitution by another duplicate File. Preview failure does not undo analysis or archival. + +Restore selects FileVersions and permits confirmed authorized destinations without previous analysis. The browser chooser lists Preferred Locations, remembers the last chosen directory and revalidates it when reopened; it does not collect browsed files. The [Restore contract](../architecture/jobs.md#restore) owns candidates, frozen paths, collisions, actual-content verification and automatic association. Existing originals are never displaced; ignored outputs stay unlinked. Explicit damaged-copy recovery never labels mismatched bytes as a verified version. + +## Restore Saved Versions + +Add files or directories from multiple folders to the Restore waitlist. Ordinary +selections follow **Latest saved version**, or **At or before** a local date/time +chosen in the Restore dialog's application-owned calendar. Double-click a regular waitlist entry or use **Change version** +to choose explicit content. Custom versions remain selected when the global time +changes; **Apply current policy to all** returns them to automatic selection. + +Files without a matching dated version remain in the waitlist. Change the cutoff, +choose an explicit version, remove the entry, or explicitly skip unmatched entries +before preparing. A version with no usable copy remains an error; skipping a date +mismatch never substitutes newer content or bypasses a copy failure. Selecting a +date chooses recorded file content, not a historical directory snapshot. Older +metadata may retain only first/last archive dates, not every intermediate save. + +The CLI uses RFC3339 with an explicit timezone; the cutoff is inclusive: + +```shell +yatm-cli restore estimate --file-id 12 --before '2026-09-01T23:59:59+08:00' +yatm-cli restore create --file-id 12 --before '2026-09-01T23:59:59+08:00' --skip-unmatched-versions --target-location 7 +yatm-cli restore create 42 --file-id 12 --before '2026-09-01T23:59:59+08:00' --target-location 7 +``` + +The last example pins version 42 for its File while other Files under selection +12 follow the cutoff. Omit `--skip-unmatched-versions` to keep missing time matches +blocking. Creation prepares a frozen Job; later waitlist/policy changes do not +change that Job's selected content. [Restore execution](../architecture/jobs.md#restore) +owns output and retry rules. + +## Scan + +Use **Scan** for selected files, directories, a Location or Media. Choose the signature policy and result handling independently; Preview is an optional stage of the same Job. Refresh only reads the current directory. + + yatm-cli scan create --location-id 2 --path photos --signature fill-missing --result originals + yatm-cli job wait 9 --wait-timeout 2m + yatm-cli scan results 9 --limit 100 + +`--signature known-only` reuses applicable facts or cache entries without hashing on a miss; `fill-missing` reads unknown or changed files; `force-read` reads every selected ordinary file. `--result report` leaves Library facts unchanged, while `originals` publishes successful original observations. `--compare-library` looks up matching content. `--preview-policy missing-only` generates missing derivatives; `regenerate-all` also replaces existing ones, and `none` disables generation. Without a usable content identity, known-only skips Preview rather than hashing implicitly. See [Scan policies and publication boundaries](../architecture/jobs.md#scan). + +`analyze create` and `preview create` are convenient presets for this same Scan service and Job kind, not separate runners. `analyze create --mode basic/incremental/force` selects known-only/fill-missing/force-read and publishes originals; `preview create` publishes originals with Preview enabled. Use `scan create --help` for the full combination of controls. + +Jobs continue after navigation. Scan stops at the first failed Location and leaves that Location's prior records intact; earlier fully published Locations remain committed. A failed or cancelled preparation is terminal. Create and explicitly submit a new Scan to read the complete selection again; the earlier Job can be retained with its evidence. A Scan is neither a filesystem snapshot nor a durable archive. + +## Scan Signatures and Previews + +Preview generation is an optional Scan stage. Archive can request a companion Scan from its frozen input manifest. Scan keeps source/range, Results and Previews visible; Advanced holds signature acquisition and optional archived-copy counting. Its summary explains the effective choices. Signature policy selects cache-only reuse, filling missing content facts, or uncached reads. Preview policy independently selects no generation, missing-only generation, or regeneration of every eligible bundle. Cache-only misses are skipped rather than hashed; choose reading missing facts when new files need Previews. Known-only can still read known content for decoding. Results-only avoids catalog publication but still performs requested reads and Preview writes. Comparison reports recorded archived copies, not fresh health checks. Existing Preview metadata is read independently with `preview get`. See [Preview](../architecture/preview.md) for bundle ownership and [ACP integrity](../architecture/media-io.md#acp-and-integrity) for cache versus verification guarantees. + +Installing the helper does not enable generation. In **Settings → Preview**, enable generation, choose Image or Video and edit the corresponding extensions and output options. Shared controls set concurrent files, timeout, input-pixel limit and an optional helper command override. Disabled controls preserve their values. Existing Preview assets remain accessible when generation is disabled or the helper is absent. Check installation independently with `yatm-cli preview capabilities`. + +## Media Inventory and Integrity + +- Media Scan selects `--result inventory` to publish inventory, or `--result verify` to check recorded copies. Both use the same pipeline; verification forces actual reads against the saved baseline. Unknown archives require explicit **Add to Library**. +- `scan media ` is the inventory preset; `verify create ` is the integrity preset. Mounted Volume work runs automatically. Tape waits for an explicitly selected matching device through `scan run --device `; the CLI never chooses or formats a device. +- Inventory does not claim healthy contents, and integrity findings do not accept damaged checksums as new expectations. See [historical checks and Media health](../architecture/library.md#copy-health). + +## Job Execution Settings + +**Settings → Jobs** holds the pipeline limits that every new Archive, Restore and Scan attempt reads +when it starts: manifest batch, queued reads, queued results, results per update and the update +interval. Queue limits count items, not bytes or memory. A running attempt keeps the limits it +started with. From the CLI, read the group with `yatm-cli settings job` and replace it with +`--execution-json FILE`. The [execution settings contract](../architecture/jobs.md#execution-settings) +lists the defaults and valid ranges. + diff --git a/docs/operations/library.md b/docs/operations/library.md index 63c97d1..58f3021 100644 --- a/docs/operations/library.md +++ b/docs/operations/library.md @@ -1,6 +1,6 @@ -# Library and Mounted Volume Workflows +# Library and Mounted Volumes -These workflows cover Library browsing and archive operations. See the [Location guide](locations.md) for registered originals, live browsing, and Library-selected Archive. +These workflows cover Library browsing, organization and search, mounted Volumes, metadata backup and identical-file tools under the [daily-use constraints](index.md#daily-use). [Archive, Restore and Scan](jobs.md) owns those Jobs; the [Location guide](locations.md) owns registered originals and live browsing. ## Browse from the CLI @@ -26,7 +26,7 @@ path such as `'/Photos/100% ready?#'` can be written as the URI `'library:///Photos/100%25 ready%3F%23'`; plain paths keep those characters literally. Names containing a colon are also literal: `yatm-cli ls library:2026` selects the same directory as `yatm-cli ls /library:2026`. -The [CLI path rules](../architecture/cli.md#directory-paths) define decoding, +The [CLI path rules](cli.md#directory-paths) define decoding, root containment and selector conflicts. An ordinary listing reads the complete directory. `--query` and Library `--recursive` @@ -52,7 +52,7 @@ The OS owns mount/unmount/eject. `yatm-cli volume candidates` lists the same dis Tags and notes belong to logical File/directory identity, not content or shared archive copies. Rename, move, and Trash preserve annotations without changing physical files. [Library](../architecture/library.md) defines normalization, limits, versions and cleanup protection. -Search runs in the active pane and keeps the other pane available for **Locate in Other Pane**. Fields are `name`, `tag`, `note`, `type`, `size`, `mtime`, `location`, and `has`; bare terms match name, Tag, or note. Use `location:` with a positive Location ID; the [Library search contract](../architecture/library.md#annotation-search-and-cleanup) defines Location and content-presence predicates. Adjacent conditions mean AND. Boolean operators, parentheses, and `*`/`?` wildcards use the query parser's native rules. Use decimal bytes and quoted RFC3339 timestamps: +Search runs in the active pane and keeps the other pane available for **Locate in Other Pane**. Fields are `name`, `tag`, `note`, `type`, `size`, `mtime`, `location`, and `has`; bare terms match name, Tag, or note. Use `location:` with a positive Location ID; `has:original`, `has:archive` (known archive coverage, not a health check), `has:unknown` and `has:duplicates` select by content presence; the [Library search contract](../architecture/library.md#annotation-search-and-cleanup) defines them. Adjacent conditions mean AND. Boolean operators, parentheses, and `*`/`?` wildcards use the query parser's native rules. Use decimal bytes and quoted RFC3339 timestamps: ```text name:report AND (tag:finance OR note:"final") @@ -60,9 +60,42 @@ type:file AND size:>=1048576 mtime:>="2026-01-01T00:00:00Z" AND NOT tag:obsolete ``` -Export creates a Library metadata backup, not a copy of physical content or Settings. Import replaces declared entity types transactionally, always preserves the target installation's Settings, and preserves prior Library data on invalid references or unique-key conflicts; review the [backup contract](../architecture/library.md#backup-and-legacy-compatibility) before replacement. +## Library Settings -Open **Tools → Identical files**, or **Find identical files** from Files. Nothing is queried until **Find** runs in the tool; changing the source or scope clears the results. Library matches current and saved-version content, including transitive relationships. Results use one scrollable grouped list and load visible rows automatically. Select one target and choose **Merge into** to combine the whole group and its version history. The Locations source searches complete selected Locations; search by name and see each selection as a chip. Choose **Keep this** for one survivor or **Delete** for a selection. Physical removals use each Location's Trash. Completed removals disappear without repeating Find; a changed group needs another explicit Find before Keep or Merge. Click a member for the ordinary right-side Inspector. +**Settings → Library** separates two default-on preferences: + +- **Include unbacked files** affects only Library visibility. Saved-only mode retains editable logical directories and Files with any saved version, including versions whose last copy is lost. Location browsing and duplicate groups are independent. +- **Confirm Delete** controls the browser's move-to-Trash dialog. Server checks always run; a CLI mutation reports its resolved plan under `--dryrun` and writes when the flag is omitted. + + yatm-cli settings library + yatm-cli settings library --confirm-remove=false + yatm-cli ls --file-id 0 --scope saved + +The CLI reads the current Library group before applying supplied fields, so omitted flags preserve other preferences. Startup never silently traverses existing roots. There is no watcher or periodic full scan. + +## Trim + +`yatm-cli library trim --positions [--files] [--dryrun]` removes invalid Positions and, with +`--files`, Files with neither an original nor saved history. Review the dry-run report first; saved version history is +kept. The [Trim contract](../architecture/library.md#trim) owns the exact rules. + +## Metadata Backup + +Export creates a Library metadata backup, including Locations, original links and saved Media encryption keys, not physical content or global preferences. Protect it like the keys it contains. Import replaces the entity types declared by that backup, including Location configuration when present. It preserves global Library/Preview/Job/Device preferences and rolls back on invalid references or unique-key conflicts; review the [backup contract](../architecture/library.md#backup-and-legacy-compatibility) before replacement. + +Use `yatm-cli library import --input library.jsonl --dryrun` to validate a downloaded backup without +replacing Library data. Downloads remain streamed on all supported HTTP versions. With an HTTP/1.0 +connection, including a proxy's upstream connection, a truncated download can appear successful; +import validation rejects the missing JSONL end record. + +Import is a quiesced operator action taken with no Job running. It validates live relationships, +keeps imported registrations immediately usable, and imported facts cannot authorize local retries. + +## Identical Files + +Open **Tools → Identical files**, or **Find identical files** from Files. Nothing is queried until **Find** runs in the tool; changing the source or scope clears the results. Library matches current and saved-version content, including transitive relationships. Results use one scrollable grouped list and load visible rows automatically. Select one target and choose **Merge into** to combine the whole group and its version history. The chosen File retains its original; other File identities move to Library Trash and lose their original links while their disk bytes remain. The Locations source searches complete selected Locations; search by name and see each selection as a chip. Choose **Keep this** for one survivor or **Delete** for a selection. Physical removals use each Location's Trash. Completed removals disappear without repeating Find; a changed group needs another explicit Find before Keep or Merge. Click a member for the ordinary right-side Inspector. + +**Duplicates in Locations** groups known signatures without automatic full-disk hashing. Expanded pages check members and distinguish confirmed, changed and unavailable entries; unopened ranges remain unchecked. Scan selected ranges to improve coverage. Unknown content does not mean unique content. Duplicate Files keep independent organization, tags and Note. ```shell yatm-cli identical groups --source library --limit 20 @@ -79,37 +112,4 @@ yatm-cli identical keep --source locations --root 12 --root 18 --group --fi yatm-cli files remove-version --file-id --version-id [--dryrun] ``` -The first groups or members reply includes a `result_id`; pass it as `--result` with the returned `--cursor` to continue without recalculating. `find` also returns a result ID and row counts for offset reads; `close` releases it early. A result expires after 30 minutes idle or a service restart, and then requires a new Find. Reuse the exact source/root scope and current fingerprint for mutations; changed groups require refresh. Keep processes unloaded members too. Version removal deletes the catalog record only; later existing coverage reconciliation can recreate it. - -## Restore Versions - -Add files or directories from multiple folders to the Restore waitlist. Ordinary -selections follow **Latest saved version**, or **At or before** a local date/time -chosen in the Restore dialog's application-owned calendar. Double-click a regular waitlist entry or use **Change version** -to choose explicit content. Custom versions remain selected when the global time -changes; **Apply current policy to all** returns them to automatic selection. - -Files without a matching dated version remain in the waitlist. Change the cutoff, -choose an explicit version, remove the entry, or explicitly skip unmatched entries -before preparing. A version with no usable copy remains an error; skipping a date -mismatch never substitutes newer content or bypasses a copy failure. Selecting a -date chooses recorded file content, not a historical directory snapshot. Older -metadata may retain only first/last archive dates, not every intermediate save. - -The CLI uses RFC3339 with an explicit timezone; the cutoff is inclusive: - -```shell -yatm-cli restore estimate --file-id 12 --before '2026-09-01T23:59:59+08:00' -yatm-cli restore create --file-id 12 --before '2026-09-01T23:59:59+08:00' --skip-unmatched-versions --target-location 7 -yatm-cli restore create 42 --file-id 12 --before '2026-09-01T23:59:59+08:00' --target-location 7 -``` - -The last example pins version 42 for its File while other Files under selection -12 follow the cutoff. Omit `--skip-unmatched-versions` to keep missing time matches -blocking. Creation prepares a frozen Job; later waitlist/policy changes do not -change that Job's selected content. [Restore execution](../architecture/jobs.md#restore) -owns output and retry rules. - -## Preview and Hash Cache - -Preview generation is an optional Scan stage. Archive can request a companion Scan from its frozen input manifest. Signature policy selects cache-only reuse, filling missing content facts, or uncached reads. Preview policy independently selects no generation, missing-only generation, or regeneration of every eligible bundle. Cache-only misses do not trigger hashing. Existing Preview metadata is read independently with `preview get`. See [Preview](../architecture/preview.md) for bundle ownership and [ACP integrity](../architecture/media-io.md#acp-and-integrity) for cache versus verification guarantees. +The first groups or members reply includes a `result_id`; pass it as `--result` with the returned `--cursor` to continue without recalculating. `find` also returns a result ID and row counts for offset reads; `close` releases it early. A result expires after 30 minutes idle or a service restart, and then requires a new Find. Reuse the exact source/root scope and current fingerprint for mutations; changed groups require refresh. Keep acts on the whole current group, including members not yet loaded. Version removal deletes the catalog record only; later existing coverage reconciliation can recreate it. diff --git a/docs/operations/locations.md b/docs/operations/locations.md index c942fd8..1adad19 100644 --- a/docs/operations/locations.md +++ b/docs/operations/locations.md @@ -1,13 +1,13 @@ -# Locations, Originals and File Versions +# Locations and Originals -Status: v1 development workflow. [Library](../architecture/library.md) owns organization and content; [Jobs](../architecture/jobs.md) owns execution guarantees. +These workflows follow the [daily-use constraints](index.md#daily-use). [Library](../architecture/library.md) owns organization and content; [Jobs](../architecture/jobs.md) owns execution guarantees. ## Register and Browse 1. Open **Settings → Locations → Add location** and choose a name and authorized server directory. **Browse directories** opens a read-only chooser; choosing a directory changes the draft and does not register it until Save. **Restore destination** recommends it in the chooser; it is not write permission. -2. Configure **Ignore** with gitignore text. Matching entries are hidden from live browsing, excluded from measurement and from Scan/Archive selections (including an explicitly named path), and never become new originals or Restore links. Existing associations and writes are unaffected: explicit admission remains available, and Restore still writes its output. Required runtime/archive exclusions cannot be bypassed. +2. Configure **Ignore** with gitignore text. For a new Location, blank uses the server's default YATM-managed-entry rules; nonempty custom text replaces that default set. Later edits preserve intentionally empty rules. Matching entries are hidden from live browsing, excluded from measurement and from Scan/Archive selections (including an explicitly named path), and never become new originals or Restore links. Existing associations and writes are unaffected: explicit admission remains available, and Restore still writes its output. Required runtime/archive exclusions cannot be bypassed. Under **Advanced**, optionally enable **Use mmap for content reads**. Mmap is off by default and applies when Archive or Scan reads original file content, including Archive preparation hashes; it does not change ordinary directory browsing. -3. Save. The directory is immediately browsable and usable, whether registered or imported. Registration does not collect files or create a Job. +3. Save. Changing the root or Ignore keeps existing records; future reads use the current configuration and existing Jobs cannot be redirected. The directory is immediately browsable and usable, whether registered or imported. Registration does not collect files or create a Job. 4. Choose the Location in Files. Both panes use the same source while retaining separate directories. Pages show actual files, empty/dot directories and optional Library associations. An unreadable entry stays visible with its error and cannot be operated on; other rows remain usable. A directory-level read failure clears that pane and provides Retry, never an offline listing. File names preserve legal UTF-8 text, including backslashes, leading/trailing spaces and @@ -18,7 +18,7 @@ still apply when creating or restoring a name. yatm-cli location create --name Documents --root /allowed/documents --ignore-file ./ignore.txt yatm-cli ls location://Documents/photos - yatm-cli ls --location-id 2 --path photos --query "type:file AND size:>1M" --limit 100 + yatm-cli ls --location-id 2 --path photos --query "type:file AND size:>1048576" --limit 100 yatm-cli files get --location-id 2 --path photos/example.jpg `ls` accepts the Location's exact display name in its URI, or the existing ID flags. @@ -28,34 +28,7 @@ quoting and duplicate Location names. A listing reads the directory completely and writes one JSON Lines message per batch, so it takes no page size and no cursor. A query is a paged Search that keeps both: continuing it uses the cursor it returned. Reload after navigation or when the directory itself is replaced. Paths refer to the server except CLI input/output files. `ls` is a pure, minimal read; `-l` requests attributes and `--status` requests availability. **Add to Library** opens Scan with known-only signatures and original publication. `files metadata` admits an uncollected entry when saving annotations. File details do not open full bytes through HTTP. See the [shared Files contract](../architecture/library.md#shared-organization-interface). -## Independent Settings - -**Settings → Library** separates two default-on preferences: - -- **Include unbacked files** affects only Library visibility. Saved-only mode retains editable logical directories and Files with any saved version, including versions whose last copy is lost. Location browsing and duplicate groups are independent. -- **Confirm Delete** controls the browser's move-to-Trash dialog. Server checks always run; a CLI mutation reports its resolved plan under `--dryrun` and writes when the flag is omitted. - - yatm-cli settings library - yatm-cli settings library --confirm-remove=false - yatm-cli ls --file-id 0 --scope saved - -The CLI reads the current Library group before applying supplied fields, so omitted flags preserve other preferences. Startup never silently traverses existing roots. There is no watcher or periodic full scan. - -## Scan and Find Duplicates - -Use **Scan** for selected files, directories, a Location or Media. Choose the signature policy and result handling independently; Preview is an optional stage of the same Job. Refresh only reads the current directory. - - yatm-cli scan create --location-id 2 --path photos --signature fill-missing --result originals - yatm-cli job wait 9 --wait-timeout 2m - yatm-cli scan results 9 --limit 100 - -`--signature known-only` reuses applicable facts or cache entries without hashing on a miss; `fill-missing` reads unknown or changed files; `force-read` reads every selected ordinary file. `--result report` leaves Library facts unchanged, while `originals` publishes successful original observations. `--compare-library` looks up matching content. `--preview-policy missing-only` generates missing derivatives; `regenerate-all` also replaces existing ones, and `none` disables generation. Without a usable content identity, known-only skips Preview rather than hashing implicitly. See [Scan policies and publication boundaries](../architecture/jobs.md#scan). - -`analyze create` and `preview create` are convenient presets for this same Scan service and Job kind, not separate runners. `analyze create --mode basic/incremental/force` selects known-only/fill-missing/force-read and publishes originals; `preview create` publishes originals with Preview enabled. Use `scan create --help` for the full combination of controls. - -Jobs continue after navigation. Scan stops at the first failed Location and leaves that Location's prior records intact; earlier fully published Locations remain committed. A failed or cancelled preparation is terminal: delete it and create a new Scan, which reads the complete selection again. A Scan is neither a filesystem snapshot nor a durable archive. - -**Duplicates in Locations** groups known signatures without automatic full-disk hashing. Expanded pages check members and distinguish confirmed, changed and unavailable entries; unopened ranges remain unchecked. Scan selected ranges to improve coverage. Unknown content does not mean unique content. Duplicate Files keep independent organization, tags and Note. +Unregistering a Location removes executable original/tracking associations, not disk bytes, Files, versions, copies or Preview. Failed access alone removes nothing. ## Physical Operations and Organization @@ -75,25 +48,4 @@ These actions use the [shared organization engine](../architecture/library.md#sh **Locate original** changes an association, not a disk path. It refuses a target owned by another File. Use it when limited observations cannot reliably reconnect an external move. Automatic continuity follows path → valid available signature → native identity within the observed range; it cannot prove every move-versus-copy history. -## Archive, Preview and Restore - -Archive and Scan with Preview accept Library selections or live Location entries without prior content analysis. Accumulate Archive files/directories across folders in the Chonky waitlist. A Library File without an original association has no Add to Archive action; a directory can be added without checking its descendants. For Library selections, the estimate counts only Files with original associations and reports skipped Files separately; live Location selections count their observed ordinary files. Preparation skips those same Files and requires at least one eligible regular File. Expansion is bounded, overlap is deduplicated, and unknown sizes are reported separately. Archive preparation establishes File identities and actual content and freezes Library-relative archive targets. Physical selection does not imply mirrored archive paths. - - yatm-cli archive create --location 2:photos --location 3:reports - yatm-cli archive create --file-id 42 --file-id 43 - yatm-cli files versions 42 - yatm-cli restore create 17 18 --target-location 3 --directory recovered - -Preparation alone is not Archive completion. Choose Media on the resulting Job to copy and verify. An associated original that cannot be read is an error, never a substitution by another duplicate File. Preview failure does not undo analysis or archival. - -Restore selects FileVersions and permits confirmed authorized destinations without previous analysis. The browser chooser lists Preferred Locations, remembers the last chosen directory and revalidates it when reopened; it does not collect browsed files. The [Restore contract](../architecture/jobs.md#restore) owns candidates, frozen paths, collisions, actual-content verification and automatic association. Existing originals are never displaced; ignored outputs stay unlinked. Explicit damaged-copy recovery never labels mismatched bytes as a verified version. - -## Inventory, Integrity and Metadata Backup - -- Media Scan selects `--result inventory` to publish inventory, or `--result verify` to check recorded copies. Both use the same pipeline; verification forces actual reads against the saved baseline. Unknown archives require explicit **Add to Library**. -- `scan media ` is the inventory preset; `verify create ` is the integrity preset. Mounted Volume work runs automatically. Tape waits for an explicitly selected matching device through `scan run --device `; the CLI never chooses or formats a device. -- Inventory does not claim healthy contents, and integrity findings do not accept damaged checksums as new expectations. See [historical checks and Media health](../architecture/media-io.md#copy-health). -- Unregistering a Location removes executable original/tracking associations, not disk bytes, Files, versions, copies or Preview. Failed access alone removes nothing. -- Library metadata export includes Library management metadata, not contents, Settings, operation results or Job bundles. Import is a quiesced operator action taken with no Job running; it validates live relationships, rolls back on conflicts, preserves the target Settings and keeps imported registrations immediately usable. Imported facts cannot authorize local retries. - The [compatibility policy](../README.md#pre-stable-compatibility) preserves v0.1.x migration/import. Incompatible pre-stable v1 stores are rejected before modification and require an explicitly chosen conversion or reinstall. Cross-Location transfer, uploads, editing, permissions, scheduling, multi-Executor transport and snapshots are outside this workflow. diff --git a/docs/operations/migration.md b/docs/operations/migration.md index 2b836b8..710814a 100644 --- a/docs/operations/migration.md +++ b/docs/operations/migration.md @@ -10,14 +10,14 @@ Obtain the versioned installer rather than reusing a script from the old install ```shell curl --fail --location --output install-release.sh \ - https://raw.githubusercontent.com/samuelncui/yatm/v1.0.0-alpha.2/install-release.sh -sudo bash install-release.sh --version v1.0.0-alpha.2 --check -sudo bash install-release.sh --version v1.0.0-alpha.2 + https://raw.githubusercontent.com/samuelncui/yatm/v1.0.0-alpha.3/install-release.sh +sudo bash install-release.sh --version v1.0.0-alpha.3 --check +sudo bash install-release.sh --version v1.0.0-alpha.3 ``` The default channel remains stable; Alpha requires `--version`. See [installation](install.md#install-or-update) for prerequisites, custom installation/unit paths, and local candidate archives. `--check` downloads and verifies the candidate and reads installation metadata without stopping the service, quiescing Jobs, creating databases, or leaving an upgrade attempt in the installation. -Automatic upgrade supports Linux amd64/systemd with SQLite and configuration, Job storage, scripts, helpers and captured indexes contained in the installation. External databases or storage require coordinated manual backup. The preflight reports passed checks, manual checks and blockers. Script existence is not evidence of Tape compatibility. Busy operations, unknown versions, missing migration instructions, invalid packages and unsupported layouts stop before service interruption. Originals, Restore output and archive Media are not software migration data; mixed layouts that cannot be classified must be resolved first. +Automatic upgrade supports Linux amd64/systemd with SQLite and configuration, Job storage, scripts, helpers and captured indexes contained in the installation. The [offline migration procedure](offline-migration.md) has the same contained-layout requirement. External resources need coordinated backup and a layout-specific migration procedure before Prepare or Commit; this guide does not provide that procedure. The preflight reports passed checks, manual checks and blockers. Script existence is not evidence of Tape compatibility. Busy operations, unknown versions, missing migration instructions, invalid packages and unsupported layouts stop before service interruption. Originals, Restore output and archive Media are not software migration data; mixed layouts that cannot be classified must be resolved first. The actual upgrade shows its retained report directory before the first confirmation. Review the current/target versions, findings, backup scope, configuration diff and Settings summary. Cancel or EOF at that prompt leaves the service unchanged. A second confirmation approves a legacy database migration report after the service has stopped and its complete backup exists. @@ -29,63 +29,28 @@ Creating empty Settings tables during migration preserves the approved configura - Each historical Job becomes a bundle containing its manifest, execution metadata and archived log. A submitted Archive item requires an unambiguous recorded physical copy; pending Restore selections require valid candidates. Legacy Restore output roots retain their original working-directory-relative meaning and are frozen for later execution. - Migration writes the current Catalog and Job bundle revision 1 directly, including the current Position parent index and signed nanosecond timestamps. Legacy instants retain their precision, lower digits are zero-filled, and out-of-range dates fail conversion instead of wrapping. Existing pre-stable v1 data is not upgraded in place; the [timestamp contract](../architecture/persistence.md#timestamps) defines current encoding and unknown values. - Captured legacy LTFS indexes supply validated order/extents for exact path/size matches. Missing or mismatched inventory is reported. Physical storage profiles retain their own `ltfs_v0`/`ltfs_v1` identities. -- The reviewed conversion moves legacy `paths.source` and `paths.target` to Locations and removes them from YAML; the old target becomes a preferred Restore destination. Existing registrations and user removals take precedence. This registration does not scan contents or move Library organization. [Location configuration](install.md#location-configuration-and-migration) owns authorization and subsequent editing. +- The reviewed conversion moves legacy `paths.source` and `paths.target` to Locations and removes them from YAML; the old target becomes a preferred Restore destination. Existing registrations and user removals take precedence. This registration does not scan contents or move Library organization. [Location configuration](locations.md) owns authorization and subsequent editing. - Preview generator preferences move to Settings when absent; saved preferences take precedence. Deployment paths, scripts, helpers, permissions, service working directory and unrelated configuration remain in place. Release-owned programs, frontend, documentation, licenses, Skill and templates are replaced completely. Default script templates are deployed only for a fresh installation. ## Tape Script Adaptation -Scripts are the environment adaptation boundary: they own LTFS executable paths, device mapping, vendor options and local helper logic. Preserve that customization. Installation checks never run Tape scripts, mount a cartridge, format Media or establish physical Tape readiness. - **Action required before the first Tape Job:** v1 requires a completed captured Index at `TAPE_DIR/.schema`. A legacy script that writes elsewhere or captures no Index is incompatible until adapted. Keep the old capture directory through migration, then update the mount output and ensure unmount waits for final Index output and device release as described below. The installer displays this notice before upgrade and preserves active scripts for manual adaptation. -Device-information scripts write JSON containing `barcode` to `OUT`. The service accepts the -historical full cartridge label (for example `ABC001L5`), takes its first six characters and normalizes -the identity to uppercase. Explicit user-supplied Tape identities remain six-character values. -Successful `{"barcode":""}` output means the electronic barcode is unassigned: FORMAT uses the -explicit user-supplied barcode and writes it through the formatting script. APPEND and Restore -require an existing device identity. An unreadable field or failed probe exits nonzero rather than -reporting an unassigned barcode. FORMAT still erases existing content regardless of barcode state. -Check this distinction in customized probes: historical shell pipelines can hide a failed helper -and return an empty barcode with exit status zero. Preserve local device/helper choices while -propagating failures; the [bundled probe](../../scripts/readinfo) shows an explicit field check. -Scripts run from the service working directory. Migration resolves literal relative capture options -from that same directory, independently of `paths.work`; offline backup validation maps this path -into the preserved installation. - Complete data migration **before changing the legacy mount script's captured-index output directory**. Prepare reads the old script/configuration to locate existing index evidence. Keep that directory and its contents if any retained script still uses it. After migration, adapt scripts manually and validate them before the first Tape Job. -| Input | Meaning | -| --- | --- | -| `DEVICE` | Configured Tape device; helpers may resolve its SCSI generic mapping. | -| `KEY_FILE` | Encryption key file supplied to the encryption script. | -| `MOUNT_POINT` | Mount destination for the current operation. | -| `TAPE_BARCODE`, `TAPE_NAME` | Cartridge identity/name passed to formatting and encryption scripts. | -| `TAPE_DIR` | Per-Job, per-cartridge artifact directory once identity is known; write captured indexes and LTFS diagnostics here. | -| `OUT` | Output filename for the device-information script's JSON result. | +Update the mount output destination while retaining vendor-specific commands: replace the fixed +`-o work_directory=/opt/yatm/captured_indices` with `-o work_directory="${TAPE_DIR}"` and use the +index capture arguments for the installed LTFS distribution from the +[mount template table](tape-scripts.md#mount-and-unmount). -The mount script must make the current cartridge's captured Index available as `TAPE_DIR/.schema`, where `` is the mounted cartridge identity. Mount/unmount receive `TAPE_DIR`, not the formatting script's barcode/name inputs. The final Index must describe the completed write before YATM publishes it. Retain vendor-specific commands while updating their output destination; for an LTFS implementation supporting the bundled options, the relevant change is: +The installer preserves customized active scripts; apply the relevant [bundled template](../../scripts/mount) +changes manually. [Tape scripts](tape-scripts.md) owns the complete input, output, mount and unmount contract. -```diff -- -o work_directory=/opt/yatm/captured_indices -o capture_index -+ -o work_directory="${TAPE_DIR}" -o capture_index -``` - -Create `TAPE_DIR`, then resolve it to an absolute path before passing LTFS its directory options, so background execution does not reinterpret a relative `paths.work`: - -```shell -mkdir -p -- "${TAPE_DIR}" -TAPE_DIR=$(cd -- "${TAPE_DIR}" && pwd -P) -``` - -The [bundled mount template](../../scripts/mount) illustrates this contract; its flags are not universal across LTFS distributions. The installer preserves customized active scripts; apply the relevant template changes manually. - -Unmount must return only after final Index output is complete, the mount has ended, and the Tape device has been released. A successful `umount` invocation alone may precede background LTFS cleanup. An ejecting setup must also wait for its configured eject completion condition. The [bundled unmount template](../../scripts/umount) shows one bounded wait implementation; retain the equivalent behavior appropriate to the local environment. Nonzero exit or timeout reports failure rather than claiming successful publication. - -Service startup acceptance and Tape acceptance are separate outcomes. A successful installer does not certify customized scripts or physical hardware. Use the [physical Tape procedure](physical-tape-e2e.md) only with explicitly assigned scratch Media. +Service startup acceptance and Tape acceptance are separate outcomes. A successful installer does not certify customized scripts or physical hardware. ## Backup Layout and Upgrade Phases @@ -131,43 +96,10 @@ The installer has no durable phase-resume protocol. A previously unfinished `.ya ## Configuration Review -`yatm-migrate -phase config-plan -plan-file ` generates a review without changing the installation. It contains a unified YAML diff, proposed Location imports, preserved Settings and resolved Preview preferences. The installer displays these before consent. `config-check` rechecks the exact file, all three Settings groups and relevant Locations after stop; any presence or value change invalidates the review. `config-apply -service-stopped --confirm` publishes the reviewed preferences and rewrites YAML after the backup and required legacy migration. +`yatm-migrate -phase config-plan -plan-file ` generates a review without changing the installation. It contains a unified YAML diff, proposed Location imports, preserved Settings and resolved Preview preferences. The installer displays these before consent. `config-check -service-stopped -plan-file ` rechecks the exact file, all three Settings groups and relevant Locations after stop; any presence or value change invalidates the review. `config-apply -service-stopped -plan-file --confirm` publishes the reviewed preferences and rewrites YAML after the backup and required legacy migration. All phases use the same `-config` and service working directory. Missing or null `paths.access` is replaced with explicit roots equivalent to the legacy source/target authorization. Explicit access rules, including `[]`, are unchanged. Relative paths retain their original service-working-directory meaning. Equal source/target roots share one registration. Successful conversion removes the legacy YAML fields; subsequent startup does not import them or recreate deleted registrations. No persistent import checkpoint is stored. Each saved Settings group is detected independently; a saved Preview group wins, while an absent Preview group receives the resolved legacy generator preferences. Location identity/name/Ignore/restore preference, unrelated YAML fields/comments and file permissions are retained. Unsupported ambiguous YAML blocks conversion before stop. An already converted configuration has no legacy fields to import; new installations use current configuration and default Settings. -## Offline Migration and Repair - -The offline sequence below supports a contained SQLite installation, with the same backup-root mapping as the installer. Stop the service and take a verified complete backup first. External databases/work roots need coordinated backup. Run the verified candidate's `yatm-migrate` from the original working directory, with its original configuration and captured indexes intact. Verify the archive checksum, then extract it with GNU tar (`--acls --xattrs --xattrs-include='*' --numeric-owner`) into a private temporary directory under `.backup/`. Use that directory as `yatm_backup_dir`; the immutable archive itself is not a `-backup-root` directory. Replace the placeholders below with the candidate, report and extracted-backup directories: - -```shell -cd /opt/yatm -"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase preflight -service-stopped -"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase prepare \ - -report-file "$yatm_reports_dir/migration.json" -# Review the report and historical manifests before continuing. -"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase commit --confirm -"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase cleanup \ - -install-root /opt/yatm -backup-root "$yatm_backup_dir" --confirm -``` - -The installer runs Cleanup once after Commit. Cleanup owns the complete validation before deleting obsolete artifacts. To inspect a committed migration and obtain a validation report separately, use the standalone phase: - -```shell -"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase validate \ - -install-root /opt/yatm -backup-root "$yatm_backup_dir" -``` - -On a failed or declined Prepare, use `-phase abort --confirm`; restart the legacy service only after Abort succeeds. Keep the report. Complete Cleanup before resuming business activity: later Library edits or Job progress can legitimately differ from the migration baseline. Apply the previously reviewed configuration only after Cleanup succeeds. Cleanup is repeatable and removes only verified obsolete active artifacts, not the archive. [Migration implementation](../../internal/migrate/legacy/migrate.go) owns conversion and validation details. - -Historical Job repair reads evidence from the explicitly selected backup, not legacy tables retained indefinitely in the active Catalog: - -```shell -./yatm-migrate -config ./config.yaml -phase repair-job -job-id 123 \ - -install-root /opt/yatm -backup-root "$yatm_backup_dir" --confirm -``` - -Stop the service and inspect the selected Job before repair. A previously frozen Restore root takes precedence over later configuration edits. A replaced Job bundle is retained beside the extracted backup under `jobs-before-repair/`. Move this repair evidence to the retained attempt directory before removing extraction scratch, and verify all repaired items before resuming the Job. The original archive remains unchanged. - ## Failure and Complete-Backup Recovery Program/resource replacement is not atomic. Interruption can leave a mixed tree; once Commit is uncertain or replacement has begun, keep the service stopped. Replacing only a binary or only a database is not a rollback. diff --git a/docs/operations/offline-migration.md b/docs/operations/offline-migration.md new file mode 100644 index 0000000..5ef7d13 --- /dev/null +++ b/docs/operations/offline-migration.md @@ -0,0 +1,46 @@ +# Offline Migration and Repair + +The support procedure for running a v0.1.x migration phase by phase outside the installer and for +repairing historical Jobs. Ordinary upgrades follow the [upgrade guide](migration.md); +use this only when that guide sends you here or a maintainer directs it. + +The offline sequence below supports a contained SQLite installation, with the same backup-root mapping as the installer. Confirm that layout before Prepare or Commit: Cleanup rejects a non-SQLite backup or configuration, Job storage, mount script or captured-index path outside the preserved installation. Additional external backups alone do not make this sequence applicable. + +Stop the service and take a verified complete backup. Run the verified candidate's `yatm-migrate` from the original working directory, with its original configuration and captured indexes intact. Verify the archive checksum, then extract it with GNU tar (`--acls --xattrs --xattrs-include='*' --numeric-owner`) into a private temporary directory under `.backup/`. Use that directory as `yatm_backup_dir`; the immutable archive itself is not a `-backup-root` directory. Set `yatm_release_dir`, `yatm_reports_dir` and `yatm_backup_dir` to existing absolute candidate, private report and extracted-backup directories. The configuration plan path below must be new: + +```shell +cd /opt/yatm +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase preflight \ + -install-root /opt/yatm -service-stopped +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase config-plan \ + -plan-file "$yatm_reports_dir/config-plan.json" +# Review the configuration diff, Location imports and preserved Settings before Prepare. +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase config-check \ + -plan-file "$yatm_reports_dir/config-plan.json" -service-stopped +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase prepare \ + -report-file "$yatm_reports_dir/migration.json" +# Review the report and historical manifests before continuing. +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase commit --confirm +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase cleanup \ + -install-root /opt/yatm -backup-root "$yatm_backup_dir" --confirm +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase config-apply \ + -plan-file "$yatm_reports_dir/config-plan.json" -service-stopped --confirm +``` + +The installer runs Cleanup once after Commit. Cleanup owns the complete validation before deleting obsolete artifacts. To inspect a committed migration and obtain a validation report separately, use the standalone phase: + +```shell +"$yatm_release_dir/yatm-migrate" -config ./config.yaml -phase validate \ + -install-root /opt/yatm -backup-root "$yatm_backup_dir" +``` + +On a failed or declined Prepare, use `-phase abort --confirm`; restart the legacy service only after Abort succeeds. Keep the report. Complete Cleanup before resuming business activity: later Library edits or Job progress can legitimately differ from the migration baseline. Apply the previously reviewed configuration only after Cleanup succeeds. Then replace release-owned files using the verified package and perform the installation guide's service-bound readiness checks before resuming use. Cleanup is repeatable and removes only verified obsolete active artifacts, not the archive. [Migration implementation](../../internal/migrate/legacy/migrate.go) owns conversion and validation details. + +Historical Job repair reads evidence from the explicitly selected backup, not legacy tables retained indefinitely in the active Catalog: + +```shell +./yatm-migrate -config ./config.yaml -phase repair-job -job-id 123 \ + -install-root /opt/yatm -backup-root "$yatm_backup_dir" --confirm +``` + +Stop the service and inspect the selected Job before repair. A previously frozen Restore root takes precedence over later configuration edits. A replaced Job bundle is retained beside the extracted backup under `jobs-before-repair/`. Move this repair evidence to the retained attempt directory before removing extraction scratch, and verify all repaired items before resuming the Job. The original archive remains unchanged. diff --git a/docs/operations/physical-tape-e2e.md b/docs/operations/physical-tape-e2e.md deleted file mode 100644 index 55dc2c9..0000000 --- a/docs/operations/physical-tape-e2e.md +++ /dev/null @@ -1,986 +0,0 @@ -# Physical Tape E2E Test Suite - -Status: Hardware release gate. See [test environment safety](testing.md); this guide does not authorize a physical run. - -This suite is a hardware gate in the [Release SOP](testing.md#release-sop). After affected source and nonphysical checks pass, it can run from a reviewed local build while release CI proceeds, following [Check Scope and Reuse](testing.md#check-scope-and-reuse). It covers behavior that the LTFS `file` backend cannot prove: drive discovery, hardware encryption, physical partition placement, tape motion, unload and reload, reads after a process restart, and a real physical full-Media boundary. Run the automated LTFS file-backend E2E first; it remains the primary coverage for injected failures, deterministic capacity exhaustion, two-Media completion, backup compatibility, and cleanup. - -## Safety and Preconditions - -`FORMAT` destroys every existing file on the loaded cartridge. Use one explicitly assigned scratch cartridge whose contents may be erased. A second scratch cartridge is needed only for the optional continuation case. Never run this suite against a production Library database, production Job directory, or a cartridge containing valid data. - -Before starting: - -1. Record the expected six-character barcode and confirm it on the physical cartridge. -2. Use an isolated YATM configuration with empty Executor and Library databases and dedicated work, source, and restore directories. -3. Confirm that the configured Tape device resolves to its SCSI generic device through Linux sysfs. The host needs Bash, Python 3, Node.js, tmux, systemd, the official LTFS programs, `mt`, `stenc`, `umount`, `fuser`, `timeout`, `openssl`, `sqlite3`, `findmnt`, and the package controller's ordinary installation/xattr tools. The main package supplies `yatm-lto-info`. -4. Confirm that the drive, cartridge generation, LTFS implementation, and encryption support are compatible. -5. Baseline preparation adds `-r 'size=1M'` only to the isolated format adapter. Capacity preparation uses ordinary packaged placement; use a separate fresh installation and state for that sequence. -6. Baseline preparation retains the package controller's 4 GiB free-space prerequisite. Capacity requires more than twice the selected payload plus 8 GiB, or payload plus the largest file plus 8 GiB with `--capacity-source-reflink`; see [capacity measurement](#continuous-boundary-capture). Budget full published Restore output before writing. -7. Record the YATM commit, ACP commit, LTFS version, drive model and firmware, cartridge generation, native cartridge capacity, barcode, and start time. - -Keep the host on stable power and reserve the drive for the complete run. Do not test process crashes or power loss as part of this suite. - -## Automated Stages - -### Prepare the Artifacts and Host Context - -Use checksum-verified Linux amd64 main and matching Preview packages from the reviewed source. -Local builds use `build/release/build.sh` and `previewworker/build/build.sh`. Reviewed locally -compiled binaries can begin affected hardware checks once their source and nonphysical checks -pass, independently of CI's final release-byte gate. This installer-based procedure uses packages -so that installation and runtime adapters are also exercised. Final release-byte acceptance and -the evidence comparison in [Check Scope and Reuse](testing.md#check-scope-and-reuse) still apply. - -On the authorized Linux hardware host, use an authorized root shell outside the agent sandbox. -Prefer a host-local executor with `--host local`; SSH transfers inputs and retrieves results. -Read applicable host operator notes first. Allocate an existing spacious, xattr-capable filesystem -for the isolated installation, fixtures and private `TMPDIR`. Keep controller inputs, `state.json` -and evidence outside the installation root that cleanup removes. - -Transfer the main archive, Preview helper archive, matching -`yatm-preview-source-linux-amd64-.tar.gz` and their `.sha256` companions. -Keep the corresponding-source archive beside the Preview helper; the package validator requires it -before installation. Transfer the reviewed controller's -`e2e/physical_package_acceptance.py` and `e2e/package*.py`, and these validators, preserving their -repository-relative layout under `CONTROLLER_ROOT`: - -- `build/release/check-release.mjs`, `archive-members.mjs` and `package-assets.mjs`; -- `build/release/check-candidate-set.mjs` and `check-preview-sources.mjs`. - -Include the operational dependencies of every selected check, beyond its entrypoint. For the -Linux virtual-unmount regression, transfer `e2e/test_package_tape_scripts.py` and -`e2e/ltfs-file-backend/umount` plus `get_device`. Broader Tape-script unit selections additionally -need `scripts/mkfs` and `scripts/get_device`. Source-based LTFS E2E uses `e2e/testdata` adapters; -its `umount-file-failures.sh` calls `umount-file.sh`, which delegates to the same -`e2e/ltfs-file-backend/umount` and `get_device`. Keep those relative paths and executable modes. -Source-based Go E2E remains a local/CI check under the [E2E guide](e2e-test.md), separate from -the host's installed-package procedure. - -Supply absolute host-local values for `CONTROLLER_ROOT`, `TEST_PARENT`, `EVIDENCE_ROOT`, -`HOST_TMP`, `MAIN_ARCHIVE`, `PREVIEW_ARCHIVE`, `STATE` and the official LTFS executables. -Set `STATE` to the new run's `state.json`. `VERSION` and `COMMIT` identify the archives; -the controller's reviewed source identity is recorded separately. Supply `SCRATCH_DEVICE` and -`SCRATCH_BARCODE` from the explicit assignment. Keep actual host, device, barcode and local path -values in the private handoff and evidence. - -Run from the absolute controller directory in every new host shell: - -```bash -set -euo pipefail -cd -- "$CONTROLLER_ROOT" -umask 077 -mkdir -p -- "$EVIDENCE_ROOT" "$HOST_TMP" -export TMPDIR="$HOST_TMP" -sha256sum -- "$MAIN_ARCHIVE" "$PREVIEW_ARCHIVE" > "$EVIDENCE_ROOT/packages.sha256" -sha256sum e2e/physical_package_acceptance.py e2e/package*.py \ - build/release/check-release.mjs build/release/archive-members.mjs \ - build/release/package-assets.mjs build/release/check-candidate-set.mjs \ - build/release/check-preview-sources.mjs > "$EVIDENCE_ROOT/controller.sha256" -uname -a > "$EVIDENCE_ROOT/kernel.txt" -id > "$EVIDENCE_ROOT/executor.txt" -findmnt -T "$TEST_PARENT" > "$EVIDENCE_ROOT/filesystem.txt" -df -Pk "$TEST_PARENT" > "$EVIDENCE_ROOT/space.txt" -``` - -Compare these hashes with the reviewed transfer manifest. Record the YATM and ACP source -identities, any reviewed working-tree changes, toolchains/build flags, extracted program hashes, -LTFS build/version, adapter hashes and any transferred regression inputs. The private host record -also identifies drive model/firmware, -cartridge generation/capacity, assigned identity, resolved Tape/SG mapping, drive reservation, -start time, and the isolated service/configuration. Package version labels alone do not establish -the executed bytes. The controller checks archive companions, package members and offline program -identity before installation. - -### Resolve LTFS Before Starting systemd - -Resolve the official LTFS installation in the host shell and supply its absolute executable paths: - -```bash -MKLTFS_BIN=$(realpath -- "$(command -v "$MKLTFS_COMMAND")") -LTFS_BIN=$(realpath -- "$(command -v "$LTFS_COMMAND")") -test -x "$MKLTFS_BIN" -test -x "$LTFS_BIN" -``` - -`MKLTFS_COMMAND` and `LTFS_COMMAND` are the reviewed executable names or absolute paths from -the handoff. Match the mount template to that LTFS distribution: - -| LTFS distribution | Mount template | Index capture arguments | -| --- | --- | --- | -| HPE or older capture-enabled builds using the bare option | `scripts/mount` | `-o work_directory="${TAPE_DIR}" -o capture_index` | -| Current official LTFS using a capture directory | `scripts/mount.openltfs` | `-o work_directory="${TAPE_DIR}" -o capture_index="${TAPE_DIR}"` | - -Both templates create `TAPE_DIR` and resolve it to an absolute directory before launching LTFS. -Normal unmount must finish the LTFS process and leave the final current-cartridge -`TAPE_DIR/.schema`. The [Tape adapter contract](migration.md#tape-script-adaptation) -owns these inputs and output. - -The physical controller below selects the openltfs template. Supply the corresponding official -LTFS build; an HPE run requires reviewed controller adapter selection matching the HPE template. -During `prepare`, `--mkltfs "$MKLTFS_BIN"` and `--ltfs-binary "$LTFS_BIN"` put absolute commands -into the isolated adapters, and the format adapter adds `-r 'size=1M'`. This happens -before the installer starts systemd. Inspect and hash the resulting adapter copies before -`baseline`. A login shell's `PATH` is insufficient evidence of the systemd service's commands. - -### Run the Scoped File-Backend Precheck - -Before the first physical operation, run the same package/controller pair through the existing -official LTFS file-backend format/append/Restore/Verify case. Use openltfs with its `file` backend -and the matching directory-valued capture option. The virtual unmount adapter waits for the -exact mount-owning LTFS process, identified by its Linux `/proc` command line, to exit as well as -for FUSE to detach. That boundary makes the final Index available before publication checks. - -Choose the exact checks and verify their transferred inputs before a tmux launch. For a changed -virtual-unmount adapter, the focused Linux regression selector is: - -```bash -cd -- "$CONTROLLER_ROOT" -python3 -m unittest -v \ - e2e.test_package_tape_scripts.VirtualUnmountTests.test_waits_for_final_index_after_fuse_detaches -``` - -This check uses a simulated LTFS process and ordinary Linux tools. Require an executed pass, -including its delayed final Index assertion. Record the selector, dependencies and result in -the handoff. The real file-backend check below selects one exact package case and its declared -prerequisite; retain that selection in its launch command and report. - -Put the resolved executables on the host shell's -`PATH`; the controller resolves them into its isolated virtual adapters before service installation: - -```bash -cd -- "$CONTROLLER_ROOT" -export TMPDIR="$HOST_TMP" -export PATH="$(dirname -- "$MKLTFS_BIN"):$(dirname -- "$LTFS_BIN"):$PATH" -test "$(realpath -- "$(command -v mkltfs)")" = "$MKLTFS_BIN" -test "$(realpath -- "$(command -v ltfs)")" = "$LTFS_BIN" -python3 e2e/package_acceptance.py \ - --host local --test-parent "$TEST_PARENT" \ - --archive "$MAIN_ARCHIVE" --preview-archive "$PREVIEW_ARCHIVE" \ - --version "$VERSION" --commit "$COMMIT" --ltfs \ - --case ltfs-format-append-restore-verify \ - --keep-root --out "$PRECHECK_EVIDENCE" -``` - -`PRECHECK_EVIDENCE` must be a new directory. The selected case includes its fresh-installation -prerequisite and creates only owned virtual-cartridge directories. Allow at least the controller's -16 GiB free-space prerequisite. `--keep-root` retains its stopped installation for review. - -Require a passed report for the selected case, with both FORMAT and APPEND partition checks: - -1. FORMAT writes a small file and a file larger than 1 MiB; APPEND adds different small content - to the same Media. This precheck and the physical fixture use the same size-only 1 MiB - placement rule, which remains effective for ACP's temporary write filenames. -2. After normal unmount, parse each Job's actual final captured `.schema`; obtain partition roles - independently from the FORMAT log. Compare exported Library Positions with actual Media paths, - sizes, complete extents and 17-byte partition/block/offset storage order. Both partition letters - must occur in the fixture. -3. The final APPEND Index must retain every earlier file and its extents on both partitions. - Earlier Positions and Media ID remain unchanged; new Positions match the final Index. -4. Restore all selected files and compare actual output hashes and sizes with the sources. - Verify reads every recorded copy and publishes matching findings and healthy observations. - -Preview acceptance resolves the PNG through its frozen Archive content signature. An Archive -selection can have a saved version without a known current Location signature; the saved identity -is the source of truth for that derivative check. - -Review `report.json`, including `case_scope`, `partition_checks`, command exits and the retained -Job artifacts. A completed copy without the final Index/Position comparison is incomplete -acceptance. Retain a failure and resolve its first unmet check before advancing to hardware. -The separate file-backend capacity case remains part of full nonphysical acceptance; this scoped -precheck provides a quick check of the relevant publication and readback path. - -### Launch One Durable Host-Local Stage - -Stages share one isolated installation and the controller's existing state: - -| Stage | Work and completion boundary | -| --- | --- | -| `prepare` | Fresh installation, deterministic sources, prepared Archive/Preview Jobs and byte-preserving Preview evidence; no Tape operation | -| `baseline` | PT-01..05: inspection, FORMAT, reload, barcode refusal, APPEND and final Index/Position/Preview checks | -| `restore` | PT-06: start the same stopped installation, restore both Archives, verify bytes and storage order | -| `performance-prepare` | Fresh separate installation, ordinary-placement setup Archive and capacity/performance observation helpers; no Tape operation | -| `performance-format` | FORMAT the setup Archive, eject and validate its final Index and Positions | -| `capacity-write` | PT-07: generate the frozen ordinary-file fixture, submit one Archive APPEND and preserve settlement, publication and failure evidence | -| `capacity-verify` | PT-07: Restore every published capacity file, verify bytes, signatures and storage order; no Archive write | -| `cleanup` | PT-08: seal and verify evidence after baseline Restore or capacity readback, delete idle test Jobs and remove the owned installation after review | - -Use `prepare → baseline → restore → cleanup` for the baseline and a separate -`performance-prepare → performance-format → capacity-write → capacity-verify → cleanup` -for capacity. Both preparation stages require absent state. Later invocations require the -preserved state, matching package identity, host, device, barcode and LTFS paths. Capacity -verification can follow a failed write only after observed settlement and device release; -cleanup requires successful full published readback. A baseline run does not prove capacity acceptance. Each successful -non-cleanup stage stops its isolated service. Give every invocation a new `--out` directory and -unique log/exit filenames. Keep `STATE` and the preparation stage's shared physical evidence -directory for the entire run. - -In a host-local Bash shell outside the sandbox, set `STAGE` to the next authorized stage and -`STAGE_OUT`, `STAGE_LOG`, `STAGE_EXIT` and `SESSION` to new paths/a unique tmux name from the -handoff. Put the log and exit file in an existing evidence directory. Recheck live drive ownership -and the assigned cartridge before physical use. Launch one stage, then review its result before -launching the next: - -```bash -cd -- "$CONTROLLER_ROOT" -sha256sum -c "$EVIDENCE_ROOT/packages.sha256" -sha256sum -c "$EVIDENCE_ROOT/controller.sha256" -common=( - --host local --test-parent "$TEST_PARENT" - --archive "$MAIN_ARCHIVE" --preview-archive "$PREVIEW_ARCHIVE" - --version "$VERSION" --commit "$COMMIT" - --physical-device "$SCRATCH_DEVICE" --physical-barcode "$SCRATCH_BARCODE" - --mkltfs "$MKLTFS_BIN" --ltfs-binary "$LTFS_BIN" --state "$STATE" -) -test ! -e "$STAGE_OUT" -test ! -e "$STAGE_LOG" -test ! -e "$STAGE_EXIT" -case "$STAGE" in - prepare|baseline|restore|cleanup) ;; - *) echo 'Select one documented physical stage' >&2; exit 1 ;; -esac -cleanup_options=() -if [[ "$STAGE" == cleanup ]]; then - # Baseline without a syscall observer; instrumented runs supply exact trace paths below. - cleanup_options=(--no-syscall-trace) -fi -printf -v stage_command '%q ' python3 e2e/physical_package_acceptance.py \ - "${common[@]}" "${cleanup_options[@]}" --physical-stage "$STAGE" --out "$STAGE_OUT" -printf -v stage_shell \ - 'cd -- %q || exit; export TMPDIR=%q; %s >%q 2>&1; stage_status=$?; printf "%%s\\n" "$stage_status" >%q; exit "$stage_status"' \ - "$CONTROLLER_ROOT" "$HOST_TMP" "$stage_command" "$STAGE_LOG" "$STAGE_EXIT" -printf -v tmux_command '%q ' bash -c "$stage_shell" -tmux new-session -d -s "$SESSION" "$tmux_command" -``` - -The host's tmux process keeps the stage alive when SSH or an agent turn ends. Read status on -that host without submitting another operation: - -```bash -if tmux has-session -t "$SESSION"; then - tmux display-message -p -t "$SESSION" '#{session_name}: #{pane_current_command}' -fi -tail -n 60 -- "$STAGE_LOG" -test ! -f "$STAGE_EXIT" || cat -- "$STAGE_EXIT" -test ! -f "$STAGE_OUT/report.json" || \ - jq '{status, error, phases, cases, cleanup}' "$STAGE_OUT/report.json" -test ! -f "$STATE" || \ - jq '{completed_stages, service, install, url, physical_evidence_dir, cleaned}' "$STATE" -``` - -An ended session normally makes `tmux has-session` exit nonzero. A stage passes only when its -exit file is `0`, its report is passed, its expected evidence assertions pass, and state records -its completed checkpoint. A missing exit file or a running/failed report requires inspection of -the existing session, service, Job phase and owned mounts. Reconnect to these records after a -disconnect; the absence of a connection does not imply the stage stopped. - -### Resume, Handoff and Review - -Advance from completed checkpoints with the same state and a new output directory. Recovery -of an interrupted invocation uses the controller's existing guards: - -- `baseline` can reuse a normally completed FORMAT only while no APPEND Job has been created. - It rechecks Media identity, encryption and FORMAT evidence before continuing. A saved APPEND - Job or incomplete/failed FORMAT requires operator diagnosis rather than another baseline write. -- `restore` can collect/check an already submitted Job after it settles - normally; it does not submit that saved Job's Media operation again. -- `capacity-write` and `capacity-verify` each persist an admission marker before work and refuse - another invocation after that marker. Read the retained result and Job state; never clear the - marker or replay Media to repair evidence. A failed write may proceed to its first explicit - `capacity-verify` after settlement and device release. -- `cleanup` resumes deletion only against the sealed evidence and unchanged remaining Job - catalog. It verifies every preserved checksum again; an active Job, uncertain lease, remaining - mount, changed catalog or changed evidence stops deletion. Preserve state, reports and shared - evidence outside the installation; host-local cleanup rejects paths that it would remove. - -For a collection-only failure, retain the failed report, review/check the collector repair, -record its changed hash and resume only where the guard above permits it. State records remain -the evidence of what happened. An unresolved extent or Index/Position mismatch remains an -unmet acceptance check even when the write or unmount completed. - -A fresh executor's private handoff includes: - -- authorized host/drive/cartridge and destructive scope, drive reservation and operator notes; -- exact source, package, program, controller and adapter hashes, LTFS selection, tools and host context; -- absolute controller/input/state/evidence/installation/TMPDIR paths, service name and CLI URL; -- exact selected checks/stage and their operational dependencies, the variables and complete - launch/status commands above, current tmux/log/exit/report locations; -- completed checkpoints and Job IDs, first unfinished check, pass criteria and failure stop condition; -- executor, result reviewer and cleanup owner, including the coordinator's decision before cleanup. - -One executor owns the drive. The coordinator reviews stage reports and compact evidence before -authorizing the separate `cleanup` invocation. Cleanup seals and verifies the shared Job evidence -before deletion, preserves state and stage reports outside the removed installation, and leaves -the cartridge ejected. After review, remove only the owned precheck/fixture resources and apply -the agreed evidence retention policy. Failed hardware remains available for diagnosis; cleanup -does not establish a pass for its failed write. - -Evidence collection copies the settled Job's original `job.log` and verifies its checksum, like -the other bundle artifacts. The [CLI output contract](../architecture/cli.md) exposes text for -inspection, not a byte-preserving log archive. Preparation exercises the real packaged Job and -collector together; parser unit tests alone do not verify that boundary. - -The configured `readinfo` script starts every physical operation with `mt -f DEVICE load`. It then spends at most about one minute polling the explicit MAM Barcode field. The script removes an attached media-generation suffix such as `L5`. A successful, explicitly empty field allows FORMAT with the assigned scratch barcode; formatting writes that identity. APPEND and Restore require the existing device identity. A failed probe or unreadable field stops the operation. The [Tape contract](../architecture/media-io.md#tape) owns these identity rules. - -Normal unmount keeps the Job attempt and device lease until the mount is gone, no process holds the resolved SG device, and `mt -f DEVICE status` reports `DR_OPEN`. This post-unmount wait shares a ten-minute deadline. A successful Job therefore exposes the drive only after eject completion; timeout follows the unmount-failure rules below. - -Capacity acceptance uses ordinary Archive sources and a complete Restore of its published -selection. FORMAT, APPEND and Restore remain separate explicit Media submissions. - -An unmount failure invalidates the attempt and keeps the device unavailable for the lifetime of that YATM process. Stop the isolated YATM service before manual cleanup, identify processes holding its recorded owned mount with `fuser -vm MOUNT_POINT`, and have them release the mount before attempting a normal `umount`. A force or lazy unmount is only a last-resort manual cleanup after the service has stopped; discard that write result and restart YATM before using the device again. Do not enter an owned LTFS mount while a physical stage is running. - -Stopping a systemd unit can also terminate its LTFS child. If the mount has already disappeared, -verify that no process holds the assigned SG device. When a normal eject reports "Medium removal -prevented", release that failed session's lock with `mt -f DEVICE unlock`, then use -`mt -f DEVICE offline` and confirm `DR_OPEN`. This cleanup never makes the failed write acceptable. - -## Fixture - -Create deterministic source files and record their size and SHA-256 before starting: - -| Path | Content | Expected LTFS partition | -| --- | --- | --- | -| `dataset/index-small.txt` | 60 KiB | Index | -| `dataset/data-small.bin` | 64 KiB | Index | -| `dataset/data-large.txt` | 2.15 MiB | Data | -| `dataset/empty.bin` | Empty | No data extent | -| `dataset/nested/payload.png` | Small PNG supported by the packaged native Preview helper | Index | -| `append/index-small.txt` | Different 65 KiB content | Index | -| `append/data.bin` | 2.34 MiB of non-sparse data | Data | - -The placement rule depends only on size: non-empty files no larger than 1 MiB go to the Index -partition, and larger files go to Data. It applies to text, binary and PNG content and to ACP's -temporary write filenames. Empty files have no data extent. Resolve the partition letters from -the FORMAT log's declared roles and compare them with the captured final Index. - -The mandatory full-Media case must reach a real physical capacity boundary with incompressible data and exercise ordinary Archive source files. Retain the path, size and SHA-256 manifest for files selected for Archive and Restore. Sparse files, zero-filled files and repeating byte patterns do not establish physical Tape consumption because filesystem holes and drive compression can reduce the bytes written. - -## Required Cases - -Run the cases in order because they share one cartridge and one isolated YATM installation. - -### PT-01: Device and Barcode Preflight - -1. Start YATM with the isolated configuration and load the scratch cartridge. -2. Inspect the configured Tape device through the packaged CLI. -3. Compare a non-empty returned barcode with the assigned scratch barcode. If the electronic barcode - is empty, inspect again with `--identity BARCODE` to check Library registration for the explicitly - assigned identity before FORMAT. - -Expected results: - -- Inspection completes without a device-busy or device-discovery error. -- A reported electronic barcode matches the assigned scratch barcode. A successfully read empty - barcode is accepted only before the initial FORMAT, which writes the assigned identity. -- The isolated Library has no Tape Media with that barcode. -- No format, mount, or Library mutation occurs during inspection. - -### PT-02: FORMAT Archive and Preview - -1. Create an Archive Job for `dataset` with Preview generation enabled. -2. Wait until indexing is complete and the Job is waiting for Media. -3. Write the Job with Tape mode `FORMAT`, the expected barcode, and a test name. -4. Wait for Archive and its follow-up preview Scan to complete and for the cartridge to eject. - -Expected results: - -- Hardware encryption is configured before formatting and the key remains in the Tape Media profile. -- LTFS formatting, mounting, ordered ACP writes, unmounting, and ejecting succeed. -- Formatting passes the assigned barcode to `mkltfs`; the subsequent reload/inspection must return it. -- Every non-empty source file has the expected size and SHA-256 in the Archive result. -- Every Archive item is `SUBMITTED`; the Library contains one `ltfs_v1` Tape Media and one Position per source file. -- The captured LTFS Index contains valid extents for every non-empty file. Each persisted Position has the same first logical extent encoded as partition, block, and byte offset in its 17-byte storage order. -- `index-small.txt`, `data-small.bin` and `payload.png` are on the index partition. - `data-large.txt` is on the data partition; `empty.bin` has no extent. -- The Job Tape directory contains the LTFS log, captured Index, Archive report, and manifest. -- The Preview bundle and its HTTP asset are readable. - -### PT-03: Reload and Inspect the Durable Tape - -1. Reload the ejected cartridge with `mt -f DEVICE load` without changing the databases. A standalone drive may take about one minute to become ready. -2. Inspect it through the packaged CLI. -3. Leave it loaded for the next Archive operation; inspection does not mount or change its contents. - -Expected results: - -- The physical barcode resolves to the Media created by PT-02. -- Media ID, name, format, file count, and written bytes match the committed Library state. -- Inspection does not format, mount, or modify the Tape or Library. - -### PT-04: Reject a Different Requested Barcode - -1. Reload the same cartridge. -2. Create a second Archive Job for `append` and wait until it is ready for Media. -3. Request a different barcode through the packaged CLI and record its preflight refusal against the loaded physical cartridge. - -Expected results: - -- The CLI reports the requested/device barcode mismatch. An unavailable MAM barcode is not a pass. -- CLI preflight refuses the request before admitting an Archive Media operation. The prepared Job, existing Tape Media and Positions remain unchanged; no encryption, formatting, mounting or copying occurs. -- Reloading and inspecting the cartridge still returns the original barcode. - -Reuse `TestTapeSessionsRejectUnverifiedIdentityBeforeMutation` for the backend's independent rejection before encryption, formatting or mounting, and `TestArchiveFailedMediaAttemptSettlesPhaseAndReopens` for a failed Media operation returning the Job to its pre-Media `READY` state. These are local/CI checks at separate boundaries; physical CLI refusal does not prove an admitted runner mismatch through RPC. - -### PT-05: Append to the Existing Tape - -1. Submit the prepared Archive Job from PT-04 against the same cartridge with mode `APPEND` and the original barcode. -2. Wait for completion and eject. - -Expected results: - -- Append reuses the original Media ID and encryption key. -- Existing files and Positions remain unchanged. -- Appended files use a collision-resistant Media path prefix and become `SUBMITTED`. -- The format-time placement policy remains effective: appended `index-small.txt` is on the index partition and `data.bin` is on the data partition. -- The final captured Index contains both Archive Jobs with valid extents and storage order. -- Within each partition, physical storage order matches the deterministic Archive item order even when source preparation completes out of order. -- Job completion and device availability occur only after the cartridge has ejected. - -### PT-06: Restart and Restore - -1. Stop YATM normally after PT-05 and start it again with the same isolated databases and work directory. -2. Create one Restore Job containing files from both Archive Jobs. -3. Sort the selected Positions by their persisted storage order and record the expected Media-path sequence. -4. Reload the cartridge and restore through the physical Tape device. -5. Wait for completion and eject. - -Expected results: - -- The restarted process reads the durable Media profile and reconfigures the drive with the stored encryption key. -- Restore reads files from both the index and data partitions. -- Position storage order is propagated unchanged into Restore Copies. The sequential-source unit test verifies that requests follow ascending partition, block, and byte offset rather than logical path order; the physical run verifies that the resulting files are restored correctly. -- Every selected file completes, including the zero-byte file. -- Restored paths, sizes, bytes, and SHA-256 values match the source fixture. -- Restored ACP signature xattrs contain the expected size and SHA-256. -- Restore logs are stored under the Restore Job's `tapes//` directory. -- Job completion and device availability occur only after the cartridge has ejected. - -### PT-07: Physical Full-Media Boundary - -Use the separate [capacity sequence](#continuous-boundary-capture) with its frozen input sizes -and disk budget. Choose enough ordinary source content to reach the measured cartridge boundary; -a completed Archive whose input fits does not establish capacity exhaustion. - -1. Run `performance-prepare` and `performance-format` with `--capacity-observe` to obtain the - setup Archive and its saved before-copy capacity preflight. -2. Run `capacity-write` once. It hashes all source files, saves the immutable size/hash manifest, - prepares one ordinary Archive and submits one APPEND. Keep continuous samples through settlement. -3. Review `capacity/write-result.json`, item order/states, progress, Library export, final Index, - Archive report and Job/LTFS logs, including before-copy and before-unmount observations. -4. Once the Job is observed idle and the device is released/ejected, run `capacity-verify` once. - It restores **every published file**, up to the 100-file bound, and verifies actual output - size/SHA-256, ACP signatures and persisted Position storage order. An empty selection fails. - -Expected results and interpretation: - -- Software refusal before creating the next file is an expected capacity boundary. Keep its - complete error chain and pending suffix. `capacity-write` still returns nonzero for a Job error; - this is evidence to review, not permission to replay the write or an automatic hardware pass. -- A late write, Close or finalization failure is a distinct failure finding. `no_space` alone - cannot distinguish pre-write refusal from partial output. Compare the logs, temporary-file - observations, final Index and publication; successful readback does not erase the write failure. -- Published items form a continuous prefix on the assigned Media. Final Index paths, sizes, - extents and storage order match their Positions. The setup Archive remains present. Pending - items have no Media/path publication; no item remains staged. Report counts match published - files and bytes. Missing or contradictory finalization evidence remains a failed check. -- Device availability and eject are checked after settlement. Uncertain ownership retains the - running service and all resources for the lifecycle owner; no device status query is made while - the Job or lease is active. No automatic unmount, eject, reformat or Media retry follows failure. -- Full published readback is mandatory. A first/last sample cannot establish this result. - -### PT-08: Cleanup and Evidence - -1. Require completed baseline Restore or capacity readback. Check the entire isolated Job catalog is idle, the device lease is released, the cartridge is ejected and no owned mounts remain. Export the Library metadata backup into preserved evidence. -2. Before deleting anything, capture each Job's catalog and bundle metadata, complete Job log, LTFS log, Archive report, captured Index, before-copy/before-unmount snapshots, and the exact `sg-trace.jsonl` and `sg-trace.jsonl.error` files when present. Capture verifies each transfer against the host SHA-256; unrelated Tape files are not swept into evidence. -3. Require the source manifests and original stage timing reports described below. Finalize every supplied syscall trace and checksum-copy it into preserved evidence. Generate a path-sorted SHA-256 manifest covering these files and all Job evidence. Missing prerequisites, active observers, capture failures or checksum mismatches stop cleanup before Job deletion. -4. Recheck preserved checksums, the saved manifest digest and unchanged Job catalog, then delete only those idle test Jobs through the API. A resumed cleanup also checks the remaining catalog against the sealed inventory and compares live Tape artifacts with their preserved checksums. An old seal that omitted an SG trace or its error file cannot authorize deletion. -5. Check mounts again, stop the owned service and remove only its installation, source files and Restore output. Keep state, stage reports, sealed evidence and the written cartridge. Cleanup issues no load, format, mount, unmount, eject or Media submission. - -Expected results: - -- The export contains the Tape profile, Files, and physical Positions needed to describe the completed run. -- The evidence checksum manifest verifies every preserved Job artifact before Job deletion. -- Job deletion removes the corresponding local Job directories. -- Cleanup does not modify or erase the physical cartridge. -- The cartridge is ejected and clearly marked as scratch or test media. - -### Cleanup Evidence Prerequisites - -Use the existing shared `physical_evidence_dir` in `STATE`. Before invoking `cleanup`, supply -the reviewed source manifests at these exact paths; do not reconstruct report timings or -substitute current source hashes for those used by an earlier stage: - -| Preserved path | Required evidence | -| --- | --- | -| `source/packages.sha256` | Original package checksum record; its main-package digest must match the preparation report | -| `source/controller.sha256` | Executed controller source hashes, including `physical_package_acceptance.py`; an instrumented SG run also includes `physical_sg_trace.c` and the executed `physical_sg_trace.so` | -| `stages//report.json` | Original completed invocation report, matching commit, version and installation, with finite nonnegative duration and the matching physical case result | - -The required baseline reports are `prepare`, `baseline` and `restore`. Capacity requires -`performance-prepare`, `performance-format`, `capacity-write` and `capacity-verify`, including -the original failed `capacity-write` report for an expected software refusal. Other required -stage reports must have passed. The current cleanup report stays in its separate `--out` -directory; it cannot be a completed prerequisite of its own invocation. - -After each entered non-cleanup stage, the controller copies the exact bytes from its final -`--out/report.json` save to `stages//report.json` under the common evidence directory. -This includes failed capacity writes, their failure reason, retention decision and final duration. -Early validation failures that never enter a stage remain only in their invocation's `--out` and -do not replace its preserved report. Keep source manifest selection explicit and reviewed: - -```bash -PHYSICAL_EVIDENCE=$(jq -er '.physical_evidence_dir' "$STATE") -mkdir -p -- "$PHYSICAL_EVIDENCE/source" -for name in packages.sha256 controller.sha256; do - cp -- "$EVIDENCE_ROOT/$name" "$PHYSICAL_EVIDENCE/source/$name" - cmp -- "$EVIDENCE_ROOT/$name" "$PHYSICAL_EVIDENCE/source/$name" -done -``` - -The same source-manifest step applies to baseline. With an SSH controller, preserved files and -stage reports are controller-local; `--syscall-trace` paths below refer to the hardware host. -The controller validates the fixed -prerequisites before sealing. Source review still establishes that the recorded hashes identify -the executed source and observer; checksum syntax alone cannot prove that history. - -Cleanup requires either repeated `--syscall-trace /absolute/host/file` arguments for **every** -separately collected syscall trace, including each `strace -ff` output, or an explicit -`--no-syscall-trace` declaration when that observer was not used. Supply exact files, not directory -or wildcard arguments. Stop/detach the observer after normal Media settlement and ensure starting -the idle service for cleanup will not restart it. Cleanup refuses symlink trace files, duplicate -filenames, open traces or an uncertain `fuser` result. It copies raw bytes into -`observers/syscall/` using the existing checksum transfer helper, whether the source -was inside or outside the owned root; the evidence seal covers every copy. An observer outside -Jobs is not discovered automatically: the cleanup owner must supply its complete file set. - -For a capacity run whose Archive and Restore used the SG observer, add `--require-sg-trace`. -This requires nonempty `jobs//tapes//sg-trace.jsonl` for both Jobs and the recorded -observer source/binary hashes. Capture preserves any accompanying `.error` even without that -flag. An error marker remains a failed observer finding for review; preserving it does not -establish complete SG measurement or a successful write. Cleanup performs no observer or -hardware operation to replace missing evidence. - -For example, pass the following options to the separate authorized cleanup invocation: - -```bash -cleanup_options=(--require-sg-trace) -for trace in "${SYSCALL_TRACES[@]}"; do - cleanup_options+=(--syscall-trace "$trace") -done -``` - -`SYSCALL_TRACES` is the explicit nonempty list of finalized host files from the handoff. If no -syscall observer was used, use `cleanup_options=(--require-sg-trace --no-syscall-trace)` for an -SG-instrumented run. Omit `--require-sg-trace` only for an uninstrumented run. Preserve source, -reports and traces before the first seal; a failed or incomplete old seal is retained for review, -not silently replaced to permit deletion. - -## Optional Physical Cases - -The first physical full-Media boundary is mandatory. The cases below either require a second cartridge or repeat destructive formatting, so they are optional unless the affected behavior changed. - -### PT-X1: Continue the Full Job on a Second Tape - -Continue the pending PT-07 Archive Job on a second scratch cartridge: - -1. Load the second scratch cartridge and write the remaining items in `FORMAT` mode with its own barcode. -2. Verify that the Archive Job completes and that every item is submitted to exactly one of the two Tape Media. -3. Restore the complete capacity logical set by loading both cartridges as requested. -4. Compare every restored size and SHA-256 with the generation manifest. - -Do not add a synthetic FUSE source or streaming-file abstraction solely for this case. The physical test should exercise ordinary source files through the production I/O path. - -### PT-X2: Explicit Delete and Reformat - -After preserving the original evidence, delete the Tape Media metadata through YATM, create a new Archive Job, and format the same physical barcode again. Verify that FORMAT is rejected before metadata deletion, succeeds after deletion, creates a new Media ID, and does not delete the old logical Files. - -## Pass Criteria and Evidence - -The release gate requires the baseline and capacity sequences plus their evidence cleanup, without manual database edits or changes to archived Tape files. A reviewed expected software refusal and successful full published readback can satisfy the capacity boundary; a late write/finalization failure remains a separate finding. PT-X1 and PT-X2 are recorded separately when run. Retain: - -- the version and hardware record; -- source and restored SHA-256 manifests; -- packaged CLI request results and the reused local/CI Session identity and failed-Media lifecycle evidence; -- every Job log and report; -- every Job's catalog and bundle metadata plus the sorted evidence SHA-256 manifest; -- captured LTFS Index files from FORMAT and APPEND; -- the immutable capacity source manifest, continuous Archive/Restore samples and write/readback results; -- all raw Job Tape SG traces and optional error markers, plus every separate syscall trace when used; -- the Library JSONL metadata backup; -- a concise result for each case, including elapsed Archive, full-Media boundary, and Restore time. - -Each invocation's `report.json` records total `duration_seconds`, per-phase setup timing, -per-case timing and command durations, including failed stages. Retain separate timings for -preparation, FORMAT/baseline, Archive, complete Restore/verification and cleanup, plus wall-clock -handoff/queue/transfer time. Capacity samples mark Media admission and settlement; write-result -timestamps include fixture generation and Archive work. Post-Restore output hashing remains part -of overall stage time. - -Reuse the already accepted ordinary-file full-capacity run when review confirms unchanged -relevant production source, dependencies, build/runtime configuration and fixture semantics. -A controller/SOP cleanup does not alone require another full Tape write. Keep the original -source/package/controller hashes and result with the reviewed diff; run affected local collector -and cleanup regressions. New release archives still need their own identity/content checks and -relevant package smoke acceptance under [Check Scope and Reuse](testing.md#check-scope-and-reuse). - -Record a failure at the first unmet expected result. Preserve the cartridge and local evidence for diagnosis instead of reformatting and retrying immediately. - -## Bounded Performance Comparison - -Use `performance-prepare`, `performance-format`, and `performance-run` with a new -state and isolated installation for each compared build. Capacity uses a fresh formatted -installation without performance-case writes. Preview is optional. Preparation creates a deterministic -64 KiB setup Archive; FORMAT uses ordinary packaged data-partition placement, -without `-r size=1M`. Measured cases APPEND, normally unmount, validate the captured -final Index and Library publication, then Restore and verify actual output bytes. - -Transfer `e2e/physical_performance_*.py` with the controller and `e2e/package*.py` modules. -`make test-package-tools` runs their local regressions with the package-controller tests and -mocked device observations, without Tape access. SVG rendering additionally needs `matplotlib`; -its rendering test reports a skip when that optional dependency is absent. Measurement parsing, -fixture checks and verification tests still run. - -Supply the explicit package, host, device, barcode and evidence arguments. Use -`--performance-source-read-mode buffered` or `mapped` consistently across stages; -mapped uses the existing Location `--use-mmap` setting. Select cases with repeated -`--performance-case`. `--performance-size-gib` accepts 4 (default), 32 or 64 GiB per -case, saved in state. The fixed order and file sizes are: - -| Case | File size | Content | -| --- | ---: | --- | -| `small-random` | 256 KiB | Deterministic SHAKE-256 bytes | -| `small-compressible` | 256 KiB | Adjacent duplicated 4 KiB random blocks | -| `large-random` | 1 GiB | Deterministic SHAKE-256 bytes | -| `large-compressible` | 1 GiB | Adjacent duplicated 4 KiB random blocks | - -At 32 GiB, a small case has 131,072 files and a large case has 32. Generation emits -an external JSON Lines SHA-256/size manifest. CLI manifest pages contain at most -1,000 rows; a private validation database checks selection, duplicates, content, -previous Positions, final Index extents and Restore order without whole-manifest -maps. Index parsing and artifact transfer stream; actual restored-byte checks use -batches of at most 64. Source/Restore fixtures are removed only after complete -verification; Job bundles and evidence remain. Interrupted cases are preserved -for explicit diagnosis. Each scene requires twice its logical size plus 8 GiB -of free evidence headroom. - -Media timing includes admission through normal finalization and publication. -Fixture generation, preparation and external content checks are outside it. Keep -input bytes/order, source mode, compression/encryption, LTFS settings and physical -layout equal. Source generation and Archive preparation affect page cache; record -cache preparation and host load rather than claiming a cold-cache measurement. - -Independent bounded pollers record application progress about every two seconds, -drive metrics about every five seconds and execution-host resources about every -five seconds. One writer streams JSONL, with request wall/monotonic start and end, -errors and timeouts. Drive reads cannot block the application polling channel. -Use `--no-performance-driver-sampling` consistently across stages for collector -calibration. The sampler reads xattrs only through the LTFS process whose exact -`work_directory` identifies the admitted Job; it never opens a competing SG reader. -Timeout cleanup kills only the sampler subprocess group. - -The controller retries only read-only Job waits when Get reports -a SQLite lock, within the original deadline and with every failed transcript kept. -Delays grow from 2 to 4, 8 and 16 seconds, then remain capped at 30 seconds; -the remaining observation budget bounds the final delay. Attempts, lock errors -and chosen delays remain in the report. There is no separate attempt-count limit. -It never repeats a Create or Media action; durable failure and other errors stop -the stage for inspection. - -Application `copied_bytes` during COPYING describes ACP source/handoff progress, -not physical commitment or final Library publication. ACP `copied_file_count` -increments when work enters content processing, so its curve is labeled file-work -entries rather than completed files. Final Index/Library and restored-byte checks -establish completed content separately. Stage changes, resets and missing samples -break interval arithmetic. Root xattrs `user.ltfs.vendor.IBM.logPage.34.00` -and `.0C.00` retain their raw bytes and request times. Binary page framing, required -widths and duplicate parameters are checked before decoding. Page 0C provides -Sequential Access byte proxies; page 34 parameter 0 reports repositions per 100 MB, -not cumulative events. Equal values do not prove freshness or its absence. - -Analysis emits observations, intervals, resources, fixed 10/30-second bins, -all candidate windows, auxiliary drive samples, controller milestones, file-work -entry rates, COPY phase-edge brackets and summary CSV/JSON plus a standalone SVG. -Phase durations retain request-boundary uncertainty; unknown edges stay absent. -The observed byte-endpoint subset rate is distinct from a full successful COPY rate. Matplotlib is needed on the controller only for -plot generation. Counters produce rates over actual sample intervals; UI smoothed -speed is not averaged again. Invalid gaps remain gaps, and zero increments mean -only that no progress was observed in that interval. - -The first observed positive ACP increment anchors candidate windows every ten -seconds. Each 180-second window has six 30-second bins. A qualifying window needs -at least 90% valid time coverage per bin, no gap longer than twice the nominal -sample period, rate coefficient of variation at most 15%, and first/last 60-second -mean drift at most 10% of the window mean. Only valid adjacent increments are -allocated by time overlap. Rates with incomplete coverage describe covered time. -These fixed engineering thresholds classify application averages; they are not -vendor guarantees, physical-streaming proof or optimization acceptance thresholds. -The first qualifying window and every later candidate remain visible, including -periodic no-progress intervals that stable averages can hide. - -Use the complete curve to determine whether a longer fixture is necessary. Fix -sampling gaps before extending duration; persistent low speed and repeated pauses -are results to diagnose. Compare one matched baseline/candidate pair first. When -noise prevents a decision, use at most one reversed affected-case pair. Keep a -change only when throughput or reposition behavior demonstrably improves without -a confirmed regression in the other measure, correctness or bounded resources. -A short successful run or failed candidate alone does not establish a hardware -limit. This flow does not claim capacity-exhaustion or release acceptance. - -## Capacity Measurement - -### Continuous Boundary Capture - -Use one planned Archive run to compare the capacity sources on the same timeline. Reuse -`performance-prepare` and `performance-format` for a fresh isolated installation, then select -`capacity-write` and `capacity-verify`. Transfer `e2e/physical_capacity_*.py` and -`e2e/physical_performance_*.py` with the controller and `e2e/package*.py` modules. -This is the PT-07 ordinary Archive and full published Restore path. - -Choose the cartridge capacity and input size before starting. A separately authorized hardware -capacity reduction can shorten an experiment: HP LTO-5 documents SET CAPACITY `0x0FB3`, roughly -92 GB, in its [integration guide, page 15](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=15). -It erases the cartridge. Verify actual drive support, beginning-of-medium position and measured -capacity. Keep that reduction during FORMAT with the implementation's `mkltfs --keep-capacity` -option; ordinary formatting can restore full capacity. Record partition sizes after formatting. -Restore full capacity only after the experiment's evidence and readback are preserved. Reduced -media establish that configuration's boundary behavior, not a full-length capacity guarantee. - -Pass `--capacity-observe` during `performance-prepare` and repeat it for each later stage. -The controller installs `physical_capacity_boundary.py` in the isolated adapters after successful -mount and before returning to YATM. Pass `--work-directory "$TAPE_DIR" --mount "$MOUNT_POINT"`, -`--phase before-copy` and `--output "$TAPE_DIR/capacity-before-copy.json"`, using the reviewed -absolute helper path. It verifies required probes before copying. At the start of normal unmount, -capture `--phase before-unmount` into `capacity-before-unmount.json`; retain a failed observation -and proceed with normal finalization. These passive hooks do not synchronize or pause an active -copy. Unload clears some HP error counters, so the final mounted snapshot is part of the evidence. - -Pass the FORMAT Job's saved `capacity-before-copy.json` as `--capacity-preflight` to -`capacity-write`. It checks the owning installation/Job and usable starting page 31, page 0C -and `statvfs` before generating payload. The live before-copy hook additionally requires raw page 17 -before permitting payload writes. Each LOG page uses one Linux xattr read with a 64 KiB buffer; -this avoids LTFS's oversized-reply error for a small caller buffer and double reads of read-cleared -counters. The default is 24 ordinary 4 GiB files, with disjoint -deterministic AES streams and an immutable SHA-256/size manifest; `--capacity-files` selects -1–100. Repeated `--capacity-file-mib` instead provides up to 100 ordered sizes, allowing large -files for the main transfer and smaller files approaching the measured boundary. Freeze this -selection before the run. Reserve twice the payload size plus 8 GiB on the source/Restore filesystem. -For a filesystem supporting Linux FICLONE, `--capacity-source-reflink` instead generates one -pseudorandom base, clones ordinary files and gives each a distinct pseudorandom prefix. Each -file receives a full size/hash manifest before Archive. This saves source-disk space; every byte -still passes through Archive and the drive. It requires the payload size plus the largest file -plus 8 GiB for source and Restore space. Check this generator on the selected filesystem with -a small fixture first; unsupported cloning fails explicitly. Repeat the flag on later stages. - -Use the stage launcher above with a separate `STATE` and new outputs for each capacity stage. -Its baseline-only stage selector must be replaced by the following selector. Keep the `common` -package/identity arguments and construct capacity options from the frozen workload before launch: - -```bash -case "$STAGE" in - performance-prepare|performance-format|capacity-write|capacity-verify|cleanup) ;; - *) echo 'Select one documented capacity stage' >&2; exit 1 ;; -esac -capacity_options=(--capacity-observe) -# Optional: add once at preparation and repeat for every subsequent invocation. -# capacity_options+=(--capacity-source-reflink) -# Set FILE_MIB to the reviewed ordered list (1–100 files, 1–65536 MiB each). -for size in "${FILE_MIB[@]}"; do - capacity_options+=(--capacity-file-mib "$size") -done -if [[ "$STAGE" == capacity-write ]]; then - capacity_options+=(--capacity-preflight "$FORMAT_PREFLIGHT") -fi -# For cleanup, set cleanup_options using Cleanup Evidence Prerequisites above; -# for other stages, use cleanup_options=(). -printf -v stage_command '%q ' python3 e2e/physical_package_acceptance.py \ - "${common[@]}" "${capacity_options[@]}" "${cleanup_options[@]}" \ - --physical-stage "$STAGE" --out "$STAGE_OUT" -``` - -`FORMAT_PREFLIGHT` is the saved FORMAT Job's `capacity-before-copy.json` from this installation; -it is not a fresh device query. `FILE_MIB` is a Bash array populated from the approved workload. -Alternatively use `--capacity-files N` for N ordinary 4 GiB files. Repeat the source mode, fixture -sizes, observation and reflink options on subsequent stages. Preview is optional for this sequence, -including `cleanup`. Preserve the written cartridge after cleanup. - -The capacity sampler records application progress and owned LTFS readings every second. Its raw -pages are 31, 0C, 17, 02 and 34, alongside `statvfs`, bounded file metadata including temporary -outputs, and resource observations. All channels use the host's wall/monotonic request times; -each sequential device query retains its own start/end time. Preserve gaps, unsupported fields, -counter resets and observation delays. Job byte progress can include rejected work; it is not -actual Tape throughput or a buffer-occupancy measurement. File-work entries and file size changes -do not establish successful Close, final publication or readable content. - -Read TapeAlert through `user.ltfs.mediaStorageAlert`, which lets LTFS retain flags in its own -cache. Reading raw page 2E independently can consume read-cleared flags before LTFS observes them. -Keep the returned bitmask and query times; this observation must not hide a warning from its owner. - -To distinguish the copyer's actual capacity reads from the independent sampler, an isolated -service can run under `strace -D -f --seccomp-bpf`, selecting `statfs` and the required file -lifecycle calls. Verify that the service's main PID remains the HTTP listener and that syscall -filtering works before admitting Media. Keep raw syscall arguments, results, durations and -process identity; a capacity reading alone does not identify its caller. Go's direct `statfs` -syscall is not intercepted by a libc preload wrapper. Keep this optional observer with the -isolated service until normal settlement, and record its potential measurement cost. Before -cleanup, finalize and explicitly supply every raw output under -[Cleanup Evidence Prerequisites](#cleanup-evidence-prerequisites); files outside Job bundles -are not captured by the Job collector. - -A capacity error returns nonzero after retaining observations, item rows, Job/LTFS logs, final -Index and Library results. Inspect those results before running `capacity-verify`; the latter -requires observed settlement and device release, then restores every published file and verifies -its bytes. Restore retains continuous samples and allows eight hours for settlement; an empty -published selection cannot pass readback acceptance. No stage automatically repeats a Media operation. A software refusal before creating -the next file, a partial-write failure, and successful readback are separate findings. - -Analyze the saved stream without accessing hardware: - -```sh -python3 -B e2e/physical_capacity_analysis.py /private/evidence/archive-samples.jsonl \ - --job-log /private/evidence/job.log --output /private/evidence/capacity -``` - -The analyzer writes observation/interval/page/event CSVs, a JSON summary and, with matplotlib, -aligned SVG/PNG curves. The optional same-host Job log adds its first error as a one-second -interval and a separate plot of the 15 seconds on either side. Request bounds remain visible; -resets, gaps and partition-coordinate changes break -curves. Compare remaining-to-EW, remaining-to-partition-end and filesystem availability -with byte counters and error events. Their overlap does not alone establish causation or an -exact physical byte position. Page 17 remains available as raw parameter data for vendor-specific -interpretation; an unsupported field is not synthesized from another source. - -### Continue to the Drive's EW Boundary - -Use [physical_ew_run.py](../../e2e/physical_ew_run.py) for an explicitly authorized -scratch-cartridge experiment that must continue beyond application admission. This is an -LTFS/drive measurement; ordinary Archive acceptance remains the workflow above. The runner -verifies the cartridge, formats once, mounts the installed LTFS and generates an incompressible -AES stream directly into ordinary files. Source data need not occupy another filesystem. -It stops new payload at the first write/Close failure, observation failure or frozen workload -bound, records partial-file cleanup, then attempts normal Index finalization and eject once. -The written cartridge and evidence remain available for review. - -Run it in the host-local tmux owner using the quoted command/exit-file construction under -[Launch One Durable Host-Local Stage](#launch-one-durable-host-local-stage). Preserve the numeric controller exit -status with a single `printf '%s\n' "$stage_status"` command; verify that the exit file contains -one integer before interpreting it. An empty file is missing evidence, not exit zero. Keep the -controller's structured lifecycle, LTFS log and captured Index as independent completion evidence. -Supply a new output directory, the verified device aliases, -installed LTFS tools, the `yatm-lto-info` executable and the explicitly authorized barcode. -Choose `--max-bytes` above the measured native capacity for this incompressible workload; -reaching that bound is a stopped experiment, not an observed EW. The optional -`--expected-native-min-bytes` catches an unintentionally shortened cartridge before format. -The CLI requires `--authorize-erase` to repeat the scratch barcode. Existing run directories -cannot replay the destructive sequence. - -Build the [test-only SG observer](../../e2e/physical_sg_trace.c) on the Linux host and run -its fake-device tests before using it: - -```sh -python3 -B -m unittest discover -s e2e -p 'test_physical_sg_trace.py' -v -cc -std=c11 -O2 -Wall -Wextra -Werror -shared -fPIC \ - e2e/physical_sg_trace.c -o /private/evidence/physical_sg_trace.so -ldl -pthread -``` - -The runner loads this library only into its foreground LTFS process. It records SG_IO command -times, raw CDBs, status, residual and sense, with bounded LOG SENSE and READ POSITION responses. -File payload is excluded. Optional short READ POSITION queries execute after clean page 31 -queries on the same device descriptor, before the owning call releases LTFS's device lock, -at most once per second. Verify this locking path in the installed LTFS build. The preflight -requires usable filesystem/page readings, initialized tracing and a successful position reply. - -A position query can receive a deferred write error. Its raw response is retained separately; -an injected error disables further injection, creates a trace error marker and stops new payload. -Such a result establishes observer interaction, not the unmodified LTFS error path. Every command -keeps its own timing, so query cost and sample gaps remain measurable. Passive snapshots use the -retained TapeAlert cache and continue briefly after payload failure; command tracing remains -active through the normal unmount. Evidence contains per-write application returns, file hashes, -Close/rename/delete results, mounted attributes, LTFS logs and captured Indexes. - -Analyze the complete or still-growing capture without accessing the drive: - -```sh -python3 -B e2e/physical_ew_analysis.py \ - --sg /private/evidence/ew-run/sg.jsonl \ - --samples /private/evidence/ew-run/capture/samples.jsonl \ - --events /private/evidence/ew-run/capture/events.jsonl \ - --metadata /private/evidence/ew-run/capture/metadata.json \ - --output /private/evidence/ew-analysis -``` - -The analyzer preserves partition attribution, unknown position/buffer fields, query intervals -and current/deferred sense. With optional Matplotlib installed it writes full-run curves and, -when EW is observed, a boundary enlargement from 120 seconds before to 30 seconds after EW. -Successful EW responses have a separate command-warning count from command errors. Compare -observations immediately before and after actual EW with -their different endpoints: page 31 remaining partition capacity, page 0C EOD-to-EW distance, -LTFS availability and logical READ POSITION. A filesystem error or EOP flag alone does not prove -a successful WRITE's EW response, and logical blocks are not physical tape length. Report what -the tested workload establishes separately from a general prediction or complete-file guarantee. - -### Interpretation and Additional Observations - -Use the [Tape capacity study](../research/tape-capacity.md#evidence-required-for-a-solution) to -choose the question and success criteria before a capacity experiment. Assess pre-write refusal, -retained file readability, cleanup, Index finalization and accurate Library publication separately. -Screen candidate policies against the [continuous data-supply constraint](../architecture/media-io.md#tape-capacity) -before hardware work. Record observation-induced stalls and reposition activity as well as throughput; -a diagnostic synchronization experiment does not qualify recurring forced drains for production. -PT-07 establishes the observed publication and readback of the tested run; it does not guarantee -that every admitted file will fit on other cartridges or workloads. - -Use one host-local hardware executor for both capacity and performance experiments. Record the -actual LTFS executable and build, backend, mount options, drive firmware and cartridge identity -with each run. Read drive information through the mounted LTFS implementation's supported -interface; its device lock coordinates the observation with active I/O. An unavailable counter -is an explicit gap, not zero and not permission for another process to access the SG device. - -For a capability investigation on an idle, unmounted Linux drive, use the maintained collector -after confirming completed Jobs, device ownership and the assigned cartridge: - -```sh -python3 -B e2e/physical_tape_observe.py --device /dev/sgN --output /private/evidence/new-observation -``` - -Substitute the resolved SG device and a new host-local evidence directory. Run under the existing -host-local executor; load and final eject remain explicit operator steps. The collector checks all -Tape aliases before each query, requests complete raw LOG SENSE and READ POSITION responses, and -records timings and errors. Ownership checks do not reserve the device: keep other users and Jobs -out for the entire experiment. Queries with unsupported fields remain recorded failures, not zeros. -The collector never positions or writes the Tape. Separately approved read-only positioning can -compare the same fields at different locations; record its exact commands and completion status. -`--cartridge-memory` additionally reads HP's documented CM buffer up to its advertised size, capped -at 8 KiB. Keep all raw output private because device pages and cartridge memory can contain -identifying information. Publish only the relevant findings in the capacity study. Fast queries -on an idle drive do not establish acceptable polling cost during Archive. - -For a physical-position comparison, add `--position-probe --cartridge-memory` immediately after -each completed authorized position change. This captures full fixed/descriptor REQUEST SENSE -replies and tests the vendor READ POSITION form before the normal observations. REQUEST SENSE can -consume pending sense information, so this mode belongs only to the isolated idle-drive experiment. -Compare beginning, an existing intermediate logical block and recorded EOD in the same partition; -confirm actual logical positions before interpreting any changing field. Preserve response lengths -and unsupported-command errors. Compare CM pages separately from live replies: unchanged stored -EOD information does not describe current head movement. - -Capture timestamped raw capacity replies and the `statvfs` block counts and units alongside -logical bytes submitted, file completion, explicit synchronization and final Index results. -Keep request start/end times: separate device queries are not one atomic observation. Interpret -partition numbers using the FORMAT log and label, and record LTFS capacity adjustments before -comparing raw drive units with filesystem available bytes. A passive sample does not synchronize -the filesystem. Synchronization experiments record the requested operation, duration and error -separately from the resulting capacity sample. - -When diagnosing an authorized capacity run, preserve its available write-error, corrected-rewrite -and TapeAlert observations before unloading. On HP LTO-5, LOG page `0x02` is specific to the current Tape, -updates on physical data-set writes and clears on unload. Its parameters count different units -(including data sets and CCQ sets), so a retry count is not a byte count. Treat reload, reset, -unsupported fields and counter decreases explicitly instead of subtracting across them. Establish -the effects of head or media degradation from the drive's documented write/rewrite mechanism and -capacity guarantees. A run on one cartridge of unknown condition cannot prove those general -properties; it supplies observations of that run only. - -Compare native capacity consumed with both logical content and corrected-write activity for -compressible and incompressible input. The [capacity study](../research/tape-capacity.md#compression-and-rewrites) -explains why a prior compression ratio or retry allowance does not prove that the next complete -file will fit. Record successful admission, rejection before output creation, and failure after -partial output as different results. End-of-media alone is not a passing pre-write refusal test. diff --git a/docs/operations/read-order.md b/docs/operations/read-order.md new file mode 100644 index 0000000..66d5157 --- /dev/null +++ b/docs/operations/read-order.md @@ -0,0 +1,116 @@ +# Faster Restores with Auto Read Order + +Restoring files scattered across a Tape is usually dominated by positioning, not reading. LTO +records data in wraps that run back and forth along the tape, so files read in recorded +(**Logical**) order can send the drive along the tape length again and again. **Auto** plans the +complete selection on the loaded cartridge before reading. It uses the cartridge's position map and +the device's read parameters (manual, learned from +earlier Jobs, or built-in) to choose the order with the lowest predicted positioning, reading and +completion time. When its inputs do not apply or no order beats Logical, it keeps Logical. The order +never changes which files are read or how they are verified. + +On one HP LTO-5 drive (Z6ED firmware) reading uncompressed tar, Auto took 47–69% less time to +restore files scattered across wraps or at wrap and band boundaries, and kept Logical when the +selected members were adjacent. The +[research record](../research/tape-read-ordering.md#software-read-ordering) has the measurements, method and limits. +These results are not a guarantee on other drives, compressed media or LTFS. Planning stops within +the **Read planning budget** (default 300 seconds) and always keeps a complete order. + +## Use Auto + +Choose **Read order → Auto** under **Settings → Devices**, or set `readOrder` to +`TAPE_READ_ORDER_AUTO` with `yatm-cli settings devices`. Saving performs no hardware operation, and +**Passive read learning** refines the parameters from ordinary Restores. Each Job log reports the +order used and any fallback reason, and **Device details** shows the last prepared read. Logical +remains the default because Auto has been measured only on the hardware above. + +## How Auto Decides + +**Read order** applies to each later read attempt on the selected device. **Logical** is the default +and preserves file order; eligible exact seek can still be used. **Auto** evaluates an alternative +order when eligible member seek, applicable geometry and a cost model are available. Otherwise it +keeps logical order and reports the fallback reason in the Job log. Saving Auto is allowed offline and +does not qualify the hardware. Matching saved READ timings can support continued reading and +record reuse in Logical mode; missing seek costs never justify reading through an avoidable gap. +Applicable positioning parameters and a usable map can enable Auto. Parameters come from manual read +parameters first, then learned read parameters, then built-in initial references. Layout/native-rate +references depend on recorded density; the generic timing seed is borrowed from a measured HP LTO-5 +configuration and is not a measurement of every device. Incomplete observations retain normal reads. +For LTFS, Auto can read partition labels before mounting. This preparation takes time even when +the final read order remains Logical; the Job log records the label-reading time separately. +Auto also considers the final position. Native tar includes its normal rewind/unload. LTFS assumes +completion does not rewrite the Index and estimates rewind/unload from the final position. This +assumption changes neither mount scripts nor write protection; an actual Index rewrite is outside +the estimate. These scores compare candidate orders rather than predict a complete ETA. +Auto retains the complete order with the lowest available predicted cost. Prediction errors remain +visible, and offline verification checks sensitivity to positioning, longitudinal movement and read +speed. These estimates do not guarantee a speedup on unmeasured hardware. + +Read preparation prefers the drive's wrap report, then the supported HP LTO-5 cartridge-memory +directory, then a matching saved position map. If none is usable, ordinary Logical reading remains +available. CM is queried once during preparation, not per file or during device-details refresh; +its source and any fallback reason appear in the Job log. It stores no per-file wrap assignment. +CM format support and physical validation have the [documented limits](../research/tape-read-ordering.md#direct-physical-coordinates). + +Migrated LTFS tapes do not need a historical XML file for Auto ordering. If selected copies lack +extents, YATM can use the current Index captured by the mount adapter to fill their planning +positions. The mounted UUID and Index generation must match. Missing or conflicting Index data +keeps ordinary Logical reads; it does not alter the saved Job or Library content. + +Current **Auto** ordering requires a Tape written with hardware compression **Off**. +Hardware-compressed tapes retain ordinary Logical reads and eligible exact seek. Changing the +drive's current compression setting does not change the cartridge's written profile. Files already +compressed before Archive are not excluded when Tape hardware compression is Off. + +## Passive Read Learning + +**Passive read learning** is On by default for native tar reads. It records completed ordinary I/O +without additional measurement commands in each Job's bundle, and after each Job refits the device's parameters +in the background from the observations of every existing Job, giving each Job equal weight so one +unusual Job has little effect. Deleting a Job removes its observations from later refits. The refit +is available to subsequent Jobs. Manual read parameters take priority and are never overwritten. Turning +learning Off stops recording; observations already recorded stay with their Jobs. LTFS can use +matching native learned read parameters as mechanical priors; separate LTFS whole-file learning is not +implemented. + +**Read planning budget (seconds)** +defaults to 300 when blank and accepts positive whole seconds. Planning completes before content +reading; the budget bounds planning and preserves a complete Logical or improved order. +It does not include device preparation or mounting. +Changing these Device settings never changes the current attempt or a cartridge's written profile. + +## Manual Read Parameters + +**Read parameters JSON** accepts measured inputs. They apply only when the actual drive identity, +firmware and density, and the cartridge's written record/compression/encryption conditions match. +Unknown conditions leave estimates unavailable. The manual-parameter dialog starts from a copy of +the page draft. **Apply** validates the JSON and changes that draft; page **Save** persists it. +Cancel leaves the page unchanged and confirms discarding edited JSON. **Use automatic parameters** clears the manual draft and +returns to automatic parameter selection after Save; it does not erase server-owned learned read parameters. +Manual inputs take precedence even when incomplete, so a partial manual model can retain Logical order. Unknown values may be omitted; do not fill them with zero to make a complete +document. Time values use seconds, rates use bytes per second, and LPOS bounds use decimal strings +to preserve unsigned 64-bit values. The source reference describes the measurement conditions. +For uncompressed linear interpolation, `longitudinal_seconds_per_wrap` means seconds to move one +complete longitudinal tape span, not seconds to read all data in a wrap. Supply it instead of +`longitudinal_seconds_per_lpos`; the two fields cannot coexist, even with a zero value. +Normalized inputs need no absolute LPOS bounds. Deriving geometry from a live wrap report requires +an explicit nominal wrap count and assumes four equal bands with alternating wrap directions. A +matching saved position map can instead supply its recorded layout. Validate the applicable layout +and measurements for the actual device. The final partial wrap retains its real end of data. +The optional `completion` object describes normal unload with `full_span_seconds` and +`fixed_seconds`. The full span is one physical tape length, independent of the serpentine +logical block count. Omitted values remain unknown; LOCATE timings do not fill them automatically. + +**From device details** uses the currently displayed report, without a new +device query, to create a parameter draft from observed identity, density and record controls. It replaces any old parameter +draft instead of assigning old measurements to new hardware. Supply applicable measured timings, +geometry and the written compression/encryption conditions separately. Current variable mode +reports no maximum record size, and current write controls do not identify a cartridge's written +profile. These draft actions do not access hardware or certify that optimization is available. + +## Last Prepared Read + +Device details show **Last prepared read** with its timestamp, Job, Media, parameter/position sources, +applicability and available validation errors. It is historical preparation information, not the +currently loaded cartridge or the final read order selected by planning. It remains available while the +drive is busy; after a service restart it is absent until another read is prepared. diff --git a/docs/operations/tape-scripts.md b/docs/operations/tape-scripts.md new file mode 100644 index 0000000..15e0620 --- /dev/null +++ b/docs/operations/tape-scripts.md @@ -0,0 +1,76 @@ +# Tape Scripts + +Configured Tape scripts adapt YATM to the local LTFS distribution, drive and helpers. This guide +owns their inputs, outputs and completion requirements; the +[Tape contract](../architecture/tape.md#tape-script-compatibility) owns the compatibility policy. + +## Responsibilities + +Scripts are the environment adaptation boundary: they own LTFS executable paths, device mapping, vendor options and local helper logic. Preserve that customization. Installation checks never run Tape scripts, mount a cartridge, format Media or establish physical Tape readiness. + +## Inputs and Outputs + +| Input | Meaning | +| --- | --- | +| `DEVICE` | Configured Tape device; helpers may resolve its SCSI generic mapping. | +| `KEY_FILE` | Temporary owner-only key file supplied to `scripts.encrypt` for both LTFS and tar; removed by operation cleanup. | +| `MOUNT_POINT` | Mount destination for the current operation. | +| `TAPE_BARCODE`, `TAPE_NAME` | Cartridge identity/name passed to formatting and encryption scripts. | +| `TAPE_DIR` | Per-Job, per-cartridge artifact directory once identity is known; write captured indexes and LTFS diagnostics here. | +| `OUT` | Output filename for the device-information script's JSON result. | + +Device-information scripts write JSON containing `barcode` to `OUT`. The service accepts the +historical full cartridge label (for example `ABC001L5`), takes its first six characters and normalizes +the identity to uppercase. Explicit user-supplied Tape identities remain six-character values. +Successful `{"barcode":""}` output means the cartridge memory barcode is unassigned: FORMAT uses the +explicit user-supplied barcode and writes it through the formatting script. APPEND and Restore +require an existing device identity. An unreadable field or failed probe exits nonzero rather than +reporting an unassigned barcode. FORMAT still erases existing content regardless of barcode state. +Check this distinction in customized probes: historical shell pipelines can hide a failed helper +and return an empty barcode with exit status zero. Preserve local device/helper choices while +propagating failures; the [bundled probe](../../scripts/readinfo) shows an explicit field check. +Scripts run from the service working directory. Migration resolves literal relative capture options +from that same directory, independently of `paths.work`; offline backup validation maps this path +into the preserved installation. + +## Mount and Unmount + +Choose the bundled mount template that matches the installed LTFS distribution: + +| LTFS distribution | Mount template | Index capture arguments | +| --- | --- | --- | +| HPE or older capture-enabled builds using the bare option | [`scripts/mount`](../../scripts/mount) | `-o work_directory="${TAPE_DIR}" -o capture_index` | +| Current official LTFS using a capture directory | [`scripts/mount.openltfs`](../../scripts/mount.openltfs) | `-o work_directory="${TAPE_DIR}" -o capture_index="${TAPE_DIR}"` | + +The mount script must make the current cartridge's captured Index available as `TAPE_DIR/.schema`, where `` is the mounted cartridge identity. Mount/unmount receive `TAPE_DIR`, not the formatting script's barcode/name inputs. The final Index must describe the completed write before YATM publishes it. + +Create `TAPE_DIR`, then resolve it to an absolute path before passing LTFS its directory options, so background execution does not reinterpret a relative `paths.work`: + +```shell +mkdir -p -- "${TAPE_DIR}" +TAPE_DIR=$(cd -- "${TAPE_DIR}" && pwd -P) +``` + +The [bundled mount template](../../scripts/mount) illustrates this contract; its flags are not universal across LTFS distributions. + +Unmount must return only after final Index output is complete, the mount has ended, and the Tape device has been released. A successful `umount` invocation alone may precede background LTFS cleanup. An ejecting setup must also wait for its configured eject completion condition. The [bundled unmount template](../../scripts/umount) shows one bounded wait implementation; retain the equivalent behavior appropriate to the local environment. Nonzero exit or timeout reports failure rather than claiming successful publication. + +## Bundled Templates + +Bundled mount scripts resolve the Job's Tape artifact directory to an absolute path before passing it to LTFS. LTFS can change its working directory when it runs in the background; captured indexes must still land in the supplied Job directory when `paths.work` is relative. + +The official LTFS file-backend adapters select a supported capture option at setup: an absolute +directory value, or bare `capture_index` with an absolute work directory for older builds. They wait +for their mount's LTFS process to exit after FUSE detaches. FUSE detachment alone can precede the final Index +write; the test adapter therefore supplies the same completed-finalization boundary as physical +device release. + +The bundled encryption script loads the drive and retries `stenc` at most 60 times. If +every attempt fails, it exits nonzero so the Tape Session cannot proceed. Tar and LTFS both use +this adapter contract; the native tar provider does not replace it with a key-management protocol. Its `stenc` arguments remain part of the environment-specific script contract. +The script includes a commented stenc 2.0.0 command, which reads the barcode descriptor +from the key file's second line instead of accepting `-kd`; operators select the form +supported by their local tool. + +Installation checks never run Tape scripts. A successful installer does not certify customized +scripts or physical hardware. diff --git a/docs/operations/tar-tape-recovery.md b/docs/operations/tar-tape-recovery.md new file mode 100644 index 0000000..9502c3f --- /dev/null +++ b/docs/operations/tar-tape-recovery.md @@ -0,0 +1,242 @@ +# Recover Tar Tape with Public Tools + +This guide describes independent content extraction for `tar_pax_v1` Tape without a YATM executable, +service, catalog, Job bundle, content index or saved file locators. Keep this guide and the cartridge's +record parameters outside the YATM installation. The raw backend currently uses fixed or variable +512-byte-aligned records, ordinary PAX tar metadata and xattrs, with optional hardware compression +and optional hardware encryption. Encrypted recovery requires its independently retained key and +a compatible public device-tool unlock procedure. For normal +Archive, Restore and Scan, use the [Device guide](devices.md). + +## Tested Reader and Required Parameters + +Physical recovery has been checked with Linux st, mt-st 1.8, GNU dd and GNU tar 1.35 for variable +and fixed records, hardware compression On and Off, and script-configured encryption on the writing +drive. That does not establish another reader or qualify every encryption adapter. The +[recovery checks](../research/tar-tape.md#public-tool-recovery-checks) record what was tested and +its limits. +Other readers must establish their own record, close/reopen, positioning and EOF/EOD/EOT behavior +before using the Tape-copy procedure below. Passive diagnostics do not supply that evidence. + +Retain the barcode, initialization UUID, record mode and record bytes from the published Media +profile. Keys use the existing Media/Job profile; runtime script key files are temporary, and +FORMAT does not create a second persistent recovery descriptor or key copy. This guide covers +completed Archive content and controlled EW results, not interrupted-FORMAT recovery. + +A retained Library JSONL export also contains the same Media profile. Before losing the installation, +use `yatm-cli library export --output library.jsonl`, or the browser's Library export. Export backs +up metadata, not Tape content or Settings. With public `jq`, select exactly one matching Media row: + +```shell +umask 077 +jq -s -e --arg barcode ABC123 ' + [.[] | select(.type == "media" and .data.identity == $barcode) | .data] + | if length == 1 then .[0] else error("expected exactly one Media") end + | select(.profile.tape.format == "tar_pax_v1") + | {identity, partition: 0, profile} +' library.jsonl > recovery.json +jq -e '.identity, .partition, .profile.tape.tar' recovery.json +``` + +The Library export retains the adapter key in `profile.tape.encryption`; protect that backup. +The optional observed `encryption_algorithm` is diagnostic metadata: its absence does not prove +plaintext. Use the configuration and qualified behavior of the adapter that wrote the cartridge +to establish whether decryption is needed. A saved key alone does not show which encryption mode +the adapter applied. Do not print a key or place it in command-line arguments. + +The descriptor uses snake-case fields such as `record_mode`, `record_bytes` and `initialization_id`; +record mode is `TAPE_RECORD_MODE_FIXED` or `TAPE_RECORD_MODE_VARIABLE`. Do not guess missing record +parameters from a cartridge generation, current drive mode or sample values. On-Tape file 0 is an +identity PAX archive containing `YATM/identity.json` and `YATM/README.txt`; it is hardware-encrypted +when the profile requires it. It is useful +for checking the recorded identity after copying, but does not replace the parameters needed to +read it with the qualified reader. + +If `hardware_compression` is true, use a reader compatible with the cartridge's hardware compression +format. Its hardware decoder must return the original PAX bytes through Linux `st`; no software +decompression command is involved. Check the reader's documented decompression setting before +copying. Typical LTO readers decode automatically; a disabled decoder requires a qualified public +device-tool procedure to enable it, verify the effective state and restore any changed session +control. The write compression setting DCE does not control read decompression DDE, so setting +`mt compression` does not establish decoder readiness. A reported current algorithm identifier +alone does not prove that another cartridge's compressed content is readable. + +## Unlock an Encrypted Cartridge + +Skip this section when the writing adapter and reader establish that decryption is unnecessary. +The tested public-reader example uses `AES-256-GCM-128` with a 32-byte key and stenc 2.0.1; +it is not a native YATM algorithm selector or a promise for every configured adapter. Establish its actual +device/driver path, read modes and clearing behavior on each reader before recovery; the physical +check above covers only its stated combination. Public-tool source findings are in the +[tar study](../research/tar-tape.md#hardware-encryption-controls). +Check `stenc --version` before querying the drive. Older interfaces exposing only `-e` cannot +request these independent encryption/decryption modes, and their status command may read records. + +For explicit independent recovery, produce `recovery.json` from the protected Library export +as above, then create a protected key file with public `jq`. This is an operator export, not an +automatic runtime copy. It writes the key directly to the file without printing it or passing it +as an argument: + +```shell +set -eu +umask 077 +KEY_FILE=recovery.key +(set -C; jq -er ' + .profile.tape.encryption + | if type == "string" and test("^v1:[0-9a-fA-F]{64}$") + then .[3:] else error("invalid retained encryption key") end +' recovery.json > "$KEY_FILE") +``` + +Stop all other drive owners and write-protect the cartridge. Query the reader with +`stenc -f "$TAPE_DEVICE"`. It must support the public algorithm, the 32-byte key and decrypt-only +mode, with application-managed control and a known empty initial key state. Do not replace a +foreign key or take over external key management. Select the algorithm index from this reader's +reported mapping; an index saved or observed on the writer is not portable. + +After configuring the record policy and positioning for the selected Tape file, unlock before +`dd`. Replace `READER_INDEX` with the confirmed index. Specify both modes: `-d on` alone can also +enable write encryption in the studied stenc version. + +```shell +READER_INDEX=confirmed-index # Replace with this reader's reported AES-256-GCM-128 index. +stenc -f "$TAPE_DEVICE" -e off -d on -k "$KEY_FILE" -a "$READER_INDEX" +stenc -f "$TAPE_DEVICE" +``` + +stenc 2.0.1 writes its normal settings-change and success messages to stderr. Check the exit +status and effective encryption off/decryption on; an empty stderr is not a success requirement. +An accepted key-load command does not validate the key: a wrong, correctly sized key can load +successfully and fail only on the encrypted read. Missing/wrong keys are not evidence of damaged +tar bytes. Recheck key persistence after positioning, close/reopen or other reader-specific events; +if an event clears it, stop and establish the reader's required unlock sequence before another read. Stop on ambiguous +ownership or state. + +After copying, explicitly clear the key through the qualified public-tool release procedure and +verify its non-secret state. For the studied stenc version, both modes off send an empty key: + +```shell +stenc -f "$TAPE_DEVICE" -e off -d off -a "$READER_INDEX" +stenc -f "$TAPE_DEVICE" +``` + +Successful descriptor close is not proof of key clearing. The reader qualification must establish +that this command releases its loaded key and leaves both modes off; preserve errors and stop using +the drive if clearing cannot be confirmed. Keep the independent key file for future recovery. + +## Copy One Physical Tape File + +Use a qualified Linux non-rewinding `st` device and public `mt`, GNU `dd`, GNU tar and `jq` tools. +Stop YATM and other users of the drive first. Physically write-protect the recovery cartridge +before loading it. Confirm its identity and select the recorded +partition using the qualified reader's documented partition procedure; this format uses partition +zero. Do not run format, erase, write, filemark or unload commands during recovery. Required +public-tool read unlocking and clearing follow the scoped procedure above. +Provide enough disk space for ordinary copies and extracted files. + +Read the descriptor explicitly extracted from the Library export and configure the recorded record policy. The following commands are +operator examples, not a hardware qualification test. Replace the device and descriptor paths with +the confirmed values: + +```shell +set -eu +export LC_ALL=C +umask 077 +TAPE_DEVICE=/dev/nst0 +PARAMETERS=recovery.json +RECOVERY_DIR=recovered-tape +RECORD_MODE=$(jq -er '.profile.tape.tar.record_mode' "$PARAMETERS") +RECORD_BYTES=$(jq -er '.profile.tape.tar.record_bytes' "$PARAMETERS") +mkdir -m 700 "$RECOVERY_DIR" +case "$RECORD_MODE" in + TAPE_RECORD_MODE_VARIABLE) mt -f "$TAPE_DEVICE" setblk 0 ;; + TAPE_RECORD_MODE_FIXED) mt -f "$TAPE_DEVICE" setblk "$RECORD_BYTES" ;; + *) echo 'Unknown record mode' >&2; exit 1 ;; +esac +``` + +For variable records, the `dd` read buffer must be at least the stored maximum record size; for +fixed records it must obey the driver's qualified fixed-block read rules. Use the recorded byte +size and verify the reader's bounds. `iflag=fullblock` can combine separate record reads across a +filemark, so it is excluded. Never add `conv=noerror`, `conv=sync` or other read-error padding. + +Copy each physical Tape file separately. File 0 is identity; file 1 and later files are independent +content archives. Always rewind and skip from BOT for each copy so the previous command's implicit +close positioning cannot choose the next file. Choose a new `N` and filename each time: + +```shell +N=0 +mt -f "$TAPE_DEVICE" rewind +if [ "$N" -gt 0 ]; then mt -f "$TAPE_DEVICE" fsf "$N"; fi +# For an encrypted cartridge, perform and verify the qualified read-only unlock here. +DD_STATUS=0 +dd if="$TAPE_DEVICE" of="$RECOVERY_DIR/archive-$N.tar" \ + bs="$RECORD_BYTES" conv=excl status=progress \ + 2>"$RECOVERY_DIR/archive-$N.dd.log" || DD_STATUS=$? +MT_STATUS=0 +mt -f "$TAPE_DEVICE" status >"$RECOVERY_DIR/archive-$N.status" \ + 2>"$RECOVERY_DIR/archive-$N.status.log" || MT_STATUS=$? +printf 'dd=%s mt-status=%s\n' "$DD_STATUS" "$MT_STATUS" \ + >"$RECOVERY_DIR/archive-$N.result" +``` + +A positioning failure stops before copying. A read or status error stops enumeration; retain the error and any partial disk copy for diagnosis. +This procedure does not salvage an interrupted Archive or repair its Tape tail. Do not pipe Tape input into tar or hide +command exit statuses. `conv=excl` prevents overwriting a retained copy. + +Interpret the fresh post-read status only according to the qualified driver/drive mapping, including +its behavior across `dd` close and the status command's reopen: + +- Successful copy plus a confirmed filemark/EOF without conflicting EOD/EOT or errors completes one + physical Tape file. Even an empty Tape file is a boundary; continue with `N + 1` from BOT. +- Confirmed recorded EOD without conflicting EOT or errors ends enumeration. Bytes reaching EOD + without a terminating filemark are an unterminated tail; stop without declaring that Archive recovered. +- EOT, unknown, stale or conflicting flags, a failed status query, or unqualified EOF/EOD semantics + mean incomplete enumeration. Stop and qualify the reader; do not infer EOD from zero bytes, a + successful `dd`, a tar terminator, or an empty file. + +## Extract the Disk Copies + +Run GNU tar against each retained ordinary disk copy in a fresh directory. First inspect file 0 and +compare `YATM/identity.json` with the retained barcode, initialization UUID and record policy. It +uses a label-local `profile` with the same snake-case record field names. Extract content archives +individually; their member paths preserve the Archive's Tape path prefix. This is a separate +invocation using an existing disk copy; inspecting it does not require another Tape read. + +```shell +set -eu +export LC_ALL=C +RECOVERY_DIR=recovered-tape +N=1 +EXTRACT_DIR="$RECOVERY_DIR/extracted-$N" +mkdir -m 700 "$EXTRACT_DIR" +LIST_STATUS=0 +tar --list --verbose --file="$RECOVERY_DIR/archive-$N.tar" \ + --xattrs --warning=missing-zero-blocks --quoting-style=shell-escape-always \ + >"$RECOVERY_DIR/archive-$N.list" 2>"$RECOVERY_DIR/archive-$N.list.log" || LIST_STATUS=$? +TAR_STATUS=0 +tar --extract --file="$RECOVERY_DIR/archive-$N.tar" \ + --directory="$EXTRACT_DIR" --same-permissions --no-same-owner \ + --xattrs --warning=missing-zero-blocks \ + 2>"$RECOVERY_DIR/archive-$N.tar.log" || TAR_STATUS=$? +printf 'list=%s extract=%s\n' "$LIST_STATUS" "$TAR_STATUS" \ + >"$RECOVERY_DIR/archive-$N.tar-result" +``` + +GNU tar's default xattr extraction selection covers `user.*`; other namespaces need an explicit +compatible-system selection such as `--xattrs-include='*'` and suitable privileges. Check extracted +paths, contents and required metadata on the destination filesystem. Preserve tar's diagnostics. +Do not use `--ignore-zeros` or `--multi-volume` to conceal incomplete framing. + +Missing tar end blocks can produce a warning even when every member payload is complete, and GNU +tar may still exit successfully. The explicit missing-zero-block warning reveals that incomplete +terminator; it does not prove physical Tape finalization. Do not discard the last complete file +merely because a warning exists. + +A nonzero extraction result is not a successful whole-archive recovery. Preserve the disk copy and +diagnostics. For a separately established normal EW boundary, check the published complete members +against retained sizes and hashes; an unpublished partial member is not restored content. Other +truncated headers, payloads or padding remain errors outside this procedure. Tar header checksums +do not verify payload contents. +Independent extraction recovers ordinary files and represented metadata, not Library organization, +Job state or proof of every expected archived copy. diff --git a/docs/operations/testing.md b/docs/operations/testing.md deleted file mode 100644 index b7451c3..0000000 --- a/docs/operations/testing.md +++ /dev/null @@ -1,612 +0,0 @@ -# Development Checks and Test Environments - -Run checks in proportion to changed behavior. Commands below run from the repository root; Go tests use isolated fixtures and do not authorize access to production data or hardware. - -## Test Maintenance - -Use the [existing entrypoints](#repository-layout-and-entrypoints) first for tests, regressions -and performance work: `go test`, maintained scripts, and the frontend's native Vitest and Node -test runners. Replace recurring manual agent sequences with reproducible checked-in harnesses -and fixtures in the owning test area. Extend an existing runner or script before adding another. - -Keep reusable release, test and benchmark scripts and synthetic or public fixtures in this -repository. Supply machine-specific hosts, paths and private inputs through arguments or -environment variables. Keep credentials, real backups, deployment configuration and sensitive -results outside tracked source and public artifacts. - -Maintain tests, harnesses and fixtures with the implementation. Update or remove obsolete -assertions, unsupported scenarios and unused tooling in the same change. Cover observable -behavior, invariants and failure boundaries; avoid duplicate tests at the same boundary and -assertions that merely repeat the implementation. Distinct unit, adapter and E2E boundaries -retain their purpose. Keep verification proportional; a small edit need not create new automation. - -Record source identities, exact commands, passes, failures, skips and unrun checks with their -reasons. [Performance Acceptance](#performance-acceptance) owns measurement rules. Existing -physical Media, device, migration and visual acceptance remains required where applicable: -automate repeatable setup and checks where useful, and record each manual step's result and -why it needs an operator or visual judgment. Follow [Media and Migration Safety](#media-and-migration-safety); -automation does not expand access or publication authorization. - -### Check Scope and Reuse - -During development and review, test the changed behavior and its affected callers. Choose the -scope before running commands; include unchanged code when a shared dependency, contract or -configuration change can affect it. A new commit by itself does not invalidate every earlier -check. Batch related fixes before expensive integration checks. - -| Change | Checks during development and review | -| --- | --- | -| Documentation | Document links/examples and `git diff --check` | -| Go implementation | Changed packages and affected callers; add race checks for concurrency/shared state and both drivers for SQLite behavior | -| Frontend | Affected Vitest tests and type checking; style/lint checks for touched presentation, browser interactions for affected visible behavior | -| API, persistence or shared infrastructure | Both sides of the changed boundary, generated output where relevant, and the workflows that consume it | -| Installer, packaging or test tooling | The owning script tests and the affected installation/package workflow | -| Performance-sensitive implementation | The affected native benchmarks, once per source; larger fixtures when traversal or scale behavior changes | - -Use runner selection (`go test PACKAGE -run PATTERN`, Vitest file filters and explicit script -tests) rather than creating another test framework. After fixing a failure, rerun its reproducer -and affected suite/callers. Do not restart unrelated passed suites to get another all-green run. -A failed or interrupted full run remains incomplete; record the completed checks separately. - -For slow verification, report the whole workflow's elapsed time, including setup, transfers, -queueing, retries and diagnosis, separately from test execution. Use existing runner/CI timings -to identify the costly stages; individual step timings do not explain the whole wait. - -Finish review and fixes before the final full source and package acceptance. The candidate CI -runs `make release-check` once before its platform matrix; a duplicate local full preflight is -not required. Use `make check` for an intentional full integration check, not after every edit. -If a final check finds a defect, verify the fix locally at its affected boundaries before -starting the replacement candidate. Changes to shared contracts or infrastructure may justify -wider checks; record the reason instead of automatically repeating everything. - -Retain passing evidence for unaffected components when their implementation, relevant -dependencies, configuration, fixtures, harness and environment remain unchanged. Record the -old and final source identities and the reviewed diff that establishes this. New archives still -need their own content, identity and checksum checks. Rebuilt binaries require relevant package -smoke tests; changed behavior requires its package workflows. Acceptance of earlier bytes alone -cannot certify a new archive, but a new archive does not require repeating unrelated benchmarks -or every exploratory/manual check. - -Hardware verification may use locally built binaries after the affected source checks pass; -GitHub availability and completion of the full platform matrix are not prerequisites. Build -from the reviewed commit with the repository's existing scripts and record the source, -toolchain, build flags, binary checksums and runtime adapters. Use the release package when -installation or package layout is part of the check. Follow the -[physical Tape execution procedure](physical-tape-e2e.md#automated-stages) for host context, -absolute LTFS adapter paths, the scoped two-partition file-backend precheck and durable staged -execution. Hardware work can start independently of CI's final release-byte acceptance; -that acceptance remains required for release delivery. Source CI and other platform builds can -run in parallel. Keep the final artifacts' identity, content and affected package smoke checks; -reuse expensive hardware results, including a full Tape write, when comparison confirms that -the relevant source, dependencies, build configuration and runtime inputs are unchanged. -If those inputs differ or cannot be established, repeat the affected checks and explain why. - -Reuse downloaded candidate files for subsequent checks when their run, commit and checksums -match. Transfer only the packages and fixtures needed by the selected host and cases. A new -report or another case selection does not require another download. Complete-set validation -belongs to candidate CI; use its run and artifact records for checks already completed there. - -## Repository Layout and Entrypoints - -File ownership and placement rules are maintained in [Repository Layout](../../AGENTS.md#repository-layout). Follow those rules when adding or moving files; this section owns the development commands and output overrides. - -The root [Makefile](../../Makefile) is the common development/CI entrypoint. Its default target displays help; implementation stays with the owning project or script. Frontend compilation remains in package.json. Use `make build`, `make backend`, `make backend-linux`, `make frontend`, `make release`, `make preview`, `make generate`, or `make demo` / `make demo-dev`. `make check` runs documentation and IDL naming checks, Go unit tests/vet, compilation and vet of the E2E harness, frontend checks, tool-script tests and shell syntax checks; `make test-race` and `make test-e2e` are explicit additional checks. Set `YATM_E2E_BIN_DIR` to the extracted candidate directory for `make test-e2e`; the [E2E guide](e2e-test.md) also documents source-based harness runs. Native Preview module validation remains separate from the root Go test suite. - -Scripts locate the checkout independently of the caller's working directory. `RELEASE_DIRECTORY` overrides the final archive directory; relative values resolve from the caller's working directory before scripts change directories. Build tools are source files, not build output. The root `install-release.sh` remains the user installation entrypoint, and installation package paths are independent of source layout. - -Internal packages are not an external Go API: former root implementation imports now use `github.com/samuelncui/yatm/internal/...`, without forwarding packages. The `entity` import path, public gRPC contracts, legacy wire encodings and storage formats are unchanged by this organization. - -## Remote Candidate Acceptance - -On the explicitly selected isolated acceptance host, package acceptance uses the corresponding checksum-verified release package, built locally or by CI. Earlier hardware diagnosis may use locally compiled application binaries under [Check Scope and Reuse](#check-scope-and-reuse). The maintained controller can invoke the server/CLI and installer through SSH or run on the authorized test host with `--host local`. For host-local execution, transfer the controller and its package validators alongside the reviewed archives, checksums, runtime adapters, fixture data and operating instructions. Application execution still uses packaged programs; do not deploy the application checkout, separately compiled test binaries such as `e2e.test`, or an in-process test server. Existing systemd, LTFS/FUSE and other host dependencies remain environmental prerequisites. - -Choose an existing, spacious, xattr-capable mounted filesystem and create a uniquely scoped test directory beneath it. Place packages, installation copies, backups, Restore outputs and a private `TMPDIR` inside that root; export `TMPDIR` for remote installer and migration invocations as well as the controller. Check available space before extraction and each capacity fixture; the test owns only that directory and its explicitly registered test resources. A temporary pathname does not imply a disposable filesystem or sufficient capacity. - -Local development and CI may compile and run Go test harnesses. Those results are separate from remote packaged-binary acceptance. If a fault-injection scenario requires private functions or a custom in-process server, run it locally/CI and record that limit; do not describe it as remote candidate acceptance. The [E2E guide](e2e-test.md#remote-packaged-binary-acceptance) owns the remote procedure. - -For official LTFS file-backend acceptance, explicitly configure the optional `templates/testing/ltfs-file-backend` adapters from the package. They operate only on scoped virtual-cartridge directories and are not default installation scripts; their README describes identity and capacity fixtures. Physical Tape remains a separate acceptance boundary. - -## Local Checks - -Select checks using [Check Scope and Reuse](#check-scope-and-reuse). For example, a legacy -migration repair checks its implementation and installer caller: - -```shell -go test ./internal/migrate/legacy ./cmd/migrate -go vet ./internal/migrate/legacy ./cmd/migrate -go test -race ./internal/migrate/legacy ./cmd/migrate -CGO_ENABLED=0 go test ./internal/migrate/legacy ./cmd/migrate -git diff --check -``` - -The coverage below is an inventory for choosing affected checks, not a requirement to run -every listed suite after each change. The [Release SOP](#release-sop) owns full verification. - -For content-model changes, cover opaque nullable signatures, unique File/version pairs, independent same-content Files, one original per File, repeated-content version reuse, shared-copy lookup, cross-batch import rollback, and evidence-based legacy migration with source preservation. These checks do not repair or maintain compatibility with old Draft databases. - -Restore association checks include existing original associations, ignored outputs, multi-version candidate freezing, exact-content adoption, namespace collisions, Media finalization failure and explicit Library/Job publication failure. Scan verification checks include real uncached reads, per-item versus device errors, absent baselines and content-guarded health publication. Run both SQLite drivers and race checks for `./internal/executor/restore`, `./internal/executor/scan` and `./internal/library`; imported restored Files and original associations must remain intact, and imported bad-copy observations must not become healthy or unchecked accidentally. - -For Files and Location changes, inspect callers in `./internal/executor/scan`, `./internal/executor/observation`, `./internal/executor/fileops`, `./internal/treeops`, `./internal/ignore`, `./internal/demo`, and `./cmd/yatm-cli`; run the affected unit/race checks and select the affected [live-file, Volume, or Preview CLI E2E](e2e-test.md). Cover optional File identity, pure browsing versus explicit collection, complete directory List, paged Search and changefeed departures, Ignore versus authorization, guarded references, physical-operation results without persistent receipts and ordered continuity where changed. Shared organization changes run the contract against its affected Library, filesystem and object-storage-semantic adapters: directory merges, retained target associations, duplicate selections, conflicts, cancellation, bounded traversal and partial publication. Do not add user Copy acceptance to the supported move/mkdir/delete contract. - -Scan tests exercise one pipeline with known-only, fill-missing and force-read policies, optional comparison/Preview and each publication policy. Known-only cache misses must not hash; verification must neither trust cache nor replace its saved baseline. Preview failures must preserve otherwise valid observations, while source-read and physical-identity failures prevent publication of the current scope; a new Scan reads its complete selection. `analyze`, `preview create` and `verify` CLI presets must resolve to this same Scan service, not separate runners. - -Capacity checks cross multiple filesystem and manifest pages, cover the 256-level traversal boundary, reject an over-depth tree, enforce the 1,000-row public result window and cancel while a recursive directory stack is open. On platforms exposing `/dev/fd`, the cancellation test records descriptor counts before traversal and after unwinding and requires no retained descriptor. The test suite does not claim a platform-independent peak RSS or fixed Job-database size; any such release evidence must be measured against the exact packaged candidate and stated with its fixture, filesystem and SQLite mode. - -Canonical CLI subprocess tests cover minimal/projected `ls`, `files get/metadata`, `mv/rm/mkdir`, independent `preview get`, typed Archive/Restore estimates and `scan create/run/results`; protocol tests cover Preview URLs and reject full-file byte routes. Read tests assert request counts, SQL cost, omitted computation and absence of collection. The [E2E coverage matrix](e2e-test.md) records complete server/CLI workflows; subprocess transport tests alone do not establish full business acceptance. Ignore tests compare against Git in an isolated temporary repository when Git is available; production matching does not invoke Git. Check the SQLite paths with both the default CGO driver and `CGO_ENABLED=0 go test ./internal/library ./entity ./internal/executor/... ./internal/apis ./internal/migrate/legacy ./internal/demo ./internal/preview/...`. Regenerate IDL with `bash dev/generate.sh` and verify a second generation produces no changes. - -Replacing a workflow starts from an inventory of the baseline implementation's UI/CLI actions, results and edge cases; each maps to its replacement and regression coverage or to an explicitly approved removal. Acceptance verifies end-to-end behavior, not only the new interface shape, and stale mocks are updated with their real consumers; passing tests without capability coverage do not prove feature parity. - -For frontend changes, run the scripts in [package.json](../../frontend/package.json), regenerate protobuf Go/TypeScript outputs when IDL changes, and keep the lockfile reproducible. Vitest owns tests under `frontend/src`; Node tooling tests under `frontend/scripts` run through their explicit `node --test` command in the frontend checks. A test belongs to one runner. Use active-LTS Node and supported stable frontend dependencies, and keep the initial bundle bounded. Check shell syntax for changed scripts and run builds/generation checks when those paths change. Documentation changes run `node dev/check-documents.mjs`; the check uses Node and Git only. - -## Performance Acceptance - -Use fixed, representative benchmarks to record current interface performance and detect -regressions. Compare a repair with its preserved pre-repair source, a pull request with its exact -base, and a release with its recorded accepted source baseline. Do not silently replace that -baseline or require comparison with the first Alpha release. - -Run each standard Go benchmark **once per source**, with `-count=1 -benchmem`. Go calibrates the -iterations within that run; there is no minimum number of separate samples or confidence-interval -gate. Use the same reviewed harness, fixture sizes, Go patch version, SQLite mode, CPU setting -and filesystem for both sources. Measure baseline and candidate serially, with other builds, -tests and agents stopped during timing. Record source identities, commands and raw results; -a skipped case, missing metric or incomplete run is not evidence. - -The [suite manifest](../../dev/performance/suites.json) selects the appropriate scope: - -| Tier | Coverage | Use | -| --- | --- | --- | -| `fast` | 10,000-entry List, Ignore, Search and Scan benchmarks | Related pull requests and local changes | -| `critical` | The same workloads with 100,000 entries | Traversal, SQL, projection and buffering changes | -| `release` | Critical benchmarks plus candidate-only 1,000,000-entry correctness checks | Release acceptance | -| `migration` | 10,000 Files and 20,000 physical copies on disk SQLite | Legacy archived-content reconciliation changes | - -For List/Scan choose one of `fast`, `critical` or `release`; do not repeat the smaller tiers inside -a release run. Use `migration` separately when its reconciliation changes. Large-data correctness -checks remain separate from timing and run once. Frontend changes also use the relevant list -interaction and virtualization tests described in [Local Checks](#local-checks), and the -[frontend benchmark](../../frontend/README.md#frontend-benchmarks) uses Vitest's native comparison -with one invocation per source. Browser tests cover actual rendering and interaction. Shared loading or virtual-list changes -include a cold route, deliberately delayed code/data responses, an error and explicit Retry, -and forward/reverse/distant scrolling with long rows. Verify request counts, selected menus -and real scroll geometry: a mocked virtualizer cannot establish those layout properties. - -List/Search fixtures associate half the entries with catalog Files and request attributes, -status, operations and navigation. They cover random/reverse names, 16 directory levels, -102 Ignore rules with exclusion and negation, and the first and next 100-row Search pages. -List records first-batch and complete-response latency with a receiver retaining all rows; -Search records each page's response latency. Both report allocations. Setup is outside the timer -and Go's benchmark loop reuses the fixture during calibration. The million-entry List check -verifies completeness, ordering and bounded batches with a receiver retaining only counters and -the previous name. Go `B/op` excludes native SQLite memory; these checks do not claim a peak RSS bound. - -Scan covers one selection across one, three and nine Locations, a small physical selection in a -large recorded Location, and renamed originals across three Locations. Each case reports Create, -completion, first-result-page and total latency, plus allocations. Scan intentionally uses -`-benchtime=1x -count=1`: it modifies catalog state, so a second operation on that same catalog -would measure a different workload. Catalog setup and Job deletion remain outside the timer. -The million-entry Scan check verifies completion and all result pages once on the candidate. - -Result browsing uses `go test ./internal/executor/scan -run '^$' -bench '^BenchmarkScanListEntries$' --benchmem -count=1`: a retained 100,000-entry manifest, first-page counts, forward/reverse -cursors and far offsets, with warm or reopened runners and Preview presence checks. Fixture -creation is outside timing. This isolates the public handler from Scan execution, RPC transport -and browser rendering. Keep its small fixture correctness check in the regular Go suite. - -Legacy migration changes use the same collector with `--tier migration`; it runs -`BenchmarkPrepareArchivedVersions` once on each source with `-benchtime=1x`, records identities -and feeds the same comparison command. Each operation consumes fresh on-disk staging tables, -including duplicate copies across pages; setup and result-count checks remain outside the timer. -This focused check measures archived-content reconciliation, while complete migration acceptance -uses the packaged installer and a copied legacy backup through the -[package controller](e2e-test.md#remote-packaged-binary-acceptance). - -Use repeatable `--package ./internal/apis` or `--package ./internal/executor/scan` to compare -only that package's suites within the selected tier. Package names must match the manifest; -structural checks and common harness files follow the selected packages. `pair.json` records -requested, selected, executed and skipped packages and structural scope. A completed subset is -not a full release comparison. Baseline and candidate use independent mutable temporary fixtures. -The manifest and shared harness are captured once and applied identically to both sources. - -Fixture-only changes can measure preparation separately with -`go test ./internal/apis -run '^$' -bench '^BenchmarkFilesFixtureSetup$' -benchtime=1x -benchmem -count=1`. -It measures fresh file creation and catalog seeding at 10,000 and 100,000 entries; deferred cleanup -is outside the timer. It is not a List/Search runtime comparison. Reuse their existing measurements -when their implementation and equivalent input distribution remain unchanged. - -The collector snapshots the supplied sources and installs the same reviewed harness in both. -It records commits and actual source hashes, including dirty changes, then compiles both suites -before measuring. Each suite executes its standard Go test binary once per source. This is the -same benchmark mechanism as `go test -run '^$' -bench PATTERN -benchmem -benchtime=1s -count=1`; -precompilation keeps compilation out of the measurement window. It adds no custom sampling loop. - -```shell -node --test dev/check-performance.test.mjs dev/performance/*.test.mjs - -PERF_OUTPUT="$(mktemp -d)/pair" -node dev/performance/collect.mjs \ - --baseline "$PERF_BASELINE" --candidate "$PERF_CANDIDATE" \ - --harness "$PERF_HARNESS" --out "$PERF_OUTPUT" \ - --tier fast --environment "$PERF_ENVIRONMENT" --idle -node dev/check-performance.mjs "$PERF_OUTPUT" -``` - -Set the checkout paths and environment label explicitly. Output must be a new directory outside -all source checkouts. Defaults are `--cpu 2`, `--benchtime 1s` and `--cgo 1`; use `--cgo 0` for -pure-Go SQLite. The harness's `go.mod` selects the exact Go patch toolchain. Only ordinary source -files are supported; symlinks/submodules and incompatible harnesses fail instead of silently -changing the inputs. Sources are measured on the same host, not compared with unrelated old -machine timings. Ambient Go configuration and workspace/module edits are disabled. - -The [comparison](../../dev/check-performance.mjs) reports each measured time/allocation value and -percentage change. An increase above 10% flags the case for review; it does not prove a code -regression or fail a completed comparison. CI highlights these flags as warnings. Missing, invalid -or failed measurements still fail the command. Review the flagged implementation and measurement -conditions before accepting the change; if necessary, rerun only that case on both sources. Keep -the initial result and explain the finding. Confirmed regressions must be fixed; unexplained flags -remain open acceptance work. Do not repeat the whole suite until it passes or automatically replace -the baseline. The comparison reports measurements, not release approval. - -Keep `pair.json`, raw `.bench` files, structural logs and `comparison.json` as verification -artifacts outside maintained documentation. An interrupted collection remains incomplete. - -The [Performance workflow](../../.github/workflows/performance.yml) measures a PR's exact head -and event base with the base revision's reviewed harness. Manual critical/release runs use the -requested full candidate commit and `PERFORMANCE_ACCEPTED_BASELINE`, an explicitly accepted full -commit SHA. Both sources run once on that runner. The workflow is read-only, retains evidence on -failure, and never updates the baseline or publishes a release. Missing tools/harness require a -reviewed change; a missing or invalid baseline fails the hosted comparison. - -Release preparation requires one completed, reviewed `release` comparison of the final source -against the recorded accepted source baseline, collected locally or through the hosted workflow. -Both sides must use the same host, tools and reviewed harness under the conditions above. Evidence -must identify the exact baseline, candidate and harness commits and content hashes, including any -dirty changes, with the commands, raw results and flag resolutions. Complete the structural checks -and relevant frontend checks as well. Resolve every flag before candidate or source-push approval; -an incomplete comparison or unexplained flag remains open work. A completed local comparison -satisfies this requirement without uploading its baseline or repeating it on a hosted runner. -Candidate assembly runs its source and package gates without automatically invoking Performance. - -After measurement, review the diff against each measured operation. Retain its evidence when -the relevant runtime source, dependencies, harness, fixtures and measurement settings are -unchanged, recording the measured and final commits. Documentation or unrelated runtime/tooling -changes do not require repeating that operation. Rerun only affected benchmarks and resolve their -flags; do not repeat the whole release tier for a change to one component. The final source -preflight and acceptance of the exact release archives remain required. - -## Large Identical Results - -Use the native benchmark for before/after Find and retained-page costs with the same 10,000- -and 100,000-File fixtures. Fixture creation is outside the measured operations: - -```shell -go test ./internal/library -run '^$' -bench '^BenchmarkIdenticalRetainedResult$' -benchmem -count=1 -``` - -The opt-in performance fixture builds isolated 10,000- and 100,000-File catalogs with many pairs, -a large group, a long transitive chain, saved versions, hidden names and unknown sizes. It measures -one complete Find and retained 200-row reads, then checks every sort direction, random offsets and -File ID position lookups against an independent result. Page checks verify that catalog collection -and component construction do not run again. - -```shell -YATM_IDENTICAL_PERF=1 go test -count=1 ./internal/library \ - -run '^TestIdenticalLargeCatalogPerformance$' -v -timeout 30m -CGO_ENABLED=0 YATM_IDENTICAL_PERF=1 go test -count=1 ./internal/library \ - -run '^TestIdenticalLargeCatalogPerformance$' -v -timeout 30m -``` - -These measurements cover query staging and retained SQL reads, not API member observation or -browser rendering. Go allocation measurements exclude fixture construction and native SQLite -memory; use an external process measurement when peak memory matters. The ordinary unit suite -skips this fixture, and the default Demo remains small enough for interactive review. - -## Local Demo - -`make demo-dev` prepares the same Demo fixture and builds the backend incrementally, then serves -Vite hot reload at `http://localhost:5173`, proxying the selected backend listen address. -`YATM_DEMO_FRONTEND_PORT` overrides the strict frontend port. A port conflict fails startup; -both children stop on signals or either child's exit. Fixture reset remains explicit through -`YATM_DEMO_RESET=1`. `make demo` retains the production frontend build for packaged-browser checks. - -`make demo` builds and serves the disposable review fixture at `http://127.0.0.1:18080`, normally -under the system temporary directory. Restarting preserves reviewer changes. Select an isolated -temporary root and reset it explicitly after fixture or pre-stable format changes. Stop its server -before resetting that root; concurrent reviews use distinct roots: - -```shell -YATM_DEMO_ROOT=/private/tmp/yatm-demo-review \ -YATM_DEMO_LISTEN=127.0.0.1:18090 \ -YATM_DEMO_RESET=1 make demo -``` - -Reset replaces only a validated Demo root whose basename begins with `yatm-demo` and which is below -a temporary directory. Never place valuable files there. The fixture uses normal Library tables, -Job runners, Volume markers and ACP transfers; it has no production mock branch and must never be -used for a physical Tape operation. - -The default fixture keeps five Locations and thirteen Jobs. Setup retains only Jobs needed by current -Location links or distinct review results; saved versions and Preview assets remain. A complete -Documents Scan supplies its LastSync link while later partial Scans supply current results. -Archive preparation collects Incoming's sources directly, and one Scan publishes their content -observations and generates Previews. The retained examples cover these review paths: - -| Fixture | Review coverage | -| --- | --- | -| Library | Annotations, mixed-field search, 101 invoices spanning two 100-result pages, logical organization and Trash, saved and unknown legacy archive dates | -| Documents | Original-only, unknown and covered content under the `archive-review` tag, unarchived edits, three text/image saved versions and date cutoffs, Ignore and dot directories, unadmitted arrivals, empty folders, failed Scan with a reconnected source | -| Shared files | One 201-member group with three dot files, three additional pairs, three/four-member groups, arbitrary scrolling, collapse, hidden rows and independent Delete; a.txt/b.txt connect transitively to unavailable c.txt with different current sizes | -| Unavailable originals | Retained originals under an inaccessible root, cross-Location duplicates and saved-version evidence | -| Incoming | Four Archive sources across two directories, a pending Archive and a completed Scan with image/video Previews | -| Restored files | Preferred Restore destination, pending multi-file Restore and completed adoption of identical existing output | -| Media | Random-write HDD, sequential-write HM-SMR, offline Volume, uninitialized disk and metadata-only Tape probe; inventory additions/changes/removals, healthy/mismatched/missing Verify findings and a Tape-bound pending check | - -For large browser reviews, opt in with `YATM_DEMO_IDENTICAL_FILES`, forwarded to `cmd/demo` as -`-identical-files`. The integer is the **total number of members across Shared files' Large group -and Many groups together**, including the default 207 members; it excludes the three/four-member -groups, transitive-history example and other Demo files. Values below 207 are rejected before -reset or writes. Any explicit count, including 207, requires reset when reusing an existing root; -omit the option on later restarts to preserve the fixture and reviewer changes. - -```shell -YATM_DEMO_ROOT=/private/tmp/yatm-demo-identical-10k \ -YATM_DEMO_IDENTICAL_FILES=10000 YATM_DEMO_RESET=1 make demo - -YATM_DEMO_ROOT=/private/tmp/yatm-demo-identical-100k \ -YATM_DEMO_IDENTICAL_FILES=100000 YATM_DEMO_RESET=1 make demo -``` - -Run one server at a time or set distinct listen addresses. At 10,000 members there is one -5,000-member group and 2,500 pairs; at 100,000 there are 50,000 members in that group and 25,000 -pairs. Smaller totals retain at least the original 201-member group, with rounding assigned to -that group so every other group is a complete pair. Every eightieth member of the large group -has a dot name. Existing feature examples and the five Locations/thirteen Jobs remain intact. -Setup writes added files in bounded batches, then uses one normal complete Scan of Shared files -to publish signatures and replace its setup Job. Cancellation stops pending batches and drains -active Jobs before closing the catalog. Large setup duration and disk use depend on the requested -count and host; it has no fixed 20-second Scan deadline or maximum fixture count. - -For each size, use explicit Find with Library and with Locations selecting Shared files. Review -far scrolling within the large group and across the pairs, collapse/expand, size sorting in -both directions under the [Library sorting contract](../architecture/library.md#identical-file-tools), -hidden files, and independent Delete/Keep in the disposable Location. Repeat Find -after mutations. The small saved-history example still exercises transitive Library grouping. - -Use an explicit Find for either identical source; Locations uses current records rather than -saved-version links. `Photos/preview-video.mp4` is a licensed, silent five-minute Big Buck Bunny -excerpt at 640×360 with an actual native-worker poster, thirty distinct 320×180 timeline frames -in a three-row sprite and VTT spanning 00:00–05:00. The timeline uses the production defaults of -320×180 tiles, a 10-second interval and at most 30 frames. These bundled -assets pass through normal content-addressed Preview publication and require no installed helper. -Its File Note displays the credit and license; [asset attribution and reproduction](../../licenses/demo-video.LICENSE) -records the exact source, tools, commands and checksums. Bundled assets do not establish acceptance -of a current native-helper release. - -`YATM_DEMO_VIDEO=/path/to/clip.mp4 YATM_DEMO_RESET=1 make demo` preserves the optional MP4 override. -It generates derivatives with `yatm-preview` on `PATH` or beside the executable and requires that -native helper. To reproduce the bundled output and check override creation and reuse: - -```shell -YATM_TEST_PREVIEW_HELPER=/path/to/yatm-preview go test -count=1 ./internal/demo \ - -run 'TestBundledVideoAssetsReproduceWithNativeWorker|TestPrepareVideoOverrideUsesNativeWorker' -``` - -Frontend-visible changes update `internal/demo` and its semantic tests. Keep fixtures deterministic, -bounded and representative of actionable, unavailable, empty and error states. Verify with -`go test ./internal/demo`, `bash -n dev/demo.sh`, an explicit reset and a browser smoke test. Demo -review supplements semantic and E2E tests; the exact visible behavior belongs to the UI architecture -and automated tests rather than a fixture transcript. - -## Release SOP - -Run this procedure for every release. The release workflow enforces the automatic gates below; -source review and physical acceptance retain their own explicit decisions. - -1. **Review scope and repository content.** Compare the public baseline with the final source by - Files/Locations, Identical, Jobs, Preview, installation/migration and API/CLI. Map retained - capabilities to their consumers and meaningful tests; resolve unclear removals before freezing - source. Check Demo coverage, including large results and real licensed media where relevant. - Review terminology and affected operator/architecture instructions against actual commands and - behavior. Update the existing owner, remove superseded claims and close verified issue records. - Check the [Tape script compatibility contract](../architecture/media-io.md#tape-script-compatibility) - against the published adapters: run unchanged historical scripts through the current caller, - and verify that upgrade preserves customized scripts/helpers. For an explicitly approved - incompatibility, check the actionable installer notice and the adapted script boundary. - Cover the [Tape lifecycle](../architecture/media-io.md#tape-lifecycle), including empty electronic - barcode FORMAT, subsequent identity checks, APPEND and Restore. - Keep these checks in the automated suite; a new-template hardware pass alone does not cover upgrades. - Inspect modules, tools and temporary files for a current runtime, generator, test or packaging - consumer before deleting them. Review ignored local directories separately; CI cannot inspect - workstation leftovers. Accepted limitations remain documented and do not become release blockers. -2. **Audit the history and squash unpublished development.** Scan the development range before - squashing, including intermediate source and commit messages. Review credentials, internal - services, personal paths, real data, configuration and private notes manually as well as with - the pinned scanner. Rotate an exposed real credential before attempting public-history cleanup; - history rewriting needs separate authorization. Recheck the public branch tip, retain a local - backup and squash only unpublished commits into one release commit whose parent is that tip. - Do not publish the backup branch or rewrite already public history. Review the final diff and - commit message again; squashing does not sanitize final content. Finish required dependency - releases and update their pins before freezing YATM. -3. **Complete review and fixes with affected checks.** Review the integrated source and finish - focused fixes/verification until no unresolved, unaccepted finding remains. Use - [Check Scope and Reuse](#check-scope-and-reuse); do not run a full preflight per fix. - Complete the [performance acceptance](#performance-acceptance) before source-push approval, - retaining unaffected measurements and checking only changed operations again. - Start affected hardware verification from a reviewed local build as soon as its source and - nonphysical checks pass, following [Check Scope and Reuse](#check-scope-and-reuse). Do not - wait for GitHub or the full release matrix to investigate or verify a hardware fix. - For workflow edits, validate GitHub syntax and expression contexts before pushing with - `go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 -shellcheck= -pyflakes=`; - script diagnostics remain covered by the existing shell checks. -4. **Run the final source preflight on the clean release commit.** The candidate workflow owns - the final full run before its platform matrix. A local full run is available for an intentional - integration check or diagnosis, but is not a prerequisite that CI must then repeat. - Put Bash 4.4 or newer on `PATH` for - the Linux installer unit tests; macOS's bundled Bash 3.2 is insufficient. Install the frozen frontend dependencies, - pinned scanner and the protobuf tools used by current generated source: protoc 26.0, - protoc-gen-go 1.33.0 and protoc-gen-go-grpc 1.3.0. Preview acceptance also requires a separate - FFmpeg fixture generator with libx264, 10-bit libx265, PNG, lavfi and display-rotation support. - Native release CI also prepares the checksum-pinned CC0 camera samples from the - [worker fixture inventory](../../previewworker/testdata/raw-fixtures.json), so real-camera - acceptance runs for every advertised RAW family instead of skipping for missing local data. - `node build/release/check-preview-fixtures.mjs` generates tiny real inputs and fails on missing - tools or features. Linux CI builds the pinned generator with - `bash build/release/build-test-ffmpeg.sh OUTPUT_DIRECTORY` after installing pkg-config, - libx264-dev, libx265-dev and zlib1g-dev, and adds its `bin` directory to `PATH`. This is a test - dependency; installed YATM and its packaged helper do not need it. Use a complete Git checkout and run: - - ```shell - RELEASE_VERSION=v1.0.0-alpha.2 RELEASE_COMMIT="$(git rev-parse HEAD)" \ - GITLEAKS_BIN=/path/to/scanner/gitleaks make release-check - ``` - - The preflight rejects dirty or mismatched inputs, incomplete history, unowned source paths, - runtime databases/logs, local configuration, compiled output, archives and scratch files. - It scans current source and all history reachable from HEAD, then runs documentation/IDL, - Go/frontend/build-tool checks, full race checks, both SQLite paths and two clean generations. - Native Preview race tests and vet run with its private libraries during each runnable helper build. - Its corresponding-source archive has a separate inventory that retains required vendor/native - sources. Filename gates do not prove that a module is used or documentation is semantically correct. -5. **Build and accept one candidate set.** GitHub Actions requires the exact reviewed commit to - exist in this repository's remote. Obtain explicit authorization for that source push before - dispatch; this does not authorize a tag, Release or asset upload. Dispatch the Release candidate - workflow with `publish=false`. Its platform matrix waits for `make release-check`. Candidate CI - validates the complete [package set](#release-backend-builds), public archive content, packaged documents, licenses, - identities and checksums. Check the exact extracted bytes at affected package boundaries - for installation/upgrade, supported `v0.1.x` migration, Archive, Restore, Scan, Volume/LTFS - and native Preview. Apply [Check Scope and Reuse](#check-scope-and-reuse) to retain valid - earlier behavior and hardware evidence instead of repeating every workflow after a rebuild. - Use the [local package controller](e2e-test.md#remote-packaged-binary-acceptance) with the exact - Linux main/Preview archives, `--ltfs` and a new private evidence directory for its named workflows. - Its report does not replace separate browser, legacy-backup or physical-media acceptance. - Complete the [Media and migration gates](#media-and-migration-safety), including separately - authorized scratch Tape acceptance. A skipped or unavailable required check remains open. -6. **Approve and publish the accepted bytes.** Review the final public source, commit message, - Release text and all assets. Obtain publication approval before tag or Release creation and - asset upload; any further source push also needs explicit authorization. - Follow the existing [upload procedure](#release-backend-builds) with the successful candidate - run ID; it rechecks provenance, content and checksums without rebuilding. Verify public asset - checksums against the accepted files after upload. -7. **Record evidence and clean owned resources.** Report passes, failures, accepted limitations and - unrun checks accurately. Keep raw verification output in CI artifacts and sensitive evidence - private. Remove only temporary processes, directories and test resources owned by that run. - -## Release Backend Builds - -Linux backend builds require Zig **0.15.2** and always enable CGO, selecting the existing mattn C SQLite driver. `build/backend/build.sh` preserves version/commit metadata and uses external static musl linking. `build/backend/cross-linux.sh` defaults to amd64; set `GOARCH` and, for ARM, `GOARM` explicitly. Non-Linux release builds retain their existing `CGO_ENABLED=0` strategy. - -| Go target | Zig target | CPU selection | -| --- | --- | --- | -| linux/amd64 | x86_64-linux-musl | default | -| linux/386 | x86-linux-musl | default | -| linux/arm64 | aarch64-linux-musl | default | -| linux/arm, GOARM=5 | unsupported | Zig 0.15.2 lacks the ARMv5 `__sync` atomics required by Go CGO | -| linux/arm, GOARM=6 | arm-linux-musleabihf | arm1176jzf_s | -| linux/arm, GOARM=7 | arm-linux-musleabihf | cortex_a7 | -| linux/s390x | s390x-linux-musl | default | - -ARMv5 and other unmapped Linux targets or ARM values fail before compilation. ARMv5 remains an explicitly allowed failure in the CI matrix and produces no candidate archive. There is no automatic pure-Go fallback. Every built Linux program passes `build/release/check-static-linux.mjs`: ELF program headers must have no interpreter or dynamic segment, Go build metadata must enable CGO, and no native Preview module may enter the main dependency graph. This excludes shared libc/SQLite/FFmpeg dependencies. Runtime acceptance remains Linux amd64; other mapped release targets are experimental until exercised on matching hardware. - -Distributed main binaries omit Go debug data and strip native debug data at the Linux external -link, so compiler and cache paths do not become public package content. Package scanning checks -the final executable strings as well as text files; trimming Go source paths alone is insufficient. - -```shell -GOARCH=amd64 ./build/backend/cross-linux.sh -node build/release/check-static-linux.mjs output/yatm-httpd -``` - -The [optional native Preview helper](../../previewworker/build/README.md) has its own module, build, platform baseline and private native libraries. Its native dependencies are never linked into the main binaries. - -The source-check job builds the frontend once and collects its locked production dependency -notices. The internal `shared-frontend` artifact records version, commit, lockfile hash and every -file checksum. Each main-platform build validates its identity and exact file inventory, then -copies the same frontend and Node notices and adds its own Go/native notices. Source maps, private -configuration, missing files and extra files are rejected. The internal manifest is never copied -into a release archive. Local `make release` calls the same collector; set -`FRONTEND_ARTIFACT_DIRECTORY` to reuse a previously collected artifact with matching inputs. -Preview native-object caching follows the [native build guide](../../previewworker/build/README.md#native-compiler-cache). - -Release validation requires exactly nine main archives, three optional helper archives and three matching source archives. Each main archive still passes `build/release/check-release.mjs`; `build/release/check-candidate-set.mjs` additionally verifies the complete platform set, matching identities, helper file boundaries and corresponding-source checksums. Linux E2E receives the extracted helper through `YATM_TEST_PREVIEW_HELPER` so native Preview acceptance cannot silently skip. - -Release builds require clean committed source and a matching `RELEASE_COMMIT`. Inputs are exported -from that commit, so ignored workstation files cannot enter recursive copies. The platform matrix -starts only after the source checks, full race suite and both SQLite paths pass. Frontend tests use -two workers to bound local and CI resource use. - -Public content checks use checksum-pinned Gitleaks 8.30.1 with its default credential rules and -additional personal-path/private-service checks. The installer also pins the default rule source; -scans retain its credential rules while removing global dependency/generated-file exclusions. -Install it with -`bash build/release/install-gitleaks.sh /path/to/scanner`, set `GITLEAKS_BIN` to that directory's -`gitleaks`, and run `node build/release/check-content.mjs source . HEAD`. A different revision range -can inspect unpublished development before squashing. `node build/release/check-content.mjs artifacts /path/to/candidates` checks -nested archives and embedded executable/library strings, normalizing npm `.tgz` names in a private -scan copy so the scanner opens them. Source maps and dependency sources are -included; the only project exception is the exact deterministic physical-Tape fixture seed at its -owned path. A repository's `.gitleaksignore` may contain explicitly approved historical fingerprints -in full `commit:file:rule:line` form. Only the Git history scan reads them; current source and package -scans always use an empty ignore file. Commit-free, wildcard and path/rule-wide exceptions are -rejected. Additions require a separate decision about the specific already published finding; -new commits remain fully scanned. Checksum-validated dependency sources have rule-specific exceptions for public source -identifiers, deterministic test values and recorded upstream build/sample paths. Each is restricted -to its exact nested member and matched value; other credentials or paths in those members still fail. Failures -report rule counts without match snippets, and temporary scanner reports are -deleted instead of uploaded as public CI evidence. Scanner regression tests use the same executable. - -The **Release candidate** workflow runs only through `workflow_dispatch`. Set `version` and leave -`publish=false` to build the candidate set at the selected commit, validate every package and run the -full Linux E2E suite against extracted candidate binaries and the native helper. A successful run -retains the complete accepted set, companion checksums and `SHA256SUMS` in `accepted-candidates`. -Build completion establishes neither performance acceptance nor publication approval. -Creating a GitHub Release does not start a build or upload. - -For local or remote package acceptance, download only the selected platform's artifacts into -one directory identified by the candidate run. For example, Linux amd64 acceptance uses: - -```shell -gh run download "$CANDIDATE_RUN_ID" --repo samuelncui/yatm \ - --name candidate-linux-amd64 --name candidate-preview-linux-amd64 \ - --dir "$CANDIDATE_DIRECTORY" -``` - -Multiple selected artifacts are extracted into per-artifact subdirectories. The Preview artifact -includes its helper and matching corresponding-source archive. Verify the -downloaded checksums and expected commit, and retain these files for subsequent cases and physical -acceptance. Record the final successful run and artifact IDs; tests started before candidate CI -finishes remain provisional until that same run succeeds. Do not download the complete -`accepted-candidates` merely to repeat passed CI inventory, content or checksum checks locally. -The publication job retrieves that complete artifact on GitHub when it needs all release assets. - -To deliver those accepted bytes, explicitly create the matching Release and tag first, then dispatch -with the same `version`, `publish=true` and its successful build `candidate_run_id`. Publishing runs -no builds or E2E; it downloads that run's immutable artifact by ID. The run must belong to this -repository and the Release candidate workflow (by workflow ID and path), identify a manual build of -that version, and have completed successfully. Its accepted artifact must still be available and unexpired. The release -tag, including an annotated tag, must resolve to the run's exact commit, and the Release prerelease -flag must match the version. Candidate identity comes from run metadata and package contents rather -than the upload checkout. Release tools come from `v1`, pinned to the same commit for preflight and -upload; no code from the candidate artifact is executed during upload. - -Upload repeats provenance and public-content checks, validates the exact 9+3+3 set and every checksum -with the existing package validator, and sends the unchanged archives and checksum files to that -existing Release. -Only the upload job has `contents: write`; candidate lookup and download use scoped `actions: read`. -The workflow neither creates a Release or tag nor replaces existing assets. An expired artifact needs -a new build and acceptance run before publication. Mocked provenance and upload tests run through -`node --test build/release/publish-candidate.test.mjs` without publishing anything. - -## Media and Migration Safety - -- Run [mounted Volume and LTFS file-backend E2E](e2e-test.md) for affected copy/publication behavior. Use the official LTFS file backend on the isolated Linux acceptance host through packaged binaries and local orchestration; Volume tests use isolated already-mounted directories. -- Run [physical Tape E2E](physical-tape-e2e.md) only with explicitly assigned scratch media and isolated databases/configuration. A documentation update is not authorization to format, write, or test a cartridge. -- Test migration only on timestamped `cp -a` copies of explicitly approved source backups. Keep private source paths and detailed reports outside the public repository. Never modify source backups or production installations during validation. -- Compare every migrated Job item-by-item, including non-empty historical Jobs, not only catalog counts. Cover prepare, abort, repeat prepare, commit, and separately confirmed cleanup. -- Clean only validated temporary test media, mounts, databases, Job bundles, and migration copies after validation. Never use production paths as cleanup targets. -- Batch confirmed review fixes before expensive E2E rather than repeating the full suite after each comment. Record actual results and unrun checks accurately; do not claim planned tests passed. - -## Delivery Gates - -The candidate preserves [independent format identities](../architecture/persistence.md#published-data-formats) and legacy migration/import. Record acceptance for the exact candidate bytes, reusing unaffected source-level evidence under [Check Scope and Reuse](#check-scope-and-reuse); earlier development results alone do not certify a rebuilt release. Preserve a forward path for supported published data. Before the first stable v1 release, replace the [pre-stable compatibility policy](../README.md#pre-stable-compatibility) with the published stable API/upgrade policy. - -Branch, push and publication gates follow [Delivery and Authorization](../../AGENTS.md#delivery-and-authorization). Release packages include only the operator documents allowlisted in the [documentation convention](../README.md#release-packages); development and architecture documents remain in the repository checkout. diff --git a/docs/release/package-acceptance.md b/docs/release/package-acceptance.md new file mode 100644 index 0000000..b20ce43 --- /dev/null +++ b/docs/release/package-acceptance.md @@ -0,0 +1,171 @@ +# Package Acceptance + +Acceptance of exact release packages: remote packaged binaries, browser, installer and legacy upgrade. + +## Release Candidates and Upgrades + +Follow the [Release SOP](release.md#release-sop) for source preflight, candidate assembly, acceptance and publication. [Release Packages](release.md#release-packages) owns the required archive set, [Release Backend Builds](release.md#release-backend-builds) owns toolchains and the [candidate and publication procedure](release.md#candidate-and-publication) owns the accepted-candidate upload. Hosted CI skips opt-in LTFS and physical Tape tests. Remote release acceptance replays the public CLI workflows with the official LTFS file backend, using packaged application binaries. Physical Tape acceptance follows the affected source and nonphysical checks and may use a reviewed local build while CI proceeds; [Evidence Reuse](release.md#evidence-reuse) owns final-artifact checks and evidence reuse. + +For local/CI harness runs, set `YATM_E2E_BIN_DIR` to an extracted candidate and run the full suite without a `-run` filter. Set `YATM_E2E_LTFS=1` only in a Linux harness environment with the official file backend. Physical Tape uses the separate [package controller](physical-tape-runbook.md#automated-stages) and explicitly assigned scratch hardware. Do not move this harness or source tree to the remote package-acceptance host. Record every pass, failure and skipped test with its reason. Use a temporary filesystem supporting user xattrs; the full-capacity LTFS fixture needs at least 12 GiB of free space for its source, virtual Media and restored bytes. + +A missing or non-executable candidate fails explicitly. Package checks reject altered checksums, missing programs, mixed identities, active configuration, development dependencies and hidden filesystem metadata. Packaged guides retain local links; source references resolve at the matching public tag. + +### Remote Packaged-Binary Acceptance + +Use repeatable `--case NAME` to select exact existing case names. The controller automatically +runs their required cases in normal order and always allocates a fresh isolated environment. +Omit `--case` for the complete selected profile. The report's `case_scope` distinguishes requested +cases, prerequisites, executed cases and skipped cases with reasons; a passing subset does not +establish full acceptance. The legacy profile supports the same selection: upgrade is required +before historical repair or JSONL roundtrip, while review/decline/abort checks do not run those actions. + +Reports use monotonic elapsed times for preparation, transfer, installation, workflows and cleanup, +plus individual commands, cases and the whole run. Independent read-only host, tool and identity +checks run in batches of at most four; each retains its result and exit status even if another fails. +Nested or concurrent durations overlap and must not be added to claim total elapsed time. + +Run the maintained controller over SSH or directly on the explicitly selected test host with `--host local`. Verify and extract the reviewed release archive into an isolated attempt directory; invoke only its shipped YATM programs. Read their offline `--version` output and compare the commit/version and archive checksum before starting. For host-local execution, transfer the controller and its package validators with the artifacts and operating instructions under [Remote Candidate Acceptance](#remote-candidate-acceptance). Do not deploy an application checkout or separately built harness. + +The [package controller](../../e2e/package_acceptance.py) requires Python 3.9 or newer and Node.js for +the existing package validators; SSH mode also requires OpenSSH. The explicitly selected +Linux amd64 test session must permit an isolated systemd installation and provide the installer's +dependencies plus `timeout`, `ss`, `setfattr` and `getfattr`. It does not install host dependencies. + +```bash +python3 e2e/package_acceptance.py \ + --host isolated-test-host --test-parent /srv/yatm-tests \ + --archive "$MAIN_ARCHIVE" --preview-archive "$PREVIEW_ARCHIVE" \ + --version "$VERSION" --commit "$COMMIT" --out "$NEW_PRIVATE_REPORT_DIRECTORY" --ltfs +``` + +Checksum companions must be beside each archive. Preview validation also requires its matching +corresponding-source archive beside the controller's Preview package; it is validated without being +installed. The controller checks package identities before preparing the host, allocates a unique +root and service name, uses real systemd and the package installer, and records CLI exits, JSON and +output-byte assertions in its evidence directory. +Its workflows cover literal UTF-8 and invalid-byte filenames, exact signed file times, Scan's +all-source preparation barrier and overlapping multi-Location selections, read-only Job creation +inputs, explicit new submission, recorded/live Search, dry run and recoverable Delete, Volume +Archive/Restore/Scan, explicit zero/pre-epoch Restore cutoffs, JSONL export/import with rejected +timestamp shapes preserving existing data, and Verify findings +for healthy, damaged, missing and explicitly repaired copies, including result paging and health. +Volume inventory checks added/changed/removed files, a stat-preserving edit under force-read, +explicit and repeated inventory import, trim, metadata-only deletion, marker registration and +reconstruction through another Scan. +With a Preview archive, it also generates an image with a literal backslash in its name and the +licensed five-minute Demo video, checks HTTP asset bytes and thirty 320x180 timeline frames, and +compares missing-only preservation with explicit regeneration. `--ltfs` adds official file-backend +format/append, finalized-prefix/pending-suffix behavior at full capacity, explicit second-Media +selection, Restore and Verify. It requires the optional LTFS dependencies, `findmnt`, `truncate` +and 16 GiB of free space including the installation backups (4 GiB without LTFS). Its cartridge paths +are newly allocated beneath the attempt root; +physical device paths are not accepted. Omitting this flag records the missing acceptance explicitly. +SQLite acceptance enables WAL on the stopped test service, writes through the CLI, restarts with +WAL disabled and compares committed Catalog data. Two same-version replacements check decline/EOF, +whole-directory backup contents, hidden files, +links, permissions, user attributes, customization and removal of obsolete managed files. A +same-root recovery retains the displaced tree and restores Catalog/Job identities without overlaying +new-only files. These cases do not simulate every installer failure stage or a cross-version upgrade. +Browser form interaction, legacy migration and physical Tape retain their separate acceptance +requirements below. Helper-only RAW/codec acceptance remains separate. + +Cleanup verifies service ownership and stops only this attempt's service. Failed runs retain their +root and transcripts; successful runs remove the root unless `--keep-root` is selected. The controller +stops the test service and unmounts any remaining owned FUSE paths before removing successful +fixtures. Unexpected mounts or unverifiable ownership retain the root and fail cleanup. +`make test-package-tools`, also included in `make check`, checks argument quoting, fixture encoding, +exit/timeout handling and cleanup without connecting to a host. +The report covers its named scenarios; it is not an overall release approval. + +`YATM_E2E_BIN_DIR` remains a local/CI selector. Some detailed Go fixtures start the checkout server +in-process even when their CLI comes from that directory. Those tests retain their source-level +coverage and do not substitute for this remote whole-package boundary. + +Select an already-mounted, xattr-capable filesystem with sufficient free space, then create one owned test root for the extracted package, fixtures, isolated installation and results. Use the archive's `install-release.sh` for remote installation acceptance; the README's exact-tag download remains the normal user bootstrap. Check free space for the complete fixture, including archive and Restore outputs, rather than relying on the system temporary directory's capacity. + +Use the same public CLI steps and observable assertions as the coverage matrix: registration, live Files operations, Scan policies, archive/restore, versions, copy verification, import/export and Job lifecycle. Compare exit codes, JSON, Job terminal results and actual output bytes from the local controller. Exercise real systemd installation/update and the official LTFS file backend. Retain invocation transcripts and artifact hashes privately. + +Low-level fault injection requiring a custom server remains local/CI coverage. Record any remote scenario that cannot be expressed using the shipped tools as unrun, with its exact reason; previous source-based or separately compiled harness runs do not satisfy this gate. Physical Tape remains a distinct explicitly authorized test. + +### Packaged browser acceptance + +The maintained Playwright suite in `frontend/e2e/` operates on disposable Demo data +and deletes two files. Create a fresh Demo root locally from the candidate source, +with `go run ./cmd/demo -root /tmp/yatm-demo-browser-UNIQUE -reset -listen 127.0.0.1:18080 +-identical-files 1001`. Then start the extracted candidate's `yatm-httpd` from its +package directory with `-config /tmp/yatm-demo-browser-UNIQUE/config.yaml`; this loads +the packaged frontend and server while all data stays inside the Demo root. The Demo +generator prepares fixtures only. Never use a regular installation or physical Tape. + +Run `YATM_BROWSER_URL=http://127.0.0.1:18080 YATM_BROWSER_COMMIT=FULL_COMMIT +pnpm --dir frontend test:e2e` after installing Chromium as described in the +[frontend guide](../../frontend/README.md#packaged-browser-acceptance). Record the main +archive checksum alongside the output. Stop the owned server after the run. CI runs +this same suite on the extracted Linux package. Browser acceptance covers actual +rendering and interaction; backend large-data tests own million-entry completion. + +The [Tape device browser fixture](../../frontend/e2e/tape-device.spec.ts) intercepts service reads +with generated gRPC messages and rejects cartridge inspection or unexpected operations. It checks +passive refresh, busy choices, diagnostic distinctions and desktop/narrow dialog layout. This is +mocked browser-consumer coverage; it does not qualify raw hardware or prove device-server behavior. + +### Installer acceptance + +Installer semantic tests cover read-only checks, version ordering, cancellation, file ownership and failure boundaries. Actual systemd acceptance uses unique `--install-dir` and `--service` values on an isolated Linux host with candidate `--archive` and `--checksum` inputs. The cases below are required acceptance, not claims that the candidate has passed them: + +| Workflow | Public command/entry point | Required assertions | +| --- | --- | --- | +| Fresh installation | Exact-version `install-release.sh`, `--config` | Shipped binaries, real systemd/API/frontend readiness, Volume-only configuration, template ownership and optional Skill. | +| Read-only review | Installer `--check`; `yatm-migrate -phase preflight` | No persistent attempt, database write, quiesce or service interruption; pass/manual/blocker distinctions; Tape scripts never run. | +| Pre-stop rejection | Installer confirmation/package/version checks | Cancel, EOF, unknown version, missing packaged migration guide, wrong checksum and Busy preserve service PID/state and active file hashes. | +| Legacy activation | Installer; `yatm-migrate` Prepare/Commit/validate/cleanup | Guide text comes from the verified archive; two explicit approvals; retained report on Prepare failure/cancel; item-by-item Library/Job/log validation before cleanup. | +| Configuration review | `yatm-migrate` config-plan/config-check/config-apply; installer | Unified YAML diff and Settings summary before stop; stale inputs rejected; explicit access preserved, legacy authorization materialized; saved/deleted registrations and Preview preferences preserved; idempotent conversion. | +| Repeated upgrade | Two successive installer updates, same-version replacement and unchanged rerun | Whole-installation tar includes hidden files, hard/symbolic links, permissions and supported xattrs; only `.backup/` excluded; archive/source verified; old checksums unchanged; successful scratch removed; unchanged rerun offers Skill without another archive. | +| Customization | Installer update | Reviewed config changes only; scripts/helpers, unit, permissions, original working-directory path meanings unchanged; complete replacement removes obsolete managed resources. | +| Failure and rollback | Installer faults plus documented in-root recovery | Backup/Prepare/Commit/config-apply/cleanup/replacement/readiness failure retains reports; uncertain mutation stays stopped; failed tree saved before whole-archive restore; `.backup/` retained. | +| Historical repair | `yatm-migrate -phase repair-job -backup-root ...` | Operates after active legacy tables are removed; every repaired item/log matches backup evidence; already frozen Restore destination survives config changes. | +| Old installation entry | `main` branch's legacy `install-release.sh` | Full parameter validation and cross-major rejection; `v0.1.x`-to-`v0.1.x` update remains available. | + +Compare configuration/script/helper/unit hashes, Catalog contents and original service state at each boundary. Inject faults only in the isolated test's candidate/environment; remove only its own service registration afterward. Persist exact candidate commit/archive hashes, stdout/stderr and each pass/fail/skip reason outside the public repository. + +Skill acceptance uses an isolated ordinary-user home and the pinned public Skills CLI from [installation](../operations/install.md#optional-agent-skill). Test the real interactive picker and existing-target cancel/replace, repeat through sudo from a root working directory, and exercise missing dependencies, non-TTY input and source-read failure. Compare installed copy hashes and ownership; cancellation must preserve existing content and optional failure must leave YATM installed. + +Real legacy archive acceptance uses an approved timestamped copy, input checksums, independent Library/Job expectations and a full CLI import/export roundtrip. Run Prepare → Abort → Prepare → Commit → validate → cleanup, repeat idempotent phases, and repair a historical Job from the unchanged backup. Compare every historical item, frozen Restore path and archived log, rather than only Catalog counts. Preserve reports privately and record absent fixture categories explicitly. Production updating follows the public guide only after candidate acceptance; it is not a migration test fixture. + +The same package controller automates this flow for a copied `v0.1.x` SQLite metadata fixture: + +```bash +python3 e2e/package_acceptance.py \ + --host isolated-test-host --test-parent /srv/yatm-tests \ + --archive "$MAIN_ARCHIVE" --version "$VERSION" --commit "$COMMIT" \ + --legacy-package "$PUBLISHED_LEGACY_LINUX_ARCHIVE" \ + --legacy-fixture "$APPROVED_METADATA_ARCHIVE" --out "$NEW_PRIVATE_REPORT_DIRECTORY" +``` + +The metadata tar contains root-level `tapes.db`, its retained WAL if present, and installation-relative +`captured_indices`, `job-logs`, `write-reports` or transitional `jobs` evidence as applicable. +The controller copies only those data members. It rejects links and escaping paths, excludes saved +configuration and scripts, and supplies an isolated configuration with no Tape devices or live +Locations. The old server never starts. The exact candidate installer exercises declined Prepare, +Abort, activation, validation and cleanup; the candidate CLI performs a complete JSONL roundtrip. +The stopped installation then exercises repeated cleanup and historical Job repair. + +[Independent metadata checks](../../e2e/legacy_metadata.py) compare copied legacy facts with the +migrated SQLite rows and Job files, including nanosecond instants and archived logs. This checker +supports completed Archive history with submitted inline sources and uniquely matched logged Media. +It rejects other retained Job kinds/states and transitional Job databases; the report records missing +Restore and other-state coverage. Compressed metadata needs `zstd` on the local `PATH`. +Run Python without optimization because these checks use assertions. A missing fixture category is +not acceptance. Real metadata, extracted copies and detailed transcripts remain in the private +report directory and must not be uploaded as public CI artifacts. Unit tests generate synthetic +inputs. The original copied archive is read-only and its checksum is rechecked after acceptance. + +## Remote Candidate Acceptance + +On the explicitly selected isolated acceptance host, package acceptance uses the corresponding checksum-verified release package, built locally or by CI. Earlier hardware diagnosis may use locally compiled application binaries under [Evidence Reuse](release.md#evidence-reuse). The maintained controller can invoke the server/CLI and installer through SSH or run on the authorized test host with `--host local`. For host-local execution, transfer the controller and its package validators alongside the reviewed archives, checksums, runtime adapters, fixture data and operating instructions. Application execution still uses packaged programs; do not deploy the application checkout, separately compiled test binaries such as `e2e.test`, or an in-process test server. Existing systemd, LTFS/FUSE and other host dependencies remain environmental prerequisites. + +Choose an existing, spacious, xattr-capable mounted filesystem and create a uniquely scoped test directory beneath it. Place packages, installation copies, backups, Restore outputs and a private `TMPDIR` inside that root; export `TMPDIR` for remote installer and migration invocations as well as the controller. Check available space before extraction and each capacity fixture; the test owns only that directory and its explicitly registered test resources. A temporary pathname does not imply a disposable filesystem or sufficient capacity. + +Local development and CI may compile and run Go test harnesses. Those results are separate from remote packaged-binary acceptance. If a fault-injection scenario requires private functions or a custom in-process server, run it locally/CI and record that limit; do not describe it as remote candidate acceptance. [Remote Packaged-Binary Acceptance](#remote-packaged-binary-acceptance) owns the remote procedure. + +For official LTFS file-backend acceptance, explicitly configure the optional `templates/testing/ltfs-file-backend` adapters from the package. They operate only on scoped virtual-cartridge directories and are not default installation scripts; their README describes identity and capacity fixtures. Physical Tape remains a separate acceptance boundary. diff --git a/docs/release/physical-measurements.md b/docs/release/physical-measurements.md new file mode 100644 index 0000000..66a7b53 --- /dev/null +++ b/docs/release/physical-measurements.md @@ -0,0 +1,400 @@ +# Physical Tape Measurements + +Repeatable hardware measurement procedures for Restore timing, performance comparison and capacity +boundaries. The [physical Tape E2E suite](physical-tape-e2e.md) owns gate cases and safety +preconditions, which apply here as well; its PT-07 boundary case uses the capacity procedure below. + +## Fixed Partial Restore Benchmark + +The shared [population](../../e2e/fixtures/restore-1p5tb-v1/manifest.json) and +[selection](../../e2e/fixtures/restore-1p5tb-v1/selection.txt) define one immutable synthetic +workload; the manifest owns its file count, sizes and paths, and units are decimal. It is a +reproducible reference, not a claim about an observed user collection. Never resample it, choose +a more favorable seed or replace failed/fallback observations; a deliberate replacement uses a new +fixture identity and preserves the previous comparison. `go test ./internal/media` checks the +frozen hashes. No maintained tool materializes the physical corpus; a physical run first needs a +reviewed materializer, the full source plus selected Restore output budget, and receipts kept +outside the corpus. Archive placement is established by actual recorded locators or the final LTFS +Index, not by manifest order; an incomplete population does not satisfy this benchmark. + +Use the same corpus, frozen selection, recorded layout, source build, parameters and completion +policy for paired LOGICAL/AUTO/AUTO/LOGICAL arms with fresh Restore destinations. Keep hardware +compression off for this smart-ordering comparison. Measure admission through normal publication +and unload, including preparation, first-file access, planning, all reads and the final return. +Report content verification separately, actual executed order, LOGICAL fallbacks, read/discard and +positioning observations where available, first-file latency, elapsed time and resource curves. +One fixed sample supports only its measured result; do not substitute the earlier sparse controls' +percentage or compare AUTO against an artificially random execution order. + +The software-only forecast uses the production tar evaluator and optimizer with the frozen paths +and sizes in ordinary Archive target-path order, assumed nominal 80-wrap uncompressed geometry +and fixed HP5-derived timing priors. It includes prepared object 2 through final unload, but no +common label-preparation time. +Its `ns/op` measures local planning, index consumption and cleanup; model seconds are predictions, +not physical Restore timings. Rejected lower central estimates remain distinct from an adopted route: + +```shell +go test ./internal/media -run '^$' -bench '^BenchmarkReadSequenceRestore1p5TB$' \ + -benchtime=1x -count=1 -benchmem +``` + +## Bounded Performance Comparison + +Use `performance-prepare`, `performance-format`, and `performance-run` with a new +state and isolated installation for each compared build. Capacity uses a fresh formatted +installation without performance-case writes. Preview is optional. Preparation creates a deterministic +64 KiB setup Archive; FORMAT uses ordinary packaged data-partition placement, +without `-r size=1M`. Measured cases APPEND, normally unmount, validate the captured +final Index and Library publication, then Restore and verify actual output bytes. + +Transfer `e2e/physical_performance_*.py` with the controller and `e2e/package*.py` modules. +`make test-package-tools` runs their local regressions with the package-controller tests and +mocked device observations, without Tape access. SVG rendering additionally needs `matplotlib`; +its rendering test reports a skip when that optional dependency is absent. Measurement parsing, +fixture checks and verification tests still run. + +Supply the explicit package, host, device, barcode and evidence arguments. Use +`--performance-source-read-mode buffered` or `mapped` consistently across stages; +mapped uses the existing Location `--use-mmap` setting. Select cases with repeated +`--performance-case`. `--performance-size-gib` accepts 4 (default), 32 or 64 GiB per +case, saved in state. The fixed order and file sizes are: + +| Case | File size | Content | +| --- | ---: | --- | +| `small-random` | 256 KiB | Deterministic SHAKE-256 bytes | +| `small-compressible` | 256 KiB | Adjacent duplicated 4 KiB random blocks | +| `large-random` | 1 GiB | Deterministic SHAKE-256 bytes | +| `large-compressible` | 1 GiB | Adjacent duplicated 4 KiB random blocks | + +At 32 GiB, a small case has 131,072 files and a large case has 32. Generation emits +an external JSON Lines SHA-256/size manifest. CLI manifest pages contain at most +1,000 rows; a private validation database checks selection, duplicates, content, +previous Positions, final Index extents and Restore order without whole-manifest +maps. Index parsing and artifact transfer stream; actual restored-byte checks use +batches of at most 64. Source/Restore fixtures are removed only after complete +verification; Job bundles and evidence remain. Interrupted cases are preserved +for explicit diagnosis. Each scene requires twice its logical size plus 8 GiB +of free evidence headroom. + +Media timing includes admission through normal finalization and publication. +Fixture generation, preparation and external content checks are outside it. Keep +input bytes/order, source mode, compression/encryption, LTFS settings and physical +layout equal. Source generation and Archive preparation affect page cache; record +cache preparation and host load rather than claiming a cold-cache measurement. + +Independent bounded pollers record application progress about every two seconds, +drive metrics about every five seconds and execution-host resources about every +five seconds. One writer streams JSONL, with request wall/monotonic start and end, +errors and timeouts. Drive reads cannot block the application polling channel. +Use `--no-performance-driver-sampling` consistently across stages to measure collector +overhead. The sampler reads xattrs only through the LTFS process whose exact +`work_directory` identifies the admitted Job; it never opens a competing SG reader. +Timeout cleanup kills only the sampler subprocess group. + +The controller retries only read-only Job waits when Get reports +a SQLite lock, within the original deadline and with every failed transcript kept. +Delays grow from 2 to 4, 8 and 16 seconds, then remain capped at 30 seconds; +the remaining observation budget bounds the final delay. Attempts, lock errors +and chosen delays remain in the report. There is no separate attempt-count limit. +It never repeats a Create or Media action; durable failure and other errors stop +the stage for inspection. + +Application `copied_bytes` during COPYING describes ACP source/handoff progress, +not physical commitment or final Library publication. ACP `copied_file_count` +increments when work enters content processing, so its curve is labeled file-work +entries rather than completed files. Final Index/Library and restored-byte checks +establish completed content separately. Stage changes, resets and missing samples +break interval arithmetic. Root xattrs `user.ltfs.vendor.IBM.logPage.34.00` +and `.0C.00` retain their raw bytes and request times. Binary page framing, required +widths and duplicate parameters are checked before decoding. Page 0C provides +Sequential Access byte proxies; page 34 parameter 0 reports repositions per 100 MB, +not cumulative events. Equal values do not prove freshness or its absence. + +Analysis emits observations, intervals, resources, fixed 10/30-second bins, +all candidate windows, auxiliary drive samples, controller milestones, file-work +entry rates, COPY phase-edge brackets and summary CSV/JSON plus a standalone SVG. +Phase durations retain request-boundary uncertainty; unknown edges stay absent. +The observed byte-endpoint subset rate is distinct from a full successful COPY rate. Matplotlib is needed on the controller only for +plot generation. Counters produce rates over actual sample intervals; UI smoothed +speed is not averaged again. Invalid gaps remain gaps, and zero increments mean +only that no progress was observed in that interval. + +The first observed positive ACP increment anchors candidate windows every ten +seconds. Each 180-second window has six 30-second bins. A qualifying window needs +at least 90% valid time coverage per bin, no gap longer than twice the nominal +sample period, rate coefficient of variation at most 15%, and first/last 60-second +mean drift at most 10% of the window mean. Only valid adjacent increments are +allocated by time overlap. Rates with incomplete coverage describe covered time. +These fixed engineering thresholds classify application averages; they are not +vendor guarantees, physical-streaming proof or optimization acceptance thresholds. +The first qualifying window and every later candidate remain visible, including +periodic no-progress intervals that stable averages can hide. + +Use the complete curve to determine whether a longer fixture is necessary. Fix +sampling gaps before extending duration; persistent low speed and repeated pauses +are results to diagnose. Compare one matched baseline/candidate pair first. When +noise prevents a decision, use at most one reversed affected-case pair. Keep a +change only when throughput or reposition behavior demonstrably improves without +a confirmed regression in the other measure, correctness or bounded resources. +A short successful run or failed candidate alone does not establish a hardware +limit. This flow does not claim capacity-exhaustion or release acceptance. + +## Capacity Measurement + +### Continuous Boundary Capture + +Use one planned Archive run to compare the capacity sources on the same timeline. Reuse +`performance-prepare` and `performance-format` for a fresh isolated installation, then select +`capacity-write` and `capacity-verify`. Transfer `e2e/physical_capacity_*.py` and +`e2e/physical_performance_*.py` with the controller and `e2e/package*.py` modules. +This is the PT-07 ordinary Archive and full published Restore path. + +Choose the cartridge capacity and input size before starting. A separately authorized hardware +capacity reduction can shorten an experiment: HP LTO-5 documents SET CAPACITY `0x0FB3`, roughly +92 GB, in its [integration guide, page 15](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=15). +It erases the cartridge. Verify actual drive support, beginning-of-medium position and measured +capacity. Keep that reduction during FORMAT with the implementation's `mkltfs --keep-capacity` +option; ordinary formatting can restore full capacity. Record partition sizes after formatting. +Restore full capacity only after the experiment's evidence and readback are preserved. Reduced +media establish that configuration's boundary behavior, not a full-length capacity guarantee. + +Pass `--capacity-observe` during `performance-prepare` and repeat it for each later stage. +The controller installs `physical_capacity_boundary.py` in the isolated adapters after successful +mount and before returning to YATM. Pass `--work-directory "$TAPE_DIR" --mount "$MOUNT_POINT"`, +`--phase before-copy` and `--output "$TAPE_DIR/capacity-before-copy.json"`, using the reviewed +absolute helper path. It verifies required probes before copying. At the start of normal unmount, +capture `--phase before-unmount` into `capacity-before-unmount.json`; retain a failed observation +and proceed with normal finalization. These passive hooks do not synchronize or pause an active +copy. Unload clears some HP error counters, so the final mounted snapshot is part of the evidence. + +Pass the FORMAT Job's saved `capacity-before-copy.json` as `--capacity-preflight` to +`capacity-write`. It checks the owning installation/Job and usable starting page 31, page 0C +and `statvfs` before generating payload. The live before-copy hook additionally requires raw page 17 +before permitting payload writes. Each LOG page uses one Linux xattr read with a 64 KiB buffer; +this avoids LTFS's oversized-reply error for a small caller buffer and double reads of read-cleared +counters. The default is 24 ordinary 4 GiB files, with disjoint +deterministic AES streams and an immutable SHA-256/size manifest; `--capacity-files` selects +1–100. Repeated `--capacity-file-mib` instead provides up to 100 ordered sizes, allowing large +files for the main transfer and smaller files approaching the measured boundary. Freeze this +selection before the run. Reserve twice the payload size plus 8 GiB on the source/Restore filesystem. +For a filesystem supporting Linux FICLONE, `--capacity-source-reflink` instead generates one +pseudorandom base, clones ordinary files and gives each a distinct pseudorandom prefix. Each +file receives a full size/hash manifest before Archive. This saves source-disk space; every byte +still passes through Archive and the drive. It requires the payload size plus the largest file +plus 8 GiB for source and Restore space. Check this generator on the selected filesystem with +a small fixture first; unsupported cloning fails explicitly. Repeat the flag on later stages. + +Use the [stage launcher](physical-tape-runbook.md#launch-one-durable-host-local-stage) with a separate `STATE` and new outputs for each capacity stage. +Its baseline-only stage selector must be replaced by the following selector. Keep the `common` +package/identity arguments and construct capacity options from the frozen workload before launch: + +```bash +case "$STAGE" in + performance-prepare|performance-format|capacity-write|capacity-verify|cleanup) ;; + *) echo 'Select one documented capacity stage' >&2; exit 1 ;; +esac +capacity_options=(--capacity-observe) +# Optional: add once at preparation and repeat for every subsequent invocation. +# capacity_options+=(--capacity-source-reflink) +# Set FILE_MIB to the reviewed ordered list (1–100 files, 1–65536 MiB each). +for size in "${FILE_MIB[@]}"; do + capacity_options+=(--capacity-file-mib "$size") +done +if [[ "$STAGE" == capacity-write ]]; then + capacity_options+=(--capacity-preflight "$FORMAT_PREFLIGHT") +fi +# For cleanup, set cleanup_options as described in the physical Tape E2E guide's Cleanup Evidence Prerequisites; +# for other stages, use cleanup_options=(). +printf -v stage_command '%q ' python3 e2e/physical_package_acceptance.py \ + "${common[@]}" "${capacity_options[@]}" "${cleanup_options[@]}" \ + --physical-stage "$STAGE" --out "$STAGE_OUT" +``` + +`FORMAT_PREFLIGHT` is the saved FORMAT Job's `capacity-before-copy.json` from this installation; +it is not a fresh device query. `FILE_MIB` is a Bash array populated from the approved workload. +Alternatively use `--capacity-files N` for N ordinary 4 GiB files. Repeat the source mode, fixture +sizes, observation and reflink options on subsequent stages. Preview is optional for this sequence, +including `cleanup`. Preserve the written cartridge after cleanup. + +The capacity sampler records application progress and owned LTFS readings every second. Its raw +pages are 31, 0C, 17, 02 and 34, alongside `statvfs`, bounded file metadata including temporary +outputs, and resource observations. All channels use the host's wall/monotonic request times; +each sequential device query retains its own start/end time. Preserve gaps, unsupported fields, +counter resets and observation delays. Job byte progress can include rejected work; it is not +actual Tape throughput or a buffer-occupancy measurement. File-work entries and file size changes +do not establish successful Close, final publication or readable content. + +Read TapeAlert through `user.ltfs.mediaStorageAlert`, which lets LTFS retain flags in its own +cache. Reading raw page 2E independently can consume read-cleared flags before LTFS observes them. +Keep the returned bitmask and query times; this observation must not hide a warning from its owner. + +To distinguish the copyer's actual capacity reads from the independent sampler, an isolated +service can run under `strace -D -f --seccomp-bpf`, selecting `statfs` and the required file +lifecycle calls. Verify that the service's main PID remains the HTTP listener and that syscall +filtering works before admitting Media. Keep raw syscall arguments, results, durations and +process identity; a capacity reading alone does not identify its caller. Go's direct `statfs` +syscall is not intercepted by a libc preload wrapper. Keep this optional observer with the +isolated service until normal settlement, and record its potential measurement cost. Before +cleanup, finalize and explicitly supply every raw output under +[Cleanup Evidence Prerequisites](physical-tape-e2e.md#cleanup-evidence-prerequisites); files outside Job bundles +are not captured by the Job collector. + +A capacity error returns nonzero after retaining observations, item rows, Job/LTFS logs, final +Index and Library results. Inspect those results before running `capacity-verify`; the latter +requires observed settlement and device release, then restores every published file and verifies +its bytes. Restore retains continuous samples and allows eight hours for settlement; an empty +published selection cannot pass readback acceptance. No stage automatically repeats a Media operation. A software refusal before creating +the next file, a partial-write failure, and successful readback are separate findings. + +Analyze the saved stream without accessing hardware: + +```sh +python3 -B e2e/physical_capacity_analysis.py /private/evidence/archive-samples.jsonl \ + --job-log /private/evidence/job.log --output /private/evidence/capacity +``` + +The analyzer writes observation/interval/page/event CSVs, a JSON summary and, with matplotlib, +aligned SVG/PNG curves. The optional same-host Job log adds its first error as a one-second +interval and a separate plot of the 15 seconds on either side. Request bounds remain visible; +resets, gaps and partition-coordinate changes break +curves. Compare remaining-to-EW, remaining-to-partition-end and filesystem availability +with byte counters and error events. Their overlap does not alone establish causation or an +exact physical byte position. Page 17 remains available as raw parameter data for vendor-specific +interpretation; an unsupported field is not synthesized from another source. + +### Continue to the Drive's EW Boundary + +Use [physical_ew_run.py](../../e2e/physical_ew_run.py) for an explicitly authorized +scratch-cartridge experiment that must continue beyond application admission. This is an +LTFS/drive measurement; ordinary Archive acceptance remains the workflow above. The runner +verifies the cartridge, formats once, mounts the installed LTFS and generates an incompressible +AES stream directly into ordinary files. Source data need not occupy another filesystem. +It stops new payload at the first write/Close failure, observation failure or frozen workload +bound, records partial-file cleanup, then attempts normal Index finalization and eject once. +The written cartridge and evidence remain available for review. + +Run it in the host-local tmux owner using the quoted command/exit-file construction under +[Launch One Durable Host-Local Stage](physical-tape-runbook.md#launch-one-durable-host-local-stage). Preserve the numeric controller exit +status with a single `printf '%s\n' "$stage_status"` command; verify that the exit file contains +one integer before interpreting it. An empty file is missing evidence, not exit zero. Keep the +controller's structured lifecycle, LTFS log and captured Index as independent completion evidence. +Supply a new output directory, the verified device aliases, +installed LTFS tools, the `yatm-lto-info` executable and the explicitly authorized barcode. +Choose `--max-bytes` above the measured native capacity for this incompressible workload; +reaching that bound is a stopped experiment, not an observed EW. The optional +`--expected-native-min-bytes` catches an unintentionally shortened cartridge before format. +The CLI requires `--authorize-erase` to repeat the scratch barcode. Existing run directories +cannot replay the destructive sequence. + +Build the [test-only SG observer](../../e2e/physical_sg_trace.c) on the Linux host and run +its fake-device tests before using it: + +```sh +python3 -B -m unittest discover -s e2e -p 'test_physical_sg_trace.py' -v +cc -std=c11 -O2 -Wall -Wextra -Werror -shared -fPIC \ + e2e/physical_sg_trace.c -o /private/evidence/physical_sg_trace.so -ldl -pthread +``` + +The runner loads this library only into its foreground LTFS process. It records SG_IO command +times, raw CDBs, status, residual and sense, with bounded LOG SENSE and READ POSITION responses. +File payload is excluded. Optional short READ POSITION queries execute after clean page 31 +queries on the same device descriptor, before the owning call releases LTFS's device lock, +at most once per second. Verify this locking path in the installed LTFS build. The preflight +requires usable filesystem/page readings, initialized tracing and a successful position reply. + +A position query can receive a deferred write error. Its raw response is retained separately; +an injected error disables further injection, creates a trace error marker and stops new payload. +Such a result establishes observer interaction, not the unmodified LTFS error path. Every command +keeps its own timing, so query cost and sample gaps remain measurable. Passive snapshots use the +retained TapeAlert cache and continue briefly after payload failure; command tracing remains +active through the normal unmount. Evidence contains per-write application returns, file hashes, +Close/rename/delete results, mounted attributes, LTFS logs and captured Indexes. + +Analyze the complete or still-growing capture without accessing the drive: + +```sh +python3 -B e2e/physical_ew_analysis.py \ + --sg /private/evidence/ew-run/sg.jsonl \ + --samples /private/evidence/ew-run/capture/samples.jsonl \ + --events /private/evidence/ew-run/capture/events.jsonl \ + --metadata /private/evidence/ew-run/capture/metadata.json \ + --output /private/evidence/ew-analysis +``` + +The analyzer preserves partition attribution, unknown position/buffer fields, query intervals +and current/deferred sense. With optional Matplotlib installed it writes full-run curves and, +when EW is observed, a boundary enlargement from 120 seconds before to 30 seconds after EW. +Successful EW responses have a separate command-warning count from command errors. Compare +observations immediately before and after actual EW with +their different endpoints: page 31 remaining partition capacity, page 0C EOD-to-EW distance, +LTFS availability and logical READ POSITION. A filesystem error or EOP flag alone does not prove +a successful WRITE's EW response, and logical blocks are not physical tape length. Report what +the tested workload establishes separately from a general prediction or complete-file guarantee. + +### Interpretation and Additional Observations + +Use the [Tape capacity study](../research/tape-capacity.md#remaining-research-boundary) to +choose the question and success criteria before a capacity experiment. Assess pre-write refusal, +retained file readability, cleanup, Index finalization and accurate Library publication separately. +Screen candidate policies against the [continuous data-supply constraint](../architecture/tape.md#tape-capacity) +before hardware work. Record observation-induced stalls and reposition activity as well as throughput; +a diagnostic synchronization experiment does not qualify recurring forced drains for production. +PT-07 establishes the observed publication and readback of the tested run; it does not guarantee +that every admitted file will fit on other cartridges or workloads. + +Use one host-local hardware executor for both capacity and performance experiments. Record the +actual LTFS executable and build, backend, mount options, drive firmware and cartridge identity +with each run. Read drive information through the mounted LTFS implementation's supported +interface; its device lock coordinates the observation with active I/O. An unavailable counter +is an explicit gap, not zero and not permission for another process to access the SG device. + +For a capability investigation on an idle, unmounted Linux drive, use the maintained collector +after confirming completed Jobs, device ownership and the assigned cartridge: + +```sh +python3 -B e2e/physical_tape_observe.py --device /dev/sgN --output /private/evidence/new-observation +``` + +Substitute the resolved SG device and a new host-local evidence directory. Run under the existing +host-local executor; load and final eject remain explicit operator steps. The collector checks all +Tape aliases before each query, requests complete raw LOG SENSE and READ POSITION responses, and +records timings and errors. Ownership checks do not reserve the device: keep other users and Jobs +out for the entire experiment. Queries with unsupported fields remain recorded failures, not zeros. +The collector never positions or writes the Tape. Separately approved read-only positioning can +compare the same fields at different locations; record its exact commands and completion status. +`--cartridge-memory` additionally reads HP's documented CM buffer up to its advertised size, capped +at 8 KiB. Keep all raw output private because device pages and cartridge memory can contain +identifying information. Publish only the relevant findings in the capacity study. Fast queries +on an idle drive do not establish acceptable polling cost during Archive. + +For a physical-position comparison, add `--position-probe --cartridge-memory` immediately after +each completed authorized position change. This captures full fixed/descriptor REQUEST SENSE +replies and tests the vendor READ POSITION form before the normal observations. REQUEST SENSE can +consume pending sense information, so this mode belongs only to the isolated idle-drive experiment. +Compare beginning, an existing intermediate logical block and recorded EOD in the same partition; +confirm actual logical positions before interpreting any changing field. Preserve response lengths +and unsupported-command errors. Compare CM pages separately from live replies: unchanged stored +EOD information does not describe current head movement. + +Capture timestamped raw capacity replies and the `statvfs` block counts and units alongside +logical bytes submitted, file completion, explicit synchronization and final Index results. +Keep request start/end times: separate device queries are not one atomic observation. Interpret +partition numbers using the FORMAT log and label, and record LTFS capacity adjustments before +comparing raw drive units with filesystem available bytes. A passive sample does not synchronize +the filesystem. Synchronization experiments record the requested operation, duration and error +separately from the resulting capacity sample. + +When diagnosing an authorized capacity run, preserve its available write-error, corrected-rewrite +and TapeAlert observations before unloading. On HP LTO-5, LOG page `0x02` is specific to the current Tape, +updates on physical data-set writes and clears on unload. Its parameters count different units +(including data sets and CCQ sets), so a retry count is not a byte count. Treat reload, reset, +unsupported fields and counter decreases explicitly instead of subtracting across them. Establish +the effects of head or media degradation from the drive's documented write/rewrite mechanism and +capacity guarantees. A run on one cartridge of unknown condition cannot prove those general +properties; it supplies observations of that run only. + +Compare native capacity consumed with both logical content and corrected-write activity for +compressible and incompressible input. The [capacity study](../research/tape-capacity.md#compression-and-rewrites) +explains why a prior compression ratio or retry allowance does not prove that the next complete +file will fit. Record successful admission, rejection before output creation, and failure after +partial output as different results. End-of-media alone is not a passing pre-write refusal test. diff --git a/docs/release/physical-tape-e2e.md b/docs/release/physical-tape-e2e.md new file mode 100644 index 0000000..f54d362 --- /dev/null +++ b/docs/release/physical-tape-e2e.md @@ -0,0 +1,313 @@ +# Physical Tape E2E Test Suite + +Status: Hardware release gate. See the [development checks](../development/testing.md); this guide does not authorize a physical run. + +This suite is a hardware gate in the [Release SOP](release.md#release-sop). After affected source and nonphysical checks pass, it can run from a reviewed local build while release CI proceeds, following [Evidence Reuse](release.md#evidence-reuse). It covers behavior that the LTFS `file` backend cannot prove: drive discovery, hardware encryption, physical partition placement, tape motion, unload and reload, reads after a process restart, and a real physical full-Media boundary. Run the automated LTFS file-backend E2E first; it remains the primary coverage for injected failures, deterministic capacity exhaustion, two-Media completion, backup compatibility, and cleanup. + +## Safety and Preconditions + +`FORMAT` destroys every existing file on the loaded cartridge. Use one explicitly assigned scratch cartridge whose contents may be erased. A second scratch cartridge is needed only for the optional continuation case. Never run this suite against a production Library database, production Job directory, or a cartridge containing valid data. + +Before starting: + +1. Record the expected six-character barcode and confirm it on the physical cartridge. +2. Use an isolated YATM configuration with empty Executor and Library databases and dedicated work, source, and restore directories. +3. Confirm that the configured Tape device resolves to its SCSI generic device through Linux sysfs. The host needs Bash, Python 3, Node.js, tmux, systemd, the official LTFS programs, `mt`, `stenc`, `umount`, `fuser`, `timeout`, `openssl`, `sqlite3`, `findmnt`, and the package controller's ordinary installation/xattr tools. The main package supplies `yatm-lto-info`. +4. Confirm that the drive, cartridge generation, LTFS implementation, and encryption support are compatible. +5. Baseline preparation adds `-r 'size=1M'` only to the isolated format adapter. Capacity preparation uses ordinary packaged placement; use a separate fresh installation and state for that sequence. +6. Baseline preparation retains the package controller's 4 GiB free-space prerequisite. Capacity requires more than twice the selected payload plus 8 GiB, or payload plus the largest file plus 8 GiB with `--capacity-source-reflink`; see [capacity measurement](physical-measurements.md#continuous-boundary-capture). Budget full published Restore output before writing. +7. Record the YATM commit, ACP commit, LTFS version, drive model and firmware, cartridge generation, native cartridge capacity, barcode, and start time. + +Keep the host on stable power and reserve the drive for the complete run. Do not test process crashes or power loss as part of this suite. + +## Fixture + +Create deterministic source files and record their size and SHA-256 before starting: + +| Path | Content | Expected LTFS partition | +| --- | --- | --- | +| `dataset/index-small.txt` | 60 KiB | Index | +| `dataset/data-small.bin` | 64 KiB | Index | +| `dataset/data-large.txt` | 2.15 MiB | Data | +| `dataset/empty.bin` | Empty | No data extent | +| `dataset/nested/payload.png` | Small PNG supported by the packaged native Preview helper | Index | +| `append/index-small.txt` | Different 65 KiB content | Index | +| `append/data.bin` | 2.34 MiB of non-sparse data | Data | + +The placement rule depends only on size: non-empty files no larger than 1 MiB go to the Index +partition, and larger files go to Data. It applies to text, binary and PNG content and to ACP's +temporary write filenames. Empty files have no data extent. Resolve the partition letters from +the FORMAT log's declared roles and compare them with the captured final Index. + +The mandatory full-Media case must reach a real physical capacity boundary with incompressible data and exercise ordinary Archive source files. Retain the path, size and SHA-256 manifest for files selected for Archive and Restore. Sparse files, zero-filled files and repeating byte patterns do not establish physical Tape consumption because filesystem holes and drive compression can reduce the bytes written. + +## Required Cases + +Run the cases in order because they share one cartridge and one isolated YATM installation. + +### PT-01: Device and Barcode Preflight + +1. Start YATM with the isolated configuration and load the scratch cartridge. +2. Inspect the configured Tape device through the packaged CLI. +3. Compare a non-empty returned barcode with the assigned scratch barcode. If the cartridge memory barcode + is empty, inspect again with `--identity BARCODE` to check Library registration for the explicitly + assigned identity before FORMAT. + +Expected results: + +- Inspection completes without a device-busy or device-discovery error. +- A reported cartridge memory barcode matches the assigned scratch barcode. A successfully read empty + barcode is accepted only before the initial FORMAT, which writes the assigned identity. +- The isolated Library has no Tape Media with that barcode. +- No format, mount, or Library mutation occurs during inspection. + +### PT-02: FORMAT Archive and Preview + +1. Create an Archive Job for `dataset` with Preview generation enabled. +2. Wait until indexing is complete and the Job is waiting for Media. +3. Write the Job with Tape mode `FORMAT`, the expected barcode, and a test name. +4. Wait for Archive and its follow-up preview Scan to complete and for the cartridge to eject. + +Expected results: + +- Hardware encryption is configured before formatting and the key remains in the Tape Media profile. +- LTFS formatting, mounting, ordered ACP writes, unmounting, and ejecting succeed. +- Formatting passes the assigned barcode to `mkltfs`; the subsequent reload/inspection must return it. +- Every non-empty source file has the expected size and SHA-256 in the Archive result. +- Every Archive item is `SUBMITTED`; the Library contains one `ltfs_v1` Tape Media and one Position per source file. +- The captured LTFS Index contains valid extents for every non-empty file. Each persisted Position has the same first logical extent encoded as partition, block, and byte offset in its 17-byte storage order. +- `index-small.txt`, `data-small.bin` and `payload.png` are on the index partition. + `data-large.txt` is on the data partition; `empty.bin` has no extent. +- The Job Tape directory contains the LTFS log, captured Index, Archive report, and manifest. +- The Preview bundle and its HTTP asset are readable. + +### PT-03: Reload and Inspect the Durable Tape + +1. Reload the ejected cartridge with `mt -f DEVICE load` without changing the databases. A standalone drive may take about one minute to become ready. +2. Inspect it through the packaged CLI. +3. Leave it loaded for the next Archive operation; inspection does not mount or change its contents. + +Expected results: + +- The physical barcode resolves to the Media created by PT-02. +- Media ID, name, format, file count, and written bytes match the committed Library state. +- Inspection does not format, mount, or modify the Tape or Library. + +### PT-04: Reject a Different Requested Barcode + +1. Reload the same cartridge. +2. Create a second Archive Job for `append` and wait until it is ready for Media. +3. Request FORMAT and then APPEND with a different, unregistered barcode through the packaged CLI and record both preflight refusals against the loaded physical cartridge. + +Expected results: + +- FORMAT reports the requested/device barcode mismatch; APPEND reports that the requested barcode has no registered Media. An unavailable cartridge memory barcode is not a pass. +- CLI preflight refuses the request before admitting an Archive Media operation. The prepared Job, existing Tape Media and Positions remain unchanged; no encryption, formatting, mounting or copying occurs. +- Reloading and inspecting the cartridge still returns the original barcode. + +Reuse `TestTapeSessionsRejectUnverifiedIdentityBeforeMutation` for the backend's independent rejection before encryption, formatting or mounting, and `TestArchiveFailedMediaAttemptSettlesPhaseAndReopens` for a failed Media operation returning the Job to its pre-Media `READY` state. These are local/CI checks at separate boundaries; physical CLI refusal does not prove an admitted runner mismatch through RPC. + +### PT-05: Append to the Existing Tape + +1. Submit the prepared Archive Job from PT-04 against the same cartridge with mode `APPEND` and the original barcode. +2. Wait for completion and eject. + +Expected results: + +- Append reuses the original Media ID and encryption key. +- Existing files and Positions remain unchanged. +- Appended files use a collision-resistant Media path prefix and become `SUBMITTED`. +- The format-time placement policy remains effective: appended `index-small.txt` is on the index partition and `data.bin` is on the data partition. +- The final captured Index contains both Archive Jobs with valid extents and storage order. +- Within each partition, physical storage order matches the deterministic Archive item order even when source preparation completes out of order. +- Job completion and device availability occur only after the cartridge has ejected. + +### PT-06: Restart and Restore + +1. Stop YATM normally after PT-05 and start it again with the same isolated databases and work directory. +2. Create one Restore Job containing files from both Archive Jobs. +3. Sort the selected Positions by their persisted storage order and record the expected Media-path sequence. +4. Reload the cartridge and restore through the physical Tape device. +5. Wait for completion and eject. + +Expected results: + +- The restarted process reads the durable Media profile and reconfigures the drive with the stored encryption key. +- Restore reads files from both the index and data partitions. +- Position storage order is propagated unchanged into Restore Copies. The sequential-source unit test verifies that requests follow ascending partition, block, and byte offset rather than logical path order; the physical run verifies that the resulting files are restored correctly. +- Every selected file completes, including the zero-byte file. +- Restored paths, sizes, bytes, and SHA-256 values match the source fixture. +- Restored ACP signature xattrs contain the expected size and SHA-256. +- Restore logs are stored under the Restore Job's `tapes//` directory. +- Job completion and device availability occur only after the cartridge has ejected. + +### PT-07: Physical Full-Media Boundary + +Use the separate [capacity sequence](physical-measurements.md#continuous-boundary-capture) with its frozen input sizes +and disk budget. Choose enough ordinary source content to reach the measured cartridge boundary; +a completed Archive whose input fits does not establish capacity exhaustion. + +1. Run `performance-prepare` and `performance-format` with `--capacity-observe` to obtain the + setup Archive and its saved before-copy capacity preflight. +2. Run `capacity-write` once. It hashes all source files, saves the immutable size/hash manifest, + prepares one ordinary Archive and submits one APPEND. Keep continuous samples through settlement. +3. Review `capacity/write-result.json`, item order/states, progress, Library export, final Index, + Archive report and Job/LTFS logs, including before-copy and before-unmount observations. +4. Once the Job is observed idle and the device is released/ejected, run `capacity-verify` once. + It restores **every published file**, up to the 100-file bound, and verifies actual output + size/SHA-256, ACP signatures and persisted Position storage order. An empty selection fails. + +Expected results and interpretation: + +- Software refusal before creating the next file is an expected capacity boundary. Keep its + complete error chain and pending suffix. `capacity-write` still returns nonzero for a Job error; + this is evidence to review, not permission to replay the write or an automatic hardware pass. +- A late write, Close or finalization failure is a distinct failure finding. `no_space` alone + cannot distinguish pre-write refusal from partial output. Compare the logs, temporary-file + observations, final Index and publication; successful readback does not erase the write failure. +- Published items form a continuous prefix on the assigned Media. Final Index paths, sizes, + extents and storage order match their Positions. The setup Archive remains present. Pending + items have no Media/path publication; no item remains staged. Report counts match published + files and bytes. Missing or contradictory finalization evidence remains a failed check. +- Device availability and eject are checked after settlement. Uncertain ownership retains the + running service and all resources for the lifecycle owner; no device status query is made while + the Job or lease is active. No automatic unmount, eject, reformat or Media retry follows failure. +- Full published readback is mandatory. A first/last sample cannot establish this result. + +### PT-08: Cleanup and Evidence + +1. Require completed baseline Restore or capacity readback. Check the entire isolated Job catalog is idle, the device lease is released, the cartridge is ejected and no owned mounts remain. Export the Library metadata backup into preserved evidence. +2. Before deleting anything, capture each Job's catalog and bundle metadata, complete Job log, LTFS log, Archive report, captured Index, before-copy/before-unmount snapshots, and the exact `sg-trace.jsonl` and `sg-trace.jsonl.error` files when present. Capture verifies each transfer against the host SHA-256; unrelated Tape files are not swept into evidence. +3. Require the source manifests and original stage timing reports described below. Finalize every supplied syscall trace and checksum-copy it into preserved evidence. Generate a path-sorted SHA-256 manifest covering these files and all Job evidence. Missing prerequisites, active observers, capture failures or checksum mismatches stop cleanup before Job deletion. +4. Recheck preserved checksums, the saved manifest digest and unchanged Job catalog, then delete only those idle test Jobs through the API. A resumed cleanup also checks the remaining catalog against the sealed inventory and compares live Tape artifacts with their preserved checksums. An old seal that omitted an SG trace or its error file cannot authorize deletion. +5. Check mounts again, stop the owned service and remove only its installation, source files and Restore output. Keep state, stage reports, sealed evidence and the written cartridge. Cleanup issues no load, format, mount, unmount, eject or Media submission. + +Expected results: + +- The export contains the Tape profile, Files, and physical Positions needed to describe the completed run. +- The evidence checksum manifest verifies every preserved Job artifact before Job deletion. +- Job deletion removes the corresponding local Job directories. +- Cleanup does not modify or erase the physical cartridge. +- The cartridge is ejected and clearly marked as scratch or test media. + +### Cleanup Evidence Prerequisites + +Use the existing shared `physical_evidence_dir` in `STATE`. Before invoking `cleanup`, supply +the reviewed source manifests at these exact paths; do not reconstruct report timings or +substitute current source hashes for those used by an earlier stage: + +| Preserved path | Required evidence | +| --- | --- | +| `source/packages.sha256` | Original package checksum record; its main-package digest must match the preparation report | +| `source/controller.sha256` | Executed controller source hashes, including `physical_package_acceptance.py`; an instrumented SG run also includes `physical_sg_trace.c` and the executed `physical_sg_trace.so` | +| `stages//report.json` | Original completed invocation report, matching commit, version and installation, with finite nonnegative duration and the matching physical case result | + +The required baseline reports are `prepare`, `baseline` and `restore`; tar also requires +`tar-clear-barcode` and `tar-fill` when they completed. +Capacity requires `performance-prepare`, `performance-format`, `capacity-write` and `capacity-verify`, including +the original failed `capacity-write` report for an expected software refusal. Other required +stage reports must have passed. The current cleanup report stays in its separate `--out` +directory; it cannot be a completed prerequisite of its own invocation. + +After each entered non-cleanup stage, the controller copies the exact bytes from its final +`--out/report.json` save to `stages//report.json` under the common evidence directory. +This includes failed capacity writes, their failure reason, retention decision and final duration. +Early validation failures that never enter a stage remain only in their invocation's `--out` and +do not replace its preserved report. Keep source manifest selection explicit and reviewed: + +```bash +PHYSICAL_EVIDENCE=$(jq -er '.physical_evidence_dir' "$STATE") +mkdir -p -- "$PHYSICAL_EVIDENCE/source" +for name in packages.sha256 controller.sha256; do + cp -- "$EVIDENCE_ROOT/$name" "$PHYSICAL_EVIDENCE/source/$name" + cmp -- "$EVIDENCE_ROOT/$name" "$PHYSICAL_EVIDENCE/source/$name" +done +``` + +The same source-manifest step applies to baseline. With an SSH controller, preserved files and +stage reports are controller-local; `--syscall-trace` paths below refer to the hardware host. +The controller validates the fixed +prerequisites before sealing. Source review still establishes that the recorded hashes identify +the executed source and observer; checksum syntax alone cannot prove that history. + +Cleanup requires either repeated `--syscall-trace /absolute/host/file` arguments for **every** +separately collected syscall trace, including each `strace -ff` output, or an explicit +`--no-syscall-trace` declaration when that observer was not used. Supply exact files, not directory +or wildcard arguments. Stop/detach the observer after normal Media settlement and ensure starting +the idle service for cleanup will not restart it. Cleanup refuses symlink trace files, duplicate +filenames, open traces or an uncertain `fuser` result. It copies raw bytes into +`observers/syscall/` using the existing checksum transfer helper, whether the source +was inside or outside the owned root; the evidence seal covers every copy. An observer outside +Jobs is not discovered automatically: the cleanup owner must supply its complete file set. + +For a capacity run whose Archive and Restore used the SG observer, add `--require-sg-trace`. +This requires nonempty `jobs//tapes//sg-trace.jsonl` for both Jobs and the recorded +observer source/binary hashes. Capture preserves any accompanying `.error` even without that +flag. An error marker remains a failed observer finding for review; preserving it does not +establish complete SG measurement or a successful write. Cleanup performs no observer or +hardware operation to replace missing evidence. + +For example, pass the following options to the separate authorized cleanup invocation: + +```bash +cleanup_options=(--require-sg-trace) +for trace in "${SYSCALL_TRACES[@]}"; do + cleanup_options+=(--syscall-trace "$trace") +done +``` + +`SYSCALL_TRACES` is the explicit nonempty list of finalized host files from the stage record. If no +syscall observer was used, use `cleanup_options=(--require-sg-trace --no-syscall-trace)` for an +SG-instrumented run. Omit `--require-sg-trace` only for an uninstrumented run. Preserve source, +reports and traces before the first seal; a failed or incomplete old seal is retained for review, +not silently replaced to permit deletion. + +## Optional Physical Cases + +The first physical full-Media boundary is mandatory. The cases below either require a second cartridge or repeat destructive formatting, so they are optional unless the affected behavior changed. + +### PT-X1: Continue the Full Job on a Second Tape + +Continue the pending PT-07 Archive Job on a second scratch cartridge: + +1. Load the second scratch cartridge and write the remaining items in `FORMAT` mode with its own barcode. +2. Verify that the Archive Job completes and that every item is submitted to exactly one of the two Tape Media. +3. Restore the complete capacity logical set by loading both cartridges as requested. +4. Compare every restored size and SHA-256 with the generation manifest. + +Do not add a synthetic FUSE source or streaming-file abstraction solely for this case. The physical test should exercise ordinary source files through the production I/O path. + +### PT-X2: Explicit Delete and Reformat + +After preserving the original evidence, delete the Tape Media metadata through YATM, create a new Archive Job, and format the same physical barcode again. Verify that FORMAT is rejected before metadata deletion, succeeds after deletion, creates a new Media ID, and does not delete the old logical Files. + +## Pass Criteria and Evidence + +The release gate requires the baseline and capacity sequences plus their evidence cleanup, without manual database edits or changes to archived Tape files. A reviewed expected software refusal and successful full published readback can satisfy the capacity boundary; a late write/finalization failure remains a separate finding. PT-X1 and PT-X2 are recorded separately when run. Retain: + +- the version and hardware record; +- source and restored SHA-256 manifests; +- packaged CLI request results and the reused local/CI Session identity and failed-Media lifecycle evidence; +- every Job log and report; +- every Job's catalog and bundle metadata plus the sorted evidence SHA-256 manifest; +- captured LTFS Index files from FORMAT and APPEND; +- the immutable capacity source manifest, continuous Archive/Restore samples and write/readback results; +- all raw Job Tape SG traces and optional error markers, plus every separate syscall trace when used; +- the Library JSONL metadata backup; +- a concise result for each case, including elapsed Archive, full-Media boundary, and Restore time. + +Each invocation's `report.json` records total `duration_seconds`, per-phase setup timing, +per-case timing and command durations, including failed stages. Retain separate timings for +preparation, FORMAT/baseline, Archive, complete Restore/verification and cleanup, plus wall-clock +handoff/queue/transfer time. Capacity samples mark Media admission and settlement; write-result +timestamps include fixture generation and Archive work. Post-Restore output hashing remains part +of overall stage time. + +Reuse the already accepted ordinary-file full-capacity run when review confirms unchanged +relevant production source, dependencies, build/runtime configuration and fixture semantics. +A controller/SOP cleanup does not alone require another full Tape write. Keep the original +source/package/controller hashes and result with the reviewed diff; run affected local collector +and cleanup regressions. New release archives still need their own identity/content checks and +relevant package smoke acceptance under [Evidence Reuse](release.md#evidence-reuse). + +Record a failure at the first unmet expected result. Preserve the cartridge and local evidence for diagnosis instead of reformatting and retrying immediately. diff --git a/docs/release/physical-tape-runbook.md b/docs/release/physical-tape-runbook.md new file mode 100644 index 0000000..f41ec30 --- /dev/null +++ b/docs/release/physical-tape-runbook.md @@ -0,0 +1,306 @@ +# Physical Tape Host Runbook + +How to run the [physical Tape E2E](physical-tape-e2e.md) and [tar acceptance](physical-tar-acceptance.md) +stages on the assigned Linux host. The gate documents own cases and pass criteria; the safety +preconditions in the E2E suite apply to every step here. + +## Automated Stages + +### Prepare the Artifacts and Host Context + +Use checksum-verified Linux amd64 main and matching Preview packages from the reviewed source. +Local builds use `build/release/build.sh` and `previewworker/build/build.sh`. Reviewed locally +compiled binaries can begin affected hardware checks once their source and nonphysical checks +pass, independently of CI's final release-byte gate. This installer-based procedure uses packages +so that installation and runtime adapters are also exercised. Final release-byte acceptance and +the evidence comparison in [Evidence Reuse](release.md#evidence-reuse) still apply. + +On the authorized Linux hardware host, use a root shell. +Prefer a host-local executor with `--host local`; SSH transfers inputs and retrieves results. +Allocate an existing spacious, xattr-capable filesystem +for the isolated installation, fixtures and private `TMPDIR`. Keep controller inputs, `state.json` +and evidence outside the installation root that cleanup removes. + +Transfer the main archive, Preview helper archive, matching +`yatm-preview-source-linux-amd64-.tar.gz` and their `.sha256` companions. +Keep the corresponding-source archive beside the Preview helper; the package validator requires it +before installation. Transfer the reviewed controller's +`e2e/physical_package_acceptance.py`, `e2e/physical_tar_recovery.py` and `e2e/package*.py`, and these validators, preserving their +repository-relative layout under `CONTROLLER_ROOT`: + +- `build/release/check-release.mjs`, `archive-members.mjs` and `package-assets.mjs`; +- `build/release/check-candidate-set.mjs` and `check-preview-sources.mjs`. + +Include the operational dependencies of every selected check, beyond its entrypoint. For the +Linux virtual-unmount regression, transfer `e2e/test_package_tape_scripts.py` and +`e2e/ltfs-file-backend/umount` plus `get_device`. Broader Tape-script unit selections additionally +need `scripts/mkfs` and `scripts/get_device`. Source-based LTFS E2E uses `e2e/testdata` adapters; +its `umount-file-failures.sh` calls `umount-file.sh`, which delegates to the same +`e2e/ltfs-file-backend/umount` and `get_device`. Keep those relative paths and executable modes. +Source-based Go E2E remains a local/CI check under the [E2E guide](../development/e2e-test.md), separate from +the host's installed-package procedure. + +Supply absolute host-local values for `CONTROLLER_ROOT`, `TEST_PARENT`, `EVIDENCE_ROOT`, +`HOST_TMP`, `MAIN_ARCHIVE`, `PREVIEW_ARCHIVE`, `STATE` and the official LTFS executables. +Set `STATE` to the new run's `state.json`. `VERSION` and `COMMIT` identify the archives; +the controller's reviewed source identity is recorded separately. Supply `SCRATCH_DEVICE` and +`SCRATCH_BARCODE` from the explicit assignment. Keep actual host, device, barcode and local path +values in the private stage record and evidence. + +Run from the absolute controller directory in every new host shell: + +```bash +set -euo pipefail +cd -- "$CONTROLLER_ROOT" +umask 077 +mkdir -p -- "$EVIDENCE_ROOT" "$HOST_TMP" +export TMPDIR="$HOST_TMP" +sha256sum -- "$MAIN_ARCHIVE" "$PREVIEW_ARCHIVE" > "$EVIDENCE_ROOT/packages.sha256" +sha256sum e2e/physical_package_acceptance.py e2e/physical_tar_recovery.py e2e/package*.py \ + build/release/check-release.mjs build/release/archive-members.mjs \ + build/release/package-assets.mjs build/release/check-candidate-set.mjs \ + build/release/check-preview-sources.mjs > "$EVIDENCE_ROOT/controller.sha256" +uname -a > "$EVIDENCE_ROOT/kernel.txt" +id > "$EVIDENCE_ROOT/executor.txt" +findmnt -T "$TEST_PARENT" > "$EVIDENCE_ROOT/filesystem.txt" +df -Pk "$TEST_PARENT" > "$EVIDENCE_ROOT/space.txt" +``` + +Compare these hashes with the reviewed transfer manifest. Record the YATM and ACP source +identities, any reviewed working-tree changes, toolchains/build flags, extracted program hashes, +LTFS build/version, adapter hashes and any transferred regression inputs. The private host record +also identifies drive model/firmware, +cartridge generation/capacity, assigned identity, resolved Tape/SG mapping, drive reservation, +start time, and the isolated service/configuration. Package version labels alone do not establish +the executed bytes. The controller checks archive companions, package members and offline program +identity before installation. + +### Resolve LTFS Before Starting systemd + +Resolve the official LTFS installation in the host shell and supply its absolute executable paths: + +```bash +MKLTFS_BIN=$(realpath -- "$(command -v "$MKLTFS_COMMAND")") +LTFS_BIN=$(realpath -- "$(command -v "$LTFS_COMMAND")") +test -x "$MKLTFS_BIN" +test -x "$LTFS_BIN" +``` + +`MKLTFS_COMMAND` and `LTFS_COMMAND` are the reviewed executable names or absolute paths from +the stage record. Choose the mount template for that LTFS distribution as the +[Tape scripts guide](../operations/tape-scripts.md#mount-and-unmount) describes. Normal unmount must +finish the LTFS process and leave the final current-cartridge `TAPE_DIR/.schema`. + +The physical controller below selects the openltfs template. Supply the corresponding official +LTFS build; an HPE run requires reviewed controller adapter selection matching the HPE template. +During `prepare`, `--mkltfs "$MKLTFS_BIN"` and `--ltfs-binary "$LTFS_BIN"` put absolute commands +into the isolated adapters, and the format adapter adds `-r 'size=1M'`. This happens +before the installer starts systemd. Inspect and hash the resulting adapter copies before +`baseline`. A login shell's `PATH` is insufficient evidence of the systemd service's commands. + +### Run the Scoped File-Backend Precheck + +Before the first physical operation, run the same package/controller pair through the existing +official LTFS file-backend format/append/Restore/Verify case. Use openltfs with its `file` backend +and the matching directory-valued capture option. The virtual unmount adapter waits for the +exact mount-owning LTFS process, identified by its Linux `/proc` command line, to exit as well as +for FUSE to detach. That boundary makes the final Index available before publication checks. + +Choose the exact checks and verify their transferred inputs before a tmux launch. For a changed +virtual-unmount adapter, the focused Linux regression selector is: + +```bash +cd -- "$CONTROLLER_ROOT" +python3 -m unittest -v \ + e2e.test_package_tape_scripts.VirtualUnmountTests.test_waits_for_final_index_after_fuse_detaches +``` + +This check uses a simulated LTFS process and ordinary Linux tools. Require an executed pass, +including its delayed final Index assertion. Record the selector, dependencies and result in +the stage record. The real file-backend check below selects one exact package case and its declared +prerequisite; retain that selection in its launch command and report. + +Put the resolved executables on the host shell's +`PATH`; the controller resolves them into its isolated virtual adapters before service installation: + +```bash +cd -- "$CONTROLLER_ROOT" +export TMPDIR="$HOST_TMP" +export PATH="$(dirname -- "$MKLTFS_BIN"):$(dirname -- "$LTFS_BIN"):$PATH" +test "$(realpath -- "$(command -v mkltfs)")" = "$MKLTFS_BIN" +test "$(realpath -- "$(command -v ltfs)")" = "$LTFS_BIN" +python3 e2e/package_acceptance.py \ + --host local --test-parent "$TEST_PARENT" \ + --archive "$MAIN_ARCHIVE" --preview-archive "$PREVIEW_ARCHIVE" \ + --version "$VERSION" --commit "$COMMIT" --ltfs \ + --case ltfs-format-append-restore-verify \ + --keep-root --out "$PRECHECK_EVIDENCE" +``` + +`PRECHECK_EVIDENCE` must be a new directory. The selected case includes its fresh-installation +prerequisite and creates only owned virtual-cartridge directories. Allow at least the controller's +16 GiB free-space prerequisite. `--keep-root` retains its stopped installation for review. + +Require a passed report for the selected case, with both FORMAT and APPEND partition checks: + +1. FORMAT writes a small file and a file larger than 1 MiB; APPEND adds different small content + to the same Media. This precheck and the physical fixture use the same size-only 1 MiB + placement rule, which remains effective for ACP's temporary write filenames. +2. After normal unmount, parse each Job's actual final captured `.schema`; obtain partition roles + independently from the FORMAT log. Compare exported Library Positions with actual Media paths, + sizes, complete extents and 17-byte partition/block/offset storage order. Both partition letters + must occur in the fixture. +3. The final APPEND Index must retain every earlier file and its extents on both partitions. + Earlier Positions and Media ID remain unchanged; new Positions match the final Index. +4. Restore all selected files and compare actual output hashes and sizes with the sources. + Verify reads every recorded copy and publishes matching findings and healthy observations. + +Preview acceptance resolves the PNG through its frozen Archive content signature. An Archive +selection can have a saved version without a known current Location signature; the saved identity +is the source of truth for that derivative check. + +Review `report.json`, including `case_scope`, `partition_checks`, command exits and the retained +Job artifacts. A completed copy without the final Index/Position comparison is incomplete +acceptance. Retain a failure and resolve its first unmet check before advancing to hardware. +The separate file-backend capacity case remains part of full nonphysical acceptance; this scoped +precheck provides a quick check of the relevant publication and readback path. + +### Launch One Durable Host-Local Stage + +Stages share one isolated installation and the controller's existing state: + +| Stage | Work and completion boundary | +| --- | --- | +| `prepare` | Fresh installation, deterministic sources, prepared Archive/Preview Jobs and byte-preserving Preview evidence; no Tape operation | +| `baseline` | PT-01..05: inspection, FORMAT, reload, barcode refusal, APPEND and final Index/Position/Preview checks | +| `restore` | PT-06: start the same stopped installation, restore both Archives, verify bytes and storage order | +| `performance-prepare` | Fresh separate installation, ordinary-placement setup Archive and capacity/performance observation helpers; no Tape operation | +| `performance-format` | FORMAT the setup Archive, eject and validate its final Index and Positions | +| `capacity-write` | PT-07: generate the frozen ordinary-file fixture, submit one Archive APPEND and preserve settlement, publication and failure evidence | +| `capacity-verify` | PT-07: Restore every published capacity file, verify bytes, signatures and storage order; no Archive write | +| `cleanup` | PT-08: seal and verify evidence after baseline Restore or capacity readback, delete idle test Jobs and remove the owned installation after review | + +Use `prepare → baseline → restore → cleanup` for the baseline and a separate +`performance-prepare → performance-format → capacity-write → capacity-verify → cleanup` +for capacity. Both preparation stages require absent state. Later invocations require the +preserved state, matching package identity, host, device, barcode and LTFS paths. Capacity +verification can follow a failed write only after observed settlement and device release; +cleanup requires successful full published readback. A baseline run does not prove capacity acceptance. Each successful +non-cleanup stage stops its isolated service. Give every invocation a new `--out` directory and +unique log/exit filenames. Keep `STATE` and the preparation stage's shared physical evidence +directory for the entire run. + +In a host-local root Bash shell, set `STAGE` to the next authorized stage and +`STAGE_OUT`, `STAGE_LOG`, `STAGE_EXIT` and `SESSION` to new paths/a unique tmux name from the +stage record. Put the log and exit file in an existing evidence directory. Recheck live drive ownership +and the assigned cartridge before physical use. Launch one stage, then review its result before +launching the next: + +```bash +cd -- "$CONTROLLER_ROOT" +sha256sum -c "$EVIDENCE_ROOT/packages.sha256" +sha256sum -c "$EVIDENCE_ROOT/controller.sha256" +common=( + --host local --test-parent "$TEST_PARENT" + --archive "$MAIN_ARCHIVE" --preview-archive "$PREVIEW_ARCHIVE" + --version "$VERSION" --commit "$COMMIT" + --physical-device "$SCRATCH_DEVICE" --physical-barcode "$SCRATCH_BARCODE" + --mkltfs "$MKLTFS_BIN" --ltfs-binary "$LTFS_BIN" --state "$STATE" +) +test ! -e "$STAGE_OUT" +test ! -e "$STAGE_LOG" +test ! -e "$STAGE_EXIT" +case "$STAGE" in + prepare|baseline|restore|cleanup) ;; + *) echo 'Select one documented physical stage' >&2; exit 1 ;; +esac +cleanup_options=() +if [[ "$STAGE" == cleanup ]]; then + # Baseline without a syscall observer; instrumented runs supply exact trace paths below. + cleanup_options=(--no-syscall-trace) +fi +printf -v stage_command '%q ' python3 e2e/physical_package_acceptance.py \ + "${common[@]}" "${cleanup_options[@]}" --physical-stage "$STAGE" --out "$STAGE_OUT" +printf -v stage_shell \ + 'cd -- %q || exit; export TMPDIR=%q; %s >%q 2>&1; stage_status=$?; printf "%%s\\n" "$stage_status" >%q; exit "$stage_status"' \ + "$CONTROLLER_ROOT" "$HOST_TMP" "$stage_command" "$STAGE_LOG" "$STAGE_EXIT" +printf -v tmux_command '%q ' bash -c "$stage_shell" +tmux new-session -d -s "$SESSION" "$tmux_command" +``` + +The host's tmux process keeps the stage alive when the SSH session ends. Read status on +that host without submitting another operation: + +```bash +if tmux has-session -t "$SESSION"; then + tmux display-message -p -t "$SESSION" '#{session_name}: #{pane_current_command}' +fi +tail -n 60 -- "$STAGE_LOG" +test ! -f "$STAGE_EXIT" || cat -- "$STAGE_EXIT" +test ! -f "$STAGE_OUT/report.json" || \ + jq '{status, error, phases, cases, cleanup}' "$STAGE_OUT/report.json" +test ! -f "$STATE" || \ + jq '{completed_stages, service, install, url, physical_evidence_dir, cleaned}' "$STATE" +``` + +An ended session normally makes `tmux has-session` exit nonzero. A stage passes only when its +exit file is `0`, its report is passed, its expected evidence assertions pass, and state records +its completed checkpoint. A missing exit file or a running/failed report requires inspection of +the existing session, service, Job phase and owned mounts. Reconnect to these records after a +disconnect; the absence of a connection does not imply the stage stopped. + +### Stage Continuation and Review + +Advance only from completed stages with the same state and a new output directory. Reconnecting +to an existing executor or collecting its completed evidence never resumes a Tape write: + +- `baseline` can reuse a normally completed FORMAT and its prepared APPEND Job only while the APPEND write has not been admitted. + It rechecks Media identity, encryption and FORMAT evidence before continuing. An admitted APPEND + write or incomplete/failed FORMAT requires operator diagnosis rather than another baseline write. +- `restore` can collect/check an already submitted Job after it settles + normally; it does not submit that saved Job's Media operation again. +- `capacity-write` and `capacity-verify` each persist an admission marker before work and refuse + another invocation after that marker. Read the retained result and Job state; never clear the + marker or replay Media to repair evidence. A failed write may proceed to its first explicit + `capacity-verify` after settlement and device release. +- `cleanup` resumes deletion only against the sealed evidence and unchanged remaining Job + catalog. It verifies every preserved checksum again; an active Job, uncertain lease, remaining + mount, changed catalog or changed evidence stops deletion. Preserve state, reports and shared + evidence outside the installation; host-local cleanup rejects paths that it would remove. + +For a collection-only failure, retain the failed report, review/check the collector repair, +record its changed hash and collect the existing result only where the guards above permit it. State records remain +the evidence of what happened. An unresolved extent or Index/Position mismatch remains an +unmet acceptance check even when the write or unmount completed. + +Before each stage, record the authorized host, drive, cartridge and destructive scope; the exact +source, package, program, controller and adapter hashes, LTFS selection, tools and host context; the +absolute controller, input, state, evidence, installation and TMPDIR paths, service name and CLI +URL; the selected stage, its launch and status commands and log, exit and report locations; and +completed checkpoints and Job IDs, the first unfinished check, pass criteria and stop condition. + +Only one executor uses the drive. Review stage reports and compact evidence before running the +separate `cleanup` invocation. Cleanup seals and verifies the shared Job evidence +before deletion, preserves state and stage reports outside the removed installation, and leaves +the cartridge ejected. After review, remove only the owned precheck/fixture resources and apply +the agreed evidence retention policy. Failed hardware remains available for diagnosis; cleanup +does not establish a pass for its failed write. + +Evidence collection copies the settled Job's original `job.log` and verifies its checksum, like +the other bundle artifacts. The [CLI output contract](../operations/cli.md) exposes text for +inspection, not a byte-preserving log archive. Preparation exercises the real packaged Job and +collector together; parser unit tests alone do not verify that boundary. + +The configured `readinfo` script starts every physical operation with `mt -f DEVICE load`. It then spends at most about one minute polling the explicit MAM Barcode field. The script removes an attached media-generation suffix such as `L5`. A successful, explicitly empty field allows FORMAT with the assigned scratch barcode; formatting writes that identity. APPEND and Restore require the existing device identity. A failed probe or unreadable field stops the operation. The [Tape contract](../architecture/tape.md#tape) owns these identity rules. + +Normal unmount keeps the Job attempt and device lease until the mount is gone, no process holds the resolved SG device, and `mt -f DEVICE status` reports `DR_OPEN`. This post-unmount wait shares a ten-minute deadline. A successful Job therefore exposes the drive only after eject completion; timeout follows the unmount-failure rules below. + +Capacity acceptance uses ordinary Archive sources and a complete Restore of its published +selection. FORMAT, APPEND and Restore remain separate explicit Media submissions. + +An unmount failure invalidates the attempt and keeps the device unavailable for the lifetime of that YATM process. Stop the isolated YATM service before manual cleanup, identify processes holding its recorded owned mount with `fuser -vm MOUNT_POINT`, and have them release the mount before attempting a normal `umount`. A force or lazy unmount is only a last-resort manual cleanup after the service has stopped; discard that write result and restart YATM before using the device again. Do not enter an owned LTFS mount while a physical stage is running. + +Stopping a systemd unit can also terminate its LTFS child. If the mount has already disappeared, +verify that no process holds the assigned SG device. When a normal eject reports "Medium removal +prevented", release that failed session's lock with `mt -f DEVICE unlock`, then use +`mt -f DEVICE offline` and confirm `DR_OPEN`. This cleanup never makes the failed write acceptable. diff --git a/docs/release/physical-tar-acceptance.md b/docs/release/physical-tar-acceptance.md new file mode 100644 index 0000000..7127f0e --- /dev/null +++ b/docs/release/physical-tar-acceptance.md @@ -0,0 +1,143 @@ +# Physical Tar Acceptance + +The tar stage sequence of the physical Tape gate, run through the installed service and CLI. +The [physical Tape E2E suite](physical-tape-e2e.md) owns safety preconditions and evidence rules, and +the [host runbook](physical-tape-runbook.md) owns stage launch. + +## Tar Acceptance + +The [LTFS cases](physical-tape-e2e.md#required-cases) exercise LTFS; their Index, partition-placement, Preview and eject +assertions do not establish tar acceptance. Tar acceptance is a separate stage sequence of the same +[controller](../../e2e/physical_package_acceptance.py) with `--physical-format tar_pax_v1`. Like +PT-01..PT-08, it drives only the installed service and `yatm-cli`, so it exercises the business +logic users run; no source-level hardware test participates. Every stage uses the configured +`encrypt` adapter, because tar has no plaintext path, and passive read learning, which `prepare` +enables explicitly in Device settings. The [public-tool reader](../../e2e/physical_tar_recovery.py) +also generates and checks the nine-file fixture oracle; the [recovery guide](../operations/tar-tape-recovery.md) +owns the operator commands. + +Assign an erasable cartridge, its exact barcode, the configured Tape device and the SCSI generic +path that sysfs resolves for it. Besides Python 3, systemd, tmux and the installer tools, the host +needs sg3_utils (`sg_raw`, `sg_read_attr`), `journalctl`, mt-st 1.8, coreutils `dd`, GNU `tar`, +`fuser`, `findmnt`, `sqlite3` and the tools the encryption adapter calls, such as `stenc`; `prepare` +checks the commands before installing. Choose explicit `fixed` or `variable` records and a +512-byte-aligned size within the observed device limits; an unknown limit or refused preparation is +an unresolved result, not permission to guess. Transfer `e2e/physical_package_acceptance.py`, +`e2e/physical_tar_recovery.py` and `e2e/package*.py`, plus `e2e/physical_performance_*.py` for +`tar-fill` and `tar-ew`. Record the package, program and tool identities, drive/firmware, driver/HBA and +cartridge density, and retain evidence outside the installation. + +Launch each stage in the host-local tmux owner described under +[Launch One Durable Host-Local Stage](physical-tape-runbook.md#launch-one-durable-host-local-stage), replacing its argument +and selector block with: + +```bash +tar_common=( + --host local --test-parent "$TEST_PARENT" + --archive "$MAIN_ARCHIVE" --version "$VERSION" --commit "$COMMIT" + --physical-format tar_pax_v1 --physical-device "$SCRATCH_DEVICE" --sg-device "$SCRATCH_SG" + --physical-barcode "$SCRATCH_BARCODE" --record-mode "$RECORD_MODE" --record-bytes "$RECORD_BYTES" + --state "$STATE" +) +case "$STAGE" in + prepare|baseline|restore) stage_options=() ;; + tar-clear-barcode) stage_options=(--confirm-clear-barcode + "CLEAR BARCODE $SCRATCH_DEVICE $SCRATCH_SG $SCRATCH_BARCODE") ;; + tar-fill|tar-ew) stage_options=() + for size in "${FILE_MIB[@]}"; do stage_options+=(--capacity-file-mib "$size"); done ;; + tar-read-order) stage_options=() ;; + cleanup) stage_options=(--no-syscall-trace) ;; + *) echo 'Select one documented tar stage' >&2; exit 1 ;; +esac +printf -v stage_command '%q ' python3 e2e/physical_package_acceptance.py \ + "${tar_common[@]}" "${stage_options[@]}" --physical-stage "$STAGE" --out "$STAGE_OUT" +``` + +The order is `prepare → [tar-clear-barcode] → baseline → restore → [tar-fill → [tar-ew → [tar-read-order]]] → cleanup`. Each +stage requires its predecessor in the preserved state, saves an admission marker before its first +device or Job operation and is never replayed. Every assertion retains stdout, stderr, exit status, +Job settlement and hashes in the evidence directory; keys are redacted from transcripts and +evidence. `tape device inspect` defines no barcode field, so the controller reads MAM attribute +0806h with `sg_read_attr --raw` on the confirmed SG path and only records the inspection. + +| Stage | Pass criteria | +| --- | --- | +| `prepare` | Fresh installation configured with only the `encrypt` adapter, fixture oracle, source and Restore Locations, Device settings with `passive_read_learning: true` and LOGICAL read order, and READY FORMAT, APPEND and mismatch Archive Jobs. No Tape access. | +| `tar-clear-barcode` | Optional preparation before FORMAT, for a cartridge whose 0806h is not empty. Requires the unregistered barcode and no holder of any Tape/SG alias. Sends WRITE ATTRIBUTE `sg_raw -s 9 -i PARAMS SG 8D 00 00 00 00 00 00 00 00 00 00 00 00 09 00 00` with the parameter list `00 00 00 05 08 06 01 00 00` (0806h, ASCII, length 0 clears it). `sg_read_attr` must then report no barcode, and `tape device inspect` none. A refusal is recorded in `mam/clear-write.json` and stops the stage; there is no fallback. | +| `baseline` | 0806h is empty before FORMAT. FORMAT settles with the requested tar profile; after reload, 0806h holds the barcode as a 32-byte space-padded value. FORMAT and then APPEND of the mismatch Job with a different barcode are refused by the CLI or by the leased Session (idle READY with its error), publish nothing and leave Media, Positions and 0806h unchanged. After reload, APPEND keeps the Media, profile and earlier Positions, adds exactly its members, logs `tar APPEND tail verified`, and neither its Job log nor the service journal mentions a sequential scan. Settled Item rows match the published locators and have cleared provisional results. | +| `restore` | After the restart, full and selected Restores without `--media-id` identify the cartridge by cartridge memory barcode; output bytes, paths, permissions, subsecond times and xattrs match the oracle. Inventory retains every member without a content finding, and Verify matches every copy. Each Restore and Scan bundle has `tapes/BARCODE/read-observations.jsonl` with a parseable line, and a learned model for the device appears in `settings devices` within 600 seconds after release. LOGICAL and AUTO Restores of the full selection record their duration, progress `elapsed_ms`, Job-log read-order decision and inspected `prepared_read`; the comparison and the model change after the second Restore are report only, with no threshold. `job delete` of the selected Restore removes its bundle. Public-tool recovery passes. | +| `tar-fill` | The tar equivalent of PT-07. Requires the reviewed `FILE_MIB` plan and optional `--capacity-source-reflink`; free space must cover the sources (the largest file with reflink), three sampled members and 8 GiB; when inspection reports remaining capacity, the payload must exceed it. One Archive APPEND writes through the normal path until capacity admission or EW stops it and settles idle READY with its capacity-stop error, complete published members and a PENDING suffix without staged storage. Earlier Positions are unchanged. A Restore of the first, middle and last published members matches their bytes; Verify matches every copy on the Media and public-tool recovery hashes every published member while streaming, so the full payload is never restored to disk. | +| `tar-ew` | Continues a filled cartridge to the drive's own EW. Requires the reviewed `FILE_MIB` plan of small members whose total exceeds the reported remaining capacity, and free space for the sources and every published member. The device's `capacity_poll_interval_seconds` is set to 0 for the write, so admission keeps only its preparation sample, and is removed again before readback. One Archive APPEND settles like `tar-fill`; earlier Positions are unchanged and every published member restores with matching bytes. The stage passes only when the Job error reports `Tape early warning`; an admission stop is recorded in `ew/write-result.json` and fails it. | +| `tar-read-order` | One LOGICAL and one AUTO Restore of the same selection on the filled cartridge: the fixture members near BOT and every other `tar-ew` member. Each records what the `restore` comparison records, in `read-order-full-comparison.json`; report only, no threshold. | +| `cleanup` | [PT-08](physical-tape-e2e.md#pt-08-cleanup-and-evidence) rules. Requires completed `restore` recovery and, once admitted, completed `tar-fill`, `tar-ew` and `tar-read-order`; the reports of every completed tar stage are required prerequisites. | + +Public-tool recovery stops the service and runs the reader with the saved key in a temporary 0600 +file and the configured `encrypt` adapter. Using only `mt`, `dd` and GNU `tar`, it extracts the +label, FORMAT and APPEND Tape files, each ending at a FILEMARK, and compares them with the oracle. +After `tar-fill` it streams the fourth Tape file through `tar --to-command` hashing: every published +member must match its size and SHA-256 exactly once, and other members may only form an unpublished +tail. Recorded EOD follows the last FILEMARK. Payloads stay on the host; only bounded receipts are +copied. Retain the unpublished tail and the reader's tar status separately from the published +members. + +Before an operation the controller requires an idle device in `tape device list` and observes +`mt status`; when it reports `DR_OPEN`, it issues one synchronous `mt load`, then verifies `ONLINE`. +No timeout wraps that load or the public reader, and a failure stops without retry. Normal Archive, +Restore and Scan unload the cartridge. `tar-fill` waits for each Job within an explicit budget of +one hour plus the payload at a conservative 35 MB/s, about 13 hours per Job for 1.5 TB (LTO-5 +nominally streams about 140 MB/s); +exceeding it stops observation and retains the active owner. Failed or interrupted operations stop +and report their errors; this suite does not salvage tails, resume interrupted writes or test +crashes. Disk-only GNU checks establish framing and metadata behavior, not drive completion. +Reload-stable seek, capacity polling cost and geometry need their own measurements; record +unknown, unsupported and refused cases explicitly. The [tar study](../research/tar-tape.md) and +[Tape contract](../architecture/tape.md) own the remaining questions and publication rules. + +### Recovery Reader and Assertion Scope + +Preparation freezes the fixture's SHA-256, modes, nanosecond times and empty/binary user xattrs. +Baseline checks FORMAT/APPEND, immutable identity/profile, exported copy hashes and unchanged +earlier Positions. After a service restart, full and selected Restores use automatic Media identification and +compare actual bytes and metadata. +Known-only inventory must remain unchecked; FORCE_READ Verify must report checked matches. +Saved physical-anchor coverage does not establish reload or seek qualification. + +Media replies carry only `encryption_key_saved`. Where the controllers need the saved key, they +read it from the matching `media` record of an in-memory Library JSONL export whose transcript is +redacted; FORMAT and APPEND/restart compare its SHA-256. For the explicitly requested public +recovery, the controller obtains the existing completed Media key this way before stopping only +its owned service. It passes record settings and the +fixture oracle to the [public reader](../../e2e/physical_tar_recovery.py), which also generated it, +with a temporary 0600 key file, adapter path, barcode, Tape name and original service directory. +The reader checks device users and mounts, then runs that adapter with the existing script inputs. +Key bytes are redacted from receipts and the caller removes the temporary file after the reader +settles. The reader requires no YATM executable, Catalog, Job recovery artifact or saved locator. `mt`/`dd` copy label, FORMAT and APPEND Tape files; GNU tar +lists/extracts them, and the disk oracle checks every fixture member. A nonempty copy requires +exclusive EOF at the expected file/block/partition after close/reopen. A separate next read +requires zero bytes and exclusive driver-reported EOD; zero bytes alone never pass. Unknown +syntax, diagnostics or boundary facts fail and retain the stopped owner and partial evidence. +The original driver block setting is restored if changed after completed read commands or failed +checks. Each attempted command is recorded before execution. If execution raises before normal +completion, its receipt remains unsettled and no restoration or status query follows. +The host-local owner waits for each public device command to return; no subprocess watchdog +kills an in-flight read or motion. An interrupted wait attempts to receive completion without +killing the child and preserves the unsettled failure. Retain and investigate that owner before further +device access. + +Encryption adapter inputs are optional as one complete set: `--encrypt-script`, `--key-file`, +`--barcode`, `--tape-name` and `--script-directory`. They do not enable a hardware compression decoder. + +These boundary assertions are scoped to Linux st's observed mt-st 1.8 status syntax and the +recorded kernel/GNU tool versions. They cover normal recorded end only, not empty physical Tape +files, physical EOT, reload, seek, EW/capacity cost measurement or arbitrary faults. Local stub checks +establish only the script-managed encryption adapter's invocation and cleanup. Other tool/driver semantics require reviewed oracles. Stages admitted to device work cannot be replayed; +cleanup requires application readback, independent recovery and sealed evidence before deleting +idle owned Jobs and the installation. It does not eject the cartridge. Keep state and evidence +outside the owned installation root. `make test-package-tools` checks this controller locally +with device/service/process boundaries stubbed; it does not run physical acceptance. + +The sampled-counter plotter's finite limits cover 24 hours at two-second app and five-second +driver cadence with ten page 0C and five page 34 parameters: 120,000 raw intervals, 45,000 +file-work intervals and 300,000 driver rows. The existing 20,000-bin and 500,000-resource-row +limits remain; larger input shapes require their own review. Plots retain observation waits and +unknown gaps without interpolation. diff --git a/docs/release/release.md b/docs/release/release.md new file mode 100644 index 0000000..eb31335 --- /dev/null +++ b/docs/release/release.md @@ -0,0 +1,268 @@ +# Release Process + +The maintainer procedure for building, accepting and publishing a release. + +## Release Packages + +Release packages must include the separate User Guide entry, all current user guides and their +required local attachments. Architecture, research, contributor rules and developer validation stay +in the repository. Packaging uses an explicit allowlist rather than copying the whole tree; update +that list in [build.sh](../../build/release/build.sh) and its validation whenever the user documentation set changes. Required user procedures +must work from the extracted package; optional implementation references may link to the exact +release's public source. + +## Release SOP + +Run this procedure for every release. The release workflow enforces the automatic gates below; +source review and physical acceptance retain their own explicit decisions. + +1. **Review scope and repository content.** Compare the public baseline with the final source by + Files/Locations, Identical, Jobs, Preview, installation/migration and API/CLI. Map retained + capabilities to their consumers and meaningful tests; resolve unclear removals before freezing + source. Check Demo coverage, including large results and real licensed media where relevant. + Review terminology and affected operator/architecture instructions against actual commands and + behavior. Update the existing owner, remove superseded claims and close verified issue records. + Check the [Tape script compatibility contract](../architecture/tape.md#tape-script-compatibility) + against the published adapters: run unchanged historical scripts through the current caller, + and verify that upgrade preserves customized scripts/helpers. For an explicitly approved + incompatibility, check the actionable installer notice and the adapted script boundary. + Cover the [Tape lifecycle](../architecture/tape.md#tape-lifecycle), including empty cartridge memory + barcode FORMAT, subsequent identity checks, APPEND and Restore. + Keep these checks in the automated suite; a new-template hardware pass alone does not cover upgrades. + Inspect modules, tools and temporary files for a current runtime, generator, test or packaging + consumer before deleting them. Review ignored local directories separately; CI cannot inspect + workstation leftovers. Accepted limitations remain documented and do not become release blockers. +2. **Audit the history and squash unpublished development.** Scan the development range before + squashing, including intermediate source and commit messages. Review credentials, internal + services, personal paths, real data, configuration and private notes manually as well as with + the pinned scanner. Rotate an exposed real credential before attempting public-history cleanup; + history rewriting needs separate authorization. Recheck the public branch tip, retain a local + backup and squash only unpublished commits into one release commit whose parent is that tip. + Do not publish the backup branch or rewrite already public history. Review the final diff and + commit message again; squashing does not sanitize final content. Finish required dependency + releases and update their pins before freezing YATM. +3. **Complete review and fixes with affected checks.** Review the integrated source and finish + focused fixes/verification until no unresolved, unaccepted finding remains. Use + [Evidence Reuse](#evidence-reuse); do not run a full preflight per fix. + Complete the [performance acceptance](../development/performance.md#performance-acceptance) before source-push approval, + retaining unaffected measurements and checking only changed operations again. + Start affected hardware verification from a reviewed local build as soon as its source and + nonphysical checks pass, following [Evidence Reuse](#evidence-reuse). Do not + wait for GitHub or the full release matrix to investigate or verify a hardware fix. + For workflow edits, validate GitHub syntax and expression contexts before pushing with + `go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 -shellcheck= -pyflakes=`; + script diagnostics remain covered by the existing shell checks. +4. **Run the final source preflight on the clean release commit.** Go commands in release gates + and package builders use `GOWORK=off` so dependencies resolve from committed module files. + Development checks can still use a local workspace. The candidate workflow owns + the final full run before its platform matrix. A local full run is available for an intentional + integration check or diagnosis, but is not a prerequisite that CI must then repeat. + Put Bash 4.4 or newer on `PATH` for + the Linux installer unit tests; macOS's bundled Bash 3.2 is insufficient. Install the frozen frontend dependencies, + pinned scanner and the protobuf tools used by current generated source: protoc 26.0, + protoc-gen-go 1.33.0 and protoc-gen-go-grpc 1.3.0. Preview acceptance also requires a separate + FFmpeg fixture generator with libx264, 10-bit libx265, PNG, lavfi and display-rotation support. + Native release CI also prepares the checksum-pinned CC0 camera samples from the + [worker fixture inventory](../../previewworker/testdata/raw-fixtures.json), so real-camera + acceptance runs for every advertised RAW family instead of skipping for missing local data. + `node build/release/check-preview-fixtures.mjs` generates tiny real inputs and fails on missing + tools or features. Linux CI builds the pinned generator with + `bash build/release/build-test-ffmpeg.sh OUTPUT_DIRECTORY` after installing pkg-config, + libx264-dev, libx265-dev and zlib1g-dev, and adds its `bin` directory to `PATH`. This is a test + dependency; installed YATM and its packaged helper do not need it. Use a complete Git checkout and run: + + ```shell + RELEASE_VERSION=vX.Y.Z RELEASE_COMMIT="$(git rev-parse HEAD)" \ + GITLEAKS_BIN=/path/to/scanner/gitleaks make release-check + ``` + + The preflight rejects dirty or mismatched inputs, incomplete history, unowned source paths, + runtime databases/logs, local configuration, compiled output, archives and scratch files. + It scans current source and all history reachable from HEAD, then runs documentation/IDL, + Go/frontend/build-tool checks, full race checks, both SQLite paths and two clean generations. + Native Preview race tests and vet run with its private libraries during each runnable helper build. + Its corresponding-source archive has a separate inventory that retains required vendor/native + sources. Filename gates do not prove that a module is used or documentation is semantically correct. +5. **Build and accept one candidate set.** GitHub Actions requires the exact reviewed commit to + exist in this repository's remote. Obtain explicit authorization for that source push before + dispatch; this does not authorize a tag, Release or asset upload. Dispatch the Release candidate + workflow with `publish=false`. Its platform matrix waits for `make release-check`. Candidate CI + validates the complete [package set](#release-backend-builds), public archive content, packaged documents, licenses, + identities and checksums. Check the exact extracted bytes at affected package boundaries + for installation/upgrade, supported `v0.1.x` migration, Archive, Restore, Scan, Volume/LTFS + and native Preview. Apply [Evidence Reuse](#evidence-reuse) to retain valid + earlier behavior and hardware evidence instead of repeating every workflow after a rebuild. + Use the [local package controller](package-acceptance.md#remote-packaged-binary-acceptance) with the exact + Linux main/Preview archives, `--ltfs` and a new private evidence directory for its named workflows. + Its report does not replace separate browser, legacy-backup or physical-media acceptance. + Complete the [Media and migration gates](#media-and-migration-safety), including separately + authorized scratch Tape acceptance. A skipped or unavailable required check remains open. +6. **Approve and publish the accepted bytes.** Review the final public source, commit message, + Release text and all assets. Obtain publication approval before tag or Release creation and + asset upload; any further source push also needs explicit authorization. + Follow the existing [upload procedure](#candidate-and-publication) with the successful candidate + run ID; it rechecks provenance, content and checksums without rebuilding. Verify public asset + checksums against the accepted files after upload. +7. **Record evidence and clean owned resources.** Report passes, failures, accepted limitations and + unrun checks accurately. Keep raw verification output in CI artifacts and sensitive evidence + private. Remove only temporary processes, directories and test resources owned by that run. + +## Release Backend Builds + +Linux backend builds require Zig **0.15.2** and always enable CGO, selecting the existing mattn C SQLite driver. `build/backend/build.sh` preserves version/commit metadata and uses external static musl linking. `build/backend/cross-linux.sh` defaults to amd64; set `GOARCH` and, for ARM, `GOARM` explicitly. Non-Linux release builds retain their existing `CGO_ENABLED=0` strategy. + +| Go target | Zig target | CPU selection | +| --- | --- | --- | +| linux/amd64 | x86_64-linux-musl | default | +| linux/386 | x86-linux-musl | default | +| linux/arm64 | aarch64-linux-musl | default | +| linux/arm, GOARM=5 | unsupported | Zig 0.15.2 lacks the ARMv5 `__sync` atomics required by Go CGO | +| linux/arm, GOARM=6 | arm-linux-musleabihf | arm1176jzf_s | +| linux/arm, GOARM=7 | arm-linux-musleabihf | cortex_a7 | +| linux/s390x | s390x-linux-musl | default | + +ARMv5 and other unmapped Linux targets or ARM values fail before compilation. ARMv5 remains an explicitly allowed failure in the CI matrix and produces no candidate archive. There is no automatic pure-Go fallback. Every built Linux program passes `build/release/check-static-linux.mjs`: ELF program headers must have no interpreter or dynamic segment, Go build metadata must enable CGO, and no native Preview module may enter the main dependency graph. This excludes shared libc/SQLite/FFmpeg dependencies. Runtime acceptance remains Linux amd64; other mapped release targets are experimental until exercised on matching hardware. + +Distributed main binaries omit Go debug data and strip native debug data at the Linux external +link, so compiler and cache paths do not become public package content. Package scanning checks +the final executable strings as well as text files; trimming Go source paths alone is insufficient. + +```shell +GOARCH=amd64 ./build/backend/cross-linux.sh +node build/release/check-static-linux.mjs output/yatm-httpd +``` + +The [optional native Preview helper](../../previewworker/build/README.md) has its own module, build, platform baseline and private native libraries. Its native dependencies are never linked into the main binaries. + +The source-check job builds the frontend once and collects its locked production dependency +notices. The internal `shared-frontend` artifact records version, commit, lockfile hash and every +file checksum. Each main-platform build validates its identity and exact file inventory, then +copies the same frontend and Node notices and adds its own Go/native notices. Source maps, private +configuration, missing files and extra files are rejected. The internal manifest is never copied +into a release archive. Local `make release` calls the same collector; set +`FRONTEND_ARTIFACT_DIRECTORY` to reuse a previously collected artifact with matching inputs. +Preview native-object caching follows the [native build guide](../../previewworker/build/README.md#native-compiler-cache). + +Release validation requires exactly nine main archives, three optional helper archives and three matching source archives. Each main archive still passes `build/release/check-release.mjs`; `build/release/check-candidate-set.mjs` additionally verifies the complete platform set, matching identities, helper file boundaries and corresponding-source checksums. Linux E2E receives the extracted helper through `YATM_TEST_PREVIEW_HELPER` so native Preview acceptance cannot silently skip. + +Release builds require clean committed source and a matching `RELEASE_COMMIT`. Inputs are exported +from that commit, so ignored workstation files cannot enter recursive copies. The platform matrix +starts only after the source checks, full race suite and both SQLite paths pass. Frontend tests use +two workers to bound local and CI resource use. + +## Content Scanning + +Public content checks use checksum-pinned Gitleaks 8.30.1 with its default credential rules and +additional personal-path/private-service checks. The installer also pins the default rule source; +scans retain its credential rules while removing global dependency/generated-file exclusions. +Install it with +`bash build/release/install-gitleaks.sh /path/to/scanner`, set `GITLEAKS_BIN` to that directory's +`gitleaks`, and run `node build/release/check-content.mjs source . HEAD`. A different revision range +can inspect unpublished development before squashing. `node build/release/check-content.mjs artifacts /path/to/candidates` checks +nested archives and embedded executable/library strings, normalizing npm `.tgz` names in a private +scan copy so the scanner opens them. Source maps and dependency sources are +included; the only project exception is the exact deterministic physical-Tape fixture seed at its +owned path. A repository's `.gitleaksignore` may contain explicitly approved historical fingerprints +in full `commit:file:rule:line` form. Only the Git history scan reads them; current source and package +scans always use an empty ignore file. Commit-free, wildcard and path/rule-wide exceptions are +rejected. Additions require a separate decision about the specific already published finding; +new commits remain fully scanned. Checksum-validated dependency sources have rule-specific exceptions for public source +identifiers, deterministic test values and recorded upstream build/sample paths. Each is restricted +to its exact nested member and matched value; other credentials or paths in those members still fail. Failures +report rule counts without match snippets, and temporary scanner reports are +deleted instead of uploaded as public CI evidence. Scanner regression tests use the same executable. + +## Candidate and Publication + +The **Release candidate** workflow runs only through `workflow_dispatch`. Set `version` and leave +`publish=false` to build the candidate set at the selected commit, validate every package and run the +full Linux E2E suite against extracted candidate binaries and the native helper. A successful run +retains the complete accepted set, companion checksums and `SHA256SUMS` in `accepted-candidates`. +Build completion establishes neither performance acceptance nor publication approval. +Creating a GitHub Release does not start a build or upload. + +For local or remote package acceptance, download only the selected platform's artifacts into +one directory identified by the candidate run. For example, Linux amd64 acceptance uses: + +```shell +gh run download "$CANDIDATE_RUN_ID" --repo samuelncui/yatm \ + --name candidate-linux-amd64 --name candidate-preview-linux-amd64 \ + --dir "$CANDIDATE_DIRECTORY" +``` + +Multiple selected artifacts are extracted into per-artifact subdirectories. The Preview artifact +includes its helper and matching corresponding-source archive. Verify the +downloaded checksums and expected commit, and retain these files for subsequent cases and physical +acceptance. Record the final successful run and artifact IDs; tests started before candidate CI +finishes remain provisional until that same run succeeds. Do not download the complete +`accepted-candidates` merely to repeat passed CI inventory, content or checksum checks locally. +The publication job retrieves that complete artifact on GitHub when it needs all release assets. + +To deliver those accepted bytes, explicitly create the matching Release and tag first, then dispatch +with the same `version`, `publish=true` and its successful build `candidate_run_id`. Publishing runs +no builds or E2E; it downloads that run's immutable artifact by ID. The run must belong to this +repository and the Release candidate workflow (by workflow ID and path), identify a manual build of +that version, and have completed successfully. Its accepted artifact must still be available and unexpired. The release +tag, including an annotated tag, must resolve to the run's exact commit, and the Release prerelease +flag must match the version. Candidate identity comes from run metadata and package contents rather +than the upload checkout. Release tools come from `v1`, pinned to the same commit for preflight and +upload; no code from the candidate artifact is executed during upload. + +Upload repeats provenance and public-content checks, validates the exact 9+3+3 set and every checksum +with the existing package validator, and sends the unchanged archives and checksum files to that +existing Release. +Only the upload job has `contents: write`; candidate lookup and download use scoped `actions: read`. +The workflow neither creates a Release or tag nor replaces existing assets. An expired artifact needs +a new build and acceptance run before publication. Mocked provenance and upload tests run through +`node --test build/release/publish-candidate.test.mjs` without publishing anything. + +## Evidence Reuse + +Finish review and fixes before the final full source and package acceptance. The candidate CI +runs `make release-check` once before its platform matrix; a duplicate local full preflight is +not required. Use `make check` for an intentional full integration check, not after every edit. +If a final check finds a defect, verify the fix locally at its affected boundaries before +starting the replacement candidate. Changes to shared contracts or infrastructure may justify +wider checks; record the reason instead of automatically repeating everything. + +Retain passing evidence for unaffected components when their implementation, relevant +dependencies, configuration, fixtures, harness and environment remain unchanged. Record the +old and final source identities and the reviewed diff that establishes this. New archives still +need their own content, identity and checksum checks. Rebuilt binaries require relevant package +smoke tests; changed behavior requires its package workflows. Acceptance of earlier bytes alone +cannot certify a new archive, but a new archive does not require repeating unrelated benchmarks +or every exploratory/manual check. + +Hardware verification may use locally built binaries after the affected source checks pass; +GitHub availability and completion of the full platform matrix are not prerequisites. Build +from the reviewed commit with the repository's existing scripts and record the source, +toolchain, build flags, binary checksums and runtime adapters. Use the release package when +installation or package layout is part of the check. Follow the +[physical Tape execution procedure](../release/physical-tape-runbook.md#automated-stages) for host context, +absolute LTFS adapter paths, the scoped two-partition file-backend precheck and durable staged +execution. Hardware work can start independently of CI's final release-byte acceptance; +that acceptance remains required for release delivery. Source CI and other platform builds can +run in parallel. Keep the final artifacts' identity, content and affected package smoke checks; +reuse expensive hardware results, including a full Tape write, when comparison confirms that +the relevant source, dependencies, build configuration and runtime inputs are unchanged. +If those inputs differ or cannot be established, repeat the affected checks and explain why. + +Reuse downloaded candidate files for subsequent checks when their run, commit and checksums +match. Transfer only the packages and fixtures needed by the selected host and cases. A new +report or another case selection does not require another download. Complete-set validation +belongs to candidate CI; use its run and artifact records for checks already completed there. + +## Media and Migration Safety + +- Run [mounted Volume and LTFS file-backend E2E](../development/e2e-test.md) for affected copy/publication behavior. Use the official LTFS file backend on the isolated Linux acceptance host through packaged binaries and local orchestration; Volume tests use isolated already-mounted directories. +- Run [physical Tape E2E](physical-tape-e2e.md) only with explicitly assigned scratch media and isolated databases/configuration. A documentation update is not authorization to format, write, or test a cartridge. +- Test migration only on timestamped `cp -a` copies of explicitly approved source backups. Keep private source paths and detailed reports outside the public repository. Never modify source backups or production installations during validation. +- Compare every migrated Job item-by-item, including non-empty historical Jobs, not only catalog counts. Cover prepare, abort, repeat prepare, commit, and separately confirmed cleanup. +- Clean only validated temporary test media, mounts, databases, Job bundles, and migration copies after validation. Never use production paths as cleanup targets. +- Batch confirmed review fixes before expensive E2E rather than repeating the full suite after each comment. Record actual results and unrun checks accurately; do not claim planned tests passed. + +## Delivery Gates + +The candidate preserves [independent format identities](../architecture/persistence.md#published-data-formats) and legacy migration/import. Record acceptance for the exact candidate bytes, reusing unaffected source-level evidence under [Evidence Reuse](#evidence-reuse); earlier development results alone do not certify a rebuilt release. Preserve a forward path for supported published data. Before the first stable v1 release, replace the [pre-stable compatibility policy](../README.md#pre-stable-compatibility) with the published stable API/upgrade policy. + +Branch, push and publication gates follow [Delivery and Authorization](../../AGENTS.md#delivery-and-authorization). Release packages include only the operator documents allowlisted in [Release Packages](#release-packages); development and architecture documents remain in the repository checkout. diff --git a/docs/research/tape-capacity.md b/docs/research/tape-capacity.md index bd10818..3564d5d 100644 --- a/docs/research/tape-capacity.md +++ b/docs/research/tape-capacity.md @@ -1,21 +1,20 @@ # Tape Capacity and Safe Completion -Status: Research; evidence reviewed on 2026-10-11. No complete solution has yet been established -for all supported drive/LTFS combinations. Possible approaches below are not implemented guarantees. -The [Media contract](../architecture/media-io.md#tape-capacity) owns current YATM behavior. +Status: Research; evidence reviewed on 2026-10-24. Findings describe the studied counters, +streaming behavior and normal capacity boundaries, not guarantees for every drive/LTFS combination. +The [Tape contract](../architecture/tape.md#tape-capacity) owns current YATM behavior. ## Objective -Prevent capacity exhaustion from leaving an unsafe Archive result: +Study what capacity counters and write observations can establish for whole-file admission and +controlled EW completion. Account for writes once, retain unknown capacity as unknown, and distinguish +complete published members from pending content. Ordinary write, cleanup and finalization failures +remain errors; partial files must not become successful Positions. -- Refuse the next complete file before creating its output when it cannot safely fit. -- Preserve completed files and the Index needed to locate and read them. -- Handle write, Close, rename, cleanup and finalization failures without publishing partial files. -- Leave a Tape that can be normally unloaded, mounted again and used for Restore. -- Report what completed, what remains pending and any cleanup/finalization failure separately. - -Explaining a particular shortfall is useful only when it establishes a mechanism needed for these -outcomes. A successful write or an accurate progress counter alone does not establish safe completion. +Normal completion and controlled EW require their own physical evidence. Restoring interrupted +Archive prefixes, repairing failed tails or guaranteeing remount after an arbitrary failure are +not product requirements of this study. A successful write or progress counter alone does not +establish physical completion. The compatibility constraints are ordinary LTFS files, existing user-owned Tape adapters and the [supported operating model](../architecture/overview.md#supported-operating-model). Replacing the @@ -24,7 +23,7 @@ reserve does not meet this scope. Device-specific capabilities may improve an op when the existing adapters and other supported configurations retain a defined behavior. Capacity management must preserve continuous data supply during normal Archive, following the -[Media constraint](../architecture/media-io.md#tape-capacity). Periodically pausing submission, +[Media constraint](../architecture/tape.md#tape-capacity). Periodically pausing submission, draining buffers, synchronizing an Index and resuming is outside this scope, including per-batch or increasingly frequent synchronization near the capacity boundary. Normal finalization when ending use of the current Tape remains necessary. @@ -38,9 +37,7 @@ that a deployed binary contains these paths. HP LTO-5 and IBM LTO manuals below own products and revisions; support and units must be checked before applying one vendor's fields to another. Community observations retain their reported hardware and software scope. -Current YATM checks available space through ACP at the actual start of each positive-size linear -output. Its finalization and Index reconciliation establish which staged results may be published; -they do not perform a content readback. See the [Media contract](../architecture/media-io.md#tape-capacity), +For how YATM currently checks LTFS space, finalizes and reconciles the Index, see the [Tape contract](../architecture/tape.md#tape-capacity), [Tape finalization](../../internal/executor/media_tape.go) and [Archive result handling](../../internal/executor/archive/stream.go). @@ -58,6 +55,13 @@ The path under examination is: `drive counters -> LTFS partition/reserve handling -> statfs -> ACP admission` +Advertised native capacity is nominal, not an exact count of available host bytes. IBM labels its +[drive capacity table](https://www.ibm.com/support/pages/tape-drive-model-characteristics-lto-and-3592-drives) +as nominal without compression. The HP manual below explicitly allows more data than its calculated +capacity, while tape defects and block packing can reduce usable capacity. A reported capacity, +advertised cartridge capacity, accepted WRITE bytes and published file payload have distinct meanings; +one must not be treated as an exact upper bound on another. + The [HP LTO-5 integration manual, page 14](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=14) describes native user-data estimates in units of 1,048,576 bytes, without a predicted compression multiplier. System areas and a retry allowance are already excluded. It describes conservative @@ -86,6 +90,68 @@ availability, and clamps availability after warning states. Reading the raw driv would omit these adjustments. The compatibility case below explains why changing all drives to page `0x17` is insufficient. +### Capacity Query Errors and Retries + +At the pinned revision, [LTFS capacity collection](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs.c#L686-L751) +returns a failed query before updating its cached capacity. That path does not itself set the +data-write error flag. The cache is used when no device is attached, rather than as a successful +response to an attached device's timeout. Capacity and data commands also share serialization; +this does not establish the behavior of concurrent native queries and writes. + +The [SG command helper](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/tape_drivers/linux/sg/sg_scsi_tape.c#L193-L385) +classifies host/driver timeouts separately from busy responses and resets. A returned timeout on +this read-only command is not itself evidence of failed data WRITE. Selected driver suggestions +can trigger one native retry; there is no general two-second sample deadline or three-failure +capacity cutoff. The LTO timeout tables for +[HP](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/tape_drivers/hp_tape.c#L86-L103), +[IBM](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/tape_drivers/ibm_tape.c#L446-L463) and +[Quantum](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/tape_drivers/quantum_tape.c#L106-L123) +specify sixty seconds for LOG SENSE; supported-command recommendations can supersede those tables. +These are implementation defaults, not universal device timing guarantees. + +These findings support separating optional capacity samples from data-write failure and +native settlement. Keeping a debited estimate and retrying on a periodic cadence is YATM's +advisory accounting policy, not an LTFS promise about concurrent sampling or counter accuracy. +A later successful query cannot establish that a reset preserved earlier buffered writes. + +### Capacity Query Concurrency + +Linux SG supports multiple requests and separate opens have separate request state, but those +requests still use the same SCSI device queue. Removing a userspace lock does not establish +independent device execution or a bound on WRITE delay. +SG request allocation can also sleep when shared host request slots are exhausted, even on its +asynchronous path. +[Linux v6.12 SG request state](https://github.com/torvalds/linux/blob/v6.12/drivers/scsi/sg.c#L128-L164), +[shared queue and request allocation](https://github.com/torvalds/linux/blob/v6.12/drivers/scsi/sg.c#L1634-L1664). + +The HP LTO-5 host reference describes ordered untagged execution and a separate FC tagged queue; +its ABORTED COMMAND examples include overlapping commands from one host. LOG SENSE checks deferred +errors, and a non-write command can report and clear an earlier write-behind error. A second Linux +descriptor therefore needs shared attempt error handling; it is not an independent SCSI initiator. +[HP EH890-90900 Volume 3, April 2011 edition 5, pages 14, 56, 189–190](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=14). +The integration guide also advises against interspersing position/log queries with long writes. +[HP software integration, page 24](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=24). + +The same HP manual reports standard INQUIRY `BQue=0, CmdQue=1` for all drives; the untagged +ordering description alone does not establish that a SAS drive cannot queue commands. +[HP INQUIRY fields, pages 38–39](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=38). + +A read-only 2026-10-19 sysfs observation on HP Ultrium 5/Z6ED with Debian Linux +`6.12.111+deb13-amd64` and `mpt3sas` reported device `queue_depth=128`, host `can_queue=6548` +and `cmd_per_lun=128`. Retained standard INQUIRY responses from an earlier serial-polling +comparison on this drive reported `BQue=0, CmdQue=1`. Upstream mpt3sas caps an untagged device's depth at one; +these observations support the configured tagged-capability path, without proving parallel +firmware execution or exact equivalence to Debian's patched driver. Host limits describe +controller resources, not the drive's query latency. +[mpt3sas v6.12.111 queue adjustment](https://github.com/gregkh/linux/blob/v6.12.111/drivers/scsi/mpt3sas/mpt3sas_scsih.c#L1676-L1723), +[Linux v6.12 INQUIRY tag discovery](https://github.com/torvalds/linux/blob/v6.12/drivers/scsi/scsi_scan.c#L916-L920). + +These findings distinguish background execution, absence of a shared userspace I/O lock and physical +noninterference. The latter requires applicable driver/device evidence. Under concurrent querying, +a before-WRITE submission counter also cannot prove that the WRITE settled before the sample; +capacity debit ordering must account for overlapping writes. Neither descriptor isolation nor a +small whole-run throughput difference resolves counter lag, native costs or whole-member fit. + ### Estimate Accuracy and Shared Inputs The HP integration manual above describes remaining capacity as calculated from EOD, expressed as @@ -156,7 +222,7 @@ A 2026-10-09 experiment used HP LTO-5 firmware Z6ED, Linux SG LTFS 2.5.0.0 (Prel 512-KiB LTFS blocks and enabled drive compression. SET CAPACITY followed by FORMAT with `--keep-capacity` shortened the scratch cartridge. Ordinary YATM Archive supplied twenty 4-GiB incompressible files, three 512-MiB files and a final 4-GiB file. The -[maintained procedure](../operations/physical-tape-e2e.md#continuous-boundary-capture) records +[maintained procedure](../release/physical-measurements.md#continuous-boundary-capture) records application progress, raw device pages, filesystem availability, file metadata and errors on the same host clock, without recurring synchronization or a separate SG controller. @@ -203,7 +269,7 @@ does not establish a full-length error bound or demonstrate that every admitted ### Full-Capacity EW Measurements A second 2026-10-09 experiment used the same HP LTO-5/Z6ED and LTFS configuration at full -cartridge capacity. The [EW procedure](../operations/physical-tape-e2e.md#continue-to-the-drives-ew-boundary) +cartridge capacity. The [EW procedure](../release/physical-measurements.md#continue-to-the-drives-ew-boundary) streamed incompressible data into ordinary 4-GiB files until LTFS refused further output. It deliberately bypassed application admission to reach the hardware warning. This was a drive/LTFS experiment, not an Archive/Restore acceptance or a production change. There were no @@ -283,13 +349,12 @@ failed-Close/failed-cleanup path that finalization and result publication must h ### Full-Capacity YATM Admission and Restore -A 2026-10-10–11 retest used ordinary YATM Archive and Restore with ACP v0.2.5 in -`v1.0.0-alpha.60020` (commit `be76d11887882cfe93109d7f2bdd993e8dffbde7`). The HP LTO-5/Z6ED, +A 2026-10-10–11 retest used ordinary YATM Archive and Restore with ACP v0.2.5. The HP LTO-5/Z6ED, LTFS 2.5.0.0 (Prelim), 512-KiB blocks and enabled drive compression were unchanged. The scratch cartridge used its full capacity. Mount options retained `sync_type=unmount` and the existing buffer pool; no recurring forced synchronization was added. The inputs were 100 ordinary incompressible files: twenty 64-GiB files followed by eighty 1-GiB files. The -[maintained capacity acceptance](../operations/physical-tape-e2e.md#continuous-boundary-capture) +[maintained capacity acceptance](../release/physical-measurements.md#continuous-boundary-capture) recorded application results, file syscalls, LTFS capacity and same-owner SG observations. **The earlier failed-tail outcome did not recur in this configuration.** Archive completed @@ -340,141 +405,37 @@ does not establish a guarantee for an arbitrary next file or every drive. It sup LTFS's capacity adjustment and the tested ACP integration; it supplies no reason to add periodic copy pauses or replace the capacity source with a more optimistic raw counter. -### Reading Position Through SG +### Full-Cartridge Tar Archive Observation -SG is a command transport; the returned field determines what can be inferred. For the HP LTO-5 -[READ POSITION contract, pages 148-150](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=148), -the position counts logical blocks and filemarks, not physical tape length or a percentage of the -partition consumed. The EOP bit indicates an existing EW-EOM condition; it is not a configurable -warning before that boundary. The old manual and the observed idle short/extended responses mark -buffer block/byte counts as unknown (`LOCU=1`, `BYCU=1`); zero-filled counters with these flags cannot -establish that the write buffer is empty. Idle observations do not establish every firmware's -behavior during writes. -Logical position alone cannot account for physical length spent rewriting the same content. +A 2026-10-21 YATM tar Archive run used HP LTO-5 firmware Z6ED, disabled drive compression, +256-KiB variable records and the production five-second capacity observer. This raw-Tape path +does not use LTFS filesystem availability or its Index allowance. Its native write journal +distinguishes full WRITE acknowledgements from application progress and complete published members. -Reading page `0x31` through SG obtains the same underlying capacity source used by the pinned LTFS -LTO-5 backend. Bypassing LTFS does not improve that source's contract and omits its partition and -Index adjustments. +The final APPEND reached current `NO SENSE` / `EOM` / `00/02` EW with confirmed full-record +acceptance. No further data WRITE followed. A non-immediate count-one filemark also reported +confirmed completion at EW; physical finalization and cleanup succeeded. The attempt published +45,592 complete members totaling 1,524,705,599,488 payload bytes. The admitted final 4,048-MiB +member remained incomplete by 389,658,112 payload bytes and was not published. This exercises +the [tar writer's completion boundary](../../internal/media/tar_write.go), rather than assuming +that every acknowledged record belongs to a complete member. -For the inspected Linux installation, `ltfs --version` reported 2.5.0.0 (Prelim) and the configured -Tape backend was `sg`. Inspection of its installed executable and libraries confirmed the relevant -call path and branches: +The last successful page `0x0C:5 - 0x0C:4` query reported 1,754,000,000 bytes remaining; +645,922,816 bytes were acknowledged from its issue through the EW record about 4.5 seconds +later. The resulting 1,108,077,184-byte discrepancy is evidence against treating that estimate +as a whole-member fit guarantee. The [cached accounting](../../internal/media/tar_capacity.go) +charged the member's complete encoded cost before its first header and debited reservations beyond +the full-return prefix captured at query issue. No missing member debit was demonstrated; even +retaining only the preparation estimate and cumulative encoded debits would have admitted it. +Full WRITE returns do not prove that the drive's remaining-space counter already reflects them. +The capture cannot apportion the discrepancy among unreflected buffered bytes, estimation error +and physical overhead. -`statfs -> ltfs_capacity_data -> tape_get_capacity -> sg_remaining_capacity -> LOG SENSE 0x31` - -The LTO-5 branch reads parameters 1/2 for remaining partition 0/1 capacity and 3/4 for their maxima. -After backend adjustments, partition mapping and unit conversion, ordinary data availability is -the remaining data-partition capacity minus half the total Index-partition capacity, clamped at -zero. Known `NO_SPACE` or `LESS_SPACE` states also force zero. The FUSE callback assigns that result -to both `f_bfree` and `f_bavail`. These binary checks establish the inspected functions' behavior, -not the complete build's source revision. - -This availability calculation does not use page `0x0C`'s EOD/EW/current-position fields, READ -POSITION coordinates or CM EOD records, and does not separately debit the live contents of LTFS -or drive buffers. A direct page `0x31` observation therefore duplicates its drive-level input, -while page `0x0C` adds boundary and position information rather than another independently verified -remaining-capacity measurement. - -**Observed HP LTO-5 capability:** firmware Z6ED returned complete page `0x0C` responses containing -parameters `0x0004` through `0x0008`, each four bytes with control byte `0x60`. The -[2011 HP LTO-5 manual, page 60](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=60) -omits these fields; that omission does not establish their absence in later firmware. The -[2012 HP LTO-6 manual, pages 64-65](https://docs.oracle.com/cd/E38452_01/en/LTO6_Vol3_E1_D20/LTO6_Vol3_E1_D20.pdf#page=64) -defines the matching layout in megabytes. The observed relationships to page 31 support a -decimal-MB interpretation on Z6ED; the table alone is not a firmware-specific unit/update guarantee: - -| Parameter | Meaning in the newer HP definition | -| --- | --- | -| `0x0004` | Approximate native capacity from partition beginning to recorded end of data | -| `0x0005` | Approximate native capacity from partition beginning to EW | -| `0x0006` | Minimum native capacity between EW and partition end | -| `0x0007` | Approximate native capacity from partition beginning to current medium position | -| `0x0008` | Maximum native capacity currently allowed in the device buffer; not its occupied bytes | - -Read-only positioning between the two partitions changed these capacity fields consistently with -page `0x17`'s partition totals and recorded use. In both partitions, `0x0004` matched recorded use -even at logical block zero. This supports an EOD interpretation rather than a count of blocks -crossed by the current reader. In the data partition, page `0x31` remaining capacity exceeded -`0x0005 - 0x0004` by approximately the reported EW-to-end region. This supports distinguishing -the two boundaries. The pinned LTFS Index deduction was larger than that difference in the same -sample, so replacing its availability with the raw EW distance would not necessarily be safer. - -A read-only comparison at confirmed data-partition logical blocks 0, 100,000 and recorded EOD -(197,013) returned `0x0007` values of 47, 26,263 and 68,738 MB respectively. Recorded EOD -(`0x0004`) remained 68,739 MB. Thus the firmware exposes a position estimate that responds to -positioning and distinguishes the current position from recorded EOD. This is native-capacity -distance, not a measured length in metres or a verified wrap/LPOS interface. An earlier observation -at logical block zero returned 22 MB. The nonzero, differing beginning-position observations leave -the relationship between logical position, physical head position and field refresh unresolved. - -These positioning observations are read-only measurements of one firmware; the newer HP definition -is an interpretation supported by them, not an explicit HP5/Z6ED update guarantee. The -[continuous Archive measurements](#continuous-archive-measurements) add evidence of delayed capacity -updates during writes. The fields remain useful observations through the LTFS owner without forced -synchronization. Neither the position test nor the size of the EW-to-end region identifies the cause -of a previous capacity shortfall. - -A physical-position field could help observe already-consumed tape length. A usable admission rule -would additionally need its partition/warning coordinates, freshness, outstanding-write accounting -and a bound on the next file's physical consumption. The observed `0x0C` fields provide a concrete -candidate for the warning-distance part of that work, while a complete admission guarantee remains -unestablished. The [HP-authored SSC proposal](https://www.t10.org/ftp/t10/document.06/06-235r0.pdf#page=2) -also qualifies its minimum EW-to-end capacity by media condition, block size and compression; -the reported buffer allowance alone is not a worst-case bound for an arbitrary next file. - -READ POSITION itself causes no tape movement. Nevertheless, the -[HP integration guidance, page 24](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=24) -advises against interleaving READ POSITION or LOG SENSE with long writes. Any observation used during -Archive must remain coordinated through the LTFS device owner and demonstrate acceptable command -latency and streaming behavior. More frequent polling alone does not prove complete-file admission. - -### Direct Physical Coordinates - -A physical location needs both the wrap and the longitudinal position (LPOS); the same longitudinal -position occurs on successive passes across the Tape. The -[Ultrium-1 format, ECMA-319](https://ecma-international.org/wp-content/uploads/ECMA-319_1st_edition_june_2001.pdf) -describes servo-encoded longitudinal position and stored EOD coordinates. It establishes the -distinction between physical and logical position, not the later HP5 command or CM-page layout. - -Some drives expose these coordinates to the host. The -[IBM SCSI reference, printed page 166](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference_GA32-0928-05%20%28EXTERNAL%29_0.pdf#page=196) -defines physical wrap and relative LPOS in vendor-specific REQUEST SENSE data. That is evidence of -an available IBM interface, not a portable SCSI or HP5 interface. The reviewed HP5 host guide does -not document the same pair. Its attribute `0x1000` contains LPOS at the fixed LP1 reference point, -which does not track current position. - -An idle HP5/Z6ED hardware comparison tested these interfaces immediately after completed positioning -at the three data-partition locations above: - -- Fixed-format REQUEST SENSE requested 255 bytes and returned 24 at each location. Its additional - length was 16 bytes; it did not contain the later offsets used by IBM's wrap/LPOS fields. -- Descriptor-format REQUEST SENSE returned 16 bytes at each location: the stream-command and - drive-error descriptors, without physical coordinates. Both sense replies were unchanged across - the positions, including after SPACE to EOD. -- READ POSITION service action `0x01`, the legacy device-specific block form, returned ILLEGAL - REQUEST / INVALID FIELD IN CDB at all three positions. The normal short, long and extended forms - succeeded and reported logical positions. - -These observations establish the tested interfaces' behavior on this firmware. They provide no -live wrap/LPOS pair to replace the native-capacity estimate. - -HP5 also documents reading cartridge memory through -[READ BUFFER, buffer ID `0x10`, pages 139-141](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=139). -The observed drive returned an 8,160-byte buffer. Pages `0x104` and `0x10E` were version 2 and -96 bytes each, with the same bytes at all three positions. The stored EOD pages therefore did not -track the read-only head movement in this experiment. Their layout differs from the Ultrium-1 -EOD page. An inspected -[community CM decoder](https://github.com/habibkaratas/hp-tape-tools/blob/3bb25b49de1864ed6392fab61f94d17cfa9d8388/lto-cm-decoder/lto_cm_decode.py) -applies older EOD offsets and labels a wrap-section field as a wrap number. Those interpretations -are not validated for HP5. Establish the versioned layout, partition mapping, validity and update -rules before treating CM bytes as coordinates. Stored EOD can locate the recorded tail; it is not -automatically live head telemetry or evidence that all submitted writes have reached that tail. - -For capacity control, relate the data partition's written tail to its writable EW boundary. Head -position by itself can change during reads, Index work and repositioning. Verified coordinates -would improve accounting for already-consumed medium, including length lost to previous rewrites; -conversion of the remaining region into a complete-file allowance still needs the bounds described -above. This separates a promising physical observation from an unproved remaining-byte guarantee. +The 49 near-end queries spanning 240 seconds completed in 1.974–12.520 ms, without a +failed query. These native request timings do not bound shared-device interference or physical +stop/reposition activity. This one drive/firmware/workload observation establishes neither a +portable capacity error bound nor complete-member readability after reload. Publication and +physical finalization remain separate from content readback and whole-file admission guarantees. ### Close and Explicit Synchronization @@ -568,15 +529,8 @@ before safe finalization: the new file, pending writes, required metadata/Index rewrite overhead. All quantities must refer to the same partition, warning boundary and completion state. An allowance already included by the drive or LTFS must be accounted for once. -The following approaches do not establish a solution under the current requirements. Their mechanisms -and limitations remain useful research evidence; they are not a recommended implementation sequence. -Combining them does not supply the missing physical bounds or remove the recovery costs. - -| Approach evaluated | Intended result | Assessment against the requirements | -| --- | --- | --- | -| Budget whole files from an initial capacity observation | Refuse an unaffordable file before creating its output, without repeated capacity queries | Does not resolve inaccurate initial capacity or unbounded additional physical consumption during writing | -| Finalize once, then remount and verify complete files | Establish which completed files remain readable after an unexpected capacity failure | Adds a full read pass over the files being accepted, with substantial Tape time; does not prevent partial output or guarantee cleanup | -| Recover through an Index, then rebuild on other Media | Salvage content when ordinary finalization or mounting fails | Repeats copying without resolving the cause; drive-related capacity loss can recur on replacement Media, so forward progress is unproved | +The accounting below remains advisory wherever its physical bounds are unknown. Public LTFS +finalization and Index findings are separate from YATM's ordinary operation contract. ### Initial Capacity and Whole-File Accounting @@ -603,110 +557,34 @@ unproved, so using estimates in this workflow improves accounting but still give Its capacity-utilization cost must be explicit; rereading space and resuming after a forced drain would reintroduce the excluded workflow. -### Finalize Once and Verify the Readable Result +### LTFS Finalization and Index Evidence -This approach would stop new outputs and settle outstanding work on unexpected exhaustion. Attempt normal -LTFS finalization once, then reload and mount normally. Compare completed candidates with the actual -Index and verify their content against the expected signatures in sequential read order. Only the -confirmed complete files become successful Archive results; other files remain pending for another -Media choice. A temporary file that cannot be deleted stays an explicitly reported failed output. - -There is a concrete stock-LTFS basis: the pinned -[unmount path](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs.c#L2079) +The pinned [unmount path](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs.c#L2079) uses Index-writing logic that tolerates `NO_SPACE` and can attempt the Index partition after a -data-partition failure. This does not prove that finalization will succeed. Its -[FUSE callback](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/ltfs_fuse.c#L1165) -also discards finalization return values, so command exit status alone is insufficient. +data-partition failure. Its [FUSE callback](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/ltfs_fuse.c#L1165) +discards finalization return values, so command exit status alone is insufficient. YATM reconciles +staged results with its captured final Index. Neither that metadata nor these source paths prove +that failed writes can be recovered, and no automatic remount/readback phase is part of Archive. -Current YATM already reconciles staged results with a captured final Index. The proposed additional -evidence is normal remount and content verification after exhaustion; captured metadata alone is not -a readback. The extra read phase starts after this Tape's writing has ended. This candidate cannot -promise that no file starts before an unexpected EW, or that the ordinary LTFS directory is free of -failed temporary files. Those are explicit product tradeoffs, not completed research objectives. - -### Recover with an Index and Rebuild Elsewhere - -This approach would retain a separate copy of the existing Index before append and any usable final -capture. These are disk copies of available metadata, not requests for periodic tape synchronization. -If normal mounting fails, use an existing on-tape generation or supported external-XML read-only -mount to recover the files represented by that Index. Verify their bytes, combine them with retained -originals for unconfirmed new files, and Archive the required files onto other Media. With one drive, -recovery to disk precedes writing the replacement Tape. - -The pinned implementation supports an +The implementation also supports an [external Index with the actual Tape for read-only access](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs.c#L1931). -Availability must be checked through each configured adapter. An old Index only describes its own -files, and a final capture can be absent or incomplete after failure. Neither an XML file nor a -successful read-only mount proves that its referenced bytes are intact. Retained originals are -therefore still needed for new content absent from a usable Index. - -This yields a normal replacement archive only after that archive's own acceptance. It does not -repair the original Tape or reclaim its tail. If the drive causes increased rewrites, changing the -cartridge while retaining that drive does not remove the mechanism. Rebuilding is therefore a possible -salvage operation, not an established way to finish Archive after capacity loss. Stock `ltfsindextool` -validates XML; it does not install -a filtered Index on Tape. Stock `ltfsck` rolls back to an existing generation, and the pinned -[`ltfsck` path](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/utils/ltfsck.c#L1280) -can return immediately when that is already the current generation. Same-Tape destructive rollback -is separate from this read-only recovery candidate. +An old Index describes only its own files; a readable XML document or mount does not prove the +referenced bytes remain intact. Stock `ltfsindextool` validates XML rather than installing a +filtered Index. The pinned [ltfsck path](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/utils/ltfsck.c#L1280) +can return immediately when the requested generation is already current. These are public-tool +findings, not a YATM repair, automatic rollback or interrupted-write recovery workflow. ### Remaining Research Boundary -The unresolved requirement is safe completion when physical capacity proves smaller than the -application expected. An applicable pre-write bound or a demonstrated way to contain exhaustion is -still needed. The latter must account for completed-file readability, failed-tail cleanup, normal -LTFS mounting, continued Archive progress and total Tape time; it would not by itself satisfy strict -pre-write refusal. Neither path may assume recurring forced synchronization, an accurate estimate, -or that replacing the cartridge eliminates a drive-related mechanism. PEW remains an optional -device-specific capability, not the compatibility baseline. No approach above is selected by this study. +The evidence does not establish a portable bound proving every admitted whole file and its +finalization will fit. Capacity estimates remain advisory where such a bound is absent. Fixed +reserves and compression/rewrite averages cannot supply an unmeasured guarantee. PEW remains an +optional device-specific capability, not the compatibility baseline. -Fixed byte/percentage reserves, subtracting only the configured buffer size, or using past -compression/rewrite averages have no demonstrated bound covering these requirements. They can move -a failure point without establishing any of the required guarantees. - -## Evidence Required for a Solution - -Start with the applicable manuals and source contracts, including original community reports that -challenge the assumptions. Resolve the remaining questions in this order: - -1. **Writable boundary:** determine whether the available counter reaches EW or a later physical - boundary, how LTFS adjusts it, when it is valid, and whether an applicable lower bound exists. -2. **Complete-file cost:** establish which pending writes, compression/formatting overhead, - corrected rewrites and final Index work are already included and which need a separate bound. -3. **Failure containment:** account for every transition below, including cleanup and finalization - failures after successful content writes. A retained older Index needs demonstrated recovery. -4. **Streaming and compatibility:** preserve continuous data supply and supported adapters. Assess - query latency and additional stop/reposition activity alongside throughput. Candidates requiring - recurring forced drains do not meet the design constraint. Retain an experimental change only - when it provides its intended benefit. - -| Failure boundary | Required result to demonstrate | -| --- | --- | -| Capacity observation fails or is unknown | No fabricated capacity or unqualified claim that the file fits | -| Next complete file is rejected | No output for that file; previously completed files still finalize normally | -| Write or Close fails after admission | Partial output never becomes a successful Position; later files remain pending | -| Rename or temporary-file removal fails | The failure is retained alongside the original cause; completed files remain accessible | -| Index synchronization or unmount fails | No success inferred from staging or an old Index; physical uncertainty remains visible | -| Reload after a full Tape | The retained Index and completed file bytes are readable; Restore verifies content | -| Continue on another Tape | Completed work is retained and only pending files continue through the normal Media choice | - -Use native Go tests and maintained scripts for deterministic admission, error propagation and -publication checks. Simulated capacity loss can validate software decisions but cannot prove a -drive's physical consumption bound. Reuse the -[physical measurement procedure](../operations/physical-tape-e2e.md#capacity-measurement) for separately -authorized experiments, with one hardware owner and observations obtained through LTFS. Short tests -can establish a specific counter behavior before a full Archive/Restore acceptance. A separately -authorized synchronization experiment can investigate observation lag; it does not authorize a -recurring synchronization policy in production. -Optional reduced-capacity cartridges only accelerate testing after their destructive setup and -restoration are approved; they do not establish behavior for differently aged hardware. - -Final physical acceptance must exercise ordinary files through YATM Archive, then normal unload, -reload and Restore. A direct giant filler, successful `df` comparison or successful unmount cannot -substitute for the combined outcome. Counter values, logical write progress and finalization evidence -must be interpreted separately. Raw run evidence belongs in acceptance artifacts. - -The remaining gap is substantive: the reviewed evidence has not established a portable pair of -bounds that guarantees every admitted whole file and its finalization will fit. Research must either -establish those bounds for a compatible approach, or identify the exact recovery behavior and -remaining risks requiring a product decision. A better estimate alone does not complete this work. +Further qualification should establish the counter's boundary and units, measured write/formatting +costs, and the cost of querying while streaming. Use native tests for admission, error propagation +and publication; use the [physical measurement procedure](../release/physical-measurements.md#capacity-measurement) +only for separately authorized hardware questions. Reuse completed evidence for unchanged paths. +Normal Archive/EW, unload and Restore must be assessed together for the tested configuration; +a giant filler, `df` comparison or unmount exit alone cannot establish that outcome. Raw failures +remain in the evidence. No arbitrary-failure recovery matrix is required to close this study. diff --git a/docs/research/tape-read-ordering.md b/docs/research/tape-read-ordering.md new file mode 100644 index 0000000..e65b54c --- /dev/null +++ b/docs/research/tape-read-ordering.md @@ -0,0 +1,967 @@ +# Tape Read Ordering + +Status: Research reviewed on 2026-10-29 for [Issue #14](https://github.com/samuelncui/yatm/issues/14). +The [read planning contract](../architecture/read-planning.md) owns current behavior; this study +supplies evidence, method and remaining questions, not a hardware-support guarantee. Physical +measurements apply only to their stated device, firmware, format and workload. + +## Objective + +Determine when reordering a Tape read selection shortens Restore and Verify, which position and +timing inputs are trustworthy enough to plan with, and how to measure the result without +overstating it for other drives, compressed media or LTFS. + +## Software Read Ordering + +CERN's 2018 presentation distinguishes estimated layout from observed physical coordinates. +Its first approximation divides total logical blocks by the nominal wrap count on an assumed +full cartridge; the presentation explicitly warns about partial occupancy, reserved wraps, +compression variation, rate adaptation and bad areas. Its second approach samples vendor-specific +REQUEST SENSE wrap/LPOS fields during reading or writing and associates observations with segment +boundaries. This observes a current position; it is not a command mapping an arbitrary logical +block to coordinates. Buffering and the command's position definition still matter when associating +that observation with a file. +[CERN LTO study, slides 9–13](https://indico.cern.ch/event/730908/contributions/3153156/attachments/1732268/2800425/LTO-CERN-HEPiX-Oct-2018-germancancio.pdf). + +CERN's February 2021 CTA investigation estimates logical-block placement within wraps from +READ END OF WRAP POSITION (REOWP) data. Its comparison used LTO-8 cartridges, identical selected +file lists, BOT starting conditions and total seek/read time. The presentation reports benefits +for both incompressible and highly compressible fixtures, while noting interpolation's suitability +for incompressible data and the need to improve non-BOT starts. +These measurements do not establish a portable timing model or a YATM speedup guarantee. +[CERN CTA RAO investigation, slides 5–10](https://indico.cern.ch/event/876801/contributions/4196223/subcontributions/325951/attachments/2186748/3694990/CERNCTARAO-preGDB-9Feb.pdf). + +CERN CTA at `7949198909ffa0d0817aee26215f28aec79c9319` selects the next file using estimated +access from the preceding file's end. Its shortest-locate-time-first algorithm is a greedy +heuristic, not an exact solution for the minimum-cost complete route. +[CTA algorithm](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/SLTFRAOAlgorithm.cpp), +[cost features](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/CTACostHeuristic.cpp). +The implementation evaluates every remaining file at each step: quadratic cost evaluations and +linear retained file/candidate state, rather than an exact permutation search or a retained +all-pairs matrix. SQL paging can bound memory while preserving the same complete eligible set. + +The seven-term prediction is `4.29 + 6.69*wrap + 3.2*band - 6.04*landingZone + +5.22*direction + 11.32*stepBack + 0.0006192*distance`. Its indicators compare the preceding end +with the next beginning; step-back is direction-dependent and applies within the same wrap. +The 2018 presentation labels the six setup coefficients in seconds and the distance coefficient +in seconds per LPOS unit. These are time-regression coefficients fitted from LTO-7M observations, +not measurements of another drive. +[CERN LTO study, slides 14–16 and 23](https://indico.cern.ch/event/730908/contributions/3153156/attachments/1732268/2800425/LTO-CERN-HEPiX-Oct-2018-germancancio.pdf). +Calculated counterexample: a forward same-wrap transition across the landing-zone boundary with +distance 1,000 predicts `4.29 - 6.04 + 0.6192 = -1.1308` seconds. This is an invalid physical +duration, not evidence that crossing the boundary takes negative time. The presentation itself +notes that short-distance and seek-to-read behavior need better modeling. Retaining the signed +prediction for candidate ranking, constraining/refitting it or adding a measured latency floor +are different choices that require review before integration. Treating it as a nonnegative +elapsed duration or silently omitting the two troublesome terms is not justified. + +With all seven terms, same-wrap step-back can be more expensive than changing wrap; +an own-wrap neighbour can hide a cheaper candidate in an aggregated band/direction group. +Inference for the published coefficients: partitioning by wrap, landing zone and side of the origin +makes the non-distance terms constant, allowing indexed neighbours to recover a minimum signed +prediction. Fewer searches are possible by aggregating different-band/direction groups, while +excluding the origin wrap from its own band's direction group. Assigning wraps local ordinal bits +provides that exclusion: every other wrap differs in at least one bit, and the union of opposite-bit +groups contains all other wraps but not the origin. Search the origin wrap separately. This trades +additional disk memberships for fewer queries. The parameterized greedy reference used for the +measurements below uses this property with all seven terms. The [full-cartridge measurement](#uncompressed-full-cartridge-measurement) +evaluates one explicit parameter set; it does not establish portable physical inputs or defaults. +It is not an index proof for full target-transfer costs. Exact candidate search still does not make the greedy route globally +optimal. Whole-route comparison must include actual +initial position, internal extent transitions and uncertainty, and must not advance through +outputs needing no read. + +LPOS units and usable bounds are separate facts. IBM's description of the LTO-1 through LTO-5 +servo format gives one LPOS increment per 7.2 mm; that does not supply a cartridge's usable endpoints. +[IBM patent US8576510B2, Background](https://patents.google.com/patent/US8576510B2/en#description). +Mathematically, with `LPOS = min + span*u`, the continuous distance term is +`c_lpos*span*abs(u1-u2)`. A directly measured coefficient in seconds per full normalized span +therefore avoids inventing absolute LPOS bounds, while retaining the seven feature terms. Full-wrap +endpoints, direction, bands and landing-zone classification still need evidence. This is not +identical to the pinned implementation: it truncates interpolated LPOS to an integer, and its +landing-zone helper compares absolute LPOS with `(max-min)/2`, without adding `min`. Continuous +coordinates or a midpoint of `u=0.5` can change ties and classifications. The approved normalized +path uses that midpoint; [read planning contract](../architecture/read-planning.md#software-read-ordering) owns its +current implementation. Measured timing accuracy and its limits appear in the +[full-cartridge measurement](#uncompressed-full-cartridge-measurement); direct physical-coordinate +accuracy remains unmeasured. +[CTA interpolation](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/InterpolationFilePositionEstimator.cpp#L75-L91), +[landing-zone calculation](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/RAOHelpers.cpp#L40-L42). + +The distinction is observable in CTA's own nonzero-origin test fixture: minimum 2,696 and maximum +171,097 yield a coded threshold of 84,200, whereas the interval midpoint is 86,896. The test +explicitly expects the coded threshold to belong to zone one; it does not establish the physical +location of a drive's speed-transition boundary. Thus the coordinate inconsistency is definite +if the intended feature is crossing the supplied interval's midpoint, but source inspection alone +does not prove that the arithmetic midpoint is the best measured physical boundary. Changing this +classification changes a tested model input and requires evaluation, not an unannounced cleanup. +The zero-origin example in the 2018 presentation does not exhibit this offset. +[CTA fixture and landing-zone test](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/RAOTest.cpp#L184-L195). + +### Standard Geometry and Timing Priors + +LTO has shared format specifications. The alliance's January 2010 announcement identifies the +generation-5 format and mechanism/cartridge interchange specifications; the current licensing +process supplies the Format Specification through a documentation package after a confidentiality +agreement. This study has not obtained the complete U-516 text and revision, so it cannot certify +its performance clauses or conclude that no such clauses exist. +[LTO generation-5 announcement](https://www.lto.org/wp-content/uploads/2015/06/LTO-Gen_5_Spec_Release.pdf), +[LTO licensing](https://www.lto.org/licensing/licensing-process/). + +Public ECMA-319, first edition June 2001, covers Ultrium-1. It specifies tape length, LPOS units, +four data bands, serpentine wrap numbering/direction and longitudinal regions in sections 9.2.1, +11.3.2, 12.8 and 15.1. Section 15.3 explicitly allows a variable number of datasets per wrap. +Its section 10.2 speed is a magnetic-test condition, not a universal seek/rewind rate. Applying +these numerical values to LTO-5 requires generation-specific evidence. +[ECMA-319](https://ecma-international.org/wp-content/uploads/ECMA-319_1st_edition_june_2001.pdf). + +HP's LTO-5 specification supplies 80 wraps, four bands, 846 m total tape, 808 m data region and +9 m/s rewind speed; full EOT-to-BOT rewind is listed as 96 seconds with a TBC qualifier. The same +reader lists 93 seconds for LTO-4 and 78 for LTO-3, also TBC, and unload excluding rewind below +19 seconds for standard cartridges. These +give useful nominal geometry and mechanical timing priors, not actual per-cartridge block boundaries. +[HP Volume 4, June 2010, pages 31–32 and 40–41](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vol4_E4.pdf). + +The existing HP5 LOCATE regression yields 92.690 seconds per longitudinal span; this is not a +directly measured full-length REWIND duration. A post-hoc sensitivity +comparison changed only this coefficient, retaining all six other fitted terms and all 48 observations: + +| Longitudinal coefficient source | Seconds per span | RMS error on the existing 16 held-out observations | +| --- | ---: | ---: | +| Frozen measured regression | 92.690 | 3.552 s | +| HP data-region length / rewind speed | 89.778 | 4.028 s | +| HP total length / rewind speed | 94.000 | 3.394 s | +| HP full rewind specification, marked TBC | 96.000 | 3.238 s | + +The mechanical estimates are within about 3.6% of the fitted coefficient. Their similar scale +supports using specifications as initial estimates and consistency checks. A pure transport time, +complete REWIND duration and LOCATE regression slope have different timing boundaries; selecting +the lowest error after observing these results would not be independent validation. No production +coefficient was changed. Geometry sources and timing sources should remain separate inputs. + +The [read planning contract](../architecture/read-planning.md#software-read-ordering) defines the +current parameter sources. Automatic initial values are not calibrated measurements of the current +reader. Match the drive and loaded medium, and +distinguish average from full-span rewind. One full-span timing cannot identify setup, direction or +short-movement costs. Passive updates can reuse settled operations with known physical spans and +timing boundaries; an EOD-to-BOT +operation need not traverse a full longitudinal span on serpentine media. Exclude flush, unload, +queueing and recovery time before fitting motion, and validate updated estimates on separate +observations rather than repeatedly fitting and judging the same samples. + +Progressive LOCATE fitting can learn operational latency without absolute LPOS, using normalized +layout and qualified logical endpoints. It cannot identify physical position or winding speed from +timing alone: density error and a distance coefficient can compensate for one another. Collect +bounded observations during ordinary operations, separate native LOCATE from subsequent READ, and +check support for short movement, backward movement and boundaries. AUTO preferentially observes +its chosen transitions, so low residuals on those routes do not validate alternatives it never +executes. Evaluate proposed updates on later observations and apply them to subsequent attempts; +keep an admitted attempt's parameters frozen. The current native passive-learning path instead +records observations in each Job bundle and refits with equal Job weight after each Job; the coverage and +physical-identifiability limits above still apply. Separate LTFS whole-file learning is not implemented. + +### Position Sources and Coverage + +The pinned CTA interpolator finds the first reported endpoint at or beyond a file's block ID. +For adjacent endpoints `P` and `E`, it estimates progress as `(block-P)/(E-P)`, reflecting that +progress for odd-numbered wraps. Configured minimum/maximum LPOS supply the longitudinal scale; +the nominal total wrap count supplies four equal band groups. This assumes dense ordered wrap IDs, +one relevant partition, the declared alternating orientation and approximately uniform object +density within each full wrap. REOWP alone supplies none of the LPOS scale or timing coefficients. +YATM's recorded member anchors can supply real logical starts/ends instead of CTA's +`fileSize/262144` end-block estimate. +[CTA position estimator](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/InterpolationFilePositionEstimator.cpp), +[layout helpers](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/RAOHelpers.cpp). + +Inference for review: with compression disabled and nearly uniform record lengths, logical-object +progress is a stronger approximation of longitudinal progress. Prefer interpolation between each +wrap's observed boundaries to repeatedly adding one nominal blocks-per-wrap constant; the latter +accumulates boundary error. Variable record mode does not prevent uniform submitted record lengths, +but arbitrary record sizes and filemarks prevent equating every logical object with the same byte +count. Compression variation can distort progress inside a wrap even when both endpoints are known. +The February 2021 investigation explicitly describes interpolation as suitable for incompressible +files; it is not a compression-specific workaround. + +A retrospective HP LTO-5/Z6ED observation supports approximate uniformity but not a universal +constant: in the prior compression-off, 256 KiB full-write run, 77 of 78 interior adjacent REOWP +endpoint differences were 73,543–73,647 objects; the difference ending at reported wrap 32 was +37,993. This excludes the initial interval and final EOD interval. The shorter interval's physical +cause is unresolved; it must not be silently removed from the map or assumed to span the same +physical length. These historical observations do not qualify intra-wrap interpolation, the current +cartridge initialization, or the failed Archive's publication. + +The last reported endpoint may be EOD partway through a wrap. Stretching that interval across the +whole configured LPOS span is unjustified. CTA's last-wrap correction is also heuristic: for +`N` descriptors it replaces the last endpoint with `E[N-2] + (E[N-1]-E[0])/N`, using integer +division. The calculated example `[1000,1500]` becomes `[1000,1250]`, causing a formerly covered +block 1400 to exceed the corrected table. A single descriptor receives no correction and is treated +as wrap zero. These are source-level counterexamples, not hardware measurements. A replacement +must distinguish reported boundaries from inferred full-wrap spans and retain the actual logical +locator regardless of estimation quality. +[CTA factory](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/FilePositionEstimatorFactory.cpp), +[last-wrap correction](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/RAOHelpers.cpp#L12-L24). + +The production factory applies this correction before constructing the estimator. Its existing +test only checks that a short final interval is extended: endpoints +`[208310,416271,624562,633521]` become `[208310,416271,624562,730864]`. It does not test preservation +of all observed addresses. The sum omits the first wrap's length, includes the final partial +interval and divides `N-1` differences by `N`. For equal full-wrap lengths `W` and a last partial +length `p`, the inferred final length is `floor(((N-2)*W+p)/N)`, which can be smaller than `p`. +Correcting an average would still leave compression and variable object density as estimation +limitations. An estimated full-wrap length must not replace the reported EOD as an address-validity +limit; rejecting that replacement does not require rejecting all partial-wrap estimation methods. +This distinction applies with compression disabled too. A partial interval containing half the +typical complete-wrap record count should not be normalized to a complete traversal merely because +it ends at recorded EOD. Applicable complete intervals could supply an estimated denominator, with +explicit uncertainty. The current normalized adapter uses the median complete interval length, +bounded below by the observed partial length. The full-cartridge measurement includes partial-wrap +targets, but does not independently measure their physical longitudinal coordinates. +[CTA tail test](https://github.com/cern-cta/CTA/blob/7949198909ffa0d0817aee26215f28aec79c9319/taped/rao/RAOTest.cpp#L167-L173). + +Support is vendor/model/firmware-specific, not an LTO-generation switch. The reviewed documents +establish the following evidence, rather than a promise about every installed firmware: + +| Drive family | Direct wrap/LPOS through REQUEST SENSE | REOWP endpoint table | +| --- | --- | --- | +| IBM LTO-1–4 | Defined in the 2008 reference; actual sense length must cover the fields | Not established by that reference | +| IBM LTO-5–6 | Defined in the current IBM reference | Explicitly unsupported in its generation table | +| IBM LTO-7–8 | Defined | Supported in the generation table | +| IBM LTO-9–10 | Defined with an additional wrap high bit | Supported in the generation table | +| HP LTO-5 / Z6ED | The reviewed guide and tested responses provide no wrap/LPOS pair | Short and long forms observed on this firmware, although absent from the reviewed 2011 command list | +| HPE LTO-7–9 | Defined in the August 2026 guides; LTO-9 adds a wrap high field | Defined in those guides; not physically tested by this study | + +IBM sources: [2008 REQUEST SENSE, pages 194–198](https://publibfi.boulder.ibm.com/epubs/pdf/a3204509.pdf#page=212), +[2026 physical fields, section 5.2.31](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference%20GA32-0928-08%20(EXTERNAL).pdf#page=191), +[generation table A.2](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference%20GA32-0928-08%20(EXTERNAL).pdf#page=509). +HPE REQUEST SENSE definitions: +[LTO-7 edition 5](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006909en_us&page=GUID-D7147C7F-2016-0901-0922-0000000006B7.html), +[LTO-8 edition 5](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006916en_us&page=GUID-D7147C7F-2016-0901-0922-0000000006B7.html), +[LTO-9 edition 3](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00001234en_us&page=GUID-D7147C7F-2016-0901-0922-0000000006B7.html). +HP5 direct-coordinate observations and alternative interfaces belong to +[Direct Physical Coordinates](#direct-physical-coordinates). + +These sources are complementary. REOWP supports whole-selection interpolation; direct coordinates +can check its assumptions or provide sampled anchors. A direct current coordinate alone does not +locate every selected file, and moving to every file merely to discover coordinates can erase the +benefit of planning. Media specifications and user-provided parameters can supply applicable +geometry and costs, but not an unobserved cartridge's actual wrap boundaries. Standard READ POSITION +supplies logical/buffer state, not a replacement physical-coordinate pair. Physical samples, host-next +state and retained application bytes must keep their distinct meanings. + +IBM GA32-0928-08 §5.2.20 documents the A3h/1Fh/45h command, short/long response framing and +populated-wrap endpoints. The long form is a reporting form, not proof that every physical wrap +has usable geometry. The documented table describes load-time state and can become stale after +writes; successful querying does not establish freshness for an already-loaded cartridge. +[IBM LTO SCSI Reference, section 5.2.20](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference%20GA32-0928-08%20(EXTERNAL).pdf). + +HPE's [LTO-7 Host Interface Guide](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006909en_us), +part 20-STG-L7-TRMV3-ED5, edition 5 (August 2026), documents the same maintenance CDB family +and a separate response contract: + +| Form | Byte layout | +| --- | --- | +| Short, 10 bytes | 0–1: response data length `0008h`; 2–3: reserved zero; 4–9: 48-bit BLOCK NUMBER | +| Long | 0–1: response data length, excluding those two bytes; 2–3: reserved zero; 12-byte wrap descriptors from byte 4 | +| Each wrap descriptor | 0–1: WRAP NUMBER; 2–3: PARTITION; 4–5: reserved zero; 6–11: 48-bit BLOCK NUMBER | + +The short-form endpoint is the cumulative record and filemark count within the specified +partition through and including the specified wrap, not a byte offset. The short response has +no partition/wrap identifiers. A complete long response's length yields its descriptor count; +that count does not prove full physical-wrap coverage. The studied pages do not specify descriptor +ordering. HPE warns that data may be invalid while writing and primarily intends queries when not +writing, such as after loading; this does not establish a complete freshness lifecycle or make +IBM's write-invalidation rule an HPE guarantee. +[HPE command](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006909en_us&docLocale=en_US&page=GUID-D7147C7F-2016-0901-0922-000000000684.html), +[short response](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006909en_us&page=GUID-D7147C7F-2016-0901-0922-00000000067B.html), +[long response and descriptor](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006909en_us&page=GUID-D7147C7F-2016-0901-0922-00000000067E.html). + +A controlled 2026-10-16 observation on HP Ultrium5-SCSI firmware Z6ED accepted both short and +report-all CDBs with GOOD status, returning 10 and 16 bytes respectively. The raw shapes match +the HPE framing, with one long-form descriptor and an endpoint consistent with the known logical +EOD. Standard long READ POSITION was unchanged afterward; ownership, production-state and control +guards passed. This establishes acceptance and response shape for that tested firmware, not an +LTO-5 portability guarantee, full-map coverage or freshness qualification. +Unknown vendor families also need their own applicable command/response evidence. + +### Passive Write Timing + +Host WRITE timing is a possible auxiliary source of turn-around candidates. It needs no extra +position command when the existing writer already records issue/completion and supplied bytes. +It cannot by itself distinguish turn-around from buffer backpressure, supply starvation, speed +changes or error recovery. Buffering can also hide a turn-around or shift its visible pause. +HP's Data Rate Matching explicitly adjusts tape speed to host supply; the distinction between +host rate and physical media motion is therefore material. +[HPE Data Rate Matching](https://www.hpe.com/emea_europe/en/collaterals/collateral.c04123270.html), +[IBM speed matching and backhitching](https://www.ibm.com/docs/en/ts2900tapeautoloader?topic=features-speed-matching). + +HP5's Performance Data page `34h`, parameter zero, reports repositions per 100 MB and updates after +each 100 MB is physically written. It is a normalized activity measure, not a cumulative wrap-change +counter or current wrap ID. Other page fields describe buffer/native data rates, not a longitudinal +coordinate. +[HP5 Host Interface Guide, page 81](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=81). + +Evaluation should compare passive timing candidates with independently reported endpoints or +documented coordinate observations, retaining write-buffer lag, missed events and unrelated stalls. +Use distinct streaming, supply-limited, compressible and APPEND workloads before assessing a +fallback estimator. A repeated pattern in one continuous write is useful evidence of correlation; +it does not identify physical coordinates universally. Monitoring must not introduce periodic +flushes, synchronous per-file position queries or a dependency from writing to metric persistence. +Missing telemetry remains missing and never changes the recorded logical seek address. + +An offline comparison reused the uncompressed full-cartridge experiment below. Two fixed uniform +models were compared with its reported wrap boundaries. Advertised native capacity divided by +80 wraps and record size assigned 73.56% of covered logical addresses to a different wrap and +covered only 98.12% of recorded addresses. CERN's 2018 total-blocks/nominal-wraps approximation, +using the independently observed final READ POSITION rather than REOWP as input, covered all +addresses but assigned 30.29% to a different wrap. The final partial wrap and a short full interval +explain why full-cartridge occupancy does not imply equal intervals. These are logical-address +fractions on this cartridge, not an error rate measured across user files. + +A fixed detector using one-second successful WRITE-completion bins found 80 low-throughput events +against 79 internal wrap boundaries. Its threshold was half the median byte rate, with at least +two low bins, gaps of at most one second joined, and five-second edge exclusions. No REOWP boundary +was used to fit those choices. At a 2,048-object matching tolerance, all 79 boundaries matched and +one extra event remained; at 1,024 objects, 76 matched, with four extras and three misses. +Matched event midpoints lagged the boundaries by 156–1,556 records, averaging about 216 MB of +host-acknowledged data. This is consistent with buffering and one-second binning, not a measured +constant buffer size. Threshold crossings supply candidates, not wrap IDs: an estimator must find +recurring sequences and distinguish unrelated stalls, rather than count every dip. + +Recurrence should be analyzed against cumulative acknowledged WRITE bytes, not a fixed elapsed-time +period. A byte-domain study redistributed each one-second bin's elapsed time across 64 MiB byte +bins while conserving duration, retaining zero-progress time at its byte coordinate. The signal +was elapsed seconds/GiB during the observed write phase, not summed command latency. It used mean +removal, a Hann window and FFT before filtering events. LTO-5's nominal +1.5 TB and 80 wraps imply a 17.462 GiB mean as a harmonic-selection prior, not a required size +for every wrap. The selected recurrence was approximately 17.97 GiB in the whole stream and both +byte-halves; 128 MiB bins and half-bin origin shifts gave the same selected peak. The globally +strongest spectral peak was approximately 8.99 GiB, a second harmonic: taking the maximum blindly +would double the inferred transition rate. Autocorrelation and observed event intervals support +the approximately 18 GiB recurrence rather than a second event every 9 GiB. + +Native capacity is [nominal rather than a hard upper bound](tape-capacity.md#capacity-is-not-one-interchangeable-number). +This uncompressed trace contains about 1.5286 TB of GOOD-acknowledged WRITE bytes, including tar +framing, about 1.9% above the 1.5 TB label. The typical observed interval is about 2.9% above the +nominal per-wrap mean. Neither percentage is a portable tolerance. A spectral prior must allow +deviations on both sides, then use local observations to refine intervals without equating every +wrap to the cartridge mean. + +A local interval check used the first byte-half's spectral period without elapsed-time spacing or +REOWP input. For adjacent candidates less than half a period apart, it compared retaining either +against the two neighboring byte intervals, requiring each interval to support one cycle +(0.5–1.5 periods). Unsupported or tied cases remained unresolved; this is a study heuristic, not a +standard tolerance. It removed one extra event and retained 79 candidates, whose median gap was +17.974 GiB. All matched the 79 reported boundaries within the existing 512 MiB tolerance. The +remaining host-visible lag was 39–389 MiB, median 202.75 MiB; no buffer correction was fitted. +With the known initial wrap and ordered matches, the estimated boundaries misclassified about +1.12% of this cartridge's logical address length, versus 30.29% for the uniform total-blocks model. +Neither result measures errors across user files or independently observed physical LPOS. + +These rules were developed after inspecting the cartridge, so split consistency and grid checks +are post-hoc evidence, not independent qualification. The 64 MiB grid oversamples the retained +one-second observations; zero padding cannot improve independent frequency resolution. Byte +recurrence must adapt to data density: compression changes host bytes per wrap. Missed cycles, +short intervals, split events and phase changes need explicit treatment. A periodic host stall +can also create this signal; spectral and interval agreement alone cannot exclude that cause. +Applicable layout and a known starting position can constrain an estimate, but must not force an +event count or turn ambiguous evidence into a reported physical endpoint. + +A subsequent native-command replay separated completed WRITE durations from host supply gaps, +with a fixed 65,536-bin memory bound. It froze the inferred map before comparing the retained +REOWP reference: all 79 boundaries matched, with maximum displacement 923 records (230.75 MiB +at this cartridge's 256 KiB record setting). The minimum empirical endpoint allowance was 3,687 +records; excluding ambiguous boundary neighborhoods retained stable wrap classification over +90.009% of the logical address range. These are same-cartridge post-hoc results, not independent +qualification or a specified physical tolerance. A periodic supply bottleneck or continuously +slow/backhitching source can leave no usable signal, and must not produce invented endpoints. +The [read planning contract](../architecture/read-planning.md#software-read-ordering) describes current production +observation, Media map persistence and uncertainty propagation. Estimated maps affect costs; +the recorded logical locators still determine actual reads. + +### Reading Through Small Gaps + +The [tar reader](../../internal/media/tar_read.go) already reuses a retained containing record, +can read the next record without LOCATE, and has a sequential path that reads and discards +intervening members. Eligible seeks otherwise use LOCATE. These source findings establish available +mechanisms, not a measured crossover between them. + +Physical inference: a small forward gap can cost less to read through than to reposition and restart +reading. Compare the remaining gap bytes at an applicable sequential transfer rate, including +startup and wrap turnarounds, with LOCATE plus target-read startup. Credit only bytes actually +retained in the application's record buffer; nominal drive buffer capacity does not establish its +read-ahead contents. Continuing across APPEND boundaries requires known filemark counts and their +measured cost; absent boundary timing cannot justify a read-through estimate. +Arbitrarily divided variable records cannot be converted exactly using +`object difference × maximum record size`. Compression, +record mode, transfer size and the current streaming state affect the comparison; no reviewed source +establishes a universal byte cutoff. +[HP LTO-5 integration, pages 23–25](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=23), +[performance specifications, pages 39–41](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vol4_E4.pdf#page=39). +An ordering model must cost the access method execution actually chooses. Discarded gaps are not +verified selected content, and writable-capacity/EW admission rules do not decide read movement. +Measurement boundaries must also match: an Open-to-header-available observation already includes +the header READ. Treating it as seek-only time and then adding all target READ bytes counts that +READ twice. Separate settled LOCATE time from READ startup and transfer, or explicitly account +for the included work before composing a complete transition. +Likewise, a first READ's completion time includes its entire record transfer; it is not a standalone +startup constant. A continuous-read sample supplies conditional rate observations, not by itself +a reusable post-LOCATE overhead. Preserve command issue/completion times and host gaps when +testing that decomposition. A rate measured over elapsed time already includes those gaps. + +Model inference: evaluate each next access as the cheaper eligible action, continued reading or +seek followed by read startup. Continued reading includes exact extra bytes, applicable transfer +rate and nonnegative startup/turn costs; retained bytes require no additional device READ. +Closely spaced selections can then favor continuous reading through the same cost comparison, +without a selected-file percentage or an independent gap-size cutoff. A crossover distance is a +consequence of measured costs, not a universal constant. Preserve the full logical-order route as +a candidate evaluated under the same action model; nearest-next greediness alone cannot establish +the best complete route. + +This changes the model's inputs. The current `AccessCost(from, to)` receives logical positions, +not Tape-file identity or retained-record state. Arbitrarily divided variable records and PAX chains +would prevent recovering absolute member ends from object counts and payload size. That parser +capability is not itself a requirement to import external archives. The +[current writer](../../internal/media/tar_write.go) submits full records across members and only +emits a final short variable record after the normal trailer. A controlled EW stop discards the +unsubmitted buffer; an interrupted operation adds no completion record. APPEND opens another Tape file. + +Under this writer's layout, paired physical anchors give the member end as +`headerOffset-headerInRecord+(endObject-headerObject)*R+endInRecord`, where `R` is the configured +record size, using checked arithmetic within one Tape file. The containing final record's exact +length remains unknown before reading, but pricing unread records at `R` overestimates each +same-Tape-file read leg by less than one record. That byte allowance does not cover positioning +or timing-model error. Cache eligibility can use the retained record's identity; it must not treat +an estimated length as actual buffered bytes. Runtime already owns the real record length. +These source findings support evaluating a model using existing locators, without making two new +persistent endpoint fields a prerequisite. Execution must use the same action policy with actual +settled state; no action-selection API is established by this research. + +Dense whole-Tape selection can span multiple APPEND archives. The current writer terminates each +archive with one filemark, and the reader advances one logical object for each data record or +filemark. Given these qualified semantics, current Tape file `f` and host-next object `P`, a forward +target in Tape file `g` ending at exclusive record object `q` needs `g-f` filemark READs and +`q-P-(g-f)` data-record READs. A nominal-size estimate can then overestimate bytes by less than +`(g-f+1)*R`, since multiple archives can end in short records. Filemarks have zero payload but +nonzero command/transition time; their effective cost and error need observations distinct from +post-LOCATE overhead. Reuse actual boundary handling without an inventory scan. Keeping a +same-Tape-file restriction would force seeks between archives even for dense selection. + +Order selection and read-action selection are distinct. Keeping LOGICAL file order can still +benefit from the same cost-based read-through decisions; reverting to unconditional LOCATE when a +reordered route is rejected would prevent that benefit. Existing retained-record reuse does not +need a timing-model switch. + +An indexed candidate search needs a proof for the cost it actually minimizes. At a fixed applicable +transfer rate, forward read-through cost is monotone in the target's exclusive physical-record end. +Complete tar members within one Tape file are disjoint or duplicate the same stored member, so +their header order also orders those ends; the first eligible successor remains sufficient for +that branch under nonnegative cumulative startup/turn costs. Seek followed by a complete target +read is different: variable target lengths can make the nearest header more expensive than a +farther one. A retained record can serve an earlier header, but only its actual retained bytes +avoid device reads. Exact minimum selection under this richer cost needs an appropriate index +or a justified search bound. Keeping CERN's +access heuristic as a candidate generator and separately pricing complete routes is another +possible extension; it must not be described as exact greedy minimization of full read time. +Neither approach establishes globally optimal scheduling. + +The positioning-score greedy baseline used in the original comparison can split a dense selection +into separate visits even when continuing through its small gaps would be cheaper. Comparing that +route with LOGICAL alone does not search a hybrid route. The current +[planner](../architecture/read-planning.md#software-read-ordering) instead improves directed continuous +runs from LOGICAL with complete action costs. This remains a bounded heuristic, with exact small +instances and mixed-group counterexamples for verification; it does not establish a global optimum. + +For equivalent target-read work, a cost-derived continuation boundary satisfies +`gap bytes / read rate + boundary costs < LOCATE cost + post-LOCATE overhead`. LOCATE itself +depends on the gap and layout, so even this simplified boundary is not `rate * mean seek time`. +Full action quotes remain necessary when cache or final state differs. At the measured 144.18 MB/s, +the held-out 2.86-second mean absolute positioning error equals about 412 MB of sequential reading; +the 9.12-second maximum equals about 1.32 GB. These equivalents are not error bounds on a fitted +gap threshold. Near the crossover, validation must measure action-choice mistakes and their time +penalty, not only positioning RMS error. + +A cheap forward link does not prove global adjacency: an incoming cursor already at the second +file can make a forced visit to the first file worse. Complete stateful quotes must include reading, +cache state and required completion; reversal is never free. The +[read planning contract](../architecture/read-planning.md#software-read-ordering) defines the implemented heuristic +and its planning budget; neither local improvement nor a separated estimate proves a global +optimum or physical speedup. + +LTFS at `4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb` reuses its last containing record when cached, +issues READ directly when the target is its current next record, and otherwise calls its seek path. +The studied Linux SG backend submits LOCATE(16); a forward gap is not automatically read and +discarded. The drive may satisfy a LOCATE through firmware/cache behavior, but command submission +does not establish which mechanical movement occurred. The studied implementation has no +cost-based small-gap discard choice. +LTFS must own any additional read-through operation; an application cannot obtain that benefit by +issuing competing raw SG commands while LTFS is mounted. +[LTFS raw reads](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs_fsops_raw.c#L630). +Its `ltfs.indexLocation` attribute reports the stored Index start, not the current head or logical +cursor. The studied xattr handlers expose block size, partition roles and medium identity, but no +live cursor or wrap geometry. They therefore cannot supply a mounted initial-position observation. +[LTFS xattrs](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/xattr.c#L709), +[IBM Storage Archive SDE 2.4.8 virtual attributes](https://www.ibm.com/docs/en/storage-archive-sde/2.4.8?topic=attributes-virtual-extended). + +That limitation does not prevent application-level file ordering. YATM already freezes every LTFS +extent and has a complete-selection planner that accepts native-path sources. Its LTFS provider +forecasts each file's extents in logical file-offset order, generates a derived execution order and +submits those same native paths to ACP; LTFS performs ordinary reads. Neither a new ACP scheduler +nor an LTFS sorting implementation is required. The mounted `ltfs.volumeBlocksize` attribute supplies +block size, while `ltfs.partitionMap` supplies index/data roles, not the complete numerical physical +partition mapping or wrap layout. A geometry source must keep that distinction. + +An unknown mounted cursor need not prevent comparing routes: retain the same first file in both, +cancel its common initial access/read cost, and compare the remaining routes from its forecast +final extent. This is an inferred planning state, not an observed head or cache; the existing +observed-state interface must not silently claim otherwise. Fragmented files retain all their +internal transitions. Cache or read-through actions specific to the raw tar reader cannot be +credited to ordinary LTFS reads without corresponding behavior. The current provider normalizes +extents, requires compatible partition geometry and fixes that common first file. Pre-mount paired +Labels can establish physical partition mapping; mounted role letters alone cannot. The +[read planning contract](../architecture/read-planning.md#software-read-ordering) defines those boundaries. + +HPE's LTO-7 unload description specifies rewind, unthread and eject. A shared destination is not +evidence of equal travel time from different final physical positions. The studied LTFS version's +`ltfs_unmount` can also write an Index when dirty or when its current Index is not on the Index +partition; it skips that write for a clean Index already there. Consequently, a terminal estimate +must follow the actual adapter and reader policy, rather than treating every completion as an +identical LOCATE to block zero. These sources establish lifecycle behavior, not a measured terminal +cost function. Native tar now unloads after successful normal Archive, Restore and Scan completion, +and its planner includes a separate fixed-plus-physical-distance terminal estimate. Built-in values +borrow the provisional HP references above; they are not measured bounds. Successful ordinary UNLOAD +commands supply separate passive learning observations. The current LTFS read model assumes no Index rewrite and prices only the +direct rewind/unload from the final forecast position. That assumption matches reading a cartridge +kept read-only after writing; it is not inferred from a mounted dirty flag or a drive write-protect +observation. Writable or unknown mounts use the same assumption. Source-cache refresh may change +LTFS metadata, and configured adapters retain their own completion behavior; a resulting Index +rewrite is outside this estimate. Actual data-partition EOD remains a coverage boundary, and its +previous DP Index start is not the append/EOD address. No read of `ltfs.sync` is used, because +the studied GET handler itself syncs the Index. Source analysis and software tests establish these +comparison boundaries; the retained HPE 3.5 controls do not certify the assumption for every mount +or prove a physical benefit from the later implementation. +[HPE LTO-7 unload scenarios](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006907en_us&docLocale=en_US&page=GUID-108EFC4D-4940-4DC7-9FEB-DBE3C0A0BC86.html), +[LTFS unmount](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs.c#L1938-L2036). +[DP-first Index writing](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/ltfs.c#L2522-L2536), +[append-position handling](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/tape.c#L794-L865), +[Index pointers](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/xattr.c#L728-L731), +[sync GET](https://github.com/LinearTapeFileSystem/ltfs/blob/4c0e6faae661e0bfd50bc9707f59ecd88c90b7eb/src/libltfs/xattr.c#L883-L885). + +Physical evaluation compares complete matched Restore selections using observed initial logical +positions and actual READ/LOCATE choices. The uncompressed measurement below includes gaps and wrap +crossings; compression differences and other devices remain unqualified. Applicable measured +transfer and seek/startup intervals supply the crossover; a guessed threshold is not acceptance +evidence. + +Logical addresses remain authoritative seek inputs. Wrap direction, band membership, endpoint +conventions, full-wrap coverage and motion costs need separate evidence. A non-ejecting reload and +new reported-map observation do not establish physical freshness. A performance estimate must +preserve that limitation without substituting estimated coordinates for authoritative logical +addresses. Session ownership and invalidation must preserve the distinction between an observed +logical map and qualified physical geometry. A borrowed timing prior permits comparison but does not establish physical qualification. Complete routes +must account for uncertainty and actual source boundaries; nominal capacity and guessed parity +are insufficient. The [read planning contract](../architecture/read-planning.md#software-read-ordering) +defines the current parameter admission, scheduler and fallback behavior. + +### Search Objectives and Established Optimization Methods + +This review studies Helsgaun's 2009 LKH paper, Cook/Held/Helsgaun's LKH-AMZ work, Bertsimas/Sim +(2004), Ganesh/Maggs/Panigrahi (2020), and the optimization-benchmarking survey v2 (2020). +Sources were checked on 2026-10-23. These findings evaluate alternatives; they do not change the +[current admission policy](../architecture/read-planning.md#software-read-ordering). + +**Sourced findings.** LKH limits proposed edges with small candidate sets and performs compound +exchanges. Its study also shows that deeper search and apparently better candidate coverage can +waste work on clustered inputs; larger neighbourhoods are not automatically better. +[Helsgaun, General k-opt submoves (2009)](https://www2.seas.gwu.edu/~simhaweb/champalg/tsp/papers/Helsgaun2.pdf). +LKH-AMZ uses directed 3/4-opt exchanges, preserves segment orientation and applies iterated local +search. Its point-to-point costs are static. The authors distribute that particular implementation +under MIT; this does not establish the licence of every LKH version. +[Author paper](https://www.math.uwaterloo.ca/~bico/papers/jptreport.pdf), +[implementation and licence](https://www.math.uwaterloo.ca/tsp/amz/code.html). + +Robust optimization makes the uncertainty set and performance criterion explicit. Bertsimas/Sim +study controlling conservatism, including shared uncertainty factors affecting many coefficients. +They do not prescribe requiring every new route to dominate the previously accepted route. +[The Price of Robustness, especially section 5](https://www.mit.edu/~dbertsim/papers/melvyn/The-Price-Of-Robustness-OR52.pdf). +Minimax regret instead minimizes the worst difference from the optimal route for each scenario. +It adds computational difficulty; the cited TSP guarantees assume undirected nonnegative graphs, +so they do not transfer to YATM's directed stateful costs. +[Robust Algorithms for TSP and Steiner Tree (2020)](https://arxiv.org/abs/2005.08137). + +Benchmark guidance distinguishes fixed-budget quality, time to a target quality, and the whole +quality-versus-time curve. It also distinguishes evaluation counts from elapsed time and requires +the actual parameter configuration to be verified. +[Benchmarking in Optimization, v2, sections 4–5](https://arxiv.org/html/2007.03488v2). +CERN's 2021 comparison similarly uses the same selected files for each ordering and measures +both positioning and reading, rather than only predicted seek scores. +[CTA investigation, slide 7](https://indico.cern.ch/event/876801/contributions/4196223/subcontributions/325951/attachments/2186748/3694990/CERNCTARAO-preGDB-9Feb.pdf). + +**Application inferences and unresolved evidence.** Historical-route dominance is a conservative +engineering policy, not an established optimum criterion: an early accepted route can prevent a +later nominal improvement with mixed scenario costs. Explaining a rejection by that rule proves +implementation consistency, not that the tradeoff is desirable. A fixed risk objective or fixed +reference envelope would remove this dependence from admission, but its benefit must be compared +with the approved policy before replacement. Six one-factor stress checks also do not establish +protection against simultaneous coefficient errors or a statistical confidence level. + +The comparison's quoted scope matters when choosing that objective. Common unquoted LTFS work +cancels from scenario-wise differences, but not from ratios to LOGICAL: adding the same time to +two routes changes their ratios and can change the worst-scenario ranking. A ratio therefore +describes the quoted comparison costs, not the complete Restore workflow. The fixed-reference +alternative `max_s(C_s(route) - C_s(LOGICAL))` compares the least saving in seconds and is invariant +to genuinely route-independent additive terms within each scenario. It is not minimax regret +against an unknown optimum. Route-dependent completion branches still require the complete +provider evaluation; they cannot be dropped as common work. Either objective needs explicit +review of its per-scenario tradeoffs before replacing production admission. + +Restore planning is measured by the Go scenario benchmarks and quality tests in +[Performance Acceptance](../development/performance.md#performance-acceptance). +A static nominal optimum would not prove robust or physical optimality. + +For further search changes, separately measure candidate generation, evaluator calls, allocation, +full replay and total planning. Compare time to reach the current route quality as well as quality +at the same budget, retaining the fixed development/acceptance inputs and adverse cases. Candidate +lists and bounded directed exchanges are supported techniques to investigate; geometry-only +neighbours, free reversal, and assumptions that a faster solver fixes an unsuitable objective are not. + +### Uncompressed Full-Cartridge Measurement + +A 2026-10-21 measurement of the normalized YATM development implementation used an HP Ultrium +LTO-5 drive with Z6ED firmware, compression and encryption off, and 256 KiB variable records. +Three Archive Jobs published 45,846 complete members, about 1.525 TB of payload. EW stopped the last +append and its incomplete tail remained unpublished. The observed map contained 80 wraps, including +a short complete interval and a partial final interval; neither was removed from the model. +This study measures the frozen experimental build's full-cartridge write and selected Restores, +not a full-cartridge readback or later optimizer revisions. + +The seven-term fit used 32 directed positioning trials; 16 separate held-out trials were not +used for fitting. Training mean absolute/RMS errors were 1.53/1.79 seconds; held-out mean +absolute/RMS errors were 2.86/3.55 seconds, with maximum absolute error 9.12 seconds. +Four held-out errors exceeded the 3.31-second training residual allowance, and only one training +trial exercised step-back. Thus the model is an approximation and its supplied allowance is not +a guaranteed error bound. Post-hoc checks of the unchanged model on the first sparse routes +overpredicted LOGICAL's total LOCATE time by 5.68% and underpredicted AUTO's by 4.03%; these routes +do not independently identify all seven coefficients. + +The worst held-out case moved backward four logical objects: native LOCATE took 0.072 seconds, +while the model predicted 9.194 seconds. Drive cache and prefetch were not controlled, so this +does not identify the physical cause. The fitted base term is 3.224 seconds, but setup indicators +and a linear distance term do not model acceleration, deceleration, current velocity or a separate +short-seek regime. They can absorb effects empirically without identifying mechanical constants. +The declared direction feature compares wrap orientations, not an observed transport velocity. +Native LOCATE timing excludes the subsequent READ; its startup and transfer costs are measured +separately. Aggregate route agreement therefore does not establish accurate short-gap decisions +or a confidence bound for choosing between nearly equal alternatives. + +A chronological replay reserved the last eight Restore Jobs for comparison. Their 70 LOCATEs +have 3.507-second mean and 9.179-second maximum absolute error; the original allowance covers only +38 of them. First-READ mean/max errors are 2.359/7.768 milliseconds, separating this discrepancy +from READ startup. A diagnostic same-wrap segment within four records fits one earlier forward +two-record observation. It reduces later overall mean error to 3.147 seconds, but leaves the maximum +unchanged. Earlier same-wrap training distances have no observations between two and 16,491 records; +the breakpoint and short backward regime are therefore unidentified. The candidate is not adopted. +Normal passive observations at varied short distances and unused later Jobs are needed before +qualifying that additional model; a better narrow mean alone is insufficient. + +Each matched selection ran LOGICAL, AUTO, AUTO, LOGICAL with frozen parameters and fresh +destinations. All 16 Restore Jobs and 2,028 repeated file content/metadata checks passed. +Elapsed times below run from Media admission through Job settlement, including preparation; +Job indexing and independent destination verification are outside that interval. + +| Selection | Files | LOGICAL observations | Requested AUTO observations | Observed route | +| --- | ---: | ---: | ---: | --- | +| Sparse across wraps | 80 | 3,208 / 3,201 s | 1,042 / 975 s | Reordered; mean elapsed time fell 68.5%, native LOCATE time 70.1% | +| Adjacent members | 381 | 262 / 194 s | 193 / 193 s | Both AUTO requests retained LOGICAL; one LOCATE per arm | +| Wrap/band boundaries | 30 | 566 / 658 s | 314 / 329 s | Reordered; mean elapsed time fell 47.4%, native LOCATE time 59.7% | +| APPEND boundaries | 16 | 92 / 9 s | 9 / 9 s | Both AUTO requests retained LOGICAL; two LOCATE commands per arm | + +The two fallback cases had an active model but no estimated improvement separated by its +uncertainty allowance. Their first LOGICAL runs had much longer label preparation; their elapsed +differences are not sorting gains. Boundary AUTO increased LOCATE count from 13 to 18 while +reducing total LOCATE time, demonstrating why command count alone is an unsuitable objective. +The adjacent case read one contiguous record span with retained-record reuse. A boundary LOGICAL +route also read through an approximately 12.5 MiB gap between selected members, while AUTO avoided +that segment. APPEND boundaries still used LOCATE; these trials do not validate sequential +filemark crossing. Runtime cache/discard counters were not captured, so gap and reuse evidence +comes from selected anchors, serialized native actions and the corresponding reader behavior. + +Restore process RSS sampled every five seconds peaked at 56.2 MiB. Separate disk-backed planning +observations used about 37 MiB RSS for both 100,000 and 1,000,000 selections, but took 13.6 and +236 seconds with 105 MB and 1.059 GB temporary indexes. Cumulative allocation was 4.31 and +54.83 GB respectively. These single observations used different wrap occupancy; they do not +establish a general scaling law, and RSS excludes filesystem cache. + +Final endpoint/selection identities depended on the post-write map and were frozen before their +timing trials, rather than before writing. No held-out refitting or observation removal was used. +Two repetitions on one cartridge do not control internal drive cache or prove physical LPOS, +layout assumptions, portable coefficients, compressed-media behavior or LTFS benefit. The observed +uncompressed benefit supports this implementation's tested use; broader cost-model, prospective +cartridge and LTFS qualification remain open. Passive read learning does not require user measurement. + +## Fixed Half-Cartridge Selected Restore + +A later HP LTO-5/Z6ED experiment used 512 KiB variable records with hardware compression and +hardware encryption off. A fixed prefix of the mixed-size 1.5 TB benchmark, plus its small boundary +members, wrote 1,444 files / 699,841,147,558 payload bytes through FORMAT and two APPENDs. The current +single-partition CM directory covered 37 wraps. One frozen selection contained 72 files / +23,946,906,636 bytes: 4.986% of written files and 3.422% of written bytes, touching 32 recorded wraps. + +Each order ran once with fresh destinations, the same frozen model and logical locators, and a +fresh CM directory. REOWP was suppressed only by the test transport; CM acquisition and all Tape +reads were real. The generated execution indexes and normalized geometry were retained, and both +Restores passed inline and independent local content/metadata checks with normal unload. + +| Measure | LOGICAL | AUTO | +| --- | ---: | ---: | +| Complete model estimate, including finish | 2,281.95 s | 890.37 s | +| Actual execution after preparation/planning | 2,521.35 s | 930.81 s | +| Media admission through settlement | 2,527.61 s | 936.10 s | +| Planning | 0 s | 1.17 s | +| Native LOCATE count / combined service time | 56 / 2,217.67 s | 58 / 673.03 s | +| Host-returned READ bytes, including framing and read-through | 31,300,523,008 | 29,343,880,192 | +| Independent local verification | 57.40 s | 61.50 s | + +AUTO reduced this pair's Media elapsed time by 62.96%, despite making two more LOCATE calls. Its +shorter positioning work accounts for most of the difference. Complete execution exceeded the +model by 10.49% for LOGICAL and 4.54% for AUTO; observed READ command counts matched both forecasts. +Actual cache-hit and discarded-byte counters were not exposed. READ reservations and returned +bytes are not selected payload or a measurement of physical tape consumption. The two orders each +read the selection, so this is not a claim that total experiment I/O was 5% of the cartridge. +This one pair does not establish statistical confidence, global optimality or behavior on other +drives, compressed media or unwritten wraps. No model refit or repeated search selected a faster +reported result. + +## Reading Position Through SG + +SG is a command transport; the returned field determines what can be inferred. For the HP LTO-5 +[READ POSITION contract, pages 148-150](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=148), +the position counts logical blocks and filemarks, not physical tape length or a percentage of the +partition consumed. The EOP bit indicates an existing EW-EOM condition; it is not a configurable +warning before that boundary. The old manual and the observed idle short/extended responses mark +buffer block/byte counts as unknown (`LOCU=1`, `BYCU=1`); zero-filled counters with these flags cannot +establish that the write buffer is empty. Idle observations do not establish every firmware's +behavior during writes. +Logical position alone cannot account for physical length spent rewriting the same content. + +Reading page `0x31` through SG obtains the same underlying capacity source used by the pinned LTFS +LTO-5 backend. Bypassing LTFS does not improve that source's contract and omits its partition and +Index adjustments. + +For the inspected Linux installation, `ltfs --version` reported 2.5.0.0 (Prelim) and the configured +Tape backend was `sg`. Inspection of its installed executable and libraries confirmed the relevant +call path and branches: + +`statfs -> ltfs_capacity_data -> tape_get_capacity -> sg_remaining_capacity -> LOG SENSE 0x31` + +The LTO-5 branch reads parameters 1/2 for remaining partition 0/1 capacity and 3/4 for their maxima. +After backend adjustments, partition mapping and unit conversion, ordinary data availability is +the remaining data-partition capacity minus half the total Index-partition capacity, clamped at +zero. Known `NO_SPACE` or `LESS_SPACE` states also force zero. The FUSE callback assigns that result +to both `f_bfree` and `f_bavail`. These binary checks establish the inspected functions' behavior, +not the complete build's source revision. + +This availability calculation does not use page `0x0C`'s EOD/EW/current-position fields, READ +POSITION coordinates or CM EOD records, and does not separately debit the live contents of LTFS +or drive buffers. A direct page `0x31` observation therefore duplicates its drive-level input, +while page `0x0C` adds boundary and position information rather than another independently verified +remaining-capacity measurement. + +The later [SSC-3 draft revision 03e, Section 8.2.2, Table 58](https://citeseerx.ist.psu.edu/document?doi=baaf4ada4b6d66dd63c05f08761157577597d52d&repid=rep1&type=pdf) +defines parameters 4 and 5 as current-partition recorded-EOD/EW distances in decimal megabytes, +with all-ones invalid values and no guarantee about writable bytes before EW. Revision 01c predates +these fields. The matching four-byte HP/IBM layout can supply an estimate without a generation +allowlist; parameter presence does not establish error/lag bounds or streaming-query suitability. + +**Observed HP LTO-5 capability:** firmware Z6ED returned complete page `0x0C` responses containing +parameters `0x0004` through `0x0008`, each four bytes with control byte `0x60`. The +[2011 HP LTO-5 manual, page 60](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=60) +omits these fields; that omission does not establish their absence in later firmware. The +[2012 HP LTO-6 manual, pages 64-65](https://docs.oracle.com/cd/E38452_01/en/LTO6_Vol3_E1_D20/LTO6_Vol3_E1_D20.pdf#page=64) +defines the matching layout in megabytes. The observed relationships to page 31 support a +decimal-MB interpretation on Z6ED; the table alone is not a firmware-specific unit/update guarantee: + +| Parameter | Meaning in the newer HP definition | +| --- | --- | +| `0x0004` | Approximate native capacity from partition beginning to recorded end of data | +| `0x0005` | Approximate native capacity from partition beginning to EW | +| `0x0006` | Minimum native capacity between EW and partition end | +| `0x0007` | Approximate native capacity from partition beginning to current medium position | +| `0x0008` | Maximum native capacity currently allowed in the device buffer; not its occupied bytes | + +Read-only positioning between the two partitions changed these capacity fields consistently with +page `0x17`'s partition totals and recorded use. In both partitions, `0x0004` matched recorded use +even at logical block zero. This supports an EOD interpretation rather than a count of blocks +crossed by the current reader. In the data partition, page `0x31` remaining capacity exceeded +`0x0005 - 0x0004` by approximately the reported EW-to-end region. This supports distinguishing +the two boundaries. The pinned LTFS Index deduction was larger than that difference in the same +sample, so replacing its availability with the raw EW distance would not necessarily be safer. + +A read-only comparison at confirmed data-partition logical blocks 0, 100,000 and recorded EOD +(197,013) returned `0x0007` values of 47, 26,263 and 68,738 MB respectively. Recorded EOD +(`0x0004`) remained 68,739 MB. Thus the firmware exposes a position estimate that responds to +positioning and distinguishes the current position from recorded EOD. This is native-capacity +distance, not a measured length in metres or a verified wrap/LPOS interface. An earlier observation +at logical block zero returned 22 MB. The nonzero, differing beginning-position observations leave +the relationship between logical position, physical head position and field refresh unresolved. + +These positioning observations are read-only measurements of one firmware; the newer HP definition +is an interpretation supported by them, not an explicit HP5/Z6ED update guarantee. The +[continuous Archive measurements](tape-capacity.md#continuous-archive-measurements) add evidence of delayed capacity +updates during writes. The fields remain useful observations through the LTFS owner without forced +synchronization. Neither the position test nor the size of the EW-to-end region identifies the cause +of a previous capacity shortfall. + +A physical-position field could help observe already-consumed tape length. A usable admission rule +would additionally need its partition/warning coordinates, freshness, outstanding-write accounting +and a bound on the next file's physical consumption. The observed `0x0C` fields provide a concrete +candidate for the warning-distance part of that work, while a complete admission guarantee remains +unestablished. The [HP-authored SSC proposal](https://www.t10.org/ftp/t10/document.06/06-235r0.pdf#page=2) +also qualifies its minimum EW-to-end capacity by media condition, block size and compression; +the reported buffer allowance alone is not a worst-case bound for an arbitrary next file. + +READ POSITION itself causes no tape movement. Nevertheless, the +[HP integration guidance, page 24](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol2_E3a/LTO5_Vol2_E3a.pdf#page=24) +advises against interleaving READ POSITION or LOG SENSE with long writes. Any observation used during +Archive must remain coordinated through the LTFS device owner and demonstrate acceptable command +latency and streaming behavior. More frequent polling alone does not prove complete-file admission. + +## Direct Physical Coordinates + +A physical location needs both the wrap and the longitudinal position (LPOS); the same longitudinal +position occurs on successive passes across the Tape. The +[Ultrium-1 format, ECMA-319](https://ecma-international.org/wp-content/uploads/ECMA-319_1st_edition_june_2001.pdf) +describes servo-encoded longitudinal position and stored EOD coordinates. It establishes the +distinction between physical and logical position, not the later HP5 command or CM-page layout. + +Some drives expose these coordinates to the host. The +[IBM SCSI reference, printed page 166](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference_GA32-0928-05%20%28EXTERNAL%29_0.pdf#page=196) +defines physical wrap and relative LPOS in vendor-specific REQUEST SENSE data. That is evidence of +an available IBM interface, not a portable SCSI or HP5 interface. The reviewed HP5 host guide does +not document the same pair. Its attribute `0x1000` contains LPOS at the fixed LP1 reference point, +which does not track current position. + +An idle HP5/Z6ED hardware comparison tested these interfaces immediately after completed positioning +at the three data-partition locations above: + +- Fixed-format REQUEST SENSE requested 255 bytes and returned 24 at each location. Its additional + length was 16 bytes; it did not contain the later offsets used by IBM's wrap/LPOS fields. +- Descriptor-format REQUEST SENSE returned 16 bytes at each location: the stream-command and + drive-error descriptors, without physical coordinates. Both sense replies were unchanged across + the positions, including after SPACE to EOD. +- READ POSITION service action `0x01`, the legacy device-specific block form, returned ILLEGAL + REQUEST / INVALID FIELD IN CDB at all three positions. The normal short, long and extended forms + succeeded and reported logical positions. + +These observations establish the tested interfaces' behavior on this firmware. They provide no +live wrap/LPOS pair to replace the native-capacity estimate. + +HP5 also documents reading cartridge memory through +[READ BUFFER, buffer ID `0x10`, pages 139-141](https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol3_E5b/LTO5_Vol3_E5b.pdf#page=139). +The observed drive returned an 8,160-byte buffer. Pages `0x104` and `0x10E` were version 2 and +96 bytes each, with the same bytes at all three positions. The stored EOD pages therefore did not +track the read-only head movement in this experiment. Their layout differs from the Ultrium-1 +EOD page. An inspected +[community CM decoder](https://github.com/habibkaratas/hp-tape-tools/blob/3bb25b49de1864ed6392fab61f94d17cfa9d8388/lto-cm-decoder/lto_cm_decode.py) +applies older EOD offsets and labels a wrap-section field as a wrap number. Those interpretations +are not validated for HP5. Establish the versioned layout, partition mapping, validity and update +rules before treating CM bytes as coordinates. Stored EOD can locate the recorded tail; it is not +automatically live head telemetry or evidence that all submitted writes have reached that tail. + +The production fallback is restricted to the studied LTO-5 directory version below; it does not +implement every generation's raw CM format. ECMA-319 Annex D.2.7.5 defines the +**Ultrium-1** directory: page `0x0103`, 96 half-wrap entries, record/filemark counts and explicit +invalidation/update rules. Its four-byte integrity check is the Reed-Solomon code defined in +section 13.2, not ordinary CRC32. The reviewed HP5 guide documents retrieval without specifying +its later directory layout. IBM's +[ITDT change log](https://www3.software.ibm.com/storage/ITDT2/ListOfChanges.html) records fixes for +missing LTO2-5 directory high bytes and partitioned cartridges; this warns against reusing the +LTO1 offsets without version-specific evidence. + +A 2026-10-24 read-only HP LTO-5/Z6ED investigation obtained matching raw CM and REOWP from one +previously written, two-partition LTFS cartridge. The cartridge was physically present but +unloaded: TEST UNIT READY and READ MEDIA SERIAL NUMBER returned `3A/00`, while the CM descriptor +returned GOOD with zero capacity. A separately requested synchronous LOAD completed in 35.57 s; +thereafter the descriptor reported 8,160 bytes. Thus these unloaded responses alone cannot prove +physical cartridge absence or unsupported CM retrieval. No FORMAT, data WRITE, Tape payload READ, +LOCATE or unload was issued in this investigation. + +Two CM reads took approximately 1.28 s each and returned identical 8,160-byte images. READ +POSITION before and after showed the same partition-zero, logical-object-zero position. Current +mode data reported density `0x58` and hardware compression disabled; MAM reported two partitions. +These are current observations, not a claim about every historical write to the cartridge. + +The actual directory had version 4 and length 2,576: a 16-byte header followed by 80 32-byte units. +Its header and all 80 units passed the RS check (12+4 and 28+4 bytes respectively). Both version-2 +EOD pages, `0x0104` and `0x010e`, were 96 bytes and passed their trailing RS checks. The protected +page table passed; the unprotected table passed when its leading `0x0ffc` padding descriptor was +excluded. Including that descriptor, as in the older table-coverage assumption, failed. This +observed coverage rule is not a blanket rule to skip arbitrary padding in every CM version. + +For this sample, the sum of the four big-endian 32-bit words at unit offsets `+8`, `+12`, `+16` +and `+20`, accumulated within each partition, reproduced **all 23 REOWP endpoints exactly**. +Grouping the first two as record counts and the last two as filemark counts also reproduced the +respective 48-bit counts at EOD-page offsets `+12` and `+18`: + +| Observed partition | First / last recorded wrap | Record-count candidate | Filemark-count candidate | Host logical EOD | +| --- | --- | ---: | ---: | ---: | +| 0 | 0 / 0 | 10 | 4 | 14 | +| 1 | 4 / 25 | 786,456 | 12 | 786,468 | + +The bounded decoder takes CM plus a confirmed actual partition count. It finds maximal contiguous runs whose dataset +fields are neither `FFFFFFFF` nor `FFFFFFFE`. Each whole run must match one EOD page's terminal +wrap, write pass and dataset, with EOD validity `0001`. Pages `0x104` and `0x10e` correspond to the +first and second partitions in this studied form. A later single-partition tar FORMAT retained +the second page with validity zero: its presence is not evidence of an active partition. The +actual partition list selects active pages; retained inactive pages still require valid framing +and checksums, but supply no coordinates. It then checks record and filemark totals independently. This +derives the observed starts without using the start flag or hardcoded wrap numbers. It cannot +select a suffix merely because its counters happen to fit. Every valid run must be accounted for. + +Under the accepted low-word interpretation, each extended count contributes a nonnegative multiple +of 2^32. Exact equality between each independently bounded run's low-word sum and its EOD total +therefore establishes zero extension contribution. It does not decode or name the packed `+24` +bits, and an unexplained wider total is rejected. Sums may exceed 2^32; no file-count threshold +substitutes for this check. Cumulative counts define exclusive object boundaries from object zero, +and the partial last wrap stops at recorded EOD. The paired HP5/Z6ED density-0x58 REOWP response +uses the same convention: the first full data-partition wrap contains 36,765 records plus filemarks, +which occupy zero-based identifiers 0 through 36,764. Its next boundary is 36,765, matching REOWP; +all 21 full-wrap endpoints and both EOD totals agree. Adding one would count an extra object. +This conclusion combines the accepted independent CM counts with HP5 logical-object numbering, +not raw equality alone. Other REOWP families retain their prior convention: HPE's +[LTO6 J38W release notes](https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_7b228de1989a4b66ac419acd74) +explicitly describe the last block of the first wrap. That statement is not a definition of HP5/Z6ED. + +These rules combine the earlier public field definitions, community decoder and observed version-4/ +version-2 correspondence. They were developed after inspecting the initial LTFS sample, so its 23 +matching endpoints are consistency evidence. A subsequent single-partition tar experiment on the +same HP5/Z6ED configuration wrote approximately 674 GB, appended 256 MiB within its partial wrap, +and then appended approximately 25.7 GB across two wrap boundaries. The 35 initial endpoints and +37 final endpoints matched independently captured REOWP; the first 34 full boundaries remained +unchanged, and record/filemark totals matched each actual EOD. The production decoder consumed all +three captures using the independently reported partition count. This qualifies the observed +normal FORMAT/APPEND transitions on that configuration, not other drives or unseen update states. +An internally coherent stale directory and every other generation's flags/update rules remain +outside this qualification. Unsupported versions, interrupted-write states and inconsistent +coverage retain ordinary reads. +The later 4 KiB split-read check returned the same image in about 1.29 s with unchanged identity and +READ POSITION; it used the production two-second per-command timeout and added no Tape motion. + +The [LTO-CM-Analyzer limitation](https://github.com/Kevin-Nakamoto/LTO-CM-Analyzer/blob/a976175dfbfeed62d913983aa5945f4c043a573b/README.md#limitations) +explicitly restricts its interpretation to LTO1. A contributor's +[LTO6-labelled report and original dumps](https://github.com/Kevin-Nakamoto/LTO-CM-Analyzer/issues/2) +have a version-5, 4,400-byte directory instead: 48 bytes of header and 136 32-byte units all pass +the same RS calculation. The generation label is not independently verified, and both supplied +8,160-byte images end before the table's declared final address. These samples support framing +observations, not another qualified device map or a reason to apply the HP5 layout to later media. + +The restricted fallback uses this format-specific interpretation and rejects unexplained states. +READ BUFFER is still mediated by the drive; it does not bypass a drive lacking that command or +guarantee that every firmware exposes its raw CM. Other validity, partition and extended-counter +forms need their own evidence. Hardware-compressed interpolation and live head telemetry remain +separate questions. + +For capacity control, relate the data partition's written tail to its writable EW boundary. Head +position by itself can change during reads, Index work and repositioning. Verified coordinates +would improve accounting for already-consumed medium, including length lost to previous rewrites; +conversion of the remaining region into a complete-file allowance still needs the bounds described +above. This separates a promising physical observation from an unproved remaining-byte guarantee. diff --git a/docs/research/tar-tape.md b/docs/research/tar-tape.md new file mode 100644 index 0000000..3974f4d --- /dev/null +++ b/docs/research/tar-tape.md @@ -0,0 +1,315 @@ +# Tar on Tape + +Status: Research reviewed on 2026-10-29 for [Issue #14](https://github.com/samuelncui/yatm/issues/14). +Sources examined include ACP `v0.2.6`'s native API, Go `1.26.8`, GNU tar `1.35` documentation +and Linux `v6.12` driver documentation. Current software behavior belongs to the +[Tape contract](../architecture/tape.md); this study supplies evidence and remaining physical +questions, not a hardware-support guarantee. Physical measurements apply only to their stated +device, firmware, format and workload. + +## Objective + +Evaluate ordinary tar archives on raw Tape for drives that cannot use LTFS, including the +LTO-4 use case. Files should remain recoverable with standard tools, while YATM retains +per-file Archive, explicit FileVersion Restore, inventory and Verify behavior. Archiving a +`.tar` file as one ordinary File does not provide that integration. Supporting an older +drive does not establish support for an older CPU, every cartridge generation or every OS. + +## Integration Constraints + +The [Tape contract](../architecture/tape.md), [Job flows](../architecture/jobs.md) and +[Library](../architecture/library.md) retain ownership of physical operations, manifests and +catalog publication respectively. These consumer boundaries explain the integration requirements: + +| Area | Confirmed constraint | Consequence for tar | +| --- | --- | --- | +| Tape identity/profile | The [glossary](../../CONTEXT.md) and [Library](../architecture/library.md) own cartridge identity and immutable profiles | Record decoding parameters must belong to the cartridge rather than current device defaults | +| Content access | [Sessions](../../internal/media/backend.go) distinguish native paths from ordered content endpoints | A raw device is not a pathname for an individual tar member | +| Copy engine | [ACP v0.2.6's filesystem adapter](https://github.com/samuelncui/acp/blob/v0.2.6/filesystem.go) includes commit/discard/metadata/cache operations | A narrow stream endpoint keeps those native assumptions out of tar; source metadata and closure still precede native output completion | +| Archive publication | [Reconciliation](../../internal/executor/archive/media_finalize.go) joins staged content with physical completed-file evidence | tar needs confirmed records and filemark completion, not a synthetic LTFS Index or submitted-file list | +| Restore | Its [frozen copies](../../internal/executor/restore/init.go) retain the selected content and physical placement | Re-querying a current Library locator later would abandon that frozen-candidate boundary | +| Scan | The [pipeline](../../internal/executor/scan/pipeline.go) enumerates fresh inventory before content reads for inventory/report policies; Verify resolves frozen expected copies | Inventory plus content acquisition may need another traversal or a defined positioning strategy; Verify needs direct access to its frozen copies | + +Existing device leases, Job lifecycle, bounded manifests, content identity and Library publication +remain useful. Their ownership does not require a new Job kind or another catalog. Existing LTFS +scripts and their [compatibility contract](../architecture/tape.md#tape-script-compatibility) +remain independently supported. Replacing a mount command with `tar` cannot satisfy these consumers. + +## Format and Device Findings + +GNU tar documents several independent archives on one non-rewinding Tape, separated by tape +marks. Appending another archive at the recorded end differs from `tar --append`, which +rewrites an existing archive's ending and has device-specific limitations. Correct positioning +is essential because a new write can destroy later recorded content. +[GNU tar manual, multiple archives and append](https://www.gnu.org/software/tar/manual/tar.html). + +PAX is the stronger interoperability candidate for YATM's long UTF-8 paths, large files and +subsecond timestamps. USTAR has an approximately 8-GiB member limit and more restrictive names; +an unspecified tar implementation's default is not a format contract. +[GNU archive formats](https://www.gnu.org/software/tar/manual/html_chapter/Formats.html), +[Go tar formats](https://pkg.go.dev/archive/tar#Format). + +Tar's 512-byte blocks are not physical Tape records. Linux variable-block writes establish +records whose readers need sufficiently large buffers; fixed-block mode has alignment rules. +Record buffering therefore belongs below the tar codec. Tar byte offsets also cannot be used +unchanged as driver seek addresses. Non-immediate filemarks synchronize device buffers; tar +footer output alone does not. +[Linux v6.12 st documentation](https://github.com/torvalds/linux/blob/v6.12/Documentation/scsi/st.rst). + +Tar's checksum covers its header, not payload integrity. YATM still needs the actual transferred +SHA-256 and size for Restore and Verify. Standard listing can reconstruct physical member +inventory after catalog loss; it does not reconstruct Library organization, expected hashes +or saved FileVersion relationships. Repeated member names also require distinguishing their +archive/occurrence, rather than treating a name as a unique physical address. +[GNU tar format](https://www.gnu.org/software/tar/manual/tar.html). + +Standard logical-object addressing is distinct from vendor-specific position cookies. SSC-3 draft +r01c §4.2.5.2 assigns each recorded block/filemark a sequential per-partition identifier and permits +later LOCATE using a saved READ POSITION identifier. Sections 7.3 and 7.5 distinguish standard +identifiers from vendor-specific forms. This supports protocol-level eligibility for an exact +reported pairing, not a claim of empirical firmware conformance or physical seek performance. +Fresh exact observations, unchanged recorded content and command error handling remain necessary; +the physical acceptance matrix separately tests reloads, record modes and device combinations. +[SSC-3 r01c, sections 4.2.5.2, 7.3 and 7.5](https://www.13thmonkey.org/documentation/SCSI/ssc3r01c.pdf). + +### Native Command Submission + +Linux `v6.12.111` SG_IO submits a request before its interruptible completion wait. A signal can +leave that request orphaned and return `ERESTARTSYS`; a handler with `SA_RESTART` can then restart +the ioctl and submit another CDB. Go's Linux runtime uses `SA_RESTART`, including its SIGURG +preemption handler. Therefore one successful Go syscall return does not establish one submitted +READ or WRITE, and handling only a visible EINTR misses hidden replay. +[Linux stable SG_IO](https://github.com/gregkh/linux/blob/v6.12.111/drivers/scsi/sg.c#L931-L946), +[Linux syscall restart](https://github.com/torvalds/linux/blob/v6.12/arch/x86/kernel/signal.c#L244-L261), +[Go Linux signal handlers](https://go.dev/src/runtime/os_linux.go), +[Go preemption signals](https://go.dev/src/runtime/signal_unix.go). + +SG v3's separate header write returns after submission; its response read only receives an existing +request. Repeating a receive cannot submit another CDB. This separation supports a single-command +exchange under YATM's existing serialized ownership, with matching responses and stable native +buffer lifetime. It does not itself establish a drive's residual semantics or physical qualification. +[Linux stable submission and receive paths](https://github.com/gregkh/linux/blob/v6.12.111/drivers/scsi/sg.c#L436-L830). + +### Encrypted Full-Cartridge Observations + +A 2026-10-28/29 run through the installed package used the same HP +LTO-5/Z6ED drive, 256 KiB variable records and AES-256-GCM configured by the `encrypt` adapter. +Every operation went through the installed service and CLI. + +One Archive APPEND of 64 GiB members published 22 members, 1,512 GB, in 10,995 s. Capacity +admission then refused the next member: the drive's EOD-to-EW estimate was 17.7 GB. The Job +returned to its pre-Media state with a pending suffix, and earlier Positions were unchanged. +Restores of the first, middle and last members matched. Verify matched every member on the +cartridge at about eight minutes per 64 GiB member. After `stenc` loaded the saved key, GNU `dd` +and `tar` read this archive, which has no tar end blocks, to its filemark with tar exit 0. They +hashed all 22 published members, and the read ended at the drive-reported EOD after four +filemarks. + +A further APPEND of 256 MiB members ran with capacity polling Off, so admission kept only its +17.5 GB preparation sample. The drive reported EW (sense 0/0002) after 54 members, about +14.5 GB, roughly 3 GB before that estimate. The Job settled with 54 published and 46 pending +members, and every published member restored with matching bytes. On this encrypted cartridge, +small members therefore reach the drive's EW before the estimate is exhausted. The single sample +does not say whether encryption overhead or the estimate's own granularity causes the 3 GB gap. + +One LOGICAL and one AUTO Restore then read 36 members from both ends of the filled cartridge: the +fixture members at BOT and every other EW-tail member. They took 276.1 s and 272.0 s. The learned +model was available, and AUTO planned in 0.16 s but kept the logical route because it found no +lower complete-route estimate. This selection leaves no room for reordering and measures no gain. + +### Public-Tool Recovery Checks + +GNU tar 1.34 disk-only checks cover long paths, permissions, subsecond modification times, empty and +binary xattrs, and extraction of completed members preceding a truncated tail. Original stenc +2.0.1 parser and decrypt-only/empty-key payload checks cover the exported key representation. +These checks establish software framing only. + +Normal physical recovery has been checked with Linux st, mt-st 1.8, GNU dd and GNU tar 1.35 for +variable and fixed records, compression On and Off, and script-configured encryption. Each check +copied the identity archive and both content archives separately from BOT, extracted all nine +fixture files with their permissions, exact modification times and ordinary xattrs, observed three +terminating filemarks and recorded EOD, and restored the original driver block setting. Encrypted +recovery used the writing drive; it does not establish compatibility with a different reader or +qualify every configured encryption adapter. + +A separate reduced-capacity EW check recovered all 43 published complete members. GNU tar +returned nonzero for the following unpublished partial member; that result is retained and does +not invalidate the independently checked complete files. Empty physical Tape files, hard EOM, +encrypted EW tails and arbitrary faults are outside those results. + +## Archive Layout Evidence + +### Hardware Compression Controls + +Hardware compression preserves the host record sequence and logical positioning. Its variable +compression ratio instead weakens interpolation from logical objects to physical distance, even +with known wrap endpoints. A possible extension would use additional position samples or applicable +physical-consumption observations with explicit uncertainty; a whole-cartridge compression ratio +cannot recover intra-wrap placement. It also needs local read-rate observations for READ/discard +costs. These are research inputs, not an implemented compressed-media cost provider. + +Without physical samples or qualified physical-consumption observations, hardware-compressed +logical offsets do not uniquely determine within-wrap distance. REOWP can constrain the wrap but +does not solve its interior density. Recurring throughput events can suggest boundaries; host-side +byte periodicity is less stable when hardware compression changes. The current HP5 reports REOWP +but no verified live LPOS, while its native-capacity position estimate remains unqualified as a +physical coordinate. Such observations must not be promoted to exact locations. + +HPE describes LTO-DC as an ALDC-derived hardware compressor that switches to pass-through for +incompressible input. Modern software codecs offer different ratio/CPU tradeoffs: the Zstandard +v1.5.7 README reports a version-1.5.6 level-1 Silesia result of 2.887:1 and 510 MB/s compression on +an i7-9700K. That in-memory software comparison contains no LTO measurement and cannot establish +superiority over drive compression. At 3:1, feeding 300 MB/s of compressed bytes requires about +900 MB/s of original-data compression throughput. Compare the same files, CPU, drive and end-to-end +Archive/Restore operations before recommending either approach. +[HPE QuickSpecs, compression](https://www.hpe.com/psnow/doc/c04123270), +[Zstandard v1.5.7 benchmark documentation](https://github.com/facebook/zstd/blob/v1.5.7/README.md#benchmarks). + +The published 2:1 LTO-5 and 2.5:1 LTO-6 figures are capacity/throughput assumptions, not fixed +compression ratios for arbitrary content. For a simple streaming software-compression pipeline, +original-byte throughput is bounded by `min(source-read rate, compressor input rate, compression +ratio * stored-byte tape rate)` before staging and other overhead. A slow source disk can dominate +both hardware and software paths. Restore destination writes may likewise dominate selected-file +output, while gap discard need not decompress or write skipped files; a whole-Restore throughput +is therefore not the right rate for every discard estimate. +[IBM LTO throughput figures](https://www.ibm.com/support/pages/lto-and-3592-throughput-information-performance-troubleshooting). + +The software option considered here compresses each file independently into a public codec stream. +An ordinary `.zst` member preserves tar-member addressing; independent recovery uses tar and the +public decoder. Path and metadata representation still require a concrete format design. With +hardware compression off and uniform stored records, geometry estimates use actual stored archive +bytes, not the original file sizes; per-file compression does not itself invalidate that mapping. +Tar needs the stored member size before its header, so compression must establish the output size +before Tape submission, using bounded staging or another explicitly evaluated pass. Account for +temporary-disk traffic, capacity and pipeline backpressure rather than buffering an unbounded file +in memory. This option is not implemented; user-supplied compressed source files remain ordinary +archive inputs. + +SSC-3 draft r01c §8.3.2 distinguishes write DCE, capability DCC and read DDE. Its current algorithm +fields report the selected writer algorithm and last encountered reader algorithm; they do not +enumerate every compatible algorithm. DDE leaves uncompressed data unaffected, and RED controls +exceptions at compression boundaries. +[SSC-3 r01c, pages 106–109](https://www.13thmonkey.org/documentation/SCSI/ssc3r01c.pdf#page=122). + +IBM GA32-0928-08 §6.6.10–11 reports default algorithm 1 for LTO-5 and unregistered ID FF for later +drives. The legacy page-10 compression selector can override page-0F DCE, so changing only DCE is +insufficient evidence that a write policy became effective. +[IBM LTO SCSI Reference, pages 378–381](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference%20GA32-0928-08%20(EXTERNAL).pdf#page=407). + +HP LTO-6 Volume 3 describes algorithm 1, independently enabled read decoding, and the legacy +selector's pass-through/default choices. +[HP LTO-6 reference, pages 104–106](https://docs.oracle.com/cd/E38452_01/en/LTO6_Vol3_E1_D20/LTO6_Vol3_E1_D20.pdf#page=104). + +These sources establish control semantics for the studied versions, not physical qualification of +YATM's transport, cleanup or public-reader recovery. Unknown firmware behavior remains unqualified. + +### Hardware Encryption Controls + +SSC-3 r03e §4.2.21 specifies separate encryption/decryption parameters and I_T-nexus ownership. +Disabling encryption together with disabled or raw decryption releases the stored parameters and +changes memory containing the key. A status-page key-instance counter is an opaque observation, +not a key-presence bit or proof of the key's correctness. +[SSC-3 r03e, section 4.2.21](https://citeseerx.ist.psu.edu/document?doi=baaf4ada4b6d66dd63c05f08761157577597d52d&repid=rep1&type=pdf). + +IBM GA32-0928-08 §6.8.2–3 and HP LTO-6 Volume 3 describe Tape Data Encryption protocol pages, +algorithm descriptors, parameter management, status and plaintext-key Set Data Encryption. Their +algorithm indexes are reader-local; the public code `00010014h` identifies AES-256-GCM with a +128-bit authentication tag and a 32-byte key. Read decryption and write encryption are separate +modes. A compatible algorithm descriptor alone does not establish safe ownership or clearing. +[IBM LTO SCSI Reference, pages 451–461 and 468–473](https://www.ibm.com/support/pages/system/files/inline-files/LTO%20SCSI%20Reference%20GA32-0928-08%20(EXTERNAL).pdf), +[HP LTO-6 reference, pages 218–226](https://docs.oracle.com/cd/E38452_01/en/LTO6_Vol3_E1_D20/LTO6_Vol3_E1_D20.pdf#page=218). + +The studied public reader is stenc 2.0.1, pinned to +[`704b5dd7dd0be93390e35b286bbf4af0b24c9ff4`](https://github.com/scsitape/stenc/tree/704b5dd7dd0be93390e35b286bbf4af0b24c9ff4). +Its main.cpp reads a hexadecimal key from the first line of a file and accepts 64 digits for this +algorithm; the YATM profile prefix must be excluded. Explicit `-e off -d on` selects decrypt-only +mode, whereas `-d on` alone also enables encryption. Off/off without a key builds a zero-key Set +Data Encryption page. Its All-I_T-nexus scope differs from a Local writer's ownership but does not +change the recorded public algorithm. These are source-level interoperability findings; public +unlock after reload, actual missing/wrong-key reads, clearing and cross-reader portability still +require physical qualification of the tested writer/reader path. + +### Ordinary Archive Boundaries + +Independent uncompressed PAX archives separate tar framing from physical Tape files. +Go's streaming [archive/tar](https://pkg.go.dev/archive/tar) exposes member boundaries and per-file +metadata; GNU tar supplies an independent interoperability reader. The selected layout belongs +to the Media contract. Hardware compression and encryption need separate evidence for the reader +and writer combination. + +| Boundary | Benefit | Cost or unresolved limitation | +| --- | --- | --- | +| One archive per Media attempt | Natural existing lifecycle boundary; continuous payload streaming | A failed late member leaves the current archive incomplete; late-member Restore may read a large prefix | +| Several complete archives per attempt | More independently completed and locatable groups | More padding, filemarks and synchronization; grouping needs a demonstrated recovery/throughput requirement | +| One archive per file | Fine filemark-level selection | Filemarks and synchronization may dominate small-file workloads | + +No source establishes a universally safe group size or optimal record size. The first row is a +useful comparison baseline, not a complete capacity policy. Finishing groups periodically must +be evaluated against the [continuous-supply constraint](../architecture/tape.md#tape-capacity), +not assumed to be free. Whole-stream software compression would add another requirement to +indexed member access; the baseline above avoids that extra dependency. + +Physical addressing needs enough evidence to find a particular member, preserve required PAX +headers and order reads. Filemark-level positioning plus scanning within an archive is a baseline; +faster record-level positioning requires drive/HBA/driver validation. Neither an archive number +nor a tar byte offset alone proves portable fast access. + +## Completion Is the Hard Boundary + +Linux's early-warning behavior can return a successful write, then `ENOSPC`, then allow a trailer +write before physical exhaustion. This does not guarantee enough room to finish the member already +in progress. Buffered failures can surface later, including at physical finalization. +[Linux v6.12 st documentation](https://github.com/torvalds/linux/blob/v6.12/Documentation/scsi/st.rst). + +Non-immediate filemark completion has its own semantics. [SSC-3 revision 01c, §6.9](https://www.13thmonkey.org/documentation/SCSI/ssc3r01c.pdf#page=89) +defines an EW NO SENSE result as successful transfer of all buffered objects; SEW=0 requires EOM +without requiring VALID. A missing information count therefore differs from a valid nonzero +remainder. [Linux v6.12 filemark handling](https://github.com/torvalds/linux/blob/v6.12/drivers/scsi/st.c#L1306) +also accepts a current NO SENSE/RECOVERED ERROR EOM completion with absent or zero remainder. +These sources support interpreting the command's completed warning; they do not make SG residual +zero alone a durability proof or qualify an application's full EW cleanup/publication path. + +Go `tar.Writer` retains I/O errors; `Close` refuses an incomplete member and does not close the +underlying device. Calling it after a failed payload write is not a repair operation. +[Go 1.26.8 writer](https://github.com/golang/go/blob/go1.26.8/src/archive/tar/writer.go). + +Consequently, complete earlier members being extractable from a damaged tail is different from +proving a safely finalized Archive result. For normal EW completion, publication must establish which complete members reached the +filemark-terminated Archive and which rows remain pending. An independent failure does not +authorize prefix publication or tail repair. Earlier independently finalized archives must remain usable. +An external inventory updated while issuing writes is a candidate list, not independent proof +that buffered bytes reached Tape. Reusing the current LTFS reconciliation flag would not supply +that proof. No full-tape safe-prefix guarantee is established by this study. + +## Content-I/O Boundary + +Ordered member endpoints allow ACP to retain bounded buffering, hashing, progress and per-item +completion without exporting its private filesystem adapter. They require actual observed +metadata before native target creation and successful source closure before +output commit. A standalone `io.Copy` would also need those completion and safe-output rules; +its short body is not evidence that a second transfer engine is simpler. The +[Media I/O contract](../architecture/media-io.md#acp-and-integrity) defines the implemented boundary. + +Issue #14 also mentions STFS. Its [studied revision](https://github.com/pojntfx/stfs/tree/3aff00b57e42600cc930f99c4ef1242431c37432) +provides its own metadata inventory and Go filesystem API; that API requires a write cache sized +for the largest file, while its operation API avoids that requirement. It is a separate integration +choice, not a drop-in path adapter for ACP or evidence that YATM's lifecycle is already satisfied. + +## Unresolved Questions and Required Evidence + +- **Initialization and identity:** validate raw initialization and later reads/appends, including + devices whose electronic identity is empty. Drive encryption support cannot be assumed from age. +- **Failed tail:** demonstrate error, cancellation, short-write and finalization outcomes. The + [encrypted full-cartridge observations](#encrypted-full-cartridge-observations) covers one + admission stop and one drive EW, including standard-tool extraction of the admission-stopped + archive. +- **Addressing and inventory:** demonstrate append/reload, duplicate names, long paths, empty and + large files, catalog-free enumeration and selected Restore with the necessary locator context. +- **Scan and Verify:** measure inventory/content traversal costs. Inventory policies require fresh + physical observations; Verify reads uncached content against its deliberately frozen old Positions. +- **Throughput:** compare representative small/mixed/large files, record sizes and archive boundaries + on the selected drive. Filesystem or in-memory tar tests cannot validate Tape positioning or EOM. + +These are physical evaluation questions, not an implementation sequence or a claim of completed +qualification. Evaluation retains the +[supported operating model](../architecture/overview.md#supported-operating-model) and existing LTFS behavior. diff --git a/e2e/ltfs-file-backend/README.md b/e2e/ltfs-file-backend/README.md index 23fb6f9..cb48610 100644 --- a/e2e/ltfs-file-backend/README.md +++ b/e2e/ltfs-file-backend/README.md @@ -14,4 +14,4 @@ official LTFS uses the default `directory` form. The package controller records installed mount script, including for SSH execution, so systemd uses the selected syntax. Select the installed build's syntax during preparation and record it with the executable identity. -Use these scripts for release-package acceptance of format, append, restore and integrity checking. Fault injection remains in the local/CI E2E harness, outside the release templates. See the public [development and release guide](https://github.com/samuelncui/yatm/blob/v1/docs/operations/testing.md). +Fault injection remains in the local/CI E2E harness, outside the release templates. diff --git a/frontend/README.md b/frontend/README.md index 49ba99c..acc6ac2 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -5,7 +5,7 @@ affected Vitest files (for example `pnpm exec vitest run src/components/files-br and run `pnpm typecheck`; add the relevant style/lint, build or browser checks when those boundaries change. `pnpm check` runs all formatting, lint, tests, type checking and the production build for final integration/release. The -[check-scope policy](../docs/operations/testing.md#check-scope-and-reuse) owns evidence reuse. +[evidence reuse policy](../docs/release/release.md#evidence-reuse) owns evidence reuse. DOM-free tests declare `// @vitest-environment node`. Shared setup loads browser helpers only in jsdom. Component tests and benchmarks retain jsdom; tests default to two workers. @@ -42,8 +42,8 @@ a 100,000-row synthetic RPC response to exercise the real virtualizer, long path page completion without creating 100,000 physical files. Do not point this mutating suite at a normal installation. -The [E2E guide](../docs/operations/e2e-test.md#packaged-browser-acceptance) owns fixture -and server setup. Traces and failure screenshots go to `output/browser-results`. +[Package acceptance](../docs/release/package-acceptance.md#packaged-browser-acceptance) owns the +fixture, server and run procedure; this section owns the browser installation. Traces and failure screenshots go to `output/browser-results`. These checks complement unit tests of request interleavings and the backend's large-data checks; they do not replace physical Media acceptance. @@ -114,7 +114,7 @@ node --test frontend/scripts/compare-performance.test.mjs The cache correctness tests separately verify that a million visited rows keep a bounded cache. Cache timings exclude RPC and DOM rendering; the real browser regression checks -scroll geometry and cursor requests. The [Go result benchmarks](../docs/operations/testing.md#performance-acceptance) +scroll geometry and cursor requests. The [Go result benchmarks](../docs/development/performance.md#performance-acceptance) measure backend paging separately. ## Inline feedback @@ -126,13 +126,99 @@ rejects direct MUI error Alerts and Alerts with controls outside this component. Use the shared `errorMessage` formatter for failures; it decodes RPC status text once and preserves ordinary errors and malformed wire text without throwing. -## State and styles +## State and Styling -The [UI architecture](../docs/architecture/ui.md#state-and-styling) owns shared state, -preference persistence, styling boundaries and initialization order. Use the application -Redux hooks for shared selections, committed Settings and Jobs return snapshots; keep -form drafts and request lifetimes with their views. +Each fact has one owner. Redux stores shared application state; it does not replace the router, +controlled forms or request lifetimes. + +| State | Owner and lifetime | +| ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Archive/Restore selections and Restore version policy | Application Redux store; hydrate browser-session preferences once, then persist changes. Successful submission clears only the submitted revision, preserving later edits. | +| Committed Library/Preview/Job/Device Settings | Application Redux store; only a current Get or successful Save publishes values. Older reads cannot replace a Save or a newer read, and an initial failure leaves values unknown until explicit Retry. Editors keep local drafts. | +| Configured Locations | Application Redux store; fetch the complete catalog once, share choices with Settings, and publish successful configuration edits. Live availability and latest analysis use detail reads. | +| File list sorting and folders-first | Application Redux store, persisted to local browser storage by list identity, independently of the current path. | +| Jobs return position, loaded rows, revisions and tombstones | In-memory Redux snapshot per route filter; request lifetimes remain local and late responses cannot publish after leaving. | +| Navigation and active route filters | Router. | +| Panes, sparse Identical pages, log/result windows, dialog drafts and pending actions | The owning view or shared feature hook; discard or invalidate them when their request or source changes. | + +The [application state module](src/state/store.ts) uses serializable protobuf JSON +and memoized decoded selectors. Controllers, timers and in-flight promises remain outside Redux. +Committed Settings and Job snapshots are not persisted. Browser storage is an optional preference +boundary: failed reads use the established default, and failed writes cannot turn a successful +server operation into a failure. A preference retains its own commit rule, such as a confirmed +Restore destination or successful Scan submission; merely editing a draft does not persist it. + +Instants follow the [shared time contract](../docs/architecture/contracts.md#time-values). Decoded values remain +`bigint`; serializable state uses decimal strings. [Shared date helpers](src/tools/time.ts) +convert to `Date` only for display and date controls, with range validation and correct rounding +before the Unix epoch. An unchanged Restore cutoff keeps its original nanoseconds rather than +round-tripping through the control's lower precision. +File rows keep their original nanoseconds as the shared browser date-sort key; saved-version +ordering and Preview selection follow the Library's [latest saved version order](../docs/architecture/library.md#organization-and-content) +with exact nanosecond comparisons. A first-date display fallback does not replace an unknown last archive date in that order. + +Shared read hooks keep requests local to each consumer. Preview and saved-version reads share their +feature hooks; the [Location-choice hook](src/components/use-location-choices.ts) owns +the root menu, Library filters, Restore, Locate original and Identical scope's choices from the +same configured Location catalog as Settings. The application preloads its complete paged catalog; +opening a chooser and filtering its names use that shared cache. An initial read shows loading, +and a failed read exposes Retry. Settings Refresh explicitly reloads the catalog; successful +configuration changes update it. A successful Library import refreshes the catalog and supersedes any earlier read. Late reads cannot overwrite newer edits. Live filesystem state +and latest analysis remain separate detail reads. Selection, Restore eligibility checks and save +actions remain with their components. Library filters retain their selected Location and label +when canceled and reopened. Media search keeps its successful current query/page when reopened; +changed queries/pages and explicit Retry perform a new read. Scan Location search and the Location choices +in Jobs filters use the same configured catalog; changing a Job filter retains loaded Media choices. The RPC transport does not log request or +response bodies; request owners present failures, and obsolete cancellation is not an application error. + +Styling has four owners: + +| Concern | Implementation | +| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Application colors, type, shape and framework-wide defaults | The application theme. | +| Shared component appearance and interactive states | The owning React component, using MUI `styled`, `sx`, theme overrides and supported slots. Reuse Feedback, ActionRow, Settings shells, search popup shells and navigation instead of copying their appearance. | +| Page layout and native application-owned elements | Co-located Less/CSS; shared components keep these styles together with their markup, and no stylesheet targets another component's private DOM or state classes. | +| Measured row positions and dimensions | Dynamic styles owned by the virtual list or resizer. | + +`styled` and `sx` use MUI's Emotion styling system. Less styles page layout and application-owned +HTML; it does not reskin framework components. A shared component's native markup may retain +co-located Less, while MUI controls use the theme and component APIs above. +Components own text alignment; root layout containers leave it unset. +Settings editors keep one shared page shell across lazy code loading and the first data read, +with one progress indicator in that shell. Revisiting committed Settings uses the shared cache; +a failed read remains an actionable error until Retry. The router owns the selected tab, +including detail and trailing-slash routes. +The [entrypoint](src/main.tsx) configures MUI utility names before importing component +modules, including production chunks. Use exported utility classes or slots instead of hardcoded +`app-Mui*` selectors. The application theme wraps application controls, including controls placed +inside Chonky slots; Chonky owns its theme and browser appearance. Do not override all Chonky +buttons from an application stylesheet. The [application FileBrowser](src/components/file-browser.tsx) +owns its layout element and supplies Chonky's public theme props, so pages need no `.chonky-*` +stylesheet selectors. Chonky owns the native scroll container for ordinary, grouped and sparse +rows. Rows fill its client width; the browser allocates scrollbar space. Fixed content insets +must not stand in for a scrollbar, including when the list fits without scrolling. +Component states must work under lazy route loading, +portals, narrow layouts and nested browsers without depending on import order. + +`pnpm styles:check` and ESLint reject known boundary violations. Direct configured-Location +list reads and Settings Get/Update calls are restricted to their shared owners, as are Job result +page reads. Visual acceptance still checks +alignment, wrapping, focus, disabled/loading states and both browser panes; a static guard cannot +prove rendered appearance. + +[MUI class-name initialization](src/mui-classnames.ts) must run before component imports in both the application entry and browser test setup. The entry dynamically loads [application bootstrap](src/bootstrap.tsx) after initialization, preserving that order across production shared chunks. Cached internal selectors and rendered classes must use the same prefix; otherwise default select overflow and outlined-label styles can fail. Verify long selected values and floating labels in real rendering. + +The [application theme](src/theme.ts) owns MUI primitive appearance; `sx` and `styled` use the same Emotion engine. Shared components own repeated structure and placement, including tabs, section cards, action rows, confirmations, feedback, result rows and saved-version choices. Less owns page layout and scoped third-party adaptations; it does not restyle those shared components. Bootstrap loads the reset before importing the application and its global styles. Component styles stay scoped, so opening a lazy route cannot change another surface. Chonky retains its own Emotion styles and notifications retain their distributed CSS. + +### Checks Run `pnpm styles:check` for known style-boundary violations and `pnpm lint` for component usage rules. Include browser checks of loading/disabled states, portals, nested browsers and narrow layouts when changing shared presentation. + +## UI Prototypes + +Build UI prototypes with the existing React modules, styling and layout, and capture their rendered +pages for review. Store temporary sources and screenshots under ignored `frontend/.drafts/`, then +remove them after the accepted behavior and constraints have been recorded in the owning current +architecture document. Generated concept images are not interface specifications. diff --git a/internal/executor/media_tar_defaults.go b/internal/executor/media_tar_defaults.go index ea658f6..c0d6b48 100644 --- a/internal/executor/media_tar_defaults.go +++ b/internal/executor/media_tar_defaults.go @@ -14,7 +14,7 @@ const ( tarBuiltinHP6Reference = "https://docs.oracle.com/cd/E38452_01/en/LTO6_Vol4_E1/LTO6_Vol4_E1.pdf" tarBuiltinHP5Reference = "https://docs.oracle.com/cd/E21419_04/en/LTO5_Vol4_E4/LTO5_Vol4_E4.pdf" tarBuiltinQuantumReference = "https://cdn.allbound.com/iq-ab/2022/12/LTO-9-TB00068A.pdf" - tarBuiltinEmpiricalReference = "docs/research/tar-tape.md#uncompressed-full-cartridge-measurement" + tarBuiltinEmpiricalReference = "docs/research/tape-read-ordering.md#uncompressed-full-cartridge-measurement" ) func builtinTapeReadParameters(report *entity.InspectDeviceResponse, scope *entity.TapeReadApplicability) *entity.TapeReadParameters { diff --git a/previewworker/build/README.md b/previewworker/build/README.md index 0ce218e..ab12041 100644 --- a/previewworker/build/README.md +++ b/previewworker/build/README.md @@ -7,7 +7,7 @@ are ordinary files; the binary archive requires no symlinks. Pinned native archi inputs are distributed in the separate `yatm-preview-source---.tar.gz` asset with its own checksum. Transfer only the binary package to acceptance hosts. It does not install or replace system `ffmpeg` or `ffprobe`. The main programs never link FFmpeg; [backend -builds](../../docs/operations/testing.md#release-backend-builds) retain the static Linux C SQLite +builds](../../docs/release/release.md#release-backend-builds) retain the static Linux C SQLite implementation. ## Native Build @@ -44,7 +44,7 @@ requires their distributed notices and rejects unowned members. Runnable builds first verify a separate FFmpeg fixture generator through `node build/release/check-preview-fixtures.mjs`. It needs libx264, 10-bit libx265, PNG, lavfi and -display-rotation support. The [test guide](../../docs/operations/testing.md#release-sop) describes +display-rotation support. The [release guide](../../docs/release/release.md#release-sop) describes the pinned Linux fixture-tool build. These test encoders are not added to the Preview runtime. Repository builds require a clean HEAD: tracked edits and untracked nonignored files fail before @@ -79,7 +79,7 @@ provide the same external directory; the source package includes the inventory a without bundling camera files or requiring a download during an offline rebuild. Use the optional helper installation described in the [installation -guide](../../docs/operations/install.md#preview-preferences), or verify its checksum and extract it +guide](../../docs/operations/install.md#preview-helper), or verify its checksum and extract it in an explicitly selected installation. Configure the native helper executable path in Preview Settings when it is outside YATM's executable directory. diff --git a/previewworker/build/package-sources.test.mjs b/previewworker/build/package-sources.test.mjs index 89740e3..f4eab6d 100644 --- a/previewworker/build/package-sources.test.mjs +++ b/previewworker/build/package-sources.test.mjs @@ -85,15 +85,15 @@ test('package sources verifies module inputs and preserves offline guide links', '# Synthetic build guide', '[Build](build.sh)', '[Worker tests](../README.md#local-tests)', - '[Release checks](../../docs/operations/testing.md#release-sop)', + '[Release checks](../../docs/release/release.md#release-sop)', '[External](https://example.invalid/guide)', '[Section](#synthetic-build-guide)', '', ].join('\n'); fs.writeFileSync(path.join(build, 'README.md'), guide); // The checkout has this guide, but the corresponding source does not include it. - fs.mkdirSync(path.join(root, 'docs/operations'), { recursive: true }); - fs.writeFileSync(path.join(root, 'docs/operations/testing.md'), '# Release SOP\n'); + fs.mkdirSync(path.join(root, 'docs/release'), { recursive: true }); + fs.writeFileSync(path.join(root, 'docs/release/release.md'), '# Release SOP\n'); const environment = { ...process.env, @@ -119,7 +119,7 @@ test('package sources verifies module inputs and preserves offline guide links', assert.equal(copied.status, 0, copied.stderr); assert(fs.existsSync(path.join(cleanPackage, 'sources', 'modules', `${fixtureModule}@${fixtureVersion}`, 'LICENSE'))); const publicRoot = `https://github.com/samuelncui/yatm/blob/${commit}`; - const sourceReadme = guide.replace('](../../docs/operations/testing.md#release-sop)', `](${publicRoot}/docs/operations/testing.md#release-sop)`); + const sourceReadme = guide.replace('](../../docs/release/release.md#release-sop)', `](${publicRoot}/docs/release/release.md#release-sop)`); assert.equal(fs.readFileSync(path.join(sourceBuild, 'README.md'), 'utf8'), sourceReadme); assert.equal(fs.readFileSync(path.join(cleanPackage, 'README.md'), 'utf8'), sourceReadme .replace('](build.sh)', `](${publicRoot}/previewworker/build/build.sh)`)