390 lines
16 KiB
Markdown
390 lines
16 KiB
Markdown
# STFS
|
|
|
|
Simple Tape File System (STFS), a file system for tapes and tar files.
|
|
|
|
⚠️ STFS has not yet been audited! While we try to make it as secure as possible, it has not yet undergone a formal security audit by a third party. Please keep this in mind if you use it for security-critical applications. ⚠️
|
|
|
|
[](https://github.com/pojntfx/stfs/actions/workflows/hydrun.yaml)
|
|
[](https://pkg.go.dev/github.com/pojntfx/stfs)
|
|
[](https://matrix.to/#/#stfs:matrix.org?via=matrix.org)
|
|
[](https://github.com/pojntfx/stfs/releases)
|
|
|
|
## Overview
|
|
|
|
🚧 This project is a work-in-progress! Instructions will be added as soon as it is usable. 🚧
|
|
|
|
## Installation
|
|
|
|
Static binaries are available on [GitHub releases](https://github.com/pojntfx/stfs/releases).
|
|
|
|
On Linux, you can install them like so:
|
|
|
|
```shell
|
|
$ curl -L -o /tmp/stfs "https://github.com/pojntfx/stfs/releases/latest/download/stfs.linux-$(uname -m)"
|
|
$ sudo install /tmp/stfs /usr/local/bin
|
|
$ sudo setcap cap_net_admin+ep /usr/local/bin/stfs # This allows rootless execution
|
|
```
|
|
|
|
On macOS, you can use the following:
|
|
|
|
```shell
|
|
$ curl -L -o /tmp/stfs "https://github.com/pojntfx/stfs/releases/latest/download/stfs.darwin-$(uname -m)"
|
|
$ sudo install /tmp/stfs /usr/local/bin
|
|
```
|
|
|
|
On Windows, the following should work (using PowerShell as administrator):
|
|
|
|
```shell
|
|
PS> Invoke-WebRequest https://github.com/pojntfx/stfs/releases/latest/download/stfs.windows-x86_64.exe -OutFile \Windows\System32\stfs.exe
|
|
```
|
|
|
|
You can find binaries for more operating systems and architectures on [GitHub releases](https://github.com/pojntfx/stfs/releases).
|
|
|
|
## Usage
|
|
|
|
### 1. Generating Keys with `stfs keygen`
|
|
|
|
While not strictly required, it is recommended to generate keys to sign and encrypt your data on tape. There are multiple methods available; `PGP` and [age](https://github.com/FiloSottile/age) for encryption, and `PGP` as well as [Minisign](https://github.com/aead/minisign) for signatures. In most cases, using age for encryption and PGP for signatures is the best option. To generate the appropriate keys, run the following; make sure to save the keys in a secure location and use a secure password:
|
|
|
|
```shell
|
|
$ stfs keygen --encryption age --password mysecureencryptionpassword --identity ~/.stfs-age.priv --recipient ~/.stfs-age.pub
|
|
$ stfs keygen --signature pgp --password mysecuresignaturepassword --identity ~/.stfs-pgp.priv --recipient ~/.stfs-pgp.pub
|
|
```
|
|
|
|
For more information, see the [key generation reference](#key-generation).
|
|
|
|
### 2. Serving a Tape Read-Write with `stfs serve ftp`
|
|
|
|
The simplest way to read or write to/from the tape (or tar file) is to use the integrated FTP server. To speed up operations, caching mechanisms and compression are available. For the write cache (`--cache-write-type`) the following types are available:
|
|
|
|
- `memory`: A simple in-memory cache; should not be used in most cases due to potential RAM exhaustion when adding large files
|
|
- `file`: A on-disk cache; this is recommended in most cases, especially if a SSD is available
|
|
|
|
For the read cache (`--cache-filesystem-type`), which is especially useful when working with many small files, similar types are available (`memory` and `dir`). `dir` uses a overlay filesystem to cache files in the directory specified with `--cache-dir`.
|
|
|
|
To further speed up IO-limited read/write operations, multiple compression options are available to be selected with `--compression` and can be tuned with `--compression-level`:
|
|
|
|
- `zstandard`: A Meta-led replacement for `gzip` with very high speeds and a very good compression ratio; this is recommended for most users
|
|
- `gzip`/`parallelgzip`: The GNU format commonly used in combination with `tar`, i.e. for `.tar.gz`; reasonably fast and with a good compression ratio
|
|
- `bzip2`/`parallelbzip2`: A reliable compression format with good speeds and a better compression ratio than `gzip`.
|
|
- `lz4`: Very fast, but at the cost of a lower compression ratio
|
|
- `brotli`: A Google-led compression format with good adoption on the web platform; very high compression ratio, very slow speeds
|
|
|
|
To serve a tape (or tar file), run the following (adjust the options accordingly):
|
|
|
|
```shell
|
|
# Use `-d /dev/nst0` for your primary tape drive instead
|
|
$ stfs serve ftp \
|
|
-d ~/Downloads/drive.tar \
|
|
-m ~/Downloads/metadata.sqlite \
|
|
-e age \
|
|
--encryption-identity ~/.stfs-age.priv \
|
|
--encryption-recipient ~/.stfs-age.pub \
|
|
--encryption-password mysecureencryptionpassword \
|
|
-s pgp \
|
|
--signature-identity ~/.stfs-pgp.priv \
|
|
--signature-recipient ~/.stfs-pgp.pub \
|
|
--signature-password mysecuresignaturepassword \
|
|
--compression zstandard
|
|
{"time":1652646259,"level":"INFO","event":"FTP server listening","data":[{"laddr":":1337"}]}
|
|
{"time":1652646259,"level":"INFO","event":"Listening...","data":["address",{"IP":"::","Port":1337,"Zone":""}]}
|
|
{"time":1652646259,"level":"INFO","event":"Starting...","data":null}
|
|
```
|
|
|
|
You can now point your file manager (GNOME files on Linux, Windows Explorer on Windows and Finder on macOS all have support for it, but macOS is read-only) to `ftp://localhost:1337` and read/write files from the tape (or tape file).
|
|
|
|
For more information, see the [servers reference](#servers).
|
|
|
|
### 3. Serving a Tape Read-Only with `stfs serve http`
|
|
|
|
If you want to serve a tape (or tar file) read-only, using the integrated HTTP server is the best option. It inherits all the same options from [Serving a Tape Read-Write with `stfs serve ftp`](#2-serving-a-tape-read-write-with-stfs-serve-ftp), minus the write cache due to it being read-only. To use it, run:
|
|
|
|
```shell
|
|
# Use `-d /dev/nst0` for your primary tape drive instead
|
|
$ stfs serve http \
|
|
-d ~/Downloads/drive.tar \
|
|
-m ~/Downloads/metadata.sqlite \
|
|
-e age \
|
|
--identity ~/.stfs-age.priv \
|
|
--password mysecureencryptionpassword \
|
|
-s pgp \
|
|
--recipient ~/.stfs-pgp.pub \
|
|
--compression zstandard
|
|
{"time":1652653259,"level":"INFO","event":"HTTP server listening","data":[{"laddr":":1337"}]}
|
|
```
|
|
|
|
You can now point your web browser to `http://localhost:1337` and read files from the tape (or tape file).
|
|
|
|
For more information, see the [servers reference](#servers).
|
|
|
|
### 4. Using Optimized Operations with `stfs operation`
|
|
|
|
### 5. Managing the Index with `stfs inventory`
|
|
|
|
### 6. Recovering Data with `stfs recovery`
|
|
|
|
### 7. Managing the Drive with `stfs drive`
|
|
|
|
### 8. Embedding STFS with `fs.STFS`
|
|
|
|
🚀 **That's it!** We hope you enjoy using stfs.
|
|
|
|
## Reference
|
|
|
|
### Command Line Arguments
|
|
|
|
```shell
|
|
$ stfs --help
|
|
Simple Tape File System (STFS), a file system for tapes and tar files.
|
|
|
|
Find more information at:
|
|
https://github.com/pojntfx/stfs
|
|
|
|
Usage:
|
|
stfs [command]
|
|
|
|
Available Commands:
|
|
completion Generate the autocompletion script for the specified shell
|
|
drive Manage tape drives
|
|
help Help about any command
|
|
inventory Get contents and metadata of tape or tar file from the index
|
|
keygen Generate a encryption or signature key
|
|
operation Perform operations on tape or tar file and the index
|
|
recovery Recover tapes or tar files
|
|
serve Serve tape or tar file and the index
|
|
|
|
Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-h, --help help for stfs
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
|
|
Use "stfs [command] --help" for more information about a command.
|
|
```
|
|
|
|
<details>
|
|
<summary>Expand subcommand reference</summary>
|
|
|
|
#### Drive Management
|
|
|
|
```shell
|
|
$ stfs drive --help
|
|
Manage tape drives
|
|
|
|
Usage:
|
|
stfs drive [command]
|
|
|
|
Aliases:
|
|
drive, dri, d
|
|
|
|
Available Commands:
|
|
eject Eject tape from drive
|
|
tell Get the current record on the tape
|
|
|
|
Flags:
|
|
-h, --help help for drive
|
|
|
|
Global Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
|
|
Use "stfs drive [command] --help" for more information about a command.
|
|
```
|
|
|
|
#### Inventory
|
|
|
|
```shell
|
|
$ stfs inventory --help
|
|
Get contents and metadata of tape or tar file from the index
|
|
|
|
Usage:
|
|
stfs inventory [command]
|
|
|
|
Aliases:
|
|
inventory, inv, i
|
|
|
|
Available Commands:
|
|
find Find a file or directory on tape or tar file by matching against a regex
|
|
list List the contents of a directory on tape or tar file
|
|
stat Get information on a file or directory on tape or tar file
|
|
|
|
Flags:
|
|
-h, --help help for inventory
|
|
|
|
Global Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
|
|
Use "stfs inventory [command] --help" for more information about a command.
|
|
```
|
|
|
|
#### Key Generation
|
|
|
|
```shell
|
|
$ stfs keygen --help
|
|
Generate a encryption or signature key
|
|
|
|
Usage:
|
|
stfs keygen [flags]
|
|
|
|
Aliases:
|
|
keygen, key, k
|
|
|
|
Flags:
|
|
-h, --help help for keygen
|
|
-i, --identity string Path to write the private key to
|
|
-p, --password string Password to protect the private key with
|
|
-r, --recipient string Path to write the public key to
|
|
|
|
Global Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
```
|
|
|
|
#### Operations
|
|
|
|
```shell
|
|
$ stfs operation --help
|
|
Perform operations on tape or tar file and the index
|
|
|
|
Usage:
|
|
stfs operation [command]
|
|
|
|
Aliases:
|
|
operation, ope, o, op
|
|
|
|
Available Commands:
|
|
archive Archive a file or directory to tape or tar file
|
|
delete Delete a file or directory from tape or tar file
|
|
initialize Truncate and initalize a file or directory
|
|
move Move a file or directory on tape or tar file
|
|
restore Restore a file or directory from tape or tar file
|
|
update Update a file or directory's content and metadata on tape or tar file
|
|
|
|
Flags:
|
|
-h, --help help for operation
|
|
|
|
Global Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
|
|
Use "stfs operation [command] --help" for more information about a command.
|
|
```
|
|
|
|
#### Recovery
|
|
|
|
```shell
|
|
$ stfs recovery --help
|
|
Recover tapes or tar files
|
|
|
|
Usage:
|
|
stfs recovery [command]
|
|
|
|
Aliases:
|
|
recovery, rec, r
|
|
|
|
Available Commands:
|
|
fetch Fetch a file or directory from tape or tar file by record and block without the index
|
|
index Index contents of tape or tar file
|
|
query Query contents of tape or tar file without the index
|
|
|
|
Flags:
|
|
-h, --help help for recovery
|
|
|
|
Global Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
|
|
Use "stfs recovery [command] --help" for more information about a command.
|
|
```
|
|
|
|
#### Servers
|
|
|
|
```shell
|
|
stfs serve --help
|
|
Serve tape or tar file and the index
|
|
|
|
Usage:
|
|
stfs serve [command]
|
|
|
|
Aliases:
|
|
serve, ser, s, srv
|
|
|
|
Available Commands:
|
|
ftp Serve tape or tar file and the index over FTP (read-write)
|
|
http Serve tape or tar file and the index over HTTP (read-only)
|
|
|
|
Flags:
|
|
-h, --help help for serve
|
|
|
|
Global Flags:
|
|
-c, --compression string Compression format to use (default , available are [ gzip parallelgzip lz4 zstandard brotli bzip2 parallelbzip2])
|
|
-d, --drive string Tape or tar file to use (default "/dev/nst0")
|
|
-e, --encryption string Encryption format to use (default , available are [ age pgp])
|
|
-m, --metadata string Metadata database to use (default "/home/pojntfx/.local/share/stfs/var/lib/stfs/metadata.sqlite")
|
|
-s, --signature string Signature format to use (default , available are [ minisign pgp])
|
|
-v, --verbose int Verbosity level (default 2, available are [0 1 2 3 4]) (default 2)
|
|
|
|
Use "stfs serve [command] --help" for more information about a command.
|
|
```
|
|
|
|
</details>
|
|
|
|
### Environment Variables
|
|
|
|
All command line arguments described above can also be set using environment variables; for example, to set `--drive` to `/tmp/drive.tar` with an environment variable, use `STFS_DRIVE=/tmp/drive.tar`.
|
|
|
|
## Acknowledgements
|
|
|
|
- [aead.dev/minisign](https://github.com/aead/minisign) provides the Minisign signature implementation.
|
|
- [filippo.io/age](https://github.com/FiloSottile/age) provides the Age encryption implementation.
|
|
- [ProtonMail/gopenpgp](github.com/ProtonMail/gopenpgp) provides the PGP signature and encryption implementation.
|
|
- [fclairamb/ftpserverlib](github.com/fclairamb/ftpserverlib) provides the FTP server implementation.
|
|
- [spf13/afero](https://github.com/spf13/afero) provides the file system abstraction layer implementation.
|
|
- All the rest of the authors who worked on the dependencies used! Thanks a lot!
|
|
|
|
## Contributing
|
|
|
|
To contribute, please use the [GitHub flow](https://guides.github.com/introduction/flow/) and follow our [Code of Conduct](./CODE_OF_CONDUCT.md).
|
|
|
|
To build and start a development version of stfs locally, run the following:
|
|
|
|
```shell
|
|
$ git clone https://github.com/pojntfx/stfs.git
|
|
$ cd stfs
|
|
$ make depend
|
|
$ make && sudo make install
|
|
$ stfs serve ftp -d /tmp/drive.tar -m /tmp/dev.sqlite # Now point Nautilus to `ftp://localhost:1337`
|
|
```
|
|
|
|
Have any questions or need help? Chat with us [on Matrix](https://matrix.to/#/#stfs:matrix.org?via=matrix.org)!
|
|
|
|
## License
|
|
|
|
STFS (c) 2022 Felicitas Pojtinger and contributors
|
|
|
|
SPDX-License-Identifier: AGPL-3.0
|