From d72da6763ad54e2d7ac221338bad19df23c11267 Mon Sep 17 00:00:00 2001 From: Felicitas Pojtinger Date: Mon, 16 May 2022 23:18:40 +0200 Subject: [PATCH] docs: Add sections on operations, index, recovery, drive management and embedding --- README.md | 236 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 233 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 2aa6e99..ab3ec7a 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ Simple Tape File System (STFS), a file system for tapes and tar files. ⚠️ STFS has not yet been audited! While we try to make it as secure as possible, it has not yet undergone a formal security audit by a third party. Please keep this in mind if you use it for security-critical applications. ⚠️ [![hydrun CI](https://github.com/pojntfx/stfs/actions/workflows/hydrun.yaml/badge.svg)](https://github.com/pojntfx/stfs/actions/workflows/hydrun.yaml) +![Go Version](https://img.shields.io/badge/go%20version-%3E=1.17-61CFDD.svg) [![Go Reference](https://pkg.go.dev/badge/github.com/pojntfx/stfs.svg)](https://pkg.go.dev/github.com/pojntfx/stfs) [![Matrix](https://img.shields.io/matrix/stfs:matrix.org)](https://matrix.to/#/#stfs:matrix.org?via=matrix.org) [![Binary Downloads](https://img.shields.io/github/downloads/pojntfx/stfs/total?label=binary%20downloads)](https://github.com/pojntfx/stfs/releases) @@ -42,6 +43,8 @@ You can find binaries for more operating systems and architectures on [GitHub re ## Usage +> Please note that this is only a short overview and does not explain all configuration options. To get more info on available commands or options, use `--help`. + ### 1. Generating Keys with `stfs keygen` While not strictly required, it is recommended to generate keys to sign and encrypt your data on tape. There are multiple methods available; `PGP` and [age](https://github.com/FiloSottile/age) for encryption, and `PGP` as well as [Minisign](https://github.com/aead/minisign) for signatures. In most cases, using age for encryption and PGP for signatures is the best option. To generate the appropriate keys, run the following; make sure to save the keys in a secure location and use a secure password: @@ -119,15 +122,242 @@ For more information, see the [servers reference](#servers). ### 4. Using Optimized Operations with `stfs operation` +While the file system API is convenient because of its similarity to most filesystems, it also can't be used without a write cache. While this isn't an issue for most applications, it requires you to have a disk that is at least as large as the largest file you want to add to the tape. To get around these limitations, STFS also provides a `tar`-like interface for interacting with the tape. Please note that these operations should be used carefully, as the usual checks (such as checking if a parent directory exists before adding files to it) don't apply. + +First, initialize an empty tape: + +```shell +# Use `-d /dev/nst0` for your primary tape drive instead +$ stfs operation initialize \ + -d ~/Downloads/drive.tar \ + -m ~/Downloads/metadata.sqlite \ + -e age \ + --recipient ~/.stfs-age.pub \ + -s pgp \ + --identity ~/.stfs-pgp.priv \ + --password mysecuresignaturepassword \ + --compression zstandard +type,indexed,record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format +archive,false,-1,-1,-1,-1,53,/,,0,511,1000,1000,pojntfx,1000,2022-05-16T22:24:13+02:00,0001-01-01T00:00:00Z,0001-01-01T00:00:00Z,0,0,null,4 +archive,true,0,-1,0,-1,53,/,,0,511,1000,1000,pojntfx,1000,2022-05-16T22:24:13+02:00,0001-01-01T00:00:00Z,0001-01-01T00:00:00Z,0,0,null,4 +``` + +You can now add files to it: + +```shell +# Use `-d /dev/nst0` for your primary tape drive instead +$ stfs operation archive \ + -d ~/Downloads/drive.tar \ + -m ~/Downloads/metadata.sqlite \ + -e age \ + --recipient ~/.stfs-age.pub \ + -s pgp \ + --identity ~/.stfs-pgp.priv \ + --password mysecuresignaturepassword \ + --compression zstandard \ + . +# ... +archive,true,1480,-1,9,-1,48,pkg/tape/write.go,,1544,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyUJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF5mAP95DKo/r136fL/SKuBwmxoMNfGZ+v61bwk/xcOBQk5vrwEAs07QV2RF6h/FME+/nXxjZrbBWmFWg8pC4IGdScnJbQ4="",""STFS.UncompressedSize"":""1544""}",4 +archive,true,1480,-1,17,-1,53,pkg/utility,,0,509,1000,1000,pojntfx,pojntfx,2022-04-18T20:19:52+02:00,2022-05-15T23:36:33+02:00,2022-04-23T16:08:59+02:00,0,0,null,4 +# ... +``` + +Full CRUD support is implemented, so you can `delete`, `move`, `restore` and `update` files like this as well. For example, to restore `pkg/tape/write.go`, run the following: + +```shell +# Use `-d /dev/nst0` for your primary tape drive instead +$ stfs operation restore \ + -d ~/Downloads/drive.tar \ + -m ~/Downloads/metadata.sqlite \ + -e age \ + --identity ~/.stfs-age.priv \ + --password mysecureencryptionpassword \ + -s pgp \ + --recipient ~/.stfs-pgp.pub \ + --compression zstandard \ + --flatten \ + --from pkg/tape/write.go \ + --to write.go +type,indexed,record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format +restore,true,1480,1480,9,9,48,/pkg/tape/write.go,,1544,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyUJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF5mAP95DKo/r136fL/SKuBwmxoMNfGZ+v61bwk/xcOBQk5vrwEAs07QV2RF6h/FME+/nXxjZrbBWmFWg8pC4IGdScnJbQ4="",""STFS.UncompressedSize"":""1544""}",4 +$ file write.go +write.go: ASCII text +``` + +For more information, see the [operations reference](#operations). + ### 5. Managing the Index with `stfs inventory` +For similar reasons as described in [Using Optimized Operations with `stfs operation`](#4-using-optimized-operations-with-stfs-operation), it can make sense to take advantage of the index in order to quickly find a file or directory. For example, to list all files in the `pkg` directory, run the following: + +```shell +$ stfs inventory list \ + -m ~/Downloads/metadata.sqlite \ + --name pkg +record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format +1454,1454,14,14,53,/pkg/cache,,0,493,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,null,4 +# ... +``` + +It is also possible to search for a file by regex with `stfs inventory find`: + +```shell +$ stfs inventory find \ + -m ~/Downloads/metadata.sqlite \ + --expression '(.*).yaml' +record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format +118,118,11,11,48,/.github/workflows/hydrun.yaml,,3000,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF9nAP98fVspytLtKjTATbtH7hoVaK7tyVGKVDabY4OkOnYiygD/QcTtdWR48Eq5pIT0bR2M9u168aXTbWoWX8JVXcm7uwg="",""STFS.UncompressedSize"":""3000""}",4 +# ... +``` + +It is also possible to get information on a single file or directory using `stfs inventory stat`. For more information, see the [inventory reference](#inventory). + ### 6. Recovering Data with `stfs recovery` +In case of unfinished write operations, sudden power losses or other forms of data corruption, the integrated recovery tools can help. For example, to query a tape starting from a specific record and block, use `stfs query`: + +```shell +# Use `-d /dev/nst0` for your primary tape drive instead +$ stfs recovery query \ + -d ~/Downloads/drive.tar \ + -e age \ + --identity ~/.stfs-age.priv \ + --password mysecureencryptionpassword \ + -s pgp \ + --recipient ~/.stfs-pgp.pub \ + --compression zstandard \ + --record 118 \ + --block 11 +118,-1,11,-1,48,.github/workflows/hydrun.yaml.zst.age,,1272,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF9nAP98fVspytLtKjTATbtH7hoVaK7tyVGKVDabY4OkOnYiygD/QcTtdWR48Eq5pIT0bR2M9u168aXTbWoWX8JVXcm7uwg="",""STFS.UncompressedSize"":""3000""}",4 +119,-1,0,-1,48,.gitignore.zst.age,,220,436,1000,1000,pojntfx,pojntfx,2022-04-18T20:19:52+02:00,2022-05-15T23:38:41+02:00,2022-04-23T16:08:59+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAGAmAP9G6z9HSr5puQjDMRpYZ11Jge95wG2g3LSetF+ts4CG7wEA38qbJx92BbQN4tWmm5G3dXg+PAnGKONAkc0IU9dmtgA="",""STFS.UncompressedSize"":""4""}",4 +# ... +``` + +If you know the record and block of a file (which you can get from the index with the `inventory` commands), you can also recover it directly: + +```shell +# Use `-d /dev/nst0` for your primary tape drive instead +$ stfs recovery fetch \ + -d ~/Downloads/drive.tar \ + -e age \ + --identity ~/.stfs-age.priv \ + --password mysecureencryptionpassword \ + -s pgp \ + --recipient ~/.stfs-pgp.pub \ + --compression zstandard \ + --record 118 \ + --block 11 \ + --to hydrun.yaml +record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format +118,-1,11,-1,48,.github/workflows/hydrun.yaml.zst.age,,1272,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF9nAP98fVspytLtKjTATbtH7hoVaK7tyVGKVDabY4OkOnYiygD/QcTtdWR48Eq5pIT0bR2M9u168aXTbWoWX8JVXcm7uwg="",""STFS.UncompressedSize"":""3000""}",4 +$ file hydrun.yaml +hydrun.yaml: ASCII text +``` + +It is also possible to restore a broken index from scratch with `stfs recovery index`. For more information, see the [recovery reference](#recovery). + ### 7. Managing the Drive with `stfs drive` +STFS can also manage the physical tape drive directly without having to rely on external tools. For example, to eject a tape from the drive, use `stfs drive eject`: + +```shell +$ stfs drive eject \ + -d /dev/nst0 +``` + +It is also possible to get the current tape position with `stfs drive tell`. For more information, see the [drive management reference](#drive-management). + ### 8. Embedding STFS with `fs.STFS` -🚀 **That's it!** We hope you enjoy using stfs. +STFS at its core provides quite a few public APIs, but the easiest way to embed it is to use it's provided [`afero.FS implementation`](https://github.com/spf13/afero). This makes it possible to easily swap out the filesystem implementation with a native one, layer caching implementations and decouple your storage layer. + +Using this API is fairly straightforward: + +```go +// ... +stfs := fs.NewSTFS( + readOps, + writeOps, + + config.MetadataConfig{ + Metadata: metadataPersister, + }, + + config.CompressionLevelFastestKey, + func() (cache.WriteCache, func() error, error) { + return cache.NewCacheWrite( + *writeCacheFlag, + config.WriteCacheTypeFile, + ) + }, + false, + false, + + func(hdr *config.Header) { + l.Trace("Header transform", hdr) + }, + l, +) + +root, err := stfs.Initialize("/", os.ModePerm) +if err != nil { + panic(err) +} + +fs, err := cache.NewCacheFilesystem( + stfs, + root, + config.NoneKey, + 0, + "", +) +if err != nil { + panic(err) +} +``` + +You can now use the Afero APIs to interact with the filesystem; if you've worked with Go's `fs` package before, they should be very familiar: + +```go +log.Println("stat /") + +stat, err := fs.Stat("/") +if err != nil { + panic(err) +} + +log.Println("Result of stat /:", stat) + +log.Println("create /test.txt") + +file, err := fs.Create("/test.txt") +if err != nil { + panic(err) +} + +log.Println("Result of create /test.txt:", file) + +log.Println("writeString /test.txt") + +n, err := file.WriteString("Hello, world!") +if err != nil { + panic(err) +} + +log.Println("Result of writeString /test.txt:", n) + +if err := file.Close(); err != nil { + panic(err) +} + +// ... +``` + +Note that STFS also implements `afero.Symlinker`, so symlinks are available as well. + +For more information, check out the [Go API](https://pkg.go.dev/github.com/pojntfx/stfs) and take a look at the provided [examples](./examples), utilities, services and tests in the package for examples. + +🚀 **That's it!** We hope you enjoy using STFS. ## Reference @@ -370,14 +600,14 @@ All command line arguments described above can also be set using environment var To contribute, please use the [GitHub flow](https://guides.github.com/introduction/flow/) and follow our [Code of Conduct](./CODE_OF_CONDUCT.md). -To build and start a development version of stfs locally, run the following: +To build and start a development version of STFS locally, run the following: ```shell $ git clone https://github.com/pojntfx/stfs.git $ cd stfs $ make depend $ make && sudo make install -$ stfs serve ftp -d /tmp/drive.tar -m /tmp/dev.sqlite # Now point Nautilus to `ftp://localhost:1337` +$ stfs serve ftp -d /tmp/drive.tar -m /tmp/dev.sqlite # Now point your file explorer to `ftp://localhost:1337` ``` Have any questions or need help? Chat with us [on Matrix](https://matrix.to/#/#stfs:matrix.org?via=matrix.org)!