docs: Add sections on operations, index, recovery, drive management and embedding

README.md

@@ -5,6 +5,7 @@ Simple Tape File System (STFS), a file system for tapes and tar files.
 ⚠️ STFS has not yet been audited! While we try to make it as secure as possible, it has not yet undergone a formal security audit by a third party. Please keep this in mind if you use it for security-critical applications. ⚠️
 
 [![hydrun CI](https://github.com/pojntfx/stfs/actions/workflows/hydrun.yaml/badge.svg)](https://github.com/pojntfx/stfs/actions/workflows/hydrun.yaml)
+![Go Version](https://img.shields.io/badge/go%20version-%3E=1.17-61CFDD.svg)
 [![Go Reference](https://pkg.go.dev/badge/github.com/pojntfx/stfs.svg)](https://pkg.go.dev/github.com/pojntfx/stfs)
 [![Matrix](https://img.shields.io/matrix/stfs:matrix.org)](https://matrix.to/#/#stfs:matrix.org?via=matrix.org)
 [![Binary Downloads](https://img.shields.io/github/downloads/pojntfx/stfs/total?label=binary%20downloads)](https://github.com/pojntfx/stfs/releases)
@@ -42,6 +43,8 @@ You can find binaries for more operating systems and architectures on [GitHub re
 
 ## Usage
 
+> Please note that this is only a short overview and does not explain all configuration options. To get more info on available commands or options, use `--help`.
+
 ### 1. Generating Keys with `stfs keygen`
 
 While not strictly required, it is recommended to generate keys to sign and encrypt your data on tape. There are multiple methods available; `PGP` and [age](https://github.com/FiloSottile/age) for encryption, and `PGP` as well as [Minisign](https://github.com/aead/minisign) for signatures. In most cases, using age for encryption and PGP for signatures is the best option. To generate the appropriate keys, run the following; make sure to save the keys in a secure location and use a secure password:
@@ -119,15 +122,242 @@ For more information, see the [servers reference](#servers).
 
 ### 4. Using Optimized Operations with `stfs operation`
 
+While the file system API is convenient because of its similarity to most filesystems, it also can't be used without a write cache. While this isn't an issue for most applications, it requires you to have a disk that is at least as large as the largest file you want to add to the tape. To get around these limitations, STFS also provides a `tar`-like interface for interacting with the tape. Please note that these operations should be used carefully, as the usual checks (such as checking if a parent directory exists before adding files to it) don't apply.
+
+First, initialize an empty tape:
+
+```shell
+# Use `-d /dev/nst0` for your primary tape drive instead
+$ stfs operation initialize \
+    -d ~/Downloads/drive.tar \
+    -m ~/Downloads/metadata.sqlite \
+    -e age \
+    --recipient ~/.stfs-age.pub \
+    -s pgp \
+    --identity ~/.stfs-pgp.priv \
+    --password mysecuresignaturepassword \
+    --compression zstandard
+type,indexed,record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format
+archive,false,-1,-1,-1,-1,53,/,,0,511,1000,1000,pojntfx,1000,2022-05-16T22:24:13+02:00,0001-01-01T00:00:00Z,0001-01-01T00:00:00Z,0,0,null,4
+archive,true,0,-1,0,-1,53,/,,0,511,1000,1000,pojntfx,1000,2022-05-16T22:24:13+02:00,0001-01-01T00:00:00Z,0001-01-01T00:00:00Z,0,0,null,4
+```
+
+You can now add files to it:
+
+```shell
+# Use `-d /dev/nst0` for your primary tape drive instead
+$ stfs operation archive \
+    -d ~/Downloads/drive.tar \
+    -m ~/Downloads/metadata.sqlite \
+    -e age \
+    --recipient ~/.stfs-age.pub \
+    -s pgp \
+    --identity ~/.stfs-pgp.priv \
+    --password mysecuresignaturepassword \
+    --compression zstandard \
+    .
+# ...
+archive,true,1480,-1,9,-1,48,pkg/tape/write.go,,1544,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyUJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF5mAP95DKo/r136fL/SKuBwmxoMNfGZ+v61bwk/xcOBQk5vrwEAs07QV2RF6h/FME+/nXxjZrbBWmFWg8pC4IGdScnJbQ4="",""STFS.UncompressedSize"":""1544""}",4
+archive,true,1480,-1,17,-1,53,pkg/utility,,0,509,1000,1000,pojntfx,pojntfx,2022-04-18T20:19:52+02:00,2022-05-15T23:36:33+02:00,2022-04-23T16:08:59+02:00,0,0,null,4
+# ...
+```
+
+Full CRUD support is implemented, so you can `delete`, `move`, `restore` and `update` files like this as well. For example, to restore `pkg/tape/write.go`, run the following:
+
+```shell
+# Use `-d /dev/nst0` for your primary tape drive instead
+$ stfs operation restore \
+    -d ~/Downloads/drive.tar \
+    -m ~/Downloads/metadata.sqlite \
+    -e age \
+    --identity ~/.stfs-age.priv \
+    --password mysecureencryptionpassword \
+    -s pgp \
+    --recipient ~/.stfs-pgp.pub \
+    --compression zstandard \
+    --flatten \
+    --from pkg/tape/write.go \
+    --to write.go
+type,indexed,record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format
+restore,true,1480,1480,9,9,48,/pkg/tape/write.go,,1544,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyUJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF5mAP95DKo/r136fL/SKuBwmxoMNfGZ+v61bwk/xcOBQk5vrwEAs07QV2RF6h/FME+/nXxjZrbBWmFWg8pC4IGdScnJbQ4="",""STFS.UncompressedSize"":""1544""}",4
+$ file write.go
+write.go: ASCII text
+```
+
+For more information, see the [operations reference](#operations).
+
 ### 5. Managing the Index with `stfs inventory`
 
+For similar reasons as described in [Using Optimized Operations with `stfs operation`](#4-using-optimized-operations-with-stfs-operation), it can make sense to take advantage of the index in order to quickly find a file or directory. For example, to list all files in the `pkg` directory, run the following:
+
+```shell
+$ stfs inventory list \
+    -m ~/Downloads/metadata.sqlite \
+    --name pkg
+record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format
+1454,1454,14,14,53,/pkg/cache,,0,493,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,null,4
+# ...
+```
+
+It is also possible to search for a file by regex with `stfs inventory find`:
+
+```shell
+$ stfs inventory find \
+    -m ~/Downloads/metadata.sqlite \
+    --expression '(.*).yaml'
+record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format
+118,118,11,11,48,/.github/workflows/hydrun.yaml,,3000,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF9nAP98fVspytLtKjTATbtH7hoVaK7tyVGKVDabY4OkOnYiygD/QcTtdWR48Eq5pIT0bR2M9u168aXTbWoWX8JVXcm7uwg="",""STFS.UncompressedSize"":""3000""}",4
+# ...
+```
+
+It is also possible to get information on a single file or directory using `stfs inventory stat`. For more information, see the [inventory reference](#inventory).
+
 ### 6. Recovering Data with `stfs recovery`
 
+In case of unfinished write operations, sudden power losses or other forms of data corruption, the integrated recovery tools can help. For example, to query a tape starting from a specific record and block, use `stfs query`:
+
+```shell
+# Use `-d /dev/nst0` for your primary tape drive instead
+$ stfs recovery query \
+    -d ~/Downloads/drive.tar \
+    -e age \
+    --identity ~/.stfs-age.priv \
+    --password mysecureencryptionpassword \
+    -s pgp \
+    --recipient ~/.stfs-pgp.pub \
+    --compression zstandard \
+    --record 118 \
+    --block 11
+118,-1,11,-1,48,.github/workflows/hydrun.yaml.zst.age,,1272,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF9nAP98fVspytLtKjTATbtH7hoVaK7tyVGKVDabY4OkOnYiygD/QcTtdWR48Eq5pIT0bR2M9u168aXTbWoWX8JVXcm7uwg="",""STFS.UncompressedSize"":""3000""}",4
+119,-1,0,-1,48,.gitignore.zst.age,,220,436,1000,1000,pojntfx,pojntfx,2022-04-18T20:19:52+02:00,2022-05-15T23:38:41+02:00,2022-04-23T16:08:59+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAGAmAP9G6z9HSr5puQjDMRpYZ11Jge95wG2g3LSetF+ts4CG7wEA38qbJx92BbQN4tWmm5G3dXg+PAnGKONAkc0IU9dmtgA="",""STFS.UncompressedSize"":""4""}",4
+# ...
+```
+
+If you know the record and block of a file (which you can get from the index with the `inventory` commands), you can also recover it directly:
+
+```shell
+# Use `-d /dev/nst0` for your primary tape drive instead
+$ stfs recovery fetch \
+    -d ~/Downloads/drive.tar \
+    -e age \
+    --identity ~/.stfs-age.priv \
+    --password mysecureencryptionpassword \
+    -s pgp \
+    --recipient ~/.stfs-pgp.pub \
+    --compression zstandard \
+    --record 118 \
+    --block 11 \
+    --to hydrun.yaml
+record,lastknownrecord,block,lastknownblock,typeflag,name,linkname,size,mode,uid,gid,uname,gname,modtime,accesstime,changetime,devmajor,devminor,paxrecords,format
+118,-1,11,-1,48,.github/workflows/hydrun.yaml.zst.age,,1272,420,1000,1000,pojntfx,pojntfx,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,2022-05-15T23:41:54+02:00,0,0,"{""STFS.Signature"":""wnUEABYIACcFAmKCsyMJkGA0c/4XcV5qFqEEjWKRLHhppJ6S+ZJlYDRz/hdxXmoAAF9nAP98fVspytLtKjTATbtH7hoVaK7tyVGKVDabY4OkOnYiygD/QcTtdWR48Eq5pIT0bR2M9u168aXTbWoWX8JVXcm7uwg="",""STFS.UncompressedSize"":""3000""}",4
+$ file hydrun.yaml
+hydrun.yaml: ASCII text
+```
+
+It is also possible to restore a broken index from scratch with `stfs recovery index`. For more information, see the [recovery reference](#recovery).
+
 ### 7. Managing the Drive with `stfs drive`
 
+STFS can also manage the physical tape drive directly without having to rely on external tools. For example, to eject a tape from the drive, use `stfs drive eject`:
+
+```shell
+$ stfs drive eject \
+    -d /dev/nst0
+```
+
+It is also possible to get the current tape position with `stfs drive tell`. For more information, see the [drive management reference](#drive-management).
+
 ### 8. Embedding STFS with `fs.STFS`
 
-🚀 **That's it!** We hope you enjoy using stfs.
+STFS at its core provides quite a few public APIs, but the easiest way to embed it is to use it's provided [`afero.FS implementation`](https://github.com/spf13/afero). This makes it possible to easily swap out the filesystem implementation with a native one, layer caching implementations and decouple your storage layer.
+
+Using this API is fairly straightforward:
+
+```go
+// ...
+stfs := fs.NewSTFS(
+	readOps,
+	writeOps,
+
+	config.MetadataConfig{
+		Metadata: metadataPersister,
+	},
+
+	config.CompressionLevelFastestKey,
+	func() (cache.WriteCache, func() error, error) {
+		return cache.NewCacheWrite(
+			*writeCacheFlag,
+			config.WriteCacheTypeFile,
+		)
+	},
+	false,
+	false,
+
+	func(hdr *config.Header) {
+		l.Trace("Header transform", hdr)
+	},
+	l,
+)
+
+root, err := stfs.Initialize("/", os.ModePerm)
+if err != nil {
+	panic(err)
+}
+
+fs, err := cache.NewCacheFilesystem(
+	stfs,
+	root,
+	config.NoneKey,
+	0,
+	"",
+)
+if err != nil {
+	panic(err)
+}
+```
+
+You can now use the Afero APIs to interact with the filesystem; if you've worked with Go's `fs` package before, they should be very familiar:
+
+```go
+log.Println("stat /")
+
+stat, err := fs.Stat("/")
+if err != nil {
+	panic(err)
+}
+
+log.Println("Result of stat /:", stat)
+
+log.Println("create /test.txt")
+
+file, err := fs.Create("/test.txt")
+if err != nil {
+	panic(err)
+}
+
+log.Println("Result of create /test.txt:", file)
+
+log.Println("writeString /test.txt")
+
+n, err := file.WriteString("Hello, world!")
+if err != nil {
+	panic(err)
+}
+
+log.Println("Result of writeString /test.txt:", n)
+
+if err := file.Close(); err != nil {
+	panic(err)
+}
+
+// ...
+```
+
+Note that STFS also implements `afero.Symlinker`, so symlinks are available as well.
+
+For more information, check out the [Go API](https://pkg.go.dev/github.com/pojntfx/stfs) and take a look at the provided [examples](./examples), utilities, services and tests in the package for examples.
+
+🚀 **That's it!** We hope you enjoy using STFS.
 
 ## Reference
 
@@ -370,14 +600,14 @@ All command line arguments described above can also be set using environment var
 
 To contribute, please use the [GitHub flow](https://guides.github.com/introduction/flow/) and follow our [Code of Conduct](./CODE_OF_CONDUCT.md).
 
-To build and start a development version of stfs locally, run the following:
+To build and start a development version of STFS locally, run the following:
 
 ```shell
 $ git clone https://github.com/pojntfx/stfs.git
 $ cd stfs
 $ make depend
 $ make && sudo make install
-$ stfs serve ftp -d /tmp/drive.tar -m /tmp/dev.sqlite # Now point Nautilus to `ftp://localhost:1337`
+$ stfs serve ftp -d /tmp/drive.tar -m /tmp/dev.sqlite # Now point your file explorer to `ftp://localhost:1337`
 ```
 
 Have any questions or need help? Chat with us [on Matrix](https://matrix.to/#/#stfs:matrix.org?via=matrix.org)!