Zach BrownandZach Brown
57c7caf348
The forest code is responsible for constructing a consistent fs image out of the items spread across all the btrees written by mounts in the system. Usually readers walk a btree looking for log trees that they should read. As a mount modifies items in its dirty log tree, readers need to be sure to check that in-memory dirty log tree even though it isn't present in the btree that records persistent log trees. The code did this by setting a flag to indicate that readers using a lock should check the dirty log tree. But the flag usage wasn't properly locked and left a race where a reader and writer could race, leaving future readers to not know that they should check the dirty log tree. When we rarely hit that race we'd see item errors that made no sense, like not being able to find an inode item to update after having just created it in the current transaction. To fix this, we clean up the tree tracking in the forest code. We get rid of the static forest_root structs in the lock_private that were used to track the two special-case roots that aren't found in log tree items: the in-memory dirty log root and the final fs root. All roots are now dynamically allocated. We use a flag in the root to identify it as the dirty log root, and identify the fs root by its rid/nr. This results in a bunch of caller churn as we remove lpriv from root identifying functions. We get rid of the idea of the writer adding a static root to the list as well as marking the log as needing to read the root. Instead we make all root management happen as we refresh the list. The forest maintains a commit sequence and writers set state in the lock to indicate that the lock has dirty items in the log during this transaction. Iteration then compares the state set by the commit, writer, and the last refresh to determine if a new refresh needs to happen. Properly tracking the presence of dirty items lets us recognize when the lock no longer has dirty items in the log and we can stop locking and reading the dirty log and fall back to reading the committed stable version. The previous code didn't do that, it would lock and read the dirty root forever. While we're in here, we fix the locking around setting bloom bits and have it track the version of the log tree that was set so that we don't have to clear set bits as the log version is rotated out by the server. There was also a subtle bug where we could hit to stale errors for the same root and return -EIO because we triggering refresh returned stale. We rework the retrying logic to use a separate error code to force refreshing so that we can't accidentally trigger eio by conflating reading stale blocks and forcing refreshing. And finally, we no longer record that we need the dirty log tree in a root if we have a lock that could never read. It's a minor optimization that doesn't change functional behaviour. Signed-off-by: Zach Brown <zab@versity.com>
Introduction
scoutfs is a clustered in-kernel Linux filesystem designed and built from the ground up to support large archival systems.
Its key differentiating features are:
- Integrated consistent indexing accelerates archival maintenance operations
- Log-structured commits allow nodes to write concurrently without contention
It meets best of breed expectations:
- Fully consistent POSIX semantics between nodes
- Rich metadata to ensure the integrity of metadata references
- Atomic transactions to maintain consistent persistent structures
- First class kernel implementation for high performance and low latency
- Open GPLv2 implementation
Learn more in the white paper.
Current Status
Alpha Open Source Development
scoutfs is under heavy active development. We're developing it in the open to give the community an opportunity to affect the design and implementation.
The core architectural design elements are in place. Much surrounding functionality hasn't been implemented. It's appropriate for early adopters and interested developers, not for production use.
In that vein, expect significant incompatible changes to both the format of network messages and persistent structures. To avoid mistakes the implementation currently calculates a hash of the format and ioctl header files in the source tree. The kernel module will refuse to mount a volume created by userspace utilities with a mismatched hash, and it will refuse to connect to a remote node with a mismatched hash. This means having to unmount, mkfs, and remount everything across many functional changes. Once the format is nailed down we'll wire up forward and back compat machinery and remove this temporary safety measure.
The current kernel module is developed against the RHEL/CentOS 7.x kernel to minimize the friction of developing and testing with partners' existing infrastructure. Once we're happy with the design we'll shift development to the upstream kernel while maintaining distro compatibility branches.
Community Mailing List
Please join us on the open scoutfs-devel@scoutfs.org mailing list hosted on Google Groups for all discussion of scoutfs.
Quick Start
This following a very rough example of the procedure to get up and running, experience will be needed to fill in the gaps. We're happy to help on the mailing list.
The requirements for running scoutfs on a small cluster are:
- One or more nodes running x86-64 CentOS/RHEL 7.4 (or 7.3)
- Access to a single shared block device
- IPv4 connectivity between the nodes
The steps for getting scoutfs mounted and operational are:
- Get the kernel module running on the nodes
- Make a new filesystem on the device with the userspace utilities
- Mount the device on all the nodes
In this example we run all of these commands on three nodes. The block device name is the same on all the nodes.
-
Get the Kernel Module and Userspace Binaries
- Either use snapshot RPMs built from git by Versity:
rpm -i https://scoutfs.s3-us-west-2.amazonaws.com/scoutfs-repo-0.0.1-1.el7_4.noarch.rpm yum install scoutfs-utils kmod-scoutfs- Or use the binaries built from checked out git repositories:
yum install kernel-devel git clone git@github.com:versity/scoutfs-kmod-dev.git make -C scoutfs-kmod-dev module modprobe libcrc32c insmod scoutfs-kmod-dev/src/scoutfs.ko git clone git@github.com:versity/scoutfs-utils-dev.git make -C scoutfs-utils-dev alias scoutfs=$PWD/scoutfs-utils-dev/src/scoutfs -
Make a New Filesystem (destroys contents, no questions asked)
We specify that two of our three nodes must be present to form a quorum for the system to function.
scoutfs mkfs -Q 2 /dev/shared_block_device -
Mount the Filesystem
Each mounting node provides its local IP address on which it will run an internal server for the other mounts if it is elected the leader by the quorum.
mkdir /mnt/scoutfs mount -t scoutfs -o server_addr=$NODE_ADDR /dev/shared_block_device /mnt/scoutfs -
For Kicks, Observe the Metadata Change Index
The
meta_seqindex tracks the inodes that are changed in each transaction.scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/one; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/two; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/one; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs