Zach BrownandZach Brown
efd9763355
This btree implementation was first built for the relatively light duty of indexing segments in the LSM item implementation. We're now using it as the core metadata index. It's already using a lot of cpu to do its job with small blocks and it only gets more expensive as the block size increases. These changes reduce the CPU use of working with the btree block structures. We use a balanced binary tree to index items by key in the block. This gives us rare tree balancing cost on insertion and deletion instead of the memmove overhead of maintaining a dense array of item offsets sorted by key. The keys are stored in the item struct which are stored in an array at the front of the block so searching for an item uses contiguous cachelines. We add a trailing owner offset to values so that we can iterate through them. This is used to track space freed up by values instead of paying the memmove cost of keeping all the values at the end of the block. We occasionally reclaim the fragmented value free space instead of splitting the block. Direct item lookups use a small hash table at the end of the block which maps offsets to items. It uses linear probing and is guaranteed to have a light load factor so lookups are very likely to only need a single cache lookup. We adjust the watermark for triggering a join from half of a block down to a quarter. This results in less utilized blocks on average. But it creates distance between the join and split thresholds so we get less cpu use from constantly joining and splitting if item populations happen to hover around the previously shared threshold. While shifting the implementation we choose not to add support for some features that no longer make sense. There are no longer callers of _before and _after, and having synthetic tests to use small btree blocks no longer makes ense when we can easily create very tall trees. Both those btree interfaces and the tiny btree block support will be removed. Signed-off-by: Zach Brown <zab@versity.com>
Introduction
scoutfs is a clustered in-kernel Linux filesystem designed and built from the ground up to support large archival systems.
Its key differentiating features are:
- Integrated consistent indexing accelerates archival maintenance operations
- Log-structured commits allow nodes to write concurrently without contention
It meets best of breed expectations:
- Fully consistent POSIX semantics between nodes
- Rich metadata to ensure the integrity of metadata references
- Atomic transactions to maintain consistent persistent structures
- First class kernel implementation for high performance and low latency
- Open GPLv2 implementation
Learn more in the white paper.
Current Status
Alpha Open Source Development
scoutfs is under heavy active development. We're developing it in the open to give the community an opportunity to affect the design and implementation.
The core architectural design elements are in place. Much surrounding functionality hasn't been implemented. It's appropriate for early adopters and interested developers, not for production use.
In that vein, expect significant incompatible changes to both the format of network messages and persistent structures. To avoid mistakes the implementation currently calculates a hash of the format and ioctl header files in the source tree. The kernel module will refuse to mount a volume created by userspace utilities with a mismatched hash, and it will refuse to connect to a remote node with a mismatched hash. This means having to unmount, mkfs, and remount everything across many functional changes. Once the format is nailed down we'll wire up forward and back compat machinery and remove this temporary safety measure.
The current kernel module is developed against the RHEL/CentOS 7.x kernel to minimize the friction of developing and testing with partners' existing infrastructure. Once we're happy with the design we'll shift development to the upstream kernel while maintaining distro compatibility branches.
Community Mailing List
Please join us on the open scoutfs-devel@scoutfs.org mailing list hosted on Google Groups for all discussion of scoutfs.
Quick Start
This following a very rough example of the procedure to get up and running, experience will be needed to fill in the gaps. We're happy to help on the mailing list.
The requirements for running scoutfs on a small cluster are:
- One or more nodes running x86-64 CentOS/RHEL 7.4 (or 7.3)
- Access to a single shared block device
- IPv4 connectivity between the nodes
The steps for getting scoutfs mounted and operational are:
- Get the kernel module running on the nodes
- Make a new filesystem on the device with the userspace utilities
- Mount the device on all the nodes
In this example we run all of these commands on three nodes. The block device name is the same on all the nodes.
-
Get the Kernel Module and Userspace Binaries
- Either use snapshot RPMs built from git by Versity:
rpm -i https://scoutfs.s3-us-west-2.amazonaws.com/scoutfs-repo-0.0.1-1.el7_4.noarch.rpm yum install scoutfs-utils kmod-scoutfs- Or use the binaries built from checked out git repositories:
yum install kernel-devel git clone git@github.com:versity/scoutfs-kmod-dev.git make -C scoutfs-kmod-dev module modprobe libcrc32c insmod scoutfs-kmod-dev/src/scoutfs.ko git clone git@github.com:versity/scoutfs-utils-dev.git make -C scoutfs-utils-dev alias scoutfs=$PWD/scoutfs-utils-dev/src/scoutfs -
Make a New Filesystem (destroys contents, no questions asked)
We specify that two of our three nodes must be present to form a quorum for the system to function.
scoutfs mkfs -Q 2 /dev/shared_block_device -
Mount the Filesystem
Each mounting node provides its local IP address on which it will run an internal server for the other mounts if it is elected the leader by the quorum.
mkdir /mnt/scoutfs mount -t scoutfs -o server_addr=$NODE_ADDR /dev/shared_block_device /mnt/scoutfs -
For Kicks, Observe the Metadata Change Index
The
meta_seqindex tracks the inodes that are changed in each transaction.scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/one; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/two; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/one; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs