From 27e55eb43c5f63d6d484c2132d66de0573a7f4ff Mon Sep 17 00:00:00 2001 From: Zach Brown Date: Fri, 10 Mar 2017 15:34:29 -0800 Subject: [PATCH] Flesh out some pieces of the scoutfs.md doc Trying to keep adding coverage across the design. Signed-off-by: Zach Brown --- kmod/Documentation/scoutfs.md | 274 ++++++++++++++++++++++++++++++++++ 1 file changed, 274 insertions(+) diff --git a/kmod/Documentation/scoutfs.md b/kmod/Documentation/scoutfs.md index baf92616..e4387e7a 100644 --- a/kmod/Documentation/scoutfs.md +++ b/kmod/Documentation/scoutfs.md @@ -29,6 +29,280 @@ data residency interfaces for efficiently executing archival policies. It is deployed on a shared block fabric for high bandwidth and low latency. +## Super Block + +The super block is the anchor of all the persistent storage in the block +device. It contains volume-wide configuration information and +references to the current stable versions of persistent data structures +in the rest of the block device. The super block is stored in two 4KB +blocks at a known location at the start of the device. + +To read the current super block both block locations are read. The +valid super block with the most recent sequence number is used. Either +of the super blocks can be corrupt because they're overwritten in place +and a crash during a write could scramble the block. + +Each new version of the super block is written to the block that doesn't +contain the current super block. If this new super block write fails +then the old super block can still be used and no data is lost. + +The super block, and indeed all file system data, doesn't touch a few +blocks at the start of the device to avoid corrupting blocks that are +used by host platforms that store data inside devices to manage them. + +## Inodes + +Inodes are stored in items identified by the inode number. + + key = struct scoutfs_inode_key { + .type = SCOUTFS_INODE_KEY, + .ino, + } + + val = struct scoutfs_inode { + size, nlink, uid, gid, atime, mtime, ..., + } + +The variable length value that stores the item struct gives us dense +inode packing without having to predefine an inode storage size when the +file system is created and gives us a future expansion mechanism that +uses the item length to determine the version of the inode struct that +is written. + +Inode numbers are 64bit and are never re-used. By never re-using inode +numbers we don't need to manage an inode number allocator that would +need to be consistent across nodes. We can grant large ranges of +numbers to mount clients for allocation. Each inode number uniquely +identify the lifetime of a file and avoids having to store a seperate +generation number for each inode number. + +## Extended Attributes + +Extended attributes are stored in items on the inode at the full name of +the attribute. The attribute name is limited to 255 bytes and the +attribute values is limited to 64KB. The max xattr value size is larger +than our max item size so we can store an xattr in multiple items, but +in the common case a single xattr is efficiently stored in a single +item. + + key = struct scoutfs_xattr_key { + .type = SCOUTFS_XATTR_KEY, + .ino, + .name, + struct scoutfs_xattr_key_footer { + .null = '\0', + .part, + } + } + +Storing the null after the attribute name, which can't be found in any +name, lets us accurately locate a given name in the presence of other +names that share partial prefixes. The part identifies each key's +position in the set of keys that make up the large value. Storing the +full name in each key ensures that all the keys that make up an +attribute are stored adjacent to each other. + +Each item's value starts with a header which describes portion of the +attribute value stored in the item. + + val = struct scoutfs_xattr_val_header { + .part_len, + .last_part, + .data, + } + +The result of all this is that operations on xattrs iterate over keys +starting with the name and part 0 and stop when they hit the final part +(or error on corruption if the parts aren't consistent.) + +## Directory Entries + +Directory entry items store the target inode number referred to by a +given entry name in a parent directory. The name is limited to 255 +non-null bytes. The large keys supported by our items let us store +directory entries in items indexed by the full entry name itself. + + key = struct scoutfs_dirent_key { + .type = SCOUTFS_DIRENT_KEY, + .ino, + .name, + } + + val = struct scoutfs_dirent { + .ino, + .readdir_pos, + .type, + } + +These full precision items let us work on each item for a given name +directly rather than scrambling their sorting by storing them at a hash +value of their name. Storing at a hash value not only adds the +complexity of collisions, it critically causes entry lock attempts in a +directory between mounts to be perfectly randomly distributed and +constantly conflicting with each other. Storing and range locking the +directory entries at their full name preserves non-overlapping patterns +between mounts and gives them a chance to efficiently operate on +disjoint sets of names. + +We index the directory entry items by the full name of the entry so +there is no limit imposed on the number of entries in a directory. The +system will run out of blocks to store entries long before the index is +incapable of storing them. + +While we can satisfy lookups with a full precision index, readdir +doesn't use a full precision iterator. It forces us to describe each +entry with a small scalar directory position. We use a separate item +that's indexed by this readdir position instead of the file name. + + key = struct scoutfs_readdir_key { + .type = SCOUTFS_DIRENT_KEY, + .ino, + .readdir_pos, + } + + val = struct scoutfs_dirent { + .ino, + .readdir_pos, + .type, + .name, + } + +The key's position is allocated as each entry is created. This results +in readdir returning entries ordered by creation time. Like inode +numbers, readdir positions are never re-used so that we don't have to +risk contention by maintaining a consistent free position index across +nodes. + +## Directory Entry Link Backrefs + +The third and final item used by each directory entry is an item that is +stored at the target inode instead of in the parent directory. These +backref items can be traversed to find the full paths from the root +inode to all the entries that link to the target inode. + + key = struct scoutfs_link_backref_key { + .type = SCOUTFS_LINK_BACKREF_KEY, + .ino, + .dir_ino, + .name, + } + + /* no value */ + +Iterating over these items for a given target ino yields the parent +dir_ino and full file name of every entry that references the target +inode. The entry items in the parent dir are stored at the full file +name so the only way for us to reference them is with another copy of +the file name, brining the total to three full copies of the name stored +for every directory entry. + +Because we store the full name for these backref items they do not +impose a limit on the number of hard links to an inode. + +## Regular File Data Extents + +scoutfs stores file data in block extents at 4KB granularity. Items +describe the extents of 4KB blocks that map logical file offsets to +physical block extents in the device: + + key = struct scoutfs_extent_key { + .type = SCOUTFS_EXTENT_KEY, + .ino, + .iblock, + .blkno, + .count, + .flags, + } + + /* no value */ + +The flags field indicates the state of the extent, for example it can be +preallocated but unwritten or offline. If the extent is offline then +the blkno is unused and should be zero. + +Checksums of file data are contained in items at the physical block +offset of the checksumed blocks. Each item contains a fixed number of +checksums for a given group of blocks. + + key = struct scoutfs_checksum_key { + .type = SCOUTFS_CHECKSUM_KEY, + .blkno, + } + + val = { + .crcs[8], + } + +The checksum items are keyed by the physical block number instead of the +logical file position so that the checksum items are only written as new +data is written. The checksum items are left alone as the file data +references change: truncate, unlink, hole punching, and cloning don't +have to modify checksum items. + +With these structures in place the file read and write paths in scoutfs +look very much like most other block file systems in Linux. The generic +buffer_head support code is used and our get_blocks callback reads and +writes the extent items that reference block extents. Write and sync +patterns, with the help of delalloc, preallocation, and fallocate, +determine the physical contiguity of extent allocations. Buffered +read-ahead and O_DIRECT reads walk the extent items and build large +efficient bios if the extents are physically contiguous. + +## Allocating Regular File Data Extents + +The primary persistent allocator for blocks on the device uses an +efficient bitmap with a bit for each 1MB segment. File data allocation +wants to track extents at 4KB granularity and also index them by the +size of the free extent, neither of which the segment bitmap allocator +supports. + +We have free extent items that track free block extents in the device at +the finer 4K granularity. There are two keys for each free extent: one +indexed by the block location and one by the size of the free extent. +Modifying a free extent can thus modify three different positions in the +key namespace: the block location, the old size location, and the new +size location. LSM lets us generate and merge these disjoint items +across different mounts efficiently. + +To avoid the prohibitively expensive lock contention of modifying these +items from multiple mounts, we first create groups of free extents and +assign a given mount to a group for the lifetime of its mount. + + key = struct scoutfs_free_extent_loc_key + .type = SCOUTFS_FREE_EXTENT_LOC_KEY, + .group, + .blkno, + .count, + } + + key = struct scoutfs_free_extent_len_key + .type = SCOUTFS_FREE_EXTENT_LEN_KEY, + .group, + .count, + .blkno, + } + +Mounts are responsible for mangement of the free extent items. They're +populated with the result from requests from the manifest server for +free segment blocks. They're consumed as file data is written and +logical extents are allocated. They're repopulated as file data is +truncated and its extents are freed. They're returned to the segment +allocator when they contain aligned 1MB free extents. + +Like all persistent filesystem items, the free extent items are +protected by range locks. In the common case a single mount will be +operating on its group and having all the lock operations satisfied by +range matches. Any mount can modify any group's extents by acquiring +the right locks, but this should be limited to rare attempts to +defragment or migrate free extents between groups. + +The manifest server is responsible for tracking the assigment of mounts +to groups as mounts come and go through clean mounts and unclean crashes +and recovery. Free extents can get stranded in groups that don't have +an assigned mount. A mount scrambling to find free space in other +groups would need a mechanism to discover other groups, perhaps with a +set of keys that record the presence of extents in each group. + ## Indexing Inodes by Modification Time As files are modified archival agents need to find these modified files