mirror of
https://github.com/versity/scoutfs.git
synced 2026-02-07 19:20:44 +00:00
5bea29a168
The use of pages in the item cache got us pretty far but it fundmanetally couldn't escape the contention around the global or per-page read locks. Some loads became bottlenecked in contention in the item cache. Worse, we were seeing inconsistency in the per-cpu cached mappings of key ranges to pages. All the users of items in the cache are transitioned from searching for items in locked pages to searching for items in the cwskip list. It's fundamentally built around a seqlock-like begin/retry pattern so most of the item work gets wrapped around search and retry helpers. Without pages we no longer have a global list of dirty pages. Instead we have per-cpu lists of dirty items that are later sorted and handed to the btree insertion iterator. We take the opportunity to clean up that interface now that it's very easy for us to iterate through the stable list of dirty items. Rather than a global lru of pages we have an algorithm for maintaining items in rough groups of ages. Shrinking randomly walks the cwskip list looking for regions of sufficiently old items rather than walking a precise global lru list of pages. Signed-off-by: Zach Brown <zab@versity.com>
Introduction
scoutfs is a clustered in-kernel Linux filesystem designed to support large archival systems. It features additional interfaces and metadata so that archive agents can perform their maintenance workflows without walking all the files in the namespace. Its cluster support lets deployments add nodes to satisfy archival tier bandwidth targets.
The design goal is to reach file populations in the trillions, with the archival bandwidth to match, while remaining operational and responsive.
Highlights of the design and implementation include:
- Fully consistent POSIX semantics between nodes
- Atomic transactions to maintain consistent persistent structures
- Integrated archival metadata replaces syncing to external databases
- Dynamic seperation of resources lets nodes write in parallel
- 64bit throughout; no limits on file or directory sizes or counts
- Open GPLv2 implementation
Community Mailing List
Please join us on the open scoutfs-devel@scoutfs.org mailing list hosted on Google Groups