Enable and Disable correct unit tests for O_DIRECT

Signed-off-by: Bryant G. Duffy-Ly <bduffyly@versity.com>

ReleaseNotes.md

@@ -2,209 +2,9 @@ Versity ScoutFS Release Notes
 =============================
 
 ---
-v1.13
+v1.3-rc
 \
-*May 19, 2023*
-
-Add the quorum\_heartbeat\_timeout\_ms mount option to set the quorum
-heartbeat timeout.
-
-Change some task prioritization and allocation behavior of the quorum
-agent to help reduce delays in sending and receiving heartbeat messages.
-
----
-v1.12
-\
-*Apr 17, 2023*
-
-Add the prepare-empty-data-device scoutfs command.  A data device can be
-unused when no files have data blocks, perhaps because they're archived
-and offline.  In this case the data device can be swapped out for
-another device without changes to the metadata device.
-
-Fix an oversight which limited inode timestamps to second granularity
-for some operations.  All operations now record timestamps with full
-nanosecond precision.
-
-Fix spurious ENOENT failures when renaming from other directories into
-the root directory.
-
----
-v1.11
-\
-*Feb 2, 2023*
-
-Fixed a free extent processing error that could prevent mount from
-proceeding when free data extents were sufficiently fragmented.  It now
-properly handle very fragmented free extent maps.
-
-Fixed a statfs server processing race that could return spurious errors
-and shut down the server.  With the race closed statfs processing is
-reliable.
-
-Fixed a rare livelock in the move\_blocks ioctl.  With the right
-relationship between ioctl arguments and eventual file extent items the
-core loop in the move\_blocks ioctl could get stuck looping on an extent
-item and never return.  The loop exit conditions were fixed and the loop
-will always advance through all extents.
-
-Changed the 'print' scoutfs commands to flush the block cache for the
-devices.  It was inconvenient to expect cache flushing to be a separate
-step to ensure consistency with remote node writes.
-
----
-v1.10
-\
-*Dec 7, 2022*
-
-Fixed a potential directory entry cache management deadlock that could
-occur when many nodes performed heavy metadata write loads across shared
-directories and their child subdirectories.  The deadlock could halt
-invalidation progress on a node which could then stop use of locks that
-needed invalidation on that node which would result in almost all tasks
-hanging on those locks that would never make progress. 
-
-Fixed a circumstance where metadata change sequence index item
-modification could leave behind old stale metadata sequence items.  The
-duplication case required concurrent metadata updates across mounts with
-particular open transaction patterns so the duplicate items are rare.
-They resulted in a small amount of additional load when walking change
-indexes but had no effect on correctness.
-
-Fixed a rare case where sparse file extension might not write partial
-blocks of zeros which was found in testing.  This required using
-truncate to extend files past file sizes that end in partial blocks
-along with the right transaction commit and memory reclaim patterns.
-This never affected regular non-sparse files nor files prepopulated with
-fallocate.
-
----
-v1.9
-\
-*Oct 29, 2022*
-
-Fix VFS cached directory entry consistency verification that could cause
-spurious "no such file or directory" (ENOENT) errors from rename over
-NFS under certain conditions.  The problem was only every with the
-consistency of in-memory cached dentry objects, persistent data was
-correct and eventual eviction of the bad cached objects would stop
-generating the errors.
-
----
-v1.8
-\
-*Oct 18, 2022*
-
-Add support for Linux POSIX Access Control Lists, as described in
-acl(5).  Mount options are added to enable ("acl") and disable ("noacl")
-support.  The default is to support ACLs.  ACLs are stored in the
-existing extended attribute scheme so adding support is does not require
-a format change.
-
-Add options to control data extent preallocation.  The default behavior
-does not change.  The options can relax the limits on preallocation
-which will then trigger under more write patterns and increase the risk
-of preallocated space which is never used.  The options are described in
-scoutfs(5).
-
----
-v1.7
-\
-*Aug 26, 2022*
-
-* **Fixed possible persistent errors moving freed data extents**
-\
-  Fixed a case where the server could hit persistent errors trying to
-  move a client's freed extents in one commit.  The client had to free
-  a large number of extents that occupied distant positions in the
-  global free extent btree.  Very large fragmented files could cause
-  this.  The server now moves the freed extents in multiple commits and
-  can always ensure forward progress.
-
-* **Fixed possible persistent errors from freed duplicate extents**
-\
-  Background orphan deletion wasn't properly synchronizing with
-  foreground tasks deleting very large files.  If a deletion took long
-  enough then background deletion could also attempt to delete inode items
-  while the deletion was making progress.  This could create duplicate
-  deletions of data extent items which causes the server to abort when
-  it later discovers the duplicate extents as it merges free lists.
-
----
-v1.6
-\
-*Jul 7, 2022*
-
-* **Fix memory leaks in rare corner cases**
-\
-  Analysis tools found a few corner cases that leaked small structures,
-  generally around error handling or startup and shutdown.
-
-* **Add --skip-likely-huge scoutfs print command option**
-\
-  Add an option to scoutfs print to reduce the size of the output
-  so that it can be used to see system-wide metadata without being
-  overwhelmed by file-level details.
-
----
-v1.5
-\
-*Jun 21, 2022*
-
-* **Fix persistent error during server startup**
-\
-  Fixed a case where the server would always hit a consistent error on
-  seartup, preventing the system from mounting.  This required a rare
-  but valid state across the clients.
-
-* **Fix a client hang that would lead to fencing**
-\
-  The client module's use of in-kernel networking was missing annotation
-  that could lead to communication hanging.  The server would fence the
-  client when it stopped communicating.  This could be identified by the
-  server fencing a client after it disconnected with no attempt by the
-  client to reconnect.
-
----
-v1.4
-\
-*May 6, 2022*
-
-* **Fix possible client crash during server failover**
-\
-  Fixed a narrow window during server failover and lock recovery that
-  could cause a client mount to believe that it had an inconsistent item
-  cache and panic.  This required very specific lock state and messaging
-  patterns between multiple mounts and multiple servers which made it
-  unlikely to occur in the field.
-
----
-v1.3
-\
-*Apr 7, 2022*
-
-* **Fix rare server instability under heavy load**
-\
-  Fixed a case of server instability under heavy load due to concurrent
-  work fully exhausting metadata block allocation pools reserved for a
-  single server transaction.  This would cause brief interruption as the
-  server shutdown and the next server started up and made progress as
-  pending work was retried.
-
-* **Fix slow fencing preventing server startup**
-\
-  If a server had to process many fence requests with a slow fencing
-  mechanism it could be interrupted before it finished.  The server
-  now makes sure heartbeat messages are sent while it is making progress
-  on fencing requests so that other quorum members don't interrupt the
-  process.
-
-* **Performance improvement in getxattr and setxattr**
-\
-  Kernel allocation patterns in the getxattr and setxattr
-  implementations were causing significant contention between CPUs.  Their
-  allocation strategy was changed so that concurrent tasks can call these
-  xattr methods without degrading performance.
+*TBD*
 
 ---
 v1.2

kmod/src/Makefile

@@ -8,7 +8,6 @@ CFLAGS_scoutfs_trace.o = -I$(src) # define_trace.h double include
 -include $(src)/Makefile.kernelcompat
  
 scoutfs-y +=			\
-	acl.o			\
 	avl.o			\
 	alloc.o			\
 	block.o			\

kmod/src/Makefile.kernelcompat

@@ -34,12 +34,3 @@ endif
 ifneq (,$(shell grep 'FMODE_KABI_ITERATE' include/linux/fs.h))
 ccflags-y += -DKC_FMODE_KABI_ITERATE
 endif
-
-#
-# v4.7-rc2-23-g0d4d717f2583
-#
-# Added user_ns argument to posix_acl_valid
-#
-ifneq (,$(shell grep 'posix_acl_valid.*user_ns,' include/linux/posix_acl.h))
-ccflags-y += -DKC_POSIX_ACL_VALID_USER_NS
-endif

kmod/src/acl.c

@@ -1,355 +0,0 @@
-/*
- * Copyright (C) 2022 Versity Software, Inc.  All rights reserved.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public
- * License v2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- */
-#include <linux/kernel.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <linux/xattr.h>
-#include <linux/posix_acl.h>
-#include <linux/posix_acl_xattr.h>
-
-#include "format.h"
-#include "super.h"
-#include "scoutfs_trace.h"
-#include "xattr.h"
-#include "acl.h"
-#include "inode.h"
-#include "trans.h"
-
-/*
- * POSIX draft ACLs are stored as full xattr items with the entries
- * encoded as the kernel's posix_acl_xattr_{header,entry} value structs.
- *
- * They're accessed and modified via user facing synthetic xattrs, iops
- * calls from the kernel, during inode mode changes, and during inode
- * creation.
- *
- * ACL access devolves into xattr access which is relatively expensive
- * so we maintain the cached native form in the vfs inode.  We drop the
- * cache in lock invalidation which means that cached acl access must
- * always be performed under cluster locking.
- */
-
-static int acl_xattr_name_len(int type, char **name, size_t *name_len)
-{
-	int ret = 0;
-
-	switch (type) {
-	case ACL_TYPE_ACCESS:
-		*name = XATTR_NAME_POSIX_ACL_ACCESS;
-		if (name_len)
-			*name_len = sizeof(XATTR_NAME_POSIX_ACL_ACCESS) - 1;
-		break;
-	case ACL_TYPE_DEFAULT:
-		*name = XATTR_NAME_POSIX_ACL_DEFAULT;
-		if (name_len)
-			*name_len = sizeof(XATTR_NAME_POSIX_ACL_DEFAULT) - 1;
-		break;
-	default:
-		ret = -EINVAL;
-		break;
-	}
-
-	return ret;
-}
-
-struct posix_acl *scoutfs_get_acl_locked(struct inode *inode, int type, struct scoutfs_lock *lock)
-{
-	struct posix_acl *acl;
-	char *value = NULL;
-	char *name;
-	int ret;
-
-	if (!IS_POSIXACL(inode))
-		return NULL;
-
-	acl = get_cached_acl(inode, type);
-	if (acl != ACL_NOT_CACHED)
-		return acl;
-
-	ret = acl_xattr_name_len(type, &name, NULL);
-	if (ret < 0)
-		return ERR_PTR(ret);
-
-	ret = scoutfs_xattr_get_locked(inode, name, NULL, 0, lock);
-	if (ret > 0) {
-		value = kzalloc(ret, GFP_NOFS);
-		if (!value)
-			ret = -ENOMEM;
-		else
-			ret = scoutfs_xattr_get_locked(inode, name, value, ret, lock);
-	}
-	if (ret > 0) {
-		acl = posix_acl_from_xattr(&init_user_ns, value, ret);
-	} else if (ret == -ENODATA || ret == 0) {
-		acl = NULL;
-	} else {
-		acl = ERR_PTR(ret);
-	}
-
-	/* can set null negative cache */
-	if (!IS_ERR(acl))
-		set_cached_acl(inode, type, acl);
-
-	kfree(value);
-
-	return acl;
-}
-
-struct posix_acl *scoutfs_get_acl(struct inode *inode, int type)
-{
-	struct super_block *sb = inode->i_sb;
-	struct scoutfs_lock *lock = NULL;
-	struct posix_acl *acl;
-	int ret;
-
-	if (!IS_POSIXACL(inode))
-		return NULL;
-
-	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_READ, 0, inode, &lock);
-	if (ret < 0) {
-		acl = ERR_PTR(ret);
-	} else {
-		acl = scoutfs_get_acl_locked(inode, type, lock);
-		scoutfs_unlock(sb, lock, SCOUTFS_LOCK_READ);
-	}
-
-	return acl;
-}
-
-/*
- * The caller has acquired the locks and dirtied the inode, they'll
- * update the inode item if we return 0.
- */
-int scoutfs_set_acl_locked(struct inode *inode, struct posix_acl *acl, int type,
-			   struct scoutfs_lock *lock, struct list_head *ind_locks)
-{
-	static const struct scoutfs_xattr_prefix_tags tgs = {0,}; /* never scoutfs. prefix */
-	bool set_mode = false;
-	char *value = NULL;
-	umode_t new_mode;
-	size_t name_len;
-	char *name;
-	int size = 0;
-	int ret;
-
-	ret = acl_xattr_name_len(type, &name, &name_len);
-	if (ret < 0)
-		return ret;
-
-	switch (type) {
-	case ACL_TYPE_ACCESS:
-		if (acl) {
-			ret = posix_acl_update_mode(inode, &new_mode, &acl);
-			if (ret < 0)
-				goto out;
-			set_mode = true;
-		}
-		break;
-	case ACL_TYPE_DEFAULT:
-		if (!S_ISDIR(inode->i_mode)) {
-			ret = acl ? -EINVAL : 0;
-			goto out;
-		}
-		break;
-	}
-
-	if (acl) {
-		size = posix_acl_xattr_size(acl->a_count);
-		value = kmalloc(size, GFP_NOFS);
-		if (!value) {
-			ret = -ENOMEM;
-			goto out;
-		}
-
-		ret = posix_acl_to_xattr(&init_user_ns, acl, value, size);
-		if (ret < 0)
-			goto out;
-	}
-
-	ret = scoutfs_xattr_set_locked(inode, name, name_len, value, size, 0, &tgs,
-				       lock, NULL, ind_locks);
-	if (ret == 0 && set_mode) {
-		inode->i_mode = new_mode;
-		if (!value) {
-			/* can be setting an acl that only affects mode, didn't need xattr */
-			inode_inc_iversion(inode);
-			inode->i_ctime = CURRENT_TIME;
-		}
-	}
-
-out:
-	if (!ret)
-		set_cached_acl(inode, type, acl);
-
-	kfree(value);
-
-	return ret;
-}
-
-int scoutfs_set_acl(struct inode *inode, struct posix_acl *acl, int type)
-{
-	struct super_block *sb = inode->i_sb;
-	struct scoutfs_lock *lock = NULL;
-	LIST_HEAD(ind_locks);
-	int ret;
-
-	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_WRITE, SCOUTFS_LKF_REFRESH_INODE, inode, &lock) ?:
-	      scoutfs_inode_index_lock_hold(inode, &ind_locks, false, true);
-	if (ret == 0) {
-		ret = scoutfs_dirty_inode_item(inode, lock) ?:
-		      scoutfs_set_acl_locked(inode, acl, type, lock, &ind_locks);
-		if (ret == 0)
-			scoutfs_update_inode_item(inode, lock, &ind_locks);
-
-		scoutfs_release_trans(sb);
-		scoutfs_inode_index_unlock(sb, &ind_locks);
-	}
-
-	scoutfs_unlock(sb, lock, SCOUTFS_LOCK_WRITE);
-	return ret;
-}
-
-int scoutfs_acl_get_xattr(struct dentry *dentry, const char *name, void *value, size_t size,
-			  int type)
-{
-	struct posix_acl *acl;
-	int ret = 0;
-
-	if (!IS_POSIXACL(dentry->d_inode))
-		return -EOPNOTSUPP;
-
-	acl = scoutfs_get_acl(dentry->d_inode, type);
-	if (IS_ERR(acl))
-		return PTR_ERR(acl);
-	if (acl == NULL)
-		return -ENODATA;
-
-	ret = posix_acl_to_xattr(&init_user_ns, acl, value, size);
-	posix_acl_release(acl);
-
-	return ret;
-}
-
-int scoutfs_acl_set_xattr(struct dentry *dentry, const char *name, const void *value, size_t size,
-			  int flags, int type)
-{
-	struct posix_acl *acl = NULL;
-	int ret;
-
-	if (!inode_owner_or_capable(dentry->d_inode))
-		return -EPERM;
-
-	if (!IS_POSIXACL(dentry->d_inode))
-		return -EOPNOTSUPP;
-
-	if (value) {
-		acl = posix_acl_from_xattr(&init_user_ns, value, size);
-		if (IS_ERR(acl))
-			return PTR_ERR(acl);
-
-		if (acl) {
-			ret = kc_posix_acl_valid(&init_user_ns, acl);
-			if (ret)
-				goto out;
-		}
-	}
-
-	ret = scoutfs_set_acl(dentry->d_inode, acl, type);
-out:
-	posix_acl_release(acl);
-
-	return ret;
-}
-
-/*
- * Apply the parent's default acl to new inodes access acl and inherit
- * it as the default for new directories.  The caller holds locks and a
- * transaction.
- */
-int scoutfs_init_acl_locked(struct inode *inode, struct inode *dir,
-			    struct scoutfs_lock *lock, struct scoutfs_lock *dir_lock,
-			    struct list_head *ind_locks)
-{
-	struct posix_acl *acl = NULL;
-	int ret = 0;
-
-	if (!S_ISLNK(inode->i_mode)) {
-		if (IS_POSIXACL(dir)) {
-			acl = scoutfs_get_acl_locked(dir, ACL_TYPE_DEFAULT, dir_lock);
-			if (IS_ERR(acl))
-				return PTR_ERR(acl);
-		}
-
-		if (!acl)
-			inode->i_mode &= ~current_umask();
-	}
-
-	if (IS_POSIXACL(dir) && acl) {
-		if (S_ISDIR(inode->i_mode)) {
-			ret = scoutfs_set_acl_locked(inode, acl, ACL_TYPE_DEFAULT,
-						     lock, ind_locks);
-			if (ret)
-				goto out;
-		}
-		ret = posix_acl_create(&acl, GFP_NOFS, &inode->i_mode);
-		if (ret < 0)
-			return ret;
-		if (ret > 0)
-			ret = scoutfs_set_acl_locked(inode, acl, ACL_TYPE_ACCESS,
-						     lock, ind_locks);
-	} else {
-		cache_no_acl(inode);
-	}
-out:
-	posix_acl_release(acl);
-	return ret;
-}
-
-/*
- * Update the access ACL based on a newly set mode.  If we return an
- * error then the xattr wasn't changed.
- *
- * Annoyingly, setattr_copy has logic that transforms the final set mode
- * that we want to use to update the acl.   But we don't want to modify
- * the other inode fields while discovering the resulting mode.  We're
- * relying on acl_chmod not caring about the transformation (currently
- * just clears sgid).  It would be better if we could get the resulting
- * mode to give to acl_chmod without modifying the other inode fields.
- *
- * The caller has the inode mutex, a cluster lock, transaction, and will
- * update the inode item if we return success.
- */
-int scoutfs_acl_chmod_locked(struct inode *inode, struct iattr *attr,
-			     struct scoutfs_lock *lock, struct list_head *ind_locks)
-{
-	struct posix_acl *acl;
-	int ret = 0;
-
-	if (!IS_POSIXACL(inode) || !(attr->ia_valid & ATTR_MODE))
-		return 0;
-
-	if (S_ISLNK(inode->i_mode))
-		return -EOPNOTSUPP;
-
-	acl = scoutfs_get_acl_locked(inode, ACL_TYPE_ACCESS, lock);
-	if (IS_ERR_OR_NULL(acl))
-		return PTR_ERR(acl);
-
-	ret = posix_acl_chmod(&acl, GFP_KERNEL, attr->ia_mode);
-	if (ret)
-		return ret;
-
-	ret = scoutfs_set_acl_locked(inode, acl, ACL_TYPE_ACCESS, lock, ind_locks);
-	posix_acl_release(acl);
-	return ret;
-}

kmod/src/acl.h

@@ -1,18 +0,0 @@
-#ifndef _SCOUTFS_ACL_H_
-#define _SCOUTFS_ACL_H_
-
-struct posix_acl *scoutfs_get_acl(struct inode *inode, int type);
-struct posix_acl *scoutfs_get_acl_locked(struct inode *inode, int type, struct scoutfs_lock *lock);
-int scoutfs_set_acl(struct inode *inode, struct posix_acl *acl, int type);
-int scoutfs_set_acl_locked(struct inode *inode, struct posix_acl *acl, int type,
-			   struct scoutfs_lock *lock, struct list_head *ind_locks);
-int scoutfs_acl_get_xattr(struct dentry *dentry, const char *name, void *value, size_t size,
-			  int type);
-int scoutfs_acl_set_xattr(struct dentry *dentry, const char *name, const void *value, size_t size,
-			  int flags, int type);
-int scoutfs_acl_chmod_locked(struct inode *inode, struct iattr *attr,
-			     struct scoutfs_lock *lock, struct list_head *ind_locks);
-int scoutfs_init_acl_locked(struct inode *inode, struct inode *dir,
-			    struct scoutfs_lock *lock, struct scoutfs_lock *dir_lock,
-			    struct list_head *ind_locks);
-#endif

kmod/src/alloc.c

@@ -84,21 +84,6 @@ static u64 smallest_order_length(u64 len)
 	return 1ULL << (free_extent_order(len) * 3);
 }
 
-/*
- * An extent modification dirties three distinct leaves of an allocator
- * btree as it adds and removes the blkno and size sorted items for the
- * old and new lengths of the extent.  Dirtying the paths to these
- * leaves can grow the tree and grow/shrink neighbours at each level.
- * We over-estimate the number of blocks allocated and freed (the paths
- * share a root, growth doesn't free) to err on the simpler and safer
- * side.  The overhead is minimal given the relatively large list blocks
- * and relatively short allocator trees.
- */
-static u32 extent_mod_blocks(u32 height)
-{
-	return ((1 + height) * 2) * 3;
-}
-
 /*
  * Free extents don't have flags and are stored in two indexes sorted by
  * block location and by length order, largest first.  The location key
@@ -892,13 +877,6 @@ static int find_zone_extent(struct super_block *sb, struct scoutfs_alloc_root *r
  * -ENOENT is returned if we run out of extents in the source tree
  * before moving the total.
  *
- * If meta_budget is non-zero then -EINPROGRESS can be returned if the
- * the caller's budget is consumed in the allocator during this call
- * (though not necessarily by us, we don't have per-thread tracking of
- * allocator consumption :/).  The call can still have made progress and
- * caller is expected commit the dirty trees and examining the resulting
- * modified trees to see if they need to continue moving extents.
- *
  * The caller can specify that extents in the source tree should first
  * be found based on their zone bitmaps.  We'll first try to find
  * extents in the exclusive zones, then vacant zones, and then we'll
@@ -913,7 +891,7 @@ int scoutfs_alloc_move(struct super_block *sb, struct scoutfs_alloc *alloc,
 		       struct scoutfs_block_writer *wri,
 		       struct scoutfs_alloc_root *dst,
 		       struct scoutfs_alloc_root *src, u64 total,
-		       __le64 *exclusive, __le64 *vacant, u64 zone_blocks, u64 meta_budget)
+		       __le64 *exclusive, __le64 *vacant, u64 zone_blocks)
 {
 	struct alloc_ext_args args = {
 		.alloc = alloc,
@@ -921,8 +899,6 @@ int scoutfs_alloc_move(struct super_block *sb, struct scoutfs_alloc *alloc,
 	};
 	struct scoutfs_extent found;
 	struct scoutfs_extent ext;
-	u32 avail_start = 0;
-	u32 freed_start = 0;
 	u64 moved = 0;
 	u64 count;
 	int ret = 0;
@@ -933,9 +909,6 @@ int scoutfs_alloc_move(struct super_block *sb, struct scoutfs_alloc *alloc,
 		vacant = NULL;
 	}
 
-	if (meta_budget != 0)
-		scoutfs_alloc_meta_remaining(alloc, &avail_start, &freed_start);
-
 	while (moved < total) {
 		count = total - moved;
 
@@ -968,24 +941,6 @@ int scoutfs_alloc_move(struct super_block *sb, struct scoutfs_alloc *alloc,
 		if (ret < 0)
 			break;
 
-		if (meta_budget != 0 &&
-		    scoutfs_alloc_meta_low_since(alloc, avail_start, freed_start, meta_budget,
-						 extent_mod_blocks(src->root.height) +
-						 extent_mod_blocks(dst->root.height))) {
-			ret = -EINPROGRESS;
-			break;
-		}
-
-		/* return partial if the server alloc can't dirty any more */
-		if (scoutfs_alloc_meta_low(sb, alloc, 50 + extent_mod_blocks(src->root.height) +
-						      extent_mod_blocks(dst->root.height))) {
-			if (WARN_ON_ONCE(!moved))
-				ret = -ENOSPC;
-			else
-				ret = 0;
-			break;
-		}
-
 		/* searching set start/len, finish initializing alloced extent */
 		ext.map = found.map ? ext.start - found.start + found.map : 0;
 		ext.flags = found.flags;
@@ -1110,6 +1065,15 @@ out:
  * than completely exhausting the avail list or overflowing the freed
  * list.
  *
+ * An extent modification dirties three distinct leaves of an allocator
+ * btree as it adds and removes the blkno and size sorted items for the
+ * old and new lengths of the extent.  Dirtying the paths to these
+ * leaves can grow the tree and grow/shrink neighbours at each level.
+ * We over-estimate the number of blocks allocated and freed (the paths
+ * share a root, growth doesn't free) to err on the simpler and safer
+ * side.  The overhead is minimal given the relatively large list blocks
+ * and relatively short allocator trees.
+ *
  * The caller tells us how many extents they're about to modify and how
  * many other additional blocks they may cow manually.  And finally, the
  * caller could be the first to dirty the avail and freed blocks in the
@@ -1118,7 +1082,7 @@ out:
 static bool list_has_blocks(struct super_block *sb, struct scoutfs_alloc *alloc,
 			    struct scoutfs_alloc_root *root, u32 extents, u32 addl_blocks)
 {
-	u32 tree_blocks = extent_mod_blocks(root->root.height) * extents;
+	u32 tree_blocks = (((1 + root->root.height) * 2) * 3) * extents;
 	u32 most = 1 + tree_blocks + addl_blocks;
 
 	if (le32_to_cpu(alloc->avail.first_nr) < most) {
@@ -1354,38 +1318,6 @@ bool scoutfs_alloc_meta_low(struct super_block *sb,
 	return lo;
 }
 
-void scoutfs_alloc_meta_remaining(struct scoutfs_alloc *alloc, u32 *avail_total, u32 *freed_space)
-{
-	unsigned int seq;
-
-	do {
-		seq = read_seqbegin(&alloc->seqlock);
-		*avail_total = le32_to_cpu(alloc->avail.first_nr);
-		*freed_space = list_block_space(alloc->freed.first_nr);
-	} while (read_seqretry(&alloc->seqlock, seq));
-}
-
-/*
- * Returns true if the caller's consumption of nr from either avail or
- * freed would end up exceeding their budget relative to the starting
- * remaining snapshot they took.
- */
-bool scoutfs_alloc_meta_low_since(struct scoutfs_alloc *alloc, u32 avail_start, u32 freed_start,
-				  u32 budget, u32 nr)
-{
-	u32 avail_use;
-	u32 freed_use;
-	u32 avail;
-	u32 freed;
-
-	scoutfs_alloc_meta_remaining(alloc, &avail, &freed);
-
-	avail_use = avail_start - avail;
-	freed_use = freed_start - freed;
-
-	return ((avail_use + nr) > budget) || ((freed_use + nr) > budget);
-}
-
 bool scoutfs_alloc_test_flag(struct super_block *sb,
 			    struct scoutfs_alloc *alloc, u32 flag)
 {
@@ -1582,10 +1514,12 @@ out:
  * call the caller's callback.  This assumes that the super it's reading
  * could be stale and will retry if it encounters stale blocks.
  */
-int scoutfs_alloc_foreach(struct super_block *sb, scoutfs_alloc_foreach_cb_t cb, void *arg)
+int scoutfs_alloc_foreach(struct super_block *sb,
+			  scoutfs_alloc_foreach_cb_t cb, void *arg)
 {
 	struct scoutfs_super_block *super = NULL;
-	DECLARE_SAVED_REFS(saved);
+	struct scoutfs_block_ref stale_refs[2] = {{0,}};
+	struct scoutfs_block_ref refs[2] = {{0,}};
 	int ret;
 
 	super = kmalloc(sizeof(struct scoutfs_super_block), GFP_NOFS);
@@ -1594,18 +1528,26 @@ int scoutfs_alloc_foreach(struct super_block *sb, scoutfs_alloc_foreach_cb_t cb,
 		goto out;
 	}
 
-	do {
-		ret = scoutfs_read_super(sb, super);
-		if (ret < 0)
-			goto out;
+retry:
+	ret = scoutfs_read_super(sb, super);
+	if (ret < 0)
+		goto out;
 
-		ret = scoutfs_alloc_foreach_super(sb, super, cb, arg);
-
-		ret = scoutfs_block_check_stale(sb, ret, &saved, &super->logs_root.ref,
-						&super->srch_root.ref);
-	} while (ret == -ESTALE);
+	refs[0] = super->logs_root.ref;
+	refs[1] = super->srch_root.ref;
 
+	ret = scoutfs_alloc_foreach_super(sb, super, cb, arg);
 out:
+	if (ret == -ESTALE) {
+		if (memcmp(&stale_refs, &refs, sizeof(refs)) == 0) {
+			ret = -EIO;
+		} else {
+			BUILD_BUG_ON(sizeof(stale_refs) != sizeof(refs));
+			memcpy(stale_refs, refs, sizeof(stale_refs));
+			goto retry;
+		}
+	}
+
 	kfree(super);
 	return ret;
 }

kmod/src/alloc.h

@@ -19,11 +19,14 @@
 	(128ULL * 1024 * 1024 >> SCOUTFS_BLOCK_SM_SHIFT)
 
 /*
- * The default size that we'll try to preallocate.  This is trying to
- * hit the limit of large efficient device writes while minimizing
- * wasted preallocation that is never used.
+ * The largest aligned region that we'll try to allocate at the end of
+ * the file as it's extended.  This is also limited to the current file
+ * size so we can only waste at most twice the total file size when
+ * files are less than this.  We try to keep this around the point of
+ * diminishing returns in streaming performance of common data devices
+ * to limit waste.
  */
-#define SCOUTFS_DATA_PREALLOC_DEFAULT_BLOCKS \
+#define SCOUTFS_DATA_EXTEND_PREALLOC_LIMIT \
 	(8ULL * 1024 * 1024 >> SCOUTFS_BLOCK_SM_SHIFT)
 
 /*
@@ -128,7 +131,7 @@ int scoutfs_alloc_move(struct super_block *sb, struct scoutfs_alloc *alloc,
 		       struct scoutfs_block_writer *wri,
 		       struct scoutfs_alloc_root *dst,
 		       struct scoutfs_alloc_root *src, u64 total,
-		       __le64 *exclusive, __le64 *vacant, u64 zone_blocks, u64 meta_budget);
+		       __le64 *exclusive, __le64 *vacant, u64 zone_blocks);
 int scoutfs_alloc_insert(struct super_block *sb, struct scoutfs_alloc *alloc,
 			 struct scoutfs_block_writer *wri, struct scoutfs_alloc_root *root,
 			 u64 start, u64 len);
@@ -155,9 +158,6 @@ int scoutfs_alloc_splice_list(struct super_block *sb,
 
 bool scoutfs_alloc_meta_low(struct super_block *sb,
 			    struct scoutfs_alloc *alloc, u32 nr);
-void scoutfs_alloc_meta_remaining(struct scoutfs_alloc *alloc, u32 *avail_total, u32 *freed_space);
-bool scoutfs_alloc_meta_low_since(struct scoutfs_alloc *alloc, u32 avail_start, u32 freed_start,
-				  u32 budget, u32 nr);
 bool scoutfs_alloc_test_flag(struct super_block *sb,
 			    struct scoutfs_alloc *alloc, u32 flag);
 

kmod/src/block.c

@@ -677,7 +677,7 @@ out:
 int scoutfs_block_read_ref(struct super_block *sb, struct scoutfs_block_ref *ref, u32 magic,
 			   struct scoutfs_block **bl_ret)
 {
-	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	struct scoutfs_block_header *hdr;
 	struct block_private *bp = NULL;
 	bool retried = false;
@@ -701,7 +701,7 @@ retry:
 		set_bit(BLOCK_BIT_CRC_VALID, &bp->bits);
 	}
 
-	if (hdr->magic != cpu_to_le32(magic) || hdr->fsid != cpu_to_le64(sbi->fsid) ||
+	if (hdr->magic != cpu_to_le32(magic) || hdr->fsid != super->hdr.fsid ||
 	    hdr->seq != ref->seq || hdr->blkno != ref->blkno) {
 		ret = -ESTALE;
 		goto out;
@@ -728,36 +728,6 @@ out:
 	return ret;
 }
 
-static bool stale_refs_match(struct scoutfs_block_ref *caller, struct scoutfs_block_ref *saved)
-{
-	return !caller || (caller->blkno == saved->blkno && caller->seq == saved->seq);
-}
-
-/*
- * Check if a read of a reference that gave ESTALE should be retried or
- * should generate a hard error.  If this is the second time we got
- * ESTALE from the same refs then we return EIO and the caller should
- * stop.  As long as we keep seeing different refs we'll return ESTALE
- * and the caller can keep trying.
- */
-int scoutfs_block_check_stale(struct super_block *sb, int ret,
-			      struct scoutfs_block_saved_refs *saved,
-			      struct scoutfs_block_ref *a, struct scoutfs_block_ref *b)
-{
-	if (ret == -ESTALE) {
-		if (stale_refs_match(a, &saved->refs[0]) && stale_refs_match(b, &saved->refs[1])){
-			ret = -EIO;
-		} else {
-			if (a)
-				saved->refs[0] = *a;
-			if (b)
-				saved->refs[1] = *b;
-		}
-	}
-
-	return ret;
-}
-
 void scoutfs_block_put(struct super_block *sb, struct scoutfs_block *bl)
 {
 	if (!IS_ERR_OR_NULL(bl))
@@ -827,7 +797,7 @@ int scoutfs_block_dirty_ref(struct super_block *sb, struct scoutfs_alloc *alloc,
 			    u32 magic, struct scoutfs_block **bl_ret,
 			    u64 dirty_blkno, u64 *ref_blkno)
 {
-	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	struct scoutfs_block *cow_bl = NULL;
 	struct scoutfs_block *bl = NULL;
 	struct block_private *exist_bp = NULL;
@@ -895,7 +865,7 @@ int scoutfs_block_dirty_ref(struct super_block *sb, struct scoutfs_alloc *alloc,
 
 	hdr = bl->data;
 	hdr->magic = cpu_to_le32(magic);
-	hdr->fsid = cpu_to_le64(sbi->fsid);
+	hdr->fsid = super->hdr.fsid;
 	hdr->blkno = cpu_to_le64(bl->blkno);
 	prandom_bytes(&hdr->seq, sizeof(hdr->seq));
 

kmod/src/block.h

@@ -13,17 +13,6 @@ struct scoutfs_block {
 	void *priv;
 };
 
-struct scoutfs_block_saved_refs {
-	struct scoutfs_block_ref refs[2];
-};
-
-#define DECLARE_SAVED_REFS(name) \
-	struct scoutfs_block_saved_refs name = {{{0,}}}
-
-int scoutfs_block_check_stale(struct super_block *sb, int ret,
-			      struct scoutfs_block_saved_refs *saved,
-			      struct scoutfs_block_ref *a, struct scoutfs_block_ref *b);
-
 int scoutfs_block_read_ref(struct super_block *sb, struct scoutfs_block_ref *ref, u32 magic,
 			   struct scoutfs_block **bl_ret);
 void scoutfs_block_put(struct super_block *sb, struct scoutfs_block *bl);

kmod/src/btree.c

@@ -2449,7 +2449,7 @@ int scoutfs_btree_free_blocks(struct super_block *sb,
 			      struct scoutfs_alloc *alloc,
 			      struct scoutfs_block_writer *wri,
 			      struct scoutfs_key *key,
-			      struct scoutfs_btree_root *root, int free_budget)
+			      struct scoutfs_btree_root *root, int alloc_low)
 {
 	u64 blknos[SCOUTFS_BTREE_MAX_HEIGHT];
 	struct scoutfs_block *bl = NULL;
@@ -2459,15 +2459,11 @@ int scoutfs_btree_free_blocks(struct super_block *sb,
 	struct scoutfs_avl_node *node;
 	struct scoutfs_avl_node *next;
 	struct scoutfs_key par_next;
-	int nr_freed = 0;
 	int nr_par;
 	int level;
 	int ret;
 	int i;
 
-	if (WARN_ON_ONCE(free_budget <= 0))
-		return -EINVAL;
-
 	if (WARN_ON_ONCE(root->height > ARRAY_SIZE(blknos)))
 		return -EIO; /* XXX corruption */
 
@@ -2542,7 +2538,8 @@ int scoutfs_btree_free_blocks(struct super_block *sb,
 		while (node) {
 
 			/* make sure we can always free parents after leaves */
-			if ((nr_freed + 1 + nr_par) > free_budget) {
+			if (scoutfs_alloc_meta_low(sb, alloc,
+						   alloc_low + nr_par + 1)) {
 				ret = 0;
 				goto out;
 			}
@@ -2556,7 +2553,6 @@ int scoutfs_btree_free_blocks(struct super_block *sb,
 						le64_to_cpu(ref.blkno));
 			if (ret < 0)
 				goto out;
-			nr_freed++;
 
 			node = scoutfs_avl_next(&bt->item_root, node);
 			if (node) {
@@ -2572,7 +2568,6 @@ int scoutfs_btree_free_blocks(struct super_block *sb,
 							       blknos[i]);
 			ret = scoutfs_free_meta(sb, alloc, wri, blknos[i]);
 			BUG_ON(ret); /* checked meta low, freed should fit */
-			nr_freed++;
 		}
 
 		/* restart walk past the subtree we just freed */

kmod/src/btree.h

@@ -125,7 +125,7 @@ int scoutfs_btree_free_blocks(struct super_block *sb,
 			      struct scoutfs_alloc *alloc,
 			      struct scoutfs_block_writer *wri,
 			      struct scoutfs_key *key,
-			      struct scoutfs_btree_root *root, int free_budget);
+			      struct scoutfs_btree_root *root, int alloc_low);
 
 void scoutfs_btree_put_iref(struct scoutfs_btree_item_ref *iref);
 

kmod/src/client.c

@@ -356,6 +356,7 @@ static int client_greeting(struct super_block *sb,
 {
 	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
 	struct client_info *client = sbi->client_info;
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	struct scoutfs_net_greeting *gr = resp;
 	bool new_server;
 	int ret;
@@ -370,9 +371,9 @@ static int client_greeting(struct super_block *sb,
 		goto out;
 	}
 
-	if (gr->fsid != cpu_to_le64(sbi->fsid)) {
+	if (gr->fsid != super->hdr.fsid) {
 		scoutfs_warn(sb, "server greeting response fsid 0x%llx did not match client fsid 0x%llx",
-			     le64_to_cpu(gr->fsid), sbi->fsid);
+			     le64_to_cpu(gr->fsid), le64_to_cpu(super->hdr.fsid));
 		ret = -EINVAL;
 		goto out;
 	}
@@ -475,6 +476,7 @@ static void scoutfs_client_connect_worker(struct work_struct *work)
 						  connect_dwork.work);
 	struct super_block *sb = client->sb;
 	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &sbi->super;
 	struct scoutfs_mount_options opts;
 	struct scoutfs_net_greeting greet;
 	struct sockaddr_in sin;
@@ -506,7 +508,7 @@ static void scoutfs_client_connect_worker(struct work_struct *work)
 		goto out;
 
 	/* send a greeting to verify endpoints of each connection */
-	greet.fsid = cpu_to_le64(sbi->fsid);
+	greet.fsid = super->hdr.fsid;
 	greet.fmt_vers = cpu_to_le64(sbi->fmt_vers);
 	greet.server_term = cpu_to_le64(client->server_term);
 	greet.rid = cpu_to_le64(sbi->rid);

kmod/src/counters.h

@@ -75,6 +75,8 @@
 	EXPAND_COUNTER(data_write_begin_enobufs_retry)		\
 	EXPAND_COUNTER(dentry_revalidate_error)			\
 	EXPAND_COUNTER(dentry_revalidate_invalid)		\
+	EXPAND_COUNTER(dentry_revalidate_locked)		\
+	EXPAND_COUNTER(dentry_revalidate_orphan)		\
 	EXPAND_COUNTER(dentry_revalidate_rcu)			\
 	EXPAND_COUNTER(dentry_revalidate_root)			\
 	EXPAND_COUNTER(dentry_revalidate_valid)			\
@@ -155,7 +157,6 @@
 	EXPAND_COUNTER(orphan_scan_error)			\
 	EXPAND_COUNTER(orphan_scan_item)			\
 	EXPAND_COUNTER(orphan_scan_omap_set)			\
-	EXPAND_COUNTER(quorum_candidate_server_stopping)	\
 	EXPAND_COUNTER(quorum_elected)				\
 	EXPAND_COUNTER(quorum_fence_error)			\
 	EXPAND_COUNTER(quorum_fence_leader)			\
@@ -166,7 +167,6 @@
 	EXPAND_COUNTER(quorum_recv_resignation)			\
 	EXPAND_COUNTER(quorum_recv_vote)			\
 	EXPAND_COUNTER(quorum_send_heartbeat)			\
-	EXPAND_COUNTER(quorum_send_heartbeat_dropped)		\
 	EXPAND_COUNTER(quorum_send_resignation)			\
 	EXPAND_COUNTER(quorum_send_request)			\
 	EXPAND_COUNTER(quorum_send_vote)			\
@@ -188,6 +188,8 @@
 	EXPAND_COUNTER(srch_search_retry_empty)			\
 	EXPAND_COUNTER(srch_search_sorted)			\
 	EXPAND_COUNTER(srch_search_sorted_block)		\
+	EXPAND_COUNTER(srch_search_stale_eio)			\
+	EXPAND_COUNTER(srch_search_stale_retry)			\
 	EXPAND_COUNTER(srch_search_xattrs)			\
 	EXPAND_COUNTER(srch_read_stale)				\
 	EXPAND_COUNTER(statfs)					\

kmod/src/data.c

@@ -21,6 +21,7 @@
 #include <linux/log2.h>
 #include <linux/falloc.h>
 #include <linux/writeback.h>
+#include <linux/aio.h>
 
 #include "format.h"
 #include "super.h"
@@ -366,27 +367,27 @@ static inline u64 ext_last(struct scoutfs_extent *ext)
 
 /*
  * The caller is writing to a logical iblock that doesn't have an
- * allocated extent.  The caller has searched for an extent containing
- * iblock.  If it already existed then it must be unallocated and
- * offline.
+ * allocated extent.
  *
- * We implement two preallocation strategies.  Typically we only
- * preallocate for simple streaming writes and limit preallocation while
- * the file is small.   The largest efficient allocation size is
- * typically large enough that it would be unreasonable to allocate that
- * much for all small files.
+ * We always allocate an extent starting at the logical iblock.  The
+ * caller has searched for an extent containing iblock.  If it already
+ * existed then it must be unallocated and offline.
  *
- * Optionally, we can simply preallocate large empty aligned regions.
- * This can waste a lot of space for small or sparse files but is
- * reasonable when a file population is known to be large and dense but
- * known to be written with non-streaming write patterns.
+ * Preallocation is used if we're strictly contiguously extending
+ * writes.  That is, if the logical block offset equals the number of
+ * online blocks.  We try to preallocate the number of blocks existing
+ * so that small files don't waste inordinate amounts of space and large
+ * files will eventually see large extents.  This only works for
+ * contiguous single stream writes or stages of files from the first
+ * block.  It doesn't work for concurrent stages, releasing behind
+ * staging, sparse files, multi-node writes, etc.  fallocate() is always
+ * a better tool to use.
  */
 static int alloc_block(struct super_block *sb, struct inode *inode,
 		       struct scoutfs_extent *ext, u64 iblock,
 		       struct scoutfs_lock *lock)
 {
 	DECLARE_DATA_INFO(sb, datinf);
-	struct scoutfs_mount_options opts;
 	const u64 ino = scoutfs_ino(inode);
 	struct data_ext_args args = {
 		.ino = ino,
@@ -394,22 +395,17 @@ static int alloc_block(struct super_block *sb, struct inode *inode,
 		.lock = lock,
 	};
 	struct scoutfs_extent found;
-	struct scoutfs_extent pre = {0,};
-	bool undo_pre = false;
+	struct scoutfs_extent pre;
 	u64 blkno = 0;
 	u64 online;
 	u64 offline;
 	u8 flags;
-	u64 start;
 	u64 count;
-	u64 rem;
 	int ret;
 	int err;
 
 	trace_scoutfs_data_alloc_block_enter(sb, ino, iblock, ext);
 
-	scoutfs_options_read(sb, &opts);
-
 	/* can only allocate over existing unallocated offline extent */
 	if (WARN_ON_ONCE(ext->len &&
 			 !(iblock >= ext->start && iblock <= ext_last(ext) &&
@@ -418,106 +414,67 @@ static int alloc_block(struct super_block *sb, struct inode *inode,
 
 	mutex_lock(&datinf->mutex);
 
-	/* default to single allocation at the written block */
-	start = iblock;
-	count = 1;
-	/* copy existing flags for preallocated regions */
-	flags = ext->len ? ext->flags : 0;
+	scoutfs_inode_get_onoff(inode, &online, &offline);
 
 	if (ext->len) {
-		/*
-		 * Assume that offline writers are going to be writing
-		 * all the offline extents and try to preallocate the
-		 * rest of the unwritten extent.
-		 */
+		/* limit preallocation to remaining existing (offline) extent */
 		count = ext->len - (iblock - ext->start);
-
-	} else if (opts.data_prealloc_contig_only) {
-		/*
-		 * Only preallocate when a quick test of the online
-		 * block counts looks like we're a simple streaming
-		 * write.  Try to write until the next extent but limit
-		 * the preallocation size to the number of online
-		 * blocks.
-		 */
-		scoutfs_inode_get_onoff(inode, &online, &offline);
-		if (iblock > 1 && iblock == online) {
-			ret = scoutfs_ext_next(sb, &data_ext_ops, &args,
-					       iblock, 1, &found);
-			if (ret < 0 && ret != -ENOENT)
-				goto out;
-			if (found.len && found.start > iblock)
-				count = found.start - iblock;
-			else
-				count = opts.data_prealloc_blocks;
-
-			count = min(iblock, count);
-		}
-
+		flags = ext->flags;
 	} else {
-		/*
-		 * Preallocation of aligned regions only preallocates if
-		 * the aligned region contains no extents at all.  This
-		 * could be fooled by offline sparse extents but we
-		 * don't want to iterate over all offline extents in the
-		 * aligned region.
-		 */
-		div64_u64_rem(iblock, opts.data_prealloc_blocks, &rem);
-		start = iblock - rem;
-		count = opts.data_prealloc_blocks;
-		ret = scoutfs_ext_next(sb, &data_ext_ops, &args, start, 1, &found);
+		/* otherwise alloc to next extent */
+		ret = scoutfs_ext_next(sb, &data_ext_ops, &args,
+				       iblock, 1, &found);
 		if (ret < 0 && ret != -ENOENT)
 			goto out;
-		if (found.len && found.start < start + count)
-			count = 1;
+		if (found.len && found.start > iblock)
+			count = found.start - iblock;
+		else
+			count = SCOUTFS_DATA_EXTEND_PREALLOC_LIMIT;
+		flags = 0;
 	}
 
 	/* overall prealloc limit */
-	count = min_t(u64, count, opts.data_prealloc_blocks);
+	count = min_t(u64, count, SCOUTFS_DATA_EXTEND_PREALLOC_LIMIT);
+
+	/* only strictly contiguous extending writes will try to preallocate */
+	if (iblock > 1 && iblock == online)
+		count = min(iblock, count);
+	else
+		count = 1;
 
 	ret = scoutfs_alloc_data(sb, datinf->alloc, datinf->wri,
 				 &datinf->dalloc, count, &blkno, &count);
 	if (ret < 0)
 		goto out;
 
-	/*
-	 * An aligned prealloc attempt that gets a smaller extent can
-	 * fail to cover iblock, make sure that it does.  This is a
-	 * pathological case so we don't try to move the window past
-	 * iblock.  Just enough to cover it, which we know is safe.
-	 */
-	if (start + count <= iblock)
-		start += (iblock - (start + count) + 1);
+	ret = scoutfs_ext_set(sb, &data_ext_ops, &args, iblock, 1, blkno, 0);
+	if (ret < 0)
+		goto out;
 
 	if (count > 1) {
-		pre.start = start;
-		pre.len = count;
-		pre.map = blkno;
+		pre.start = iblock + 1;
+		pre.len = count - 1;
+		pre.map = blkno + 1;
 		pre.flags = flags | SEF_UNWRITTEN;
 		ret = scoutfs_ext_set(sb, &data_ext_ops, &args, pre.start,
 				      pre.len, pre.map, pre.flags);
-		if (ret < 0)
+		if (ret < 0) {
+			err = scoutfs_ext_set(sb, &data_ext_ops, &args, iblock,
+					      1, 0, flags);
+			BUG_ON(err); /* couldn't restore original */
 			goto out;
-		undo_pre = true;
+		}
 	}
 
-	ret = scoutfs_ext_set(sb, &data_ext_ops, &args, iblock, 1, blkno + (iblock - start), 0);
-	if (ret < 0)
-		goto out;
-
 	/* tell the caller we have a single block, could check next? */
 	ext->start = iblock;
 	ext->len = 1;
-	ext->map = blkno + (iblock - start);
+	ext->map = blkno;
 	ext->flags = 0;
 	ret = 0;
+
 out:
 	if (ret < 0 && blkno > 0) {
-		if (undo_pre) {
-			err = scoutfs_ext_set(sb, &data_ext_ops, &args,
-					      pre.start, pre.len, 0, flags);
-			BUG_ON(err); /* leaked preallocated extent */
-		}
 		err = scoutfs_free_data(sb, datinf->alloc, datinf->wri,
 				        &datinf->data_freed, blkno, count);
 		BUG_ON(err); /* leaked free blocks */
@@ -533,8 +490,68 @@ out:
 	return ret;
 }
 
+static int alloc_block_dio(struct super_block *sb, struct inode *inode,
+			   struct scoutfs_extent *ext, struct buffer_head *bh,
+			   u64 iblock, struct scoutfs_lock *lock)
+{
+	DECLARE_DATA_INFO(sb, datinf);
+	const u64 ino = scoutfs_ino(inode);
+	struct data_ext_args args = {
+		.ino = ino,
+		.inode = inode,
+		.lock = lock,
+	};
+	u64 blkno = 0;
+	u64 blocks = 0;
+	u64 count = 0;
+	u64 last;
+	u8 ext_fl = 0;
+	int ret = 0;
+	bool first = true;
+	int err;
+
+	last = (bh->b_size - 1) >> SCOUTFS_BLOCK_SM_SHIFT;
+
+	while(blocks < last) {
+		if (ext->len >= last && first)
+			count = min_t(u64, last, SCOUTFS_FALLOCATE_ALLOC_LIMIT);
+		else
+			count = min_t(u64, last - blocks, SCOUTFS_FALLOCATE_ALLOC_LIMIT);
+
+		mutex_lock(&datinf->mutex);
+		ret = scoutfs_alloc_data(sb, datinf->alloc, datinf->wri,
+					 &datinf->dalloc, count, &blkno, &count);
+		if (ret == 0) {
+			ret = scoutfs_ext_set(sb, &data_ext_ops, &args, iblock,
+					      count, blkno,
+					      ext_fl | SEF_UNWRITTEN);
+			if (ret < 0) {
+				err = scoutfs_free_data(sb, datinf->alloc,
+							datinf->wri,
+							&datinf->data_freed,
+							blkno, count);
+				BUG_ON(err); /* inconsistent */
+			}
+		}
+
+		mutex_unlock(&datinf->mutex);
+
+		if (ret < 0)
+			break;
+
+		blocks += count;
+		first = false;
+		ret = scoutfs_ext_next(sb, &data_ext_ops, &args,
+				       iblock, 1, ext);
+		if (ret < 0)
+			break;
+	}
+
+	return ret;
+}
+
 static int scoutfs_get_block(struct inode *inode, sector_t iblock,
-			     struct buffer_head *bh, int create)
+			     struct buffer_head *bh, int create, bool dio_flag)
 {
 	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
 	const u64 ino = scoutfs_ino(inode);
@@ -575,7 +592,7 @@ static int scoutfs_get_block(struct inode *inode, sector_t iblock,
 	}
 
 	/* convert unwritten to written, could be staging */
-	if (create && ext.map && (ext.flags & SEF_UNWRITTEN)) {
+	if (create && ext.map && !dio_flag && (ext.flags & SEF_UNWRITTEN)) {
 		un.start = iblock;
 		un.len = 1;
 		un.map = ext.map + (iblock - ext.start);
@@ -587,11 +604,26 @@ static int scoutfs_get_block(struct inode *inode, sector_t iblock,
 			set_buffer_new(bh);
 		}
 		goto out;
+	} else if (create && ext.map && dio_flag) {
+		un.start = iblock;
+		un.len = 1;
+		un.map = ext.map + (iblock - ext.start);
+		un.flags = ext.flags;
+		ret = scoutfs_ext_set(sb, &data_ext_ops, &args,
+				      un.start, un.len, un.map, un.flags);
+		if (ret == 0) {
+			ext = un;
+			set_buffer_new(bh);
+		}
+		goto out;
 	}
 
 	/* allocate and map blocks containing our logical block */
 	if (create && !ext.map) {
-		ret = alloc_block(sb, inode, &ext, iblock, lock);
+		if (dio_flag)
+			ret = alloc_block_dio(sb, inode, &ext, bh, iblock, lock);
+		else
+			ret = alloc_block(sb, inode, &ext, iblock, lock);
 		if (ret == 0)
 			set_buffer_new(bh);
 	} else {
@@ -625,25 +657,75 @@ static int scoutfs_get_block_read(struct inode *inode, sector_t iblock,
 	int ret;
 
 	down_read(&si->extent_sem);
-	ret = scoutfs_get_block(inode, iblock, bh, create);
+	ret = scoutfs_get_block(inode, iblock, bh, create, false);
 	up_read(&si->extent_sem);
 
 	return ret;
 }
 
-int scoutfs_get_block_write(struct inode *inode, sector_t iblock, struct buffer_head *bh,
-			    int create)
+int scoutfs_get_block_write(struct inode *inode, sector_t iblock,
+			    struct buffer_head *bh, int create)
 {
 	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
 	int ret;
 
 	down_write(&si->extent_sem);
-	ret = scoutfs_get_block(inode, iblock, bh, create);
+	ret = scoutfs_get_block(inode, iblock, bh, create, false);
 	up_write(&si->extent_sem);
 
 	return ret;
 }
 
+
+static int scoutfs_get_block_write_dio(struct inode *inode, sector_t iblock,
+				       struct buffer_head *bh, int create)
+{
+	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
+	struct super_block *sb = inode->i_sb;
+	struct scoutfs_lock *lock = NULL;
+	LIST_HEAD(ind_locks);
+	int ret;
+
+	lock = scoutfs_per_task_get(&si->pt_data_lock);
+	if (WARN_ON_ONCE(!lock)) {
+		return -EINVAL;
+	}
+
+	if (inode)
+		ret = scoutfs_inode_index_lock_hold(inode, &ind_locks,
+						    true, false);
+	else
+		ret = scoutfs_hold_trans(sb, false);
+	if (ret)
+		goto out;
+
+	if (inode)
+		ret = scoutfs_dirty_inode_item(inode, lock);
+
+	if (ret < 0)
+		goto out_unlock;
+
+	down_write(&si->extent_sem);
+	ret = scoutfs_get_block(inode, iblock, bh, create, true);
+	up_write(&si->extent_sem);
+
+	if (inode) {
+		scoutfs_inode_set_data_seq(inode);
+		scoutfs_inode_inc_data_version(inode);
+		inode_inc_iversion(inode);
+
+		if (ret > 0)
+			i_size_write(inode, ret);
+		scoutfs_update_inode_item(inode, lock, &ind_locks);
+	}
+
+out_unlock:
+	scoutfs_release_trans(sb);
+	scoutfs_inode_index_unlock(sb, &ind_locks);
+out:
+	return ret;
+}
+
 /*
  * This is almost never used.  We can't block on a cluster lock while
  * holding the page lock because lock invalidation gets the page lock
@@ -906,6 +988,154 @@ static int scoutfs_write_end(struct file *file, struct address_space *mapping,
 	return ret;
 }
 
+static s64 convert_unwritten_items(struct super_block *sb, struct inode *inode,
+			           u64 ino, u64 iblock, u64 last,
+			           struct scoutfs_lock *lock)
+{
+	struct data_ext_args args = {
+		.ino = ino,
+		.inode = inode,
+		.lock = lock,
+	};
+	struct scoutfs_extent ext;
+	struct scoutfs_extent un;
+	u64 offset;
+	s64 ret;
+	int i;
+
+	ret = 0;
+
+	for (i = 0; iblock <= last; i++) {
+		if (i == EXTENTS_PER_HOLD) {
+			ret = iblock;
+			break;
+		}
+
+		ret = scoutfs_ext_next(sb, &data_ext_ops, &args,
+				       iblock, 1, &ext);
+		if (ret < 0) {
+			if (ret == -ENOENT)
+				ret = 0;
+			break;
+		}
+
+		/* done if we went past the region */
+		if (ext.start > last) {
+			ret = 0;
+			break;
+		}
+
+		/* nothing to do when already marked written */
+		if (!(ext.flags & SEF_UNWRITTEN)) {
+			iblock = ext.start + ext.len;
+			continue;
+		}
+
+		iblock = max(ext.start, iblock);
+		offset = iblock - ext.start;
+
+		un.start = iblock;
+		un.map = ext.map ? ext.map + offset : 0;
+		un.len = min(ext.len - offset, last - iblock + 1);
+		un.flags = ext.flags & ~(SEF_OFFLINE|SEF_UNWRITTEN);
+
+		ret = scoutfs_ext_set(sb, &data_ext_ops, &args,
+				      un.start, un.len, un.map, un.flags);
+		if (ret < 0)
+			break;
+
+		iblock += un.len;
+	}
+
+	return ret;
+}
+
+static ssize_t
+convert_unwritten_extent(struct inode *inode, loff_t offset, ssize_t count)
+{
+	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
+	struct super_block *sb = inode->i_sb;
+	const u64 ino = scoutfs_ino(inode);
+	struct scoutfs_lock *lock = NULL;
+	LIST_HEAD(ind_locks);
+	u64 iblock;
+	u64 last;
+	ssize_t ret = 0;
+
+	lock = scoutfs_per_task_get(&si->pt_data_lock);
+	if (WARN_ON_ONCE(!lock)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	iblock = offset >> SCOUTFS_BLOCK_SM_SHIFT;
+	last = (offset + count - 1) >> SCOUTFS_BLOCK_SM_SHIFT;
+	while(iblock <= last) {
+		if (inode)
+			ret = scoutfs_inode_index_lock_hold(inode, &ind_locks,
+							    true, false);
+		else
+			ret = scoutfs_hold_trans(sb, false);
+		if (ret)
+			break; 
+
+		if (inode)
+			ret = scoutfs_dirty_inode_item(inode, lock);
+		else
+			ret = 0;
+
+		if (ret == 0) {
+			down_write(&si->extent_sem);
+			ret = convert_unwritten_items(sb, inode, ino, iblock,
+						      last, lock);
+			up_write(&si->extent_sem);
+		}
+
+		if (ret < 0)
+			goto out;
+
+		if (inode) {
+			scoutfs_inode_set_data_seq(inode);
+			scoutfs_inode_inc_data_version(inode);
+			inode_inc_iversion(inode);
+
+			if (ret > 0)
+				i_size_write(inode, ret);
+			scoutfs_update_inode_item(inode, lock, &ind_locks);
+		}
+		scoutfs_release_trans(sb);
+		if (inode)
+			scoutfs_inode_index_unlock(sb, &ind_locks);
+
+		if (ret <= 0)
+			break;
+
+		iblock = ret;
+	}
+
+out:
+	return ret;
+}
+
+static ssize_t
+scoutfs_direct_IO(int rw, struct kiocb *iocb, const struct iovec *iov,
+		  loff_t offset, unsigned long nr_segs)
+{
+	struct file *file = iocb->ki_filp;
+	struct address_space *mapping = file->f_mapping;
+	struct inode *inode = mapping->host;
+	ssize_t ret;
+
+	ret = blockdev_direct_IO(rw, iocb, inode, iov, offset, nr_segs,
+				 scoutfs_get_block_write_dio);
+	if (ret > 0 && (rw & WRITE))
+	{
+		ret = convert_unwritten_extent(inode, offset, ret);
+	}
+
+	return ret;
+}
+
 /*
  * Try to allocate unwritten extents for any unallocated regions of the
  * logical block extent from the caller.  The caller manages locks and
@@ -1192,9 +1422,9 @@ static void truncate_inode_pages_extent(struct inode *inode, u64 start, u64 len)
  * explained above the move_blocks ioctl argument structure definition.
  *
  * The caller has processed the ioctl args and performed the most basic
- * argument sanity and inode checks, but we perform more detailed inode
- * checks once we have the inode lock and refreshed inodes.  Our job is
- * to safely lock the two files and move the extents.
+ * inode checks, but we perform more detailed inode checks once we have
+ * the inode lock and refreshed inodes.  Our job is to safely lock the
+ * two files and move the extents.
  */
 #define MOVE_DATA_EXTENTS_PER_HOLD 16
 int scoutfs_data_move_blocks(struct inode *from, u64 from_off,
@@ -1254,15 +1484,6 @@ int scoutfs_data_move_blocks(struct inode *from, u64 from_off,
 	count = (byte_len + SCOUTFS_BLOCK_SM_MASK) >> SCOUTFS_BLOCK_SM_SHIFT;
 	to_iblock = to_off >> SCOUTFS_BLOCK_SM_SHIFT;
 
-	/* only move extent blocks inside i_size, careful not to wrap */
-	from_size = i_size_read(from);
-	if (from_off >= from_size) {
-		ret = 0;
-		goto out;
-	}
-	if (from_off + byte_len > from_size)
-		count = ((from_size - from_off) + SCOUTFS_BLOCK_SM_MASK) >> SCOUTFS_BLOCK_SM_SHIFT;
-
 	if (S_ISDIR(from->i_mode) || S_ISDIR(to->i_mode)) {
 		ret = -EISDIR;
 		goto out;
@@ -1338,8 +1559,9 @@ int scoutfs_data_move_blocks(struct inode *from, u64 from_off,
 				break;
 			}
 
-			/* done if next extent starts after moving region */
-			if (ext.start >= from_iblock + count) {
+			/* only move extents within count and i_size */
+			if (ext.start >= from_iblock + count ||
+			    ext.start >= i_size_read(from)) {
 				done = true;
 				ret = 0;
 				break;
@@ -1347,14 +1569,12 @@ int scoutfs_data_move_blocks(struct inode *from, u64 from_off,
 
 			from_start = max(ext.start, from_iblock);
 			map = ext.map + (from_start - ext.start);
-			len = min(from_iblock + count, ext.start + ext.len) - from_start;
-			to_start = to_iblock + (from_start - from_iblock);
+			len = min3(from_iblock + count,
+				   round_up((u64)i_size_read(from),
+					    SCOUTFS_BLOCK_SM_SIZE),
+				   ext.start + ext.len) - from_start;
 
-			/* we'd get stuck, shouldn't happen */
-			if (WARN_ON_ONCE(len == 0)) {
-				ret = -EIO;
-				goto out;
-			}
+			to_start = to_iblock + (from_start - from_iblock);
 
 			if (is_stage) {
 				ret = scoutfs_ext_next(sb, &data_ext_ops, &to_args,
@@ -1528,13 +1748,14 @@ int scoutfs_data_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 	iblock = start >> SCOUTFS_BLOCK_SM_SHIFT;
 	last = (start + len - 1) >> SCOUTFS_BLOCK_SM_SHIFT;
 
-	while (iblock <= last) {
+	while (true) {
 		ret = scoutfs_ext_next(sb, &data_ext_ops, &args,
 				       iblock, 1, &ext);
 		if (ret < 0) {
-			if (ret == -ENOENT)
+			if (ret == -ENOENT) {
 				ret = 0;
-			last_flags = FIEMAP_EXTENT_LAST;
+				last_flags = FIEMAP_EXTENT_LAST;
+			}
 			break;
 		}
 
@@ -1859,6 +2080,7 @@ const struct address_space_operations scoutfs_file_aops = {
 	.writepages		= scoutfs_writepages,
 	.write_begin		= scoutfs_write_begin,
 	.write_end		= scoutfs_write_end,
+	.direct_IO		= scoutfs_direct_IO,
 };
 
 const struct file_operations scoutfs_file_fops = {

kmod/src/data.h

@@ -43,15 +43,14 @@ extern const struct file_operations scoutfs_file_fops;
 struct scoutfs_alloc;
 struct scoutfs_block_writer;
 
-int scoutfs_get_block_write(struct inode *inode, sector_t iblock, struct buffer_head *bh,
-			    int create);
-
 int scoutfs_data_truncate_items(struct super_block *sb, struct inode *inode,
 				u64 ino, u64 iblock, u64 last, bool offline,
 				struct scoutfs_lock *lock);
 int scoutfs_data_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
 			u64 start, u64 len);
 long scoutfs_fallocate(struct file *file, int mode, loff_t offset, loff_t len);
+int scoutfs_get_block_write(struct inode *inode, sector_t iblock,
+			    struct buffer_head *bh, int create);
 int scoutfs_data_init_offline_extent(struct inode *inode, u64 size,
 				     struct scoutfs_lock *lock);
 int scoutfs_data_move_blocks(struct inode *from, u64 from_off,

kmod/src/dir.c

@@ -32,7 +32,6 @@
 #include "hash.h"
 #include "omap.h"
 #include "forest.h"
-#include "acl.h"
 #include "counters.h"
 #include "scoutfs_trace.h"
 
@@ -60,6 +59,8 @@
  * All the entries have a dirent struct with the full name in their
  * value.  The dirent struct contains the name hash and readdir position
  * so that any item use can reference all the items for a given entry.
+ * This is important for deleting all the items given a dentry that was
+ * populated by lookup.
  */
 
 static unsigned int mode_to_type(umode_t mode)
@@ -98,12 +99,100 @@ static unsigned int dentry_type(enum scoutfs_dentry_type type)
 	return DT_UNKNOWN;
 }
 
+/*
+ * @lock_cov: tells revalidation that the dentry is still locked and valid.
+ *
+ * @pos, @hash: lets us remove items on final unlink without having to
+ * look them up.
+ */
+struct dentry_info {
+	struct scoutfs_lock_coverage lock_cov;
+	u64 hash;
+	u64 pos;
+};
+
+static struct kmem_cache *dentry_info_cache;
+
+static void scoutfs_d_release(struct dentry *dentry)
+{
+	struct super_block *sb = dentry->d_sb;
+	struct dentry_info *di = dentry->d_fsdata;
+
+	if (di) {
+		scoutfs_lock_del_coverage(sb, &di->lock_cov);
+		kmem_cache_free(dentry_info_cache, di);
+		dentry->d_fsdata = NULL;
+	}
+}
+
 static int scoutfs_d_revalidate(struct dentry *dentry, unsigned int flags);
 
-const struct dentry_operations scoutfs_dentry_ops = {
+static const struct dentry_operations scoutfs_dentry_ops = {
+	.d_release = scoutfs_d_release,
 	.d_revalidate = scoutfs_d_revalidate,
 };
 
+static int alloc_dentry_info(struct dentry *dentry)
+{
+	struct dentry_info *di;
+
+	smp_rmb();
+	if (dentry->d_op == &scoutfs_dentry_ops)
+		return 0;
+
+	di = kmem_cache_zalloc(dentry_info_cache, GFP_NOFS);
+	if (!di)
+		return -ENOMEM;
+
+	scoutfs_lock_init_coverage(&di->lock_cov);
+
+	spin_lock(&dentry->d_lock);
+	if (!dentry->d_fsdata) {
+		dentry->d_fsdata = di;
+		smp_wmb();
+		d_set_d_op(dentry, &scoutfs_dentry_ops);
+	}
+	spin_unlock(&dentry->d_lock);
+
+	if (di != dentry->d_fsdata)
+		kmem_cache_free(dentry_info_cache, di);
+
+	return 0;
+}
+
+static void update_dentry_info(struct super_block *sb, struct dentry *dentry,
+			       u64 hash, u64 pos, struct scoutfs_lock *lock)
+{
+	struct dentry_info *di = dentry->d_fsdata;
+
+	if (WARN_ON_ONCE(di == NULL))
+		return;
+
+	scoutfs_lock_add_coverage(sb, lock, &di->lock_cov);
+	di->hash = hash;
+	di->pos = pos;
+}
+
+static u64 dentry_info_hash(struct dentry *dentry)
+{
+	struct dentry_info *di = dentry->d_fsdata;
+
+	if (WARN_ON_ONCE(di == NULL))
+		return 0;
+
+	return di->hash;
+}
+
+static u64 dentry_info_pos(struct dentry *dentry)
+{
+	struct dentry_info *di = dentry->d_fsdata;
+
+	if (WARN_ON_ONCE(di == NULL))
+		return 0;
+
+	return di->pos;
+}
+
 static void init_dirent_key(struct scoutfs_key *key, u8 type, u64 ino,
 			    u64 major, u64 minor)
 {
@@ -228,105 +317,62 @@ out:
 	return ret;
 }
 
-static int lookup_dentry_dirent(struct super_block *sb, u64 dir_ino, struct dentry *dentry,
-				struct scoutfs_dirent *dent_ret,
-				struct scoutfs_lock *lock)
-{
-	return lookup_dirent(sb, dir_ino, dentry->d_name.name, dentry->d_name.len,
-			     dirent_name_hash(dentry->d_name.name, dentry->d_name.len),
-			     dent_ret, lock);
-}
-
-static u64 dentry_parent_ino(struct dentry *dentry)
-{
-	struct dentry *parent = NULL;
-	struct inode *dir;
-	u64 dir_ino = 0;
-
-	if ((parent = dget_parent(dentry)) && (dir = parent->d_inode))
-		dir_ino = scoutfs_ino(dir);
-
-	dput(parent);
-	return dir_ino;
-}
-
-/* negative dentries return 0, our root ino is non-zero (1) */
-static u64 dentry_ino(struct dentry *dentry)
-{
-	return dentry->d_inode ? scoutfs_ino(dentry->d_inode) : 0;
-}
-
-static void set_dentry_fsdata(struct dentry *dentry, struct scoutfs_lock *lock)
-{
-	void *now = (void *)(unsigned long)lock->refresh_gen;
-	void *was;
-
-	/* didn't want to alloc :/ */
-	BUILD_BUG_ON(sizeof(dentry->d_fsdata) != sizeof(u64));
-	BUILD_BUG_ON(sizeof(dentry->d_fsdata) != sizeof(long));
-
-	do {
-		was = dentry->d_fsdata;
-	} while (cmpxchg(&dentry->d_fsdata, was, now) != was);
-}
-
-static bool test_dentry_fsdata(struct dentry *dentry, u64 refresh)
-{
-	u64 fsd = (unsigned long)ACCESS_ONCE(dentry->d_fsdata);
-
-	return fsd == refresh;
-}
-
 /*
- * Validate an operation caller's input dentry argument.  If the fsdata
- * is valid then the underlying dirent items couldn't have changed and
- * we return 0.  If fsdata is no longer protected by a lock or its
- * fields don't match then we check the dirent item.  If the dirent item
- * doesn't match what the caller expected given their dentry fields then
- * we return an error.
+ * Verify that the caller's dentry still precisely matches our dirent
+ * items.
+ *
+ * The caller has a dentry that the vfs revalidated before they acquired
+ * their locks.  If the dentry is still covered by a lock we immediately
+ * return 0.   If not, we check items and return -ENOENT if a positive
+ * dentry no longer matches the items or -EEXIST if a negative entry's
+ * name now has an item.
  */
-static int validate_dentry(struct super_block *sb, u64 dir_ino, struct dentry *dentry,
-			   struct scoutfs_lock *lock)
+static int verify_entry(struct super_block *sb, u64 dir_ino, struct dentry *dentry,
+			struct scoutfs_lock *lock)
 {
-	u64 ino = dentry_ino(dentry);
+	struct dentry_info *di = dentry->d_fsdata;
 	struct scoutfs_dirent dent = {0,};
+	const char *name;
+	u64 dentry_ino;
+	int name_len;
+	u64 hash;
 	int ret;
 
-	if (test_dentry_fsdata(dentry, lock->refresh_gen)) {
-		ret = 0;
-		goto out;
-	}
+	if (scoutfs_lock_is_covered(sb, &di->lock_cov))
+		return 0;
 
-	ret = lookup_dentry_dirent(sb, dir_ino, dentry, &dent, lock);
+	dentry_ino = dentry->d_inode ? scoutfs_ino(dentry->d_inode) : 0;
+	name = dentry->d_name.name;
+	name_len = dentry->d_name.len;
+	hash = dirent_name_hash(name, name_len);
+
+	ret = lookup_dirent(sb, dir_ino, name, name_len, hash, &dent, lock);
 	if (ret < 0 && ret != -ENOENT)
-		goto out;
+		return ret;
 
-	/* use negative zeroed dent when lookup gave -ENOENT */
-	if (!ino && dent.ino) {
-		/* caller expected negative but there was a dirent */
-		ret = -EEXIST;
-	} else if (ino && !dent.ino) {
-		/* caller expected positive but there was no dirent */
-		ret = -ENOENT;
-	} else if (ino != le64_to_cpu(dent.ino)) {
-		/* name linked to different inode than caller's */
-		ret = -ESTALE;
+	if (dentry_ino != le64_to_cpu(dent.ino) || di->hash != le64_to_cpu(dent.hash) ||
+	    di->pos != le64_to_cpu(dent.pos)) {
+		if (dentry_ino)
+			ret = -ENOENT;
+		else
+			ret = -EEXIST;
 	} else {
-		/* dirent ino matches dentry ino */
 		ret = 0;
 	}
 
-out:
-	trace_scoutfs_validate_dentry(sb, dentry, dir_ino, ino, le64_to_cpu(dent.ino),
-				      lock->refresh_gen, ret);
-
 	return ret;
 }
 
 static int scoutfs_d_revalidate(struct dentry *dentry, unsigned int flags)
 {
 	struct super_block *sb = dentry->d_sb;
-	u64 dir_ino = dentry_parent_ino(dentry);
+	struct dentry_info *di = dentry->d_fsdata;
+	struct dentry *parent = dget_parent(dentry);
+	struct scoutfs_lock *lock = NULL;
+	struct scoutfs_dirent dent;
+	bool is_covered = false;
+	struct inode *dir;
+	u64 dentry_ino;
 	int ret;
 
 	/* don't think this happens but we can find out */
@@ -348,7 +394,47 @@ static int scoutfs_d_revalidate(struct dentry *dentry, unsigned int flags)
 		goto out;
 	}
 
-	if (test_dentry_fsdata(dentry, scoutfs_lock_ino_refresh_gen(sb, dir_ino))) {
+	if (WARN_ON_ONCE(di == NULL)) {
+		ret = 0;
+		goto out;
+	}
+
+	is_covered = scoutfs_lock_is_covered(sb, &di->lock_cov);
+	if (is_covered) {
+		scoutfs_inc_counter(sb, dentry_revalidate_locked);
+		ret = 1;
+		goto out;
+	}
+
+	if (!parent || !parent->d_inode) {
+		scoutfs_inc_counter(sb, dentry_revalidate_orphan);
+		ret = 0;
+		goto out;
+	}
+	dir = parent->d_inode;
+
+	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_READ, 0, dir, &lock);
+	if (ret)
+		goto out;
+
+	ret = lookup_dirent(sb, scoutfs_ino(dir),
+			    dentry->d_name.name, dentry->d_name.len,
+			    dirent_name_hash(dentry->d_name.name,
+					     dentry->d_name.len),
+			    &dent, lock);
+	if (ret == -ENOENT) {
+		dent.ino = 0;
+		dent.hash = 0;
+		dent.pos = 0;
+	} else if (ret < 0) {
+		goto out;
+	}
+
+	dentry_ino = dentry->d_inode ? scoutfs_ino(dentry->d_inode) : 0;
+
+	if ((dentry_ino == le64_to_cpu(dent.ino))) {
+		update_dentry_info(sb, dentry, le64_to_cpu(dent.hash),
+				   le64_to_cpu(dent.pos), lock);
 		scoutfs_inc_counter(sb, dentry_revalidate_valid);
 		ret = 1;
 	} else {
@@ -357,7 +443,10 @@ static int scoutfs_d_revalidate(struct dentry *dentry, unsigned int flags)
 	}
 
 out:
-	trace_scoutfs_d_revalidate(sb, dentry, flags, dir_ino, ret);
+	trace_scoutfs_d_revalidate(sb, dentry, flags, parent, is_covered, ret);
+
+	dput(parent);
+	scoutfs_unlock(sb, lock, SCOUTFS_LOCK_READ);
 
 	if (ret < 0 && ret != -ECHILD)
 		scoutfs_inc_counter(sb, dentry_revalidate_error);
@@ -394,6 +483,10 @@ static struct dentry *scoutfs_lookup(struct inode *dir, struct dentry *dentry,
 		goto out;
 	}
 
+	ret = alloc_dentry_info(dentry);
+	if (ret)
+		goto out;
+
 	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_READ, 0, dir, &dir_lock);
 	if (ret)
 		goto out;
@@ -407,7 +500,8 @@ static struct dentry *scoutfs_lookup(struct inode *dir, struct dentry *dentry,
 		ino = le64_to_cpu(dent.ino);
 	}
 	if (ret == 0)
-		set_dentry_fsdata(dentry, dir_lock);
+		update_dentry_info(sb, dentry, le64_to_cpu(dent.hash),
+				   le64_to_cpu(dent.pos), dir_lock);
 
 	scoutfs_unlock(sb, dir_lock, SCOUTFS_LOCK_READ);
 
@@ -631,6 +725,10 @@ static struct inode *lock_hold_create(struct inode *dir, struct dentry *dentry,
 	int ret = 0;
 	u64 ino;
 
+	ret = alloc_dentry_info(dentry);
+	if (ret)
+		return ERR_PTR(ret);
+
 	ret = scoutfs_alloc_ino(sb, S_ISDIR(mode), &ino);
 	if (ret)
 		return ERR_PTR(ret);
@@ -667,8 +765,7 @@ retry:
 	if (ret)
 		goto out_unlock;
 
-	ret = scoutfs_new_inode(sb, dir, mode, rdev, ino, *inode_lock, &inode) ?:
-	      scoutfs_init_acl_locked(inode, dir, *inode_lock, *dir_lock, ind_locks);
+	ret = scoutfs_new_inode(sb, dir, mode, rdev, ino, *inode_lock, &inode);
 	if (ret < 0)
 		goto out;
 
@@ -719,7 +816,7 @@ static int scoutfs_mknod(struct inode *dir, struct dentry *dentry, umode_t mode,
 		return PTR_ERR(inode);
 	si = SCOUTFS_I(inode);
 
-	ret = validate_dentry(sb, scoutfs_ino(dir), dentry, dir_lock);
+	ret = verify_entry(sb, scoutfs_ino(dir), dentry, dir_lock);
 	if (ret < 0)
 		goto out;
 
@@ -732,7 +829,7 @@ static int scoutfs_mknod(struct inode *dir, struct dentry *dentry, umode_t mode,
 	if (ret)
 		goto out;
 
-	set_dentry_fsdata(dentry, dir_lock);
+	update_dentry_info(sb, dentry, hash, pos, dir_lock);
 
 	i_size_write(dir, i_size_read(dir) + dentry->d_name.len);
 	dir->i_mtime = dir->i_ctime = CURRENT_TIME;
@@ -806,15 +903,19 @@ static int scoutfs_link(struct dentry *old_dentry,
 	if (ret)
 		return ret;
 
-	ret = validate_dentry(sb, scoutfs_ino(dir), dentry, dir_lock);
-	if (ret < 0)
-		goto out_unlock;
-
 	if (inode->i_nlink >= SCOUTFS_LINK_MAX) {
 		ret = -EMLINK;
 		goto out_unlock;
 	}
 
+	ret = alloc_dentry_info(dentry);
+	if (ret)
+		goto out_unlock;
+
+	ret = verify_entry(sb, scoutfs_ino(dir), dentry, dir_lock);
+	if (ret < 0)
+		goto out_unlock;
+
 	dir_size = i_size_read(dir) + dentry->d_name.len;
 
 	if (inode->i_nlink == 0) {
@@ -840,7 +941,7 @@ retry:
 		goto out;
 
 	if (del_orphan) {
-		ret = scoutfs_inode_orphan_delete(sb, scoutfs_ino(inode), orph_lock, inode_lock);
+		ret = scoutfs_inode_orphan_delete(sb, scoutfs_ino(inode), orph_lock);
 		if (ret)
 			goto out;
 	}
@@ -852,11 +953,11 @@ retry:
 			      scoutfs_ino(inode), inode->i_mode, dir_lock,
 			      inode_lock);
 	if (ret) {
-		err = scoutfs_inode_orphan_create(sb, scoutfs_ino(inode), orph_lock, inode_lock);
+		err = scoutfs_inode_orphan_create(sb, scoutfs_ino(inode), orph_lock);
 		WARN_ON_ONCE(err); /* no orphan, might not scan and delete after crash */
 		goto out;
 	}
-	set_dentry_fsdata(dentry, dir_lock);
+	update_dentry_info(sb, dentry, hash, pos, dir_lock);
 
 	i_size_write(dir, dir_size);
 	dir->i_mtime = dir->i_ctime = CURRENT_TIME;
@@ -904,11 +1005,9 @@ static int scoutfs_unlink(struct inode *dir, struct dentry *dentry)
 	struct scoutfs_lock *inode_lock = NULL;
 	struct scoutfs_lock *orph_lock = NULL;
 	struct scoutfs_lock *dir_lock = NULL;
-	struct scoutfs_dirent dent;
 	LIST_HEAD(ind_locks);
 	u64 ind_seq;
-	u64 hash;
-	int ret;
+	int ret = 0;
 
 	ret = scoutfs_lock_inodes(sb, SCOUTFS_LOCK_WRITE,
 				  SCOUTFS_LKF_REFRESH_INODE,
@@ -917,7 +1016,11 @@ static int scoutfs_unlink(struct inode *dir, struct dentry *dentry)
 	if (ret)
 		return ret;
 
-	ret = validate_dentry(sb, scoutfs_ino(dir), dentry, dir_lock);
+	ret = alloc_dentry_info(dentry);
+	if (ret)
+		goto unlock;
+
+	ret = verify_entry(sb, scoutfs_ino(dir), dentry, dir_lock);
 	if (ret < 0)
 		goto unlock;
 
@@ -926,13 +1029,6 @@ static int scoutfs_unlink(struct inode *dir, struct dentry *dentry)
 		goto unlock;
 	}
 
-	hash = dirent_name_hash(dentry->d_name.name, dentry->d_name.len);
-
-	ret = lookup_dirent(sb, scoutfs_ino(dir), dentry->d_name.name, dentry->d_name.len, hash,
-			    &dent, dir_lock);
-	if (ret < 0)
-		goto out;
-
 	if (should_orphan(inode)) {
 		ret = scoutfs_lock_orphan(sb, SCOUTFS_LOCK_WRITE_ONLY, 0, scoutfs_ino(inode),
 					  &orph_lock);
@@ -951,20 +1047,21 @@ retry:
 		goto unlock;
 
 	if (should_orphan(inode)) {
-		ret = scoutfs_inode_orphan_create(sb, scoutfs_ino(inode), orph_lock, inode_lock);
+		ret = scoutfs_inode_orphan_create(sb, scoutfs_ino(inode), orph_lock);
 		if (ret < 0)
 			goto out;
 	}
 
-	ret = del_entry_items(sb, scoutfs_ino(dir), le64_to_cpu(dent.hash), le64_to_cpu(dent.pos),
-			      scoutfs_ino(inode), dir_lock, inode_lock);
+	ret = del_entry_items(sb, scoutfs_ino(dir), dentry_info_hash(dentry),
+			      dentry_info_pos(dentry), scoutfs_ino(inode),
+			      dir_lock, inode_lock);
 	if (ret) {
-		ret = scoutfs_inode_orphan_delete(sb, scoutfs_ino(inode), orph_lock, inode_lock);
+		ret = scoutfs_inode_orphan_delete(sb, scoutfs_ino(inode), orph_lock);
 		WARN_ON_ONCE(ret); /* should have been dirty */
 		goto out;
 	}
 
-	set_dentry_fsdata(dentry, dir_lock);
+	update_dentry_info(sb, dentry, 0, 0, dir_lock);
 
 	dir->i_ctime = ts;
 	dir->i_mtime = ts;
@@ -1145,11 +1242,10 @@ const struct inode_operations scoutfs_symlink_iops = {
 	.put_link       = scoutfs_put_link,
 	.getattr	= scoutfs_getattr,
 	.setattr	= scoutfs_setattr,
-	.setxattr	= generic_setxattr,
-	.getxattr	= generic_getxattr,
+	.setxattr	= scoutfs_setxattr,
+	.getxattr	= scoutfs_getxattr,
 	.listxattr	= scoutfs_listxattr,
-	.removexattr	= generic_removexattr,
-	.get_acl	= scoutfs_get_acl,
+	.removexattr	= scoutfs_removexattr,
 };
 
 /*
@@ -1177,13 +1273,17 @@ static int scoutfs_symlink(struct inode *dir, struct dentry *dentry,
 	    name_len > PATH_MAX || name_len > SCOUTFS_SYMLINK_MAX_SIZE)
 		return -ENAMETOOLONG;
 
+	ret = alloc_dentry_info(dentry);
+	if (ret)
+		return ret;
+
 	inode = lock_hold_create(dir, dentry, S_IFLNK|S_IRWXUGO, 0,
 				 &dir_lock, &inode_lock, NULL, &ind_locks);
 	if (IS_ERR(inode))
 		return PTR_ERR(inode);
 	si = SCOUTFS_I(inode);
 
-	ret = validate_dentry(sb, scoutfs_ino(dir), dentry, dir_lock);
+	ret = verify_entry(sb, scoutfs_ino(dir), dentry, dir_lock);
 	if (ret < 0)
 		goto out;
 
@@ -1201,7 +1301,7 @@ static int scoutfs_symlink(struct inode *dir, struct dentry *dentry,
 	if (ret)
 		goto out;
 
-	set_dentry_fsdata(dentry, dir_lock);
+	update_dentry_info(sb, dentry, hash, pos, dir_lock);
 
 	i_size_write(dir, i_size_read(dir) + dentry->d_name.len);
 	dir->i_mtime = dir->i_ctime = CURRENT_TIME;
@@ -1443,11 +1543,6 @@ static int item_d_ancestor(struct super_block *sb, u64 p1, u64 p2, u64 *p_ret)
 
 	*p_ret = 0;
 
-	if (p2 == SCOUTFS_ROOT_INO) {
-		ret = 0;
-		goto out;
-	}
-
 	ret = scoutfs_dir_get_backref_path(sb, p2, 0, 0, &list);
 	if (ret)
 		goto out;
@@ -1536,8 +1631,6 @@ static int scoutfs_rename_common(struct inode *old_dir,
 	struct scoutfs_lock *old_inode_lock = NULL;
 	struct scoutfs_lock *new_inode_lock = NULL;
 	struct scoutfs_lock *orph_lock = NULL;
-	struct scoutfs_dirent new_dent;
-	struct scoutfs_dirent old_dent;
 	struct timespec now;
 	bool ins_new = false;
 	bool del_new = false;
@@ -1585,18 +1678,19 @@ static int scoutfs_rename_common(struct inode *old_dir,
 	if (ret)
 		goto out_unlock;
 
-	/* make sure that the entries assumed by the argument still exist */
-	ret = validate_dentry(sb, scoutfs_ino(old_dir), old_dentry, old_dir_lock) ?:
-	      validate_dentry(sb, scoutfs_ino(new_dir), new_dentry, new_dir_lock);
-	if (ret)
-		goto out_unlock;
-
 	/* test dir i_size now that it's refreshed */
 	if (new_inode && S_ISDIR(new_inode->i_mode) && i_size_read(new_inode)) {
 		ret = -ENOTEMPTY;
 		goto out_unlock;
 	}
 
+	/* make sure that the entries assumed by the argument still exist */
+	ret = alloc_dentry_info(old_dentry) ?:
+	      alloc_dentry_info(new_dentry) ?:
+	      verify_entry(sb, scoutfs_ino(old_dir), old_dentry, old_dir_lock) ?:
+	      verify_entry(sb, scoutfs_ino(new_dir), new_dentry, new_dir_lock);
+	if (ret)
+		goto out_unlock;
 
 	if ((flags & RENAME_NOREPLACE) && (new_inode != NULL)) {
 		ret = -EEXIST;
@@ -1639,12 +1733,10 @@ retry:
 
 	/* remove the new entry if it exists */
 	if (new_inode) {
-		ret = lookup_dirent(sb, scoutfs_ino(new_dir), new_dentry->d_name.name,
-				    new_dentry->d_name.len, new_hash, &new_dent, new_dir_lock);
-		if (ret < 0)
-			goto out;
-		ret = del_entry_items(sb, scoutfs_ino(new_dir), le64_to_cpu(new_dent.hash),
-				      le64_to_cpu(new_dent.pos), scoutfs_ino(new_inode),
+		ret = del_entry_items(sb, scoutfs_ino(new_dir),
+				      dentry_info_hash(new_dentry),
+				      dentry_info_pos(new_dentry),
+				      scoutfs_ino(new_inode),
 				      new_dir_lock, new_inode_lock);
 		if (ret)
 			goto out;
@@ -1660,22 +1752,18 @@ retry:
 		goto out;
 	del_new = true;
 
-	ret = lookup_dirent(sb, scoutfs_ino(old_dir), old_dentry->d_name.name,
-			    old_dentry->d_name.len, old_hash, &old_dent, old_dir_lock);
-	if (ret < 0)
-		goto out;
-
 	/* remove the old entry */
-	ret = del_entry_items(sb, scoutfs_ino(old_dir), le64_to_cpu(old_dent.hash),
-			      le64_to_cpu(old_dent.pos), scoutfs_ino(old_inode),
+	ret = del_entry_items(sb, scoutfs_ino(old_dir),
+			      dentry_info_hash(old_dentry),
+			      dentry_info_pos(old_dentry),
+			      scoutfs_ino(old_inode),
 			      old_dir_lock, old_inode_lock);
 	if (ret)
 		goto out;
 	ins_old = true;
 
 	if (should_orphan(new_inode)) {
-		ret = scoutfs_inode_orphan_create(sb, scoutfs_ino(new_inode), orph_lock,
-						  new_inode_lock);
+		ret = scoutfs_inode_orphan_create(sb, scoutfs_ino(new_inode), orph_lock);
 		if (ret)
 			goto out;
 	}
@@ -1683,7 +1771,7 @@ retry:
 	/* won't fail from here on out, update all the vfs structs */
 
 	/* the caller will use d_move to move the old_dentry into place */
-	set_dentry_fsdata(old_dentry, new_dir_lock);
+	update_dentry_info(sb, old_dentry, new_hash, new_pos, new_dir_lock);
 
        i_size_write(old_dir, i_size_read(old_dir) - old_dentry->d_name.len);
        if (!new_inode)
@@ -1748,8 +1836,8 @@ out:
 		err = 0;
 		if (ins_old)
 			err = add_entry_items(sb, scoutfs_ino(old_dir),
-					      le64_to_cpu(old_dent.hash),
-					      le64_to_cpu(old_dent.pos),
+					      dentry_info_hash(old_dentry),
+					      dentry_info_pos(old_dentry),
 					      old_dentry->d_name.name,
 					      old_dentry->d_name.len,
 					      scoutfs_ino(old_inode),
@@ -1765,8 +1853,8 @@ out:
 
 		if (ins_new && err == 0)
 			err = add_entry_items(sb, scoutfs_ino(new_dir),
-					      le64_to_cpu(new_dent.hash),
-					      le64_to_cpu(new_dent.pos),
+					      dentry_info_hash(new_dentry),
+					      dentry_info_pos(new_dentry),
 					      new_dentry->d_name.name,
 					      new_dentry->d_name.len,
 					      scoutfs_ino(new_inode),
@@ -1837,7 +1925,7 @@ static int scoutfs_tmpfile(struct inode *dir, struct dentry *dentry, umode_t mod
 		return PTR_ERR(inode);
 	si = SCOUTFS_I(inode);
 
-	ret = scoutfs_inode_orphan_create(sb, scoutfs_ino(inode), orph_lock, inode_lock);
+	ret = scoutfs_inode_orphan_create(sb, scoutfs_ino(inode), orph_lock);
 	if (ret < 0)
 		goto out; /* XXX returning error but items created */
 
@@ -1890,14 +1978,32 @@ const struct inode_operations_wrapper scoutfs_dir_iops = {
 	.rename		= scoutfs_rename,
 	.getattr	= scoutfs_getattr,
 	.setattr	= scoutfs_setattr,
-	.setxattr	= generic_setxattr,
-	.getxattr	= generic_getxattr,
+	.setxattr	= scoutfs_setxattr,
+	.getxattr	= scoutfs_getxattr,
 	.listxattr	= scoutfs_listxattr,
-	.removexattr	= generic_removexattr,
-	.get_acl	= scoutfs_get_acl,
+	.removexattr	= scoutfs_removexattr,
 	.symlink	= scoutfs_symlink,
 	.permission	= scoutfs_permission,
 	},
 	.tmpfile	= scoutfs_tmpfile,
 	.rename2	= scoutfs_rename2,
 };
+
+void scoutfs_dir_exit(void)
+{
+	if (dentry_info_cache) {
+		kmem_cache_destroy(dentry_info_cache);
+		dentry_info_cache = NULL;
+	}
+}
+
+int scoutfs_dir_init(void)
+{
+	dentry_info_cache = kmem_cache_create("scoutfs_dentry_info",
+					      sizeof(struct dentry_info), 0,
+					      SLAB_RECLAIM_ACCOUNT, NULL);
+	if (!dentry_info_cache)
+		return -ENOMEM;
+
+	return 0;
+}

kmod/src/dir.h

@@ -8,8 +8,6 @@ extern const struct file_operations scoutfs_dir_fops;
 extern const struct inode_operations_wrapper scoutfs_dir_iops;
 extern const struct inode_operations scoutfs_symlink_iops;
 
-extern const struct dentry_operations scoutfs_dentry_ops;
-
 struct scoutfs_link_backref_entry {
 	struct list_head head;
 	u64 dir_ino;
@@ -31,4 +29,7 @@ int scoutfs_dir_add_next_linkref(struct super_block *sb, u64 ino,
 int scoutfs_symlink_drop(struct super_block *sb, u64 ino,
 			 struct scoutfs_lock *lock, u64 i_size);
 
+int scoutfs_dir_init(void);
+void scoutfs_dir_exit(void);
+
 #endif

kmod/src/file.c

@@ -47,6 +47,9 @@ ssize_t scoutfs_file_aio_read(struct kiocb *iocb, const struct iovec *iov,
 	DECLARE_DATA_WAIT(dw);
 	int ret;
 
+	if (!is_sync_kiocb(iocb))
+		return -EINVAL;
+
 retry:
 	/* protect checked extents from release */
 	mutex_lock(&inode->i_mutex);
@@ -97,6 +100,9 @@ ssize_t scoutfs_file_aio_write(struct kiocb *iocb, const struct iovec *iov,
 	DECLARE_DATA_WAIT(dw);
 	int ret;
 
+	if (!is_sync_kiocb(iocb))
+		return -EINVAL;
+
 	if (iocb->ki_left == 0) /* Does this even happen? */
 		return 0;
 

kmod/src/forest.c

@@ -78,6 +78,11 @@ struct forest_refs {
 	struct scoutfs_block_ref logs_ref;
 };
 
+/* initialize some refs that initially aren't equal */
+#define DECLARE_STALE_TRACKING_SUPER_REFS(a, b)		\
+	struct forest_refs a = {{cpu_to_le64(0),}};	\
+	struct forest_refs b = {{cpu_to_le64(1),}}
+
 struct forest_bloom_nrs {
 	unsigned int nrs[SCOUTFS_FOREST_BLOOM_NRS];
 };
@@ -131,11 +136,11 @@ static struct scoutfs_block *read_bloom_ref(struct super_block *sb, struct scout
 int scoutfs_forest_next_hint(struct super_block *sb, struct scoutfs_key *key,
 			     struct scoutfs_key *next)
 {
+	DECLARE_STALE_TRACKING_SUPER_REFS(prev_refs, refs);
 	struct scoutfs_net_roots roots;
 	struct scoutfs_btree_root item_root;
 	struct scoutfs_log_trees *lt;
 	SCOUTFS_BTREE_ITEM_REF(iref);
-	DECLARE_SAVED_REFS(saved);
 	struct scoutfs_key found;
 	struct scoutfs_key ltk;
 	bool checked_fs;
@@ -150,6 +155,8 @@ retry:
 		goto out;
 
 	trace_scoutfs_forest_using_roots(sb, &roots.fs_root, &roots.logs_root);
+	refs.fs_ref = roots.fs_root.ref;
+	refs.logs_ref = roots.logs_root.ref;
 
 	scoutfs_key_init_log_trees(&ltk, 0, 0);
 	checked_fs = false;
@@ -205,10 +212,14 @@ retry:
 		}
 	}
 
-	ret = scoutfs_block_check_stale(sb, ret, &saved, &roots.fs_root.ref, &roots.logs_root.ref);
-	if (ret == -ESTALE)
+	if (ret == -ESTALE) {
+		if (memcmp(&prev_refs, &refs, sizeof(refs)) == 0)
+			return -EIO;
+		prev_refs = refs;
 		goto retry;
+	}
 out:
+
 	return ret;
 }
 
@@ -530,8 +541,9 @@ void scoutfs_forest_dec_inode_count(struct super_block *sb)
 
 /*
  * Return the total inode count from the super block and all the
- * log_btrees it references.  ESTALE from read blocks is returned to the
- * caller who is expected to retry or return hard errors.
+ * log_btrees it references.   This assumes it's working with a block
+ * reference hierarchy that should be fully consistent.   If we see
+ * ESTALE we've hit persistent corruption.
  */
 int scoutfs_forest_inode_count(struct super_block *sb, struct scoutfs_super_block *super,
 			       u64 *inode_count)
@@ -560,6 +572,8 @@ int scoutfs_forest_inode_count(struct super_block *sb, struct scoutfs_super_bloc
 		if (ret < 0) {
 			if (ret == -ENOENT)
 				ret = 0;
+			else if (ret == -ESTALE)
+				ret = -EIO;
 			break;
 		}
 	}

kmod/src/format.h

@@ -683,19 +683,16 @@ struct scoutfs_xattr_totl_val {
 #define SCOUTFS_QUORUM_ELECT_VAR_MS	100
 
 /*
- * Once a leader is elected they send heartbeat messages to all quorum
- * members at regular intervals to force members to wait the much longer
- * heartbeat timeout.  Once the heartbeat timeout expires without
- * receiving a heartbeat message a member will start an election.
+ * Once a leader is elected they send out heartbeats at regular
+ * intervals to force members to wait the much longer heartbeat timeout.
+ * Once heartbeat timeout expires without receiving a heartbeat they'll
+ * switch over the performing elections.
  *
  * These determine how long it could take members to notice that a
- * leader has gone silent and start to elect a new leader.  The
- * heartbeat timeout can be changed at run time by options.
+ * leader has gone silent and start to elect a new leader.
  */
 #define SCOUTFS_QUORUM_HB_IVAL_MS	100
-#define SCOUTFS_QUORUM_MIN_HB_TIMEO_MS	(2 * MSEC_PER_SEC)
-#define SCOUTFS_QUORUM_DEF_HB_TIMEO_MS	(10 * MSEC_PER_SEC)
-#define SCOUTFS_QUORUM_MAX_HB_TIMEO_MS	(60 * MSEC_PER_SEC)
+#define SCOUTFS_QUORUM_HB_TIMEO_MS	(5 * MSEC_PER_SEC)
 
 /*
  * A newly elected leader will give fencing some time before giving up and

kmod/src/inode.c

@@ -19,7 +19,6 @@
 #include <linux/pagemap.h>
 #include <linux/sched.h>
 #include <linux/list_sort.h>
-#include <linux/workqueue.h>
 #include <linux/buffer_head.h>
 
 #include "format.h"
@@ -38,7 +37,6 @@
 #include "omap.h"
 #include "forest.h"
 #include "btree.h"
-#include "acl.h"
 
 /*
  * XXX
@@ -69,10 +67,8 @@ struct inode_sb_info {
 
 	struct delayed_work orphan_scan_dwork;
 
-	struct workqueue_struct *iput_workq;
 	struct work_struct iput_work;
-	spinlock_t iput_lock;
-	struct list_head iput_list;
+	struct llist_head iput_llist;
 };
 
 #define DECLARE_INODE_SB_INFO(sb, name) \
@@ -99,9 +95,7 @@ static void scoutfs_inode_ctor(void *obj)
 	init_rwsem(&si->xattr_rwsem);
 	INIT_LIST_HEAD(&si->writeback_entry);
 	scoutfs_lock_init_coverage(&si->ino_lock_cov);
-	INIT_LIST_HEAD(&si->iput_head);
-	si->iput_count = 0;
-	si->iput_flags = 0;
+	atomic_set(&si->iput_count, 0);
 
 	inode_init_once(&si->inode);
 }
@@ -143,22 +137,20 @@ void scoutfs_destroy_inode(struct inode *inode)
 static const struct inode_operations scoutfs_file_iops = {
 	.getattr	= scoutfs_getattr,
 	.setattr	= scoutfs_setattr,
-	.setxattr	= generic_setxattr,
-	.getxattr	= generic_getxattr,
+	.setxattr	= scoutfs_setxattr,
+	.getxattr	= scoutfs_getxattr,
 	.listxattr	= scoutfs_listxattr,
-	.removexattr	= generic_removexattr,
-	.get_acl	= scoutfs_get_acl,
+	.removexattr	= scoutfs_removexattr,
 	.fiemap		= scoutfs_data_fiemap,
 };
 
 static const struct inode_operations scoutfs_special_iops = {
 	.getattr	= scoutfs_getattr,
 	.setattr	= scoutfs_setattr,
-	.setxattr	= generic_setxattr,
-	.getxattr	= generic_getxattr,
+	.setxattr	= scoutfs_setxattr,
+	.getxattr	= scoutfs_getxattr,
 	.listxattr	= scoutfs_listxattr,
-	.removexattr	= generic_removexattr,
-	.get_acl	= scoutfs_get_acl,
+	.removexattr	= scoutfs_removexattr,
 };
 
 /*
@@ -331,6 +323,7 @@ int scoutfs_inode_refresh(struct inode *inode, struct scoutfs_lock *lock)
 			load_inode(inode, &sinode);
 			atomic64_set(&si->last_refreshed, refresh_gen);
 			scoutfs_lock_add_coverage(sb, lock, &si->ino_lock_cov);
+			si->drop_invalidated = false;
 		}
 	} else {
 		ret = 0;
@@ -369,16 +362,15 @@ static int set_inode_size(struct inode *inode, struct scoutfs_lock *lock,
 	if (!S_ISREG(inode->i_mode))
 		return 0;
 
-	ret = scoutfs_inode_index_lock_hold(inode, &ind_locks, true, false);
-	if (ret)
-		return ret;
-
 	scoutfs_per_task_add(&si->pt_data_lock, &pt_ent, lock);
 	ret = block_truncate_page(inode->i_mapping, new_size, scoutfs_get_block_write);
 	scoutfs_per_task_del(&si->pt_data_lock, &pt_ent);
-	if (ret < 0)
-		goto unlock;
-	scoutfs_inode_queue_writeback(inode);
+	if (ret)
+		goto out;
+
+	ret = scoutfs_inode_index_lock_hold(inode, &ind_locks, true, false);
+	if (ret)
+		goto out;
 
 	if (new_size != i_size_read(inode))
 		scoutfs_inode_inc_data_version(inode);
@@ -391,10 +383,10 @@ static int set_inode_size(struct inode *inode, struct scoutfs_lock *lock,
 	inode_inc_iversion(inode);
 	scoutfs_update_inode_item(inode, lock, &ind_locks);
 
-unlock:
 	scoutfs_release_trans(sb);
 	scoutfs_inode_index_unlock(sb, &ind_locks);
 
+out:
 	return ret;
 }
 
@@ -524,15 +516,10 @@ retry:
 	if (ret)
 		goto out;
 
-	ret = scoutfs_acl_chmod_locked(inode, attr, lock, &ind_locks);
-	if (ret < 0)
-		goto release;
-
 	setattr_copy(inode, attr);
 	inode_inc_iversion(inode);
 	scoutfs_update_inode_item(inode, lock, &ind_locks);
 
-release:
 	scoutfs_release_trans(sb);
 	scoutfs_inode_index_unlock(sb, &ind_locks);
 out:
@@ -969,8 +956,7 @@ void scoutfs_inode_init_index_key(struct scoutfs_key *key, u8 type, u64 major,
 static int update_index_items(struct super_block *sb,
 			      struct scoutfs_inode_info *si, u64 ino, u8 type,
 			      u64 major, u32 minor,
-			      struct list_head *lock_list,
-			      struct scoutfs_lock *primary)
+			      struct list_head *lock_list)
 {
 	struct scoutfs_lock *ins_lock;
 	struct scoutfs_lock *del_lock;
@@ -987,7 +973,7 @@ static int update_index_items(struct super_block *sb,
 	scoutfs_inode_init_index_key(&ins, type, major, minor, ino);
 
 	ins_lock = find_index_lock(lock_list, type, major, minor, ino);
-	ret = scoutfs_item_create_force(sb, &ins, NULL, 0, ins_lock, primary);
+	ret = scoutfs_item_create_force(sb, &ins, NULL, 0, ins_lock);
 	if (ret || !will_del_index(si, type, major, minor))
 		return ret;
 
@@ -999,7 +985,7 @@ static int update_index_items(struct super_block *sb,
 
 	del_lock = find_index_lock(lock_list, type, get_item_major(si, type),
 				   get_item_minor(si, type), ino);
-	ret = scoutfs_item_delete_force(sb, &del, del_lock, primary);
+	ret = scoutfs_item_delete_force(sb, &del, del_lock);
 	if (ret) {
 		err = scoutfs_item_delete(sb, &ins, ins_lock);
 		BUG_ON(err);
@@ -1011,8 +997,7 @@ static int update_index_items(struct super_block *sb,
 static int update_indices(struct super_block *sb,
 			  struct scoutfs_inode_info *si, u64 ino, umode_t mode,
 			  struct scoutfs_inode *sinode,
-			  struct list_head *lock_list,
-			  struct scoutfs_lock *primary)
+			  struct list_head *lock_list)
 {
 	struct index_update {
 		u8 type;
@@ -1032,7 +1017,7 @@ static int update_indices(struct super_block *sb,
 			continue;
 
 		ret = update_index_items(sb, si, ino, upd->type, upd->major,
-					 upd->minor, lock_list, primary);
+					 upd->minor, lock_list);
 		if (ret)
 			break;
 	}
@@ -1072,7 +1057,7 @@ void scoutfs_update_inode_item(struct inode *inode, struct scoutfs_lock *lock,
 	/* only race with other inode field stores once */
 	store_inode(&sinode, inode);
 
-	ret = update_indices(sb, si, ino, inode->i_mode, &sinode, lock_list, lock);
+	ret = update_indices(sb, si, ino, inode->i_mode, &sinode, lock_list);
 	BUG_ON(ret);
 
 	scoutfs_inode_init_key(&key, ino);
@@ -1341,7 +1326,7 @@ void scoutfs_inode_index_unlock(struct super_block *sb, struct list_head *list)
 
 /* this is called on final inode cleanup so enoent is fine */
 static int remove_index(struct super_block *sb, u64 ino, u8 type, u64 major,
-			u32 minor, struct list_head *ind_locks, struct scoutfs_lock *primary)
+			u32 minor, struct list_head *ind_locks)
 {
 	struct scoutfs_key key;
 	struct scoutfs_lock *lock;
@@ -1350,7 +1335,7 @@ static int remove_index(struct super_block *sb, u64 ino, u8 type, u64 major,
 	scoutfs_inode_init_index_key(&key, type, major, minor, ino);
 
 	lock = find_index_lock(ind_locks, type, major, minor, ino);
-	ret = scoutfs_item_delete_force(sb, &key, lock, primary);
+	ret = scoutfs_item_delete_force(sb, &key, lock);
 	if (ret == -ENOENT)
 		ret = 0;
 	return ret;
@@ -1367,17 +1352,16 @@ static int remove_index(struct super_block *sb, u64 ino, u8 type, u64 major,
  */
 static int remove_index_items(struct super_block *sb, u64 ino,
 			      struct scoutfs_inode *sinode,
-			      struct list_head *ind_locks,
-			      struct scoutfs_lock *primary)
+			      struct list_head *ind_locks)
 {
 	umode_t mode = le32_to_cpu(sinode->mode);
 	int ret;
 
 	ret = remove_index(sb, ino, SCOUTFS_INODE_INDEX_META_SEQ_TYPE,
-			   le64_to_cpu(sinode->meta_seq), 0, ind_locks, primary);
+			   le64_to_cpu(sinode->meta_seq), 0, ind_locks);
 	if (ret == 0 && S_ISREG(mode))
 		ret = remove_index(sb, ino, SCOUTFS_INODE_INDEX_DATA_SEQ_TYPE,
-				   le64_to_cpu(sinode->data_seq), 0, ind_locks, primary);
+				   le64_to_cpu(sinode->data_seq), 0, ind_locks);
 	return ret;
 }
 
@@ -1467,6 +1451,7 @@ int scoutfs_new_inode(struct super_block *sb, struct inode *dir, umode_t mode, d
 	si->have_item = false;
 	atomic64_set(&si->last_refreshed, lock->refresh_gen);
 	scoutfs_lock_add_coverage(sb, lock, &si->ino_lock_cov);
+	si->drop_invalidated = false;
 	si->flags = 0;
 
 	scoutfs_inode_set_meta_seq(inode);
@@ -1509,24 +1494,22 @@ static void init_orphan_key(struct scoutfs_key *key, u64 ino)
  * zone under a write only lock while the caller has the inode protected
  * by a write lock.
  */
-int scoutfs_inode_orphan_create(struct super_block *sb, u64 ino, struct scoutfs_lock *lock,
-				struct scoutfs_lock *primary)
+int scoutfs_inode_orphan_create(struct super_block *sb, u64 ino, struct scoutfs_lock *lock)
 {
 	struct scoutfs_key key;
 
 	init_orphan_key(&key, ino);
 
-	return scoutfs_item_create_force(sb, &key, NULL, 0, lock, primary);
+	return scoutfs_item_create_force(sb, &key, NULL, 0, lock);
 }
 
-int scoutfs_inode_orphan_delete(struct super_block *sb, u64 ino, struct scoutfs_lock *lock,
-				struct scoutfs_lock *primary)
+int scoutfs_inode_orphan_delete(struct super_block *sb, u64 ino, struct scoutfs_lock *lock)
 {
 	struct scoutfs_key key;
 
 	init_orphan_key(&key, ino);
 
-	return scoutfs_item_delete_force(sb, &key, lock, primary);
+	return scoutfs_item_delete_force(sb, &key, lock);
 }
 
 /*
@@ -1579,7 +1562,7 @@ retry:
 
 	release = true;
 
-	ret = remove_index_items(sb, ino, sinode, &ind_locks, lock);
+	ret = remove_index_items(sb, ino, sinode, &ind_locks);
 	if (ret)
 		goto out;
 
@@ -1594,7 +1577,7 @@ retry:
 	if (ret < 0)
 		goto out;
 
-	ret = scoutfs_inode_orphan_delete(sb, ino, orph_lock, lock);
+	ret = scoutfs_inode_orphan_delete(sb, ino, orph_lock);
 	if (ret < 0)
 		goto out;
 
@@ -1711,7 +1694,6 @@ static int try_delete_inode_items(struct super_block *sb, u64 ino)
 	struct scoutfs_lock *lock = NULL;
 	struct scoutfs_inode sinode;
 	struct scoutfs_key key;
-	bool clear_trying = false;
 	u64 group_nr;
 	int bit_nr;
 	int ret;
@@ -1731,7 +1713,6 @@ static int try_delete_inode_items(struct super_block *sb, u64 ino)
 		ret = 0;
 		goto out;
 	}
-	clear_trying = true;
 
 	/* can't delete if it's cached in local or remote mounts */
 	if (scoutfs_omap_test(sb, ino) || test_bit_le(bit_nr, ldata->map.bits)) {
@@ -1758,7 +1739,7 @@ static int try_delete_inode_items(struct super_block *sb, u64 ino)
 
 	ret = delete_inode_items(sb, ino, &sinode, lock, orph_lock);
 out:
-	if (clear_trying)
+	if (ldata)
 		clear_bit(bit_nr, ldata->trying);
 
 	scoutfs_unlock(sb, lock, SCOUTFS_LOCK_WRITE);
@@ -1768,18 +1749,18 @@ out:
 }
 
 /*
- * As we evicted an inode we need to decide to try and delete its items
- * or not, which is expensive.  We only try when we have lock coverage
- * and the inode has been unlinked.  This catches the common case of
- * regular deletion so deletion will be performed in the final unlink
- * task.  It also catches open-unlink or o_tmpfile that aren't cached on
- * other nodes.
+ * As we drop an inode we need to decide to try and delete its items or
+ * not, which is expensive.  The two common cases we want to get right
+ * both have cluster lock coverage and don't want to delete.   Dropping
+ * unused inodes during read lock invalidation has the current lock and
+ * sees a nonzero nlink and knows not to delete.  Final iput after a
+ * local unlink also has a lock, sees a zero nlink, and tries to perform
+ * item deletion in the task that dropped the last link, as users
+ * expect. 
  *
- * Inodes being evicted outside of lock coverage, by referenced dentries
- * or inodes that survived the attempt to drop them as their lock was
- * invalidated, will not try to delete.  This means that cross-mount
- * open/unlink will almost certainly fall back to the orphan scanner to
- * perform final deletion.
+ * Evicting an inode outside of cluster locking is the odd slow path
+ * that involves lock contention during use the worst cross-mount
+ * open-unlink/delete case.
  */
 void scoutfs_evict_inode(struct inode *inode)
 {
@@ -1795,7 +1776,7 @@ void scoutfs_evict_inode(struct inode *inode)
 		/* clear before trying to delete tests */
 		scoutfs_omap_clear(sb, ino);
 
-		if (scoutfs_lock_is_covered(sb, &si->ino_lock_cov) && inode->i_nlink == 0)
+		if (!scoutfs_lock_is_covered(sb, &si->ino_lock_cov) || inode->i_nlink == 0)
 			try_delete_inode_items(sb, scoutfs_ino(inode));
 	}
 
@@ -1820,56 +1801,30 @@ int scoutfs_drop_inode(struct inode *inode)
 {
 	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
 	struct super_block *sb = inode->i_sb;
-	const bool covered = scoutfs_lock_is_covered(sb, &si->ino_lock_cov);
 
 	trace_scoutfs_drop_inode(sb, scoutfs_ino(inode), inode->i_nlink, inode_unhashed(inode),
-				 covered);
+				 si->drop_invalidated);
 
-	return !covered || generic_drop_inode(inode);
+	return si->drop_invalidated || !scoutfs_lock_is_covered(sb, &si->ino_lock_cov) ||
+	       generic_drop_inode(inode);
 }
 
-
-/*
- * These iput workers can be concurrent amongst cpus.  This lets us get
- * some concurrency when these async final iputs end up performing very
- * expensive inode deletion.  Typically they're dropping linked inodes
- * that lost lock coverage and the iput will evict without deleting.
- *
- * Keep in mind that the dputs in d_prune can ascend into parents and
- * end up performing the final iput->evict deletion on other inodes.
- */
 static void iput_worker(struct work_struct *work)
 {
 	struct inode_sb_info *inf = container_of(work, struct inode_sb_info, iput_work);
 	struct scoutfs_inode_info *si;
-	struct inode *inode;
-	unsigned long count;
-	unsigned long flags;
+	struct scoutfs_inode_info *tmp;
+	struct llist_node *inodes;
+	bool more;
 
-	spin_lock(&inf->iput_lock);
-	while ((si = list_first_entry_or_null(&inf->iput_list, struct scoutfs_inode_info,
-					      iput_head))) {
-		list_del_init(&si->iput_head);
-		count = si->iput_count;
-		flags = si->iput_flags;
-		si->iput_count = 0;
-		si->iput_flags = 0;
-		spin_unlock(&inf->iput_lock);
+	inodes = llist_del_all(&inf->iput_llist);
 
-		inode = &si->inode;
-
-		/* can't touch during unmount, dcache destroys w/o locks */
-		if ((flags & SI_IPUT_FLAG_PRUNE) && !inf->stopped)
-			d_prune_aliases(inode);
-
-		while (count-- > 0)
-			iput(inode);
-
-		/* can't touch inode after final iput */
-
-		spin_lock(&inf->iput_lock);
+	llist_for_each_entry_safe(si, tmp, inodes, iput_llnode) {
+		do {
+			more = atomic_dec_return(&si->iput_count) > 0;
+			iput(&si->inode);
+		} while (more);
 	}
-	spin_unlock(&inf->iput_lock);
 }
 
 /*
@@ -1886,21 +1841,15 @@ static void iput_worker(struct work_struct *work)
  * Nothing stops multiple puts of an inode before the work runs so we
  * can track multiple puts in flight.
  */
-void scoutfs_inode_queue_iput(struct inode *inode, unsigned long flags)
+void scoutfs_inode_queue_iput(struct inode *inode)
 {
 	DECLARE_INODE_SB_INFO(inode->i_sb, inf);
 	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
-	bool should_queue;
 
-	spin_lock(&inf->iput_lock);
-	si->iput_count++;
-	si->iput_flags |= flags;
-	if ((should_queue = list_empty(&si->iput_head)))
-		list_add_tail(&si->iput_head, &inf->iput_list);
-	spin_unlock(&inf->iput_lock);
-
-	if (should_queue)
-		queue_work(inf->iput_workq, &inf->iput_work);
+	if (atomic_inc_return(&si->iput_count) == 1)
+		llist_add(&si->iput_llnode, &inf->iput_llist);
+	smp_wmb(); /* count and list visible before work executes */
+	schedule_work(&inf->iput_work);
 }
 
 /*
@@ -2104,7 +2053,7 @@ int scoutfs_inode_walk_writeback(struct super_block *sb, bool write)
 		trace_scoutfs_inode_walk_writeback(sb, scoutfs_ino(inode),
 						   write, ret);
 		if (ret) {
-			scoutfs_inode_queue_iput(inode, 0);
+			scoutfs_inode_queue_iput(inode);
 			goto out;
 		}
 
@@ -2120,7 +2069,7 @@ int scoutfs_inode_walk_writeback(struct super_block *sb, bool write)
 		if (!write)
 			list_del_init(&si->writeback_entry);
 
-		scoutfs_inode_queue_iput(inode, 0);
+		scoutfs_inode_queue_iput(inode);
 	}
 
 	spin_unlock(&inf->writeback_lock);
@@ -2145,15 +2094,7 @@ int scoutfs_inode_setup(struct super_block *sb)
 	spin_lock_init(&inf->ino_alloc.lock);
 	INIT_DELAYED_WORK(&inf->orphan_scan_dwork, inode_orphan_scan_worker);
 	INIT_WORK(&inf->iput_work, iput_worker);
-	spin_lock_init(&inf->iput_lock);
-	INIT_LIST_HEAD(&inf->iput_list);
-
-	/* re-entrant, worker locks with itself and queueing */
-	inf->iput_workq = alloc_workqueue("scoutfs_inode_iput", WQ_UNBOUND, 0);
-	if (!inf->iput_workq) {
-		kfree(inf);
-		return -ENOMEM;
-	}
+	init_llist_head(&inf->iput_llist);
 
 	sbi->inode_sb_info = inf;
 
@@ -2189,18 +2130,14 @@ void scoutfs_inode_flush_iput(struct super_block *sb)
 	DECLARE_INODE_SB_INFO(sb, inf);
 
 	if (inf)
-		flush_workqueue(inf->iput_workq);
+		flush_work(&inf->iput_work);
 }
 
 void scoutfs_inode_destroy(struct super_block *sb)
 {
 	struct inode_sb_info *inf = SCOUTFS_SB(sb)->inode_sb_info;
 
-	if (inf) {
-		if (inf->iput_workq)
-			destroy_workqueue(inf->iput_workq);
-		kfree(inf);
-	}
+	kfree(inf);
 }
 
 void scoutfs_inode_exit(void)

kmod/src/inode.h

@@ -56,16 +56,14 @@ struct scoutfs_inode_info {
 
 	struct scoutfs_lock_coverage ino_lock_cov;
 
-	struct list_head iput_head;
-	unsigned long iput_count;
-	unsigned long iput_flags;
+	/* drop if i_count hits 0, allows drop while invalidate holds coverage */
+	bool drop_invalidated;
+	struct llist_node iput_llnode;
+	atomic_t iput_count;
 
 	struct inode inode;
 };
 
-/* try to prune dcache aliases with queued iput */
-#define SI_IPUT_FLAG_PRUNE	(1 << 0)
-
 static inline struct scoutfs_inode_info *SCOUTFS_I(struct inode *inode)
 {
 	return container_of(inode, struct scoutfs_inode_info, inode);
@@ -80,7 +78,7 @@ struct inode *scoutfs_alloc_inode(struct super_block *sb);
 void scoutfs_destroy_inode(struct inode *inode);
 int scoutfs_drop_inode(struct inode *inode);
 void scoutfs_evict_inode(struct inode *inode);
-void scoutfs_inode_queue_iput(struct inode *inode, unsigned long flags);
+void scoutfs_inode_queue_iput(struct inode *inode);
 
 #define SCOUTFS_IGF_LINKED (1 << 0) /* enoent if nlink == 0 */
 struct inode *scoutfs_iget(struct super_block *sb, u64 ino, int lkf, int igf);
@@ -127,10 +125,8 @@ int scoutfs_getattr(struct vfsmount *mnt, struct dentry *dentry,
 		    struct kstat *stat);
 int scoutfs_setattr(struct dentry *dentry, struct iattr *attr);
 
-int scoutfs_inode_orphan_create(struct super_block *sb, u64 ino, struct scoutfs_lock *lock,
-				struct scoutfs_lock *primary);
-int scoutfs_inode_orphan_delete(struct super_block *sb, u64 ino, struct scoutfs_lock *lock,
-				struct scoutfs_lock *primary);
+int scoutfs_inode_orphan_create(struct super_block *sb, u64 ino, struct scoutfs_lock *lock);
+int scoutfs_inode_orphan_delete(struct super_block *sb, u64 ino, struct scoutfs_lock *lock);
 void scoutfs_inode_schedule_orphan_dwork(struct super_block *sb);
 
 void scoutfs_inode_queue_writeback(struct inode *inode);

kmod/src/item.c

@@ -1676,14 +1676,6 @@ static int lock_safe(struct scoutfs_lock *lock, struct scoutfs_key *key,
 		return 0;
 }
 
-static int optional_lock_mode_match(struct scoutfs_lock *lock, int mode)
-{
-	if (WARN_ON_ONCE(lock && lock->mode != mode))
-		return -EINVAL;
-	else
-		return 0;
-}
-
 /*
  * Copy the cached item's value into the caller's value.  The number of
  * bytes copied is returned.  A null val returns 0.
@@ -1840,19 +1832,12 @@ out:
  * also increase the seqs.  It lets us limit the inputs of item merging
  * to the last stable seq and ensure that all the items in open
  * transactions and granted locks will have greater seqs.
- *
- * This is a little awkward for WRITE_ONLY locks which can have much
- * older versions than the version of locked primary data that they're
- * operating on behalf of.  Callers can optionally provide that primary
- * lock to get the version from.   This ensures that items created under
- * WRITE_ONLY locks can not have versions less than their primary data.
  */
-static u64 item_seq(struct super_block *sb, struct scoutfs_lock *lock,
-		    struct scoutfs_lock *primary)
+static u64 item_seq(struct super_block *sb, struct scoutfs_lock *lock)
 {
 	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
 
-	return max3(sbi->trans_seq, lock->write_seq, primary ? primary->write_seq : 0);
+	return max(sbi->trans_seq, lock->write_seq);
 }
 
 /*
@@ -1887,7 +1872,7 @@ int scoutfs_item_dirty(struct super_block *sb, struct scoutfs_key *key,
 	if (!item || item->deletion) {
 		ret = -ENOENT;
 	} else {
-		item->seq = item_seq(sb, lock, NULL);
+		item->seq = item_seq(sb, lock);
 		mark_item_dirty(sb, cinf, pg, NULL, item);
 		ret = 0;
 	}
@@ -1904,10 +1889,10 @@ out:
  */
 static int item_create(struct super_block *sb, struct scoutfs_key *key,
 		       void *val, int val_len, struct scoutfs_lock *lock,
-		       struct scoutfs_lock *primary, int mode, bool force)
+		       int mode, bool force)
 {
 	DECLARE_ITEM_CACHE_INFO(sb, cinf);
-	const u64 seq = item_seq(sb, lock, primary);
+	const u64 seq = item_seq(sb, lock);
 	struct cached_item *found;
 	struct cached_item *item;
 	struct cached_page *pg;
@@ -1917,8 +1902,7 @@ static int item_create(struct super_block *sb, struct scoutfs_key *key,
 
 	scoutfs_inc_counter(sb, item_create);
 
-	if ((ret = lock_safe(lock, key, mode)) ||
-	    (ret = optional_lock_mode_match(primary, SCOUTFS_LOCK_WRITE)))
+	if ((ret = lock_safe(lock, key, mode)))
 		goto out;
 
 	ret = scoutfs_forest_set_bloom_bits(sb, lock);
@@ -1959,15 +1943,15 @@ out:
 int scoutfs_item_create(struct super_block *sb, struct scoutfs_key *key,
 			void *val, int val_len, struct scoutfs_lock *lock)
 {
-	return item_create(sb, key, val, val_len, lock, NULL,
+	return item_create(sb, key, val, val_len, lock,
 			   SCOUTFS_LOCK_READ, false);
 }
 
 int scoutfs_item_create_force(struct super_block *sb, struct scoutfs_key *key,
 			      void *val, int val_len,
-			      struct scoutfs_lock *lock, struct scoutfs_lock *primary)
+			      struct scoutfs_lock *lock)
 {
-	return item_create(sb, key, val, val_len, lock, primary,
+	return item_create(sb, key, val, val_len, lock,
 			   SCOUTFS_LOCK_WRITE_ONLY, true);
 }
 
@@ -1981,7 +1965,7 @@ int scoutfs_item_update(struct super_block *sb, struct scoutfs_key *key,
 			void *val, int val_len, struct scoutfs_lock *lock)
 {
 	DECLARE_ITEM_CACHE_INFO(sb, cinf);
-	const u64 seq = item_seq(sb, lock, NULL);
+	const u64 seq = item_seq(sb, lock);
 	struct cached_item *item;
 	struct cached_item *found;
 	struct cached_page *pg;
@@ -2041,16 +2025,12 @@ out:
  * current items so the caller always writes with write only locks.  If
  * combining the current delta item and the caller's item results in a
  * null we can just drop it, we don't have to emit a deletion item.
- *
- * Delta items don't have to worry about creating items with old
- * versions under write_only locks.  The versions don't impact how we
- * merge two items.
  */
 int scoutfs_item_delta(struct super_block *sb, struct scoutfs_key *key,
 		       void *val, int val_len, struct scoutfs_lock *lock)
 {
 	DECLARE_ITEM_CACHE_INFO(sb, cinf);
-	const u64 seq = item_seq(sb, lock, NULL);
+	const u64 seq = item_seq(sb, lock);
 	struct cached_item *item;
 	struct cached_page *pg;
 	struct rb_node **pnode;
@@ -2119,11 +2099,10 @@ out:
  * deletion item if there isn't one already cached.
  */
 static int item_delete(struct super_block *sb, struct scoutfs_key *key,
-		       struct scoutfs_lock *lock, struct scoutfs_lock *primary,
-		       int mode, bool force)
+		       struct scoutfs_lock *lock, int mode, bool force)
 {
 	DECLARE_ITEM_CACHE_INFO(sb, cinf);
-	const u64 seq = item_seq(sb, lock, primary);
+	const u64 seq = item_seq(sb, lock);
 	struct cached_item *item;
 	struct cached_page *pg;
 	struct rb_node **pnode;
@@ -2132,8 +2111,7 @@ static int item_delete(struct super_block *sb, struct scoutfs_key *key,
 
 	scoutfs_inc_counter(sb, item_delete);
 
-	if ((ret = lock_safe(lock, key, mode)) ||
-	    (ret = optional_lock_mode_match(primary, SCOUTFS_LOCK_WRITE)))
+	if ((ret = lock_safe(lock, key, mode)))
 		goto out;
 
 	ret = scoutfs_forest_set_bloom_bits(sb, lock);
@@ -2183,13 +2161,13 @@ out:
 int scoutfs_item_delete(struct super_block *sb, struct scoutfs_key *key,
 			struct scoutfs_lock *lock)
 {
-	return item_delete(sb, key, lock, NULL, SCOUTFS_LOCK_WRITE, false);
+	return item_delete(sb, key, lock, SCOUTFS_LOCK_WRITE, false);
 }
 
 int scoutfs_item_delete_force(struct super_block *sb, struct scoutfs_key *key,
-			      struct scoutfs_lock *lock, struct scoutfs_lock *primary)
+			      struct scoutfs_lock *lock)
 {
-	return item_delete(sb, key, lock, primary, SCOUTFS_LOCK_WRITE_ONLY, true);
+	return item_delete(sb, key, lock, SCOUTFS_LOCK_WRITE_ONLY, true);
 }
 
 u64 scoutfs_item_dirty_pages(struct super_block *sb)

kmod/src/item.h

@@ -15,15 +15,16 @@ int scoutfs_item_create(struct super_block *sb, struct scoutfs_key *key,
 			void *val, int val_len, struct scoutfs_lock *lock);
 int scoutfs_item_create_force(struct super_block *sb, struct scoutfs_key *key,
 			      void *val, int val_len,
-			      struct scoutfs_lock *lock, struct scoutfs_lock *primary);
+			      struct scoutfs_lock *lock);
 int scoutfs_item_update(struct super_block *sb, struct scoutfs_key *key,
 			void *val, int val_len, struct scoutfs_lock *lock);
 int scoutfs_item_delta(struct super_block *sb, struct scoutfs_key *key,
 		       void *val, int val_len, struct scoutfs_lock *lock);
 int scoutfs_item_delete(struct super_block *sb, struct scoutfs_key *key,
 			  struct scoutfs_lock *lock);
-int scoutfs_item_delete_force(struct super_block *sb, struct scoutfs_key *key,
-			      struct scoutfs_lock *lock, struct scoutfs_lock *primary);
+int scoutfs_item_delete_force(struct super_block *sb,
+				struct scoutfs_key *key,
+				struct scoutfs_lock *lock);
 
 u64 scoutfs_item_dirty_pages(struct super_block *sb);
 int scoutfs_item_write_dirty(struct super_block *sb);

kmod/src/kernelcompat.h

@@ -46,10 +46,4 @@ static inline int dir_emit_dots(struct file *file, void *dirent,
 }
 #endif
 
-#ifdef KC_POSIX_ACL_VALID_USER_NS
-#define kc_posix_acl_valid(user_ns, acl) posix_acl_valid(user_ns, acl)
-#else
-#define kc_posix_acl_valid(user_ns, acl) posix_acl_valid(acl)
-#endif
-
 #endif

kmod/src/lock.c

@@ -18,7 +18,6 @@
 #include <linux/mm.h>
 #include <linux/sort.h>
 #include <linux/ctype.h>
-#include <linux/posix_acl.h>
 
 #include "super.h"
 #include "lock.h"
@@ -130,13 +129,16 @@ static bool lock_modes_match(int granted, int requested)
  * allows deletions to be performed by unlink without having to wait for
  * remote cached inodes to be dropped.
  *
- * We kick the d_prune and iput off to async work because they can end
- * up in final iput and inode eviction item deletion which would
- * deadlock.   d_prune->dput can end up in iput on parents in different
- * locks entirely.
+ * If the cached inode was already deferring final inode deletion then
+ * we can't perform that inline in invalidation.  The locking alone
+ * deadlock, and it might also take multiple transactions to fully
+ * delete an inode with significant metadata.  We only perform the iput
+ * inline if we know that possible eviction can't perform the final
+ * deletion, otherwise we kick it off to async work.
  */
 static void invalidate_inode(struct super_block *sb, u64 ino)
 {
+	DECLARE_LOCK_INFO(sb, linfo);
 	struct scoutfs_inode_info *si;
 	struct inode *inode;
 
@@ -150,9 +152,17 @@ static void invalidate_inode(struct super_block *sb, u64 ino)
 			scoutfs_data_wait_changed(inode);
 		}
 
-		forget_all_cached_acls(inode);
+		/* can't touch during unmount, dcache destroys w/o locks */
+		if (!linfo->unmounting)
+			d_prune_aliases(inode);
 
-		scoutfs_inode_queue_iput(inode, SI_IPUT_FLAG_PRUNE);
+		si->drop_invalidated = true;
+		if (scoutfs_lock_is_covered(sb, &si->ino_lock_cov) && inode->i_nlink > 0) {
+			iput(inode);
+		} else {
+			/* defer iput to work context so we don't evict inodes from invalidation */
+			scoutfs_inode_queue_iput(inode);
+		}
 	}
 }
 
@@ -188,6 +198,16 @@ static int lock_invalidate(struct super_block *sb, struct scoutfs_lock *lock,
 	/* have to invalidate if we're not in the only usable case */
 	if (!(prev == SCOUTFS_LOCK_WRITE && mode == SCOUTFS_LOCK_READ)) {
 retry:
+		/* invalidate inodes before removing coverage */
+		if (lock->start.sk_zone == SCOUTFS_FS_ZONE) {
+			ino = le64_to_cpu(lock->start.ski_ino);
+			last = le64_to_cpu(lock->end.ski_ino);
+			while (ino <= last) {
+				invalidate_inode(sb, ino);
+				ino++;
+			}
+		}
+
 		/* remove cov items to tell users that their cache is stale */
 		spin_lock(&lock->cov_list_lock);
 		list_for_each_entry_safe(cov, tmp, &lock->cov_list, head) {
@@ -203,16 +223,6 @@ retry:
 		}
 		spin_unlock(&lock->cov_list_lock);
 
-		/* invalidate inodes after removing coverage so drop/evict aren't covered */
-		if (lock->start.sk_zone == SCOUTFS_FS_ZONE) {
-			ino = le64_to_cpu(lock->start.ski_ino);
-			last = le64_to_cpu(lock->end.ski_ino);
-			while (ino <= last) {
-				invalidate_inode(sb, ino);
-				ino++;
-			}
-		}
-
 		scoutfs_item_invalidate(sb, &lock->start, &lock->end);
 	}
 
@@ -279,7 +289,6 @@ static struct scoutfs_lock *lock_alloc(struct super_block *sb,
 	lock->sb = sb;
 	init_waitqueue_head(&lock->waitq);
 	lock->mode = SCOUTFS_LOCK_NULL;
-	lock->invalidating_mode = SCOUTFS_LOCK_NULL;
 
 	atomic64_set(&lock->forest_bloom_nr, 0);
 
@@ -657,9 +666,7 @@ struct inv_req {
  *
  * Before we start invalidating the lock we set the lock to the new
  * mode, preventing further incompatible users of the old mode from
- * using the lock while we're invalidating.  We record the previously
- * granted mode so that we can send lock recover responses with the old
- * granted mode during invalidation.
+ * using the lock while we're invalidating.
  */
 static void lock_invalidate_worker(struct work_struct *work)
 {
@@ -684,8 +691,7 @@ static void lock_invalidate_worker(struct work_struct *work)
 		if (!lock_counts_match(nl->new_mode, lock->users))
 			continue;
 
-		/* set the new mode, no incompatible users during inval, recov needs old */
-		lock->invalidating_mode = lock->mode;
+		/* set the new mode, no incompatible users during inval */
 		lock->mode = nl->new_mode;
 
 		/* move everyone that's ready to our private list */
@@ -728,8 +734,6 @@ static void lock_invalidate_worker(struct work_struct *work)
 		list_del(&ireq->head);
 		kfree(ireq);
 
-		lock->invalidating_mode = SCOUTFS_LOCK_NULL;
-
 		if (list_empty(&lock->inv_list)) {
 			/* finish if another request didn't arrive */
 			list_del_init(&lock->inv_head);
@@ -820,7 +824,6 @@ int scoutfs_lock_recover_request(struct super_block *sb, u64 net_id,
 {
 	DECLARE_LOCK_INFO(sb, linfo);
 	struct scoutfs_net_lock_recover *nlr;
-	enum scoutfs_lock_mode mode;
 	struct scoutfs_lock *lock;
 	struct scoutfs_lock *next;
 	struct rb_node *node;
@@ -841,15 +844,10 @@ int scoutfs_lock_recover_request(struct super_block *sb, u64 net_id,
 
 	for (i = 0; lock && i < SCOUTFS_NET_LOCK_MAX_RECOVER_NR; i++) {
 
-		if (lock->invalidating_mode != SCOUTFS_LOCK_NULL)
-			mode = lock->invalidating_mode;
-		else
-			mode = lock->mode;
-
 		nlr->locks[i].key = lock->start;
 		nlr->locks[i].write_seq = cpu_to_le64(lock->write_seq);
-		nlr->locks[i].old_mode = mode;
-		nlr->locks[i].new_mode = mode;
+		nlr->locks[i].old_mode = lock->mode;
+		nlr->locks[i].new_mode = lock->mode;
 
 		node = rb_next(&lock->node);
 		if (node)
@@ -1515,38 +1513,6 @@ void scoutfs_lock_flush_invalidate(struct super_block *sb)
 		flush_work(&linfo->inv_work);
 }
 
-static u64 get_held_lock_refresh_gen(struct super_block *sb, struct scoutfs_key *start)
-{
-	DECLARE_LOCK_INFO(sb, linfo);
-	struct scoutfs_lock *lock;
-	u64 refresh_gen = 0;
-
-	/* this can be called from all manner of places */
-	if (!linfo)
-		return 0;
-
-	spin_lock(&linfo->lock);
-	lock = lock_lookup(sb, start, NULL);
-	if (lock) {
-		if (lock_mode_can_read(lock->mode))
-			refresh_gen = lock->refresh_gen;
-	}
-	spin_unlock(&linfo->lock);
-
-	return refresh_gen;
-}
-
-u64 scoutfs_lock_ino_refresh_gen(struct super_block *sb, u64 ino)
-{
-	struct scoutfs_key start;
-
-	scoutfs_key_set_zeros(&start);
-	start.sk_zone = SCOUTFS_FS_ZONE;
-	start.ski_ino = cpu_to_le64(ino & ~(u64)SCOUTFS_LOCK_INODE_GROUP_MASK);
-
-	return get_held_lock_refresh_gen(sb, &start);
-}
-
 /*
  * The caller is going to be shutting down transactions and the client.
  * We need to make sure that locking won't call either after we return.

kmod/src/lock.h

@@ -39,7 +39,6 @@ struct scoutfs_lock {
 	struct list_head cov_list;
 
 	enum scoutfs_lock_mode mode;
-	enum scoutfs_lock_mode invalidating_mode;
 	unsigned int waiters[SCOUTFS_LOCK_NR_MODES];
 	unsigned int users[SCOUTFS_LOCK_NR_MODES];
 
@@ -100,8 +99,6 @@ void scoutfs_lock_del_coverage(struct super_block *sb,
 bool scoutfs_lock_protected(struct scoutfs_lock *lock, struct scoutfs_key *key,
 			    enum scoutfs_lock_mode mode);
 
-u64 scoutfs_lock_ino_refresh_gen(struct super_block *sb, u64 ino);
-
 void scoutfs_free_unused_locks(struct super_block *sb);
 
 int scoutfs_lock_setup(struct super_block *sb);

kmod/src/lock_server.c

@@ -749,7 +749,7 @@ out:
 	if (ret < 0) {
 		scoutfs_err(sb, "lock server err %d during client rid %016llx farewell, shutting down",
 			    ret, rid);
-		scoutfs_server_stop(sb);
+		scoutfs_server_abort(sb);
 	}
 
 	return ret;

kmod/src/net.c

@@ -355,7 +355,6 @@ static int submit_send(struct super_block *sb,
 		}
 		if (rid != 0) {
 			spin_unlock(&conn->lock);
-			kfree(msend);
 			return -ENOTCONN;
 		}
 	}
@@ -992,8 +991,6 @@ static void scoutfs_net_listen_worker(struct work_struct *work)
 		if (ret < 0)
 			break;
 
-		acc_sock->sk->sk_allocation = GFP_NOFS;
-
 		/* inherit accepted request funcs from listening conn */
 		acc_conn = scoutfs_net_alloc_conn(sb, conn->notify_up,
 						  conn->notify_down,
@@ -1056,8 +1053,6 @@ static void scoutfs_net_connect_worker(struct work_struct *work)
 	if (ret)
 		goto out;
 
-	sock->sk->sk_allocation = GFP_NOFS;
-
 	/* caller specified connect timeout */
 	tv.tv_sec = conn->connect_timeout_ms / MSEC_PER_SEC;
 	tv.tv_usec = (conn->connect_timeout_ms % MSEC_PER_SEC) * USEC_PER_MSEC;
@@ -1297,7 +1292,7 @@ restart:
 				if (ret) {
 					scoutfs_err(sb, "client fence returned err %d, shutting down server",
 						    ret);
-					scoutfs_server_stop(sb);
+					scoutfs_server_abort(sb);
 				}
 			}
 			destroy_conn(acc);
@@ -1346,12 +1341,10 @@ scoutfs_net_alloc_conn(struct super_block *sb,
 	if (!conn)
 		return NULL;
 
-	if (info_size) {
-		conn->info = kzalloc(info_size, GFP_NOFS);
-		if (!conn->info) {
-			kfree(conn);
-			return NULL;
-		}
+	conn->info = kzalloc(info_size, GFP_NOFS);
+	if (!conn->info) {
+		kfree(conn);
+		return NULL;
 	}
 
 	conn->workq = alloc_workqueue("scoutfs_net_%s",
@@ -1457,8 +1450,6 @@ int scoutfs_net_bind(struct super_block *sb,
 	if (ret)
 		goto out;
 
-	sock->sk->sk_allocation = GFP_NOFS;
-
 	optval = 1;
 	ret = kernel_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR,
 				(char *)&optval, sizeof(optval));

kmod/src/omap.c

@@ -157,15 +157,6 @@ static int free_rid(struct omap_rid_list *list, struct omap_rid_entry *entry)
 	return nr;
 }
 
-static void free_rid_list(struct omap_rid_list *list)
-{
-	struct omap_rid_entry *entry;
-	struct omap_rid_entry *tmp;
-
-	list_for_each_entry_safe(entry, tmp, &list->head, head)
-		free_rid(list, entry);
-}
-
 static int copy_rids(struct omap_rid_list *to, struct omap_rid_list *from, spinlock_t *from_lock)
 {
 	struct omap_rid_entry *entry;
@@ -813,10 +804,6 @@ void scoutfs_omap_server_shutdown(struct super_block *sb)
 	llist_for_each_entry_safe(req, tmp, requests, llnode)
 		kfree(req);
 
-	spin_lock(&ominf->lock);
-	free_rid_list(&ominf->rids);
-	spin_unlock(&ominf->lock);
-
 	synchronize_rcu();
 }
 
@@ -877,10 +864,6 @@ void scoutfs_omap_destroy(struct super_block *sb)
 		rhashtable_walk_stop(&iter);
 		rhashtable_walk_exit(&iter);
 
-		spin_lock(&ominf->lock);
-		free_rid_list(&ominf->rids);
-		spin_unlock(&ominf->lock);
-
 		rhashtable_destroy(&ominf->group_ht);
 		rhashtable_destroy(&ominf->req_ht);
 		kfree(ominf);

kmod/src/options.c

@@ -27,28 +27,17 @@
 #include "options.h"
 #include "super.h"
 #include "inode.h"
-#include "alloc.h"
 
 enum {
-	Opt_acl,
-	Opt_data_prealloc_blocks,
-	Opt_data_prealloc_contig_only,
 	Opt_metadev_path,
-	Opt_noacl,
 	Opt_orphan_scan_delay_ms,
-	Opt_quorum_heartbeat_timeout_ms,
 	Opt_quorum_slot_nr,
 	Opt_err,
 };
 
 static const match_table_t tokens = {
-	{Opt_acl, "acl"},
-	{Opt_data_prealloc_blocks, "data_prealloc_blocks=%s"},
-	{Opt_data_prealloc_contig_only, "data_prealloc_contig_only=%s"},
 	{Opt_metadev_path, "metadev_path=%s"},
-	{Opt_noacl, "noacl"},
 	{Opt_orphan_scan_delay_ms, "orphan_scan_delay_ms=%s"},
-	{Opt_quorum_heartbeat_timeout_ms, "quorum_heartbeat_timeout_ms=%s"},
 	{Opt_quorum_slot_nr, "quorum_slot_nr=%s"},
 	{Opt_err, NULL}
 };
@@ -117,39 +106,11 @@ static void free_options(struct scoutfs_mount_options *opts)
 #define DEFAULT_ORPHAN_SCAN_DELAY_MS	(10 * MSEC_PER_SEC)
 #define MAX_ORPHAN_SCAN_DELAY_MS	(60 * MSEC_PER_SEC)
 
-#define MIN_DATA_PREALLOC_BLOCKS	1ULL
-#define MAX_DATA_PREALLOC_BLOCKS	((unsigned long long)SCOUTFS_BLOCK_SM_MAX)
-
 static void init_default_options(struct scoutfs_mount_options *opts)
 {
 	memset(opts, 0, sizeof(*opts));
-
-	opts->data_prealloc_blocks = SCOUTFS_DATA_PREALLOC_DEFAULT_BLOCKS;
-	opts->data_prealloc_contig_only = 1;
-	opts->orphan_scan_delay_ms = -1;
-	opts->quorum_heartbeat_timeout_ms = SCOUTFS_QUORUM_DEF_HB_TIMEO_MS;
 	opts->quorum_slot_nr = -1;
-}
-
-static int set_quorum_heartbeat_timeout_ms(struct super_block *sb, int ret, u64 val)
-{
-	DECLARE_OPTIONS_INFO(sb, optinf);
-
-	if (ret < 0) {
-		scoutfs_err(sb, "failed to parse quorum_heartbeat_timeout_ms value");
-		return -EINVAL;
-	}
-	if (val < SCOUTFS_QUORUM_MIN_HB_TIMEO_MS || val > SCOUTFS_QUORUM_MAX_HB_TIMEO_MS) {
-		scoutfs_err(sb, "invalid quorum_heartbeat_timeout_ms value %llu, must be between %lu and %lu",
-			    val, SCOUTFS_QUORUM_MIN_HB_TIMEO_MS, SCOUTFS_QUORUM_MAX_HB_TIMEO_MS);
-		return -EINVAL;
-	}
-
-	write_seqlock(&optinf->seqlock);
-	optinf->opts.quorum_heartbeat_timeout_ms = val;
-	write_sequnlock(&optinf->seqlock);
-
-	return 0;
+	opts->orphan_scan_delay_ms = DEFAULT_ORPHAN_SCAN_DELAY_MS;
 }
 
 /*
@@ -161,7 +122,6 @@ static int set_quorum_heartbeat_timeout_ms(struct super_block *sb, int ret, u64
 static int parse_options(struct super_block *sb, char *options, struct scoutfs_mount_options *opts)
 {
 	substring_t args[MAX_OPT_ARGS];
-	u64 nr64;
 	int nr;
 	int token;
 	char *p;
@@ -174,44 +134,12 @@ static int parse_options(struct super_block *sb, char *options, struct scoutfs_m
 		token = match_token(p, tokens, args);
 		switch (token) {
 
-		case Opt_acl:
-			sb->s_flags |= MS_POSIXACL;
-			break;
-
-		case Opt_data_prealloc_blocks:
-			ret = match_u64(args, &nr64);
-			if (ret < 0 ||
-			    nr64 < MIN_DATA_PREALLOC_BLOCKS || nr64 > MAX_DATA_PREALLOC_BLOCKS) {
-				scoutfs_err(sb, "invalid data_prealloc_blocks option, must be between %llu and %llu",
-					    MIN_DATA_PREALLOC_BLOCKS, MAX_DATA_PREALLOC_BLOCKS);
-				if (ret == 0)
-					ret = -EINVAL;
-				return ret;
-			}
-			opts->data_prealloc_blocks = nr64;
-			break;
-
-		case Opt_data_prealloc_contig_only:
-			ret = match_int(args, &nr);
-			if (ret < 0 || nr < 0 || nr > 1) {
-				scoutfs_err(sb, "invalid data_prealloc_contig_only option, bool must only be 0 or 1");
-				if (ret == 0)
-					ret = -EINVAL;
-				return ret;
-			}
-			opts->data_prealloc_contig_only = nr;
-			break;
-
 		case Opt_metadev_path:
 			ret = parse_bdev_path(sb, &args[0], &opts->metadev_path);
 			if (ret < 0)
 				return ret;
 			break;
 
-		case Opt_noacl:
-			sb->s_flags &= ~MS_POSIXACL;
-			break;
-
 		case Opt_orphan_scan_delay_ms:
 			if (opts->orphan_scan_delay_ms != -1) {
 				scoutfs_err(sb, "multiple orphan_scan_delay_ms options provided, only provide one.");
@@ -230,13 +158,6 @@ static int parse_options(struct super_block *sb, char *options, struct scoutfs_m
 			opts->orphan_scan_delay_ms = nr;
 			break;
 
-		case Opt_quorum_heartbeat_timeout_ms:
-			ret = match_u64(args, &nr64);
-			ret = set_quorum_heartbeat_timeout_ms(sb, ret, nr64);
-			if (ret < 0)
-				return ret;
-			break;
-
 		case Opt_quorum_slot_nr:
 			if (opts->quorum_slot_nr != -1) {
 				scoutfs_err(sb, "multiple quorum_slot_nr options provided, only provide one.");
@@ -260,9 +181,6 @@ static int parse_options(struct super_block *sb, char *options, struct scoutfs_m
 		}
 	}
 
-	if (opts->orphan_scan_delay_ms == -1)
-		opts->orphan_scan_delay_ms = DEFAULT_ORPHAN_SCAN_DELAY_MS;
-
 	if (!opts->metadev_path) {
 		scoutfs_err(sb, "Required mount option \"metadev_path\" not found");
 		return -EINVAL;
@@ -332,17 +250,10 @@ int scoutfs_options_show(struct seq_file *seq, struct dentry *root)
 {
 	struct super_block *sb = root->d_sb;
 	struct scoutfs_mount_options opts;
-	const bool is_acl = !!(sb->s_flags & MS_POSIXACL);
 
 	scoutfs_options_read(sb, &opts);
 
-	if (is_acl)
-		seq_puts(seq, ",acl");
-	seq_printf(seq, ",data_prealloc_blocks=%llu", opts.data_prealloc_blocks);
-	seq_printf(seq, ",data_prealloc_contig_only=%u", opts.data_prealloc_contig_only);
 	seq_printf(seq, ",metadev_path=%s", opts.metadev_path);
-	if (!is_acl)
-		seq_puts(seq, ",noacl");
 	seq_printf(seq, ",orphan_scan_delay_ms=%u", opts.orphan_scan_delay_ms);
 	if (opts.quorum_slot_nr >= 0)
 		seq_printf(seq, ",quorum_slot_nr=%d", opts.quorum_slot_nr);
@@ -350,83 +261,6 @@ int scoutfs_options_show(struct seq_file *seq, struct dentry *root)
 	return 0;
 }
 
-static ssize_t data_prealloc_blocks_show(struct kobject *kobj, struct kobj_attribute *attr,
-					 char *buf)
-{
-	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
-	struct scoutfs_mount_options opts;
-
-	scoutfs_options_read(sb, &opts);
-
-	return snprintf(buf, PAGE_SIZE, "%llu", opts.data_prealloc_blocks);
-}
-static ssize_t data_prealloc_blocks_store(struct kobject *kobj, struct kobj_attribute *attr,
-					  const char *buf, size_t count)
-{
-	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
-	DECLARE_OPTIONS_INFO(sb, optinf);
-	char nullterm[30]; /* more than enough for octal -U64_MAX */
-	u64 val;
-	int len;
-	int ret;
-
-	len = min(count, sizeof(nullterm) - 1);
-	memcpy(nullterm, buf, len);
-	nullterm[len] = '\0';
-
-	ret = kstrtoll(nullterm, 0, &val);
-	if (ret < 0 || val < MIN_DATA_PREALLOC_BLOCKS || val > MAX_DATA_PREALLOC_BLOCKS) {
-		scoutfs_err(sb, "invalid data_prealloc_blocks option, must be between %llu and %llu",
-			    MIN_DATA_PREALLOC_BLOCKS, MAX_DATA_PREALLOC_BLOCKS);
-		return -EINVAL;
-	}
-
-	write_seqlock(&optinf->seqlock);
-	optinf->opts.data_prealloc_blocks = val;
-	write_sequnlock(&optinf->seqlock);
-
-	return count;
-}
-SCOUTFS_ATTR_RW(data_prealloc_blocks);
-
-static ssize_t data_prealloc_contig_only_show(struct kobject *kobj, struct kobj_attribute *attr,
-					 char *buf)
-{
-	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
-	struct scoutfs_mount_options opts;
-
-	scoutfs_options_read(sb, &opts);
-
-	return snprintf(buf, PAGE_SIZE, "%u", opts.data_prealloc_contig_only);
-}
-static ssize_t data_prealloc_contig_only_store(struct kobject *kobj, struct kobj_attribute *attr,
-					  const char *buf, size_t count)
-{
-	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
-	DECLARE_OPTIONS_INFO(sb, optinf);
-	char nullterm[20]; /* more than enough for octal -U32_MAX */
-	long val;
-	int len;
-	int ret;
-
-	len = min(count, sizeof(nullterm) - 1);
-	memcpy(nullterm, buf, len);
-	nullterm[len] = '\0';
-
-	ret = kstrtol(nullterm, 0, &val);
-	if (ret < 0 || val < 0 || val > 1) {
-		scoutfs_err(sb, "invalid data_prealloc_contig_only option, bool must be 0 or 1");
-		return -EINVAL;
-	}
-
-	write_seqlock(&optinf->seqlock);
-	optinf->opts.data_prealloc_contig_only = val;
-	write_sequnlock(&optinf->seqlock);
-
-	return count;
-}
-SCOUTFS_ATTR_RW(data_prealloc_contig_only);
-
 static ssize_t metadev_path_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf)
 {
 	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
@@ -479,38 +313,6 @@ static ssize_t orphan_scan_delay_ms_store(struct kobject *kobj, struct kobj_attr
 }
 SCOUTFS_ATTR_RW(orphan_scan_delay_ms);
 
-static ssize_t quorum_heartbeat_timeout_ms_show(struct kobject *kobj, struct kobj_attribute *attr,
-						char *buf)
-{
-	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
-	struct scoutfs_mount_options opts;
-
-	scoutfs_options_read(sb, &opts);
-
-	return snprintf(buf, PAGE_SIZE, "%llu", opts.quorum_heartbeat_timeout_ms);
-}
-static ssize_t quorum_heartbeat_timeout_ms_store(struct kobject *kobj, struct kobj_attribute *attr,
-						 const char *buf, size_t count)
-{
-	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
-	char nullterm[30]; /* more than enough for octal -U64_MAX */
-	u64 val;
-	int len;
-	int ret;
-
-	len = min(count, sizeof(nullterm) - 1);
-	memcpy(nullterm, buf, len);
-	nullterm[len] = '\0';
-
-	ret = kstrtoll(nullterm, 0, &val);
-	ret = set_quorum_heartbeat_timeout_ms(sb, ret, val);
-	if (ret == 0)
-		ret = count;
-
-	return ret;
-}
-SCOUTFS_ATTR_RW(quorum_heartbeat_timeout_ms);
-
 static ssize_t quorum_slot_nr_show(struct kobject *kobj, struct kobj_attribute *attr, char *buf)
 {
 	struct super_block *sb = SCOUTFS_SYSFS_ATTRS_SB(kobj);
@@ -523,11 +325,8 @@ static ssize_t quorum_slot_nr_show(struct kobject *kobj, struct kobj_attribute *
 SCOUTFS_ATTR_RO(quorum_slot_nr);
 
 static struct attribute *options_attrs[] = {
-	SCOUTFS_ATTR_PTR(data_prealloc_blocks),
-	SCOUTFS_ATTR_PTR(data_prealloc_contig_only),
 	SCOUTFS_ATTR_PTR(metadev_path),
 	SCOUTFS_ATTR_PTR(orphan_scan_delay_ms),
-	SCOUTFS_ATTR_PTR(quorum_heartbeat_timeout_ms),
 	SCOUTFS_ATTR_PTR(quorum_slot_nr),
 	NULL,
 };

kmod/src/options.h

@@ -6,12 +6,10 @@
 #include "format.h"
 
 struct scoutfs_mount_options {
-	u64 data_prealloc_blocks;
-	bool data_prealloc_contig_only;
 	char *metadev_path;
 	unsigned int orphan_scan_delay_ms;
 	int quorum_slot_nr;
-	u64 quorum_heartbeat_timeout_ms;
+
 };
 
 void scoutfs_options_read(struct super_block *sb, struct scoutfs_mount_options *opts);

kmod/src/quorum.c

@@ -100,63 +100,46 @@ struct last_msg {
 	ktime_t ts;
 };
 
-struct count_recent {
-	u64 count;
-	ktime_t recent;
-};
-
 enum quorum_role { FOLLOWER, CANDIDATE, LEADER };
 
 struct quorum_status {
 	enum quorum_role role;
 	u64 term;
-	u64 server_start_term;
-	int server_event;
 	int vote_for;
 	unsigned long vote_bits;
 	ktime_t timeout;
 };
 
-#define HB_DELAY_NR		(SCOUTFS_QUORUM_MAX_HB_TIMEO_MS / MSEC_PER_SEC)
-
 struct quorum_info {
 	struct super_block *sb;
-	struct scoutfs_quorum_config qconf;
-	struct workqueue_struct *workq;
 	struct work_struct work;
 	struct socket *sock;
 	bool shutdown;
 
 	int our_quorum_slot_nr;
+	unsigned long flags;
 	int votes_needed;
 
 	spinlock_t show_lock;
 	struct quorum_status show_status;
 	struct last_msg last_send[SCOUTFS_QUORUM_MAX_SLOTS];
 	struct last_msg last_recv[SCOUTFS_QUORUM_MAX_SLOTS];
-	struct count_recent *hb_delay;
-	unsigned long max_hb_delay;
 
 	struct scoutfs_sysfs_attrs ssa;
 };
 
+#define QINF_FLAG_SERVER 0
+
 #define DECLARE_QUORUM_INFO(sb, name) \
 	struct quorum_info *name = SCOUTFS_SB(sb)->quorum_info
 #define DECLARE_QUORUM_INFO_KOBJ(kobj, name) \
 	DECLARE_QUORUM_INFO(SCOUTFS_SYSFS_ATTRS_SB(kobj), name)
 
-static bool quorum_slot_present(struct scoutfs_quorum_config *qconf, int i)
+static bool quorum_slot_present(struct scoutfs_super_block *super, int i)
 {
 	BUG_ON(i < 0 || i > SCOUTFS_QUORUM_MAX_SLOTS);
 
-	return qconf->slots[i].addr.v4.family == cpu_to_le16(SCOUTFS_AF_IPV4);
-}
-
-static void quorum_slot_sin(struct scoutfs_quorum_config *qconf, int i, struct sockaddr_in *sin)
-{
-	BUG_ON(i < 0 || i >= SCOUTFS_QUORUM_MAX_SLOTS);
-
-	scoutfs_addr_to_sin(sin, &qconf->slots[i].addr);
+	return super->qconf.slots[i].addr.v4.family == cpu_to_le16(SCOUTFS_AF_IPV4);
 }
 
 static ktime_t election_timeout(void)
@@ -170,14 +153,15 @@ static ktime_t heartbeat_interval(void)
 	return ktime_add_ms(ktime_get(), SCOUTFS_QUORUM_HB_IVAL_MS);
 }
 
-static ktime_t heartbeat_timeout(struct scoutfs_mount_options *opts)
+static ktime_t heartbeat_timeout(void)
 {
-	return ktime_add_ms(ktime_get(), opts->quorum_heartbeat_timeout_ms);
+	return ktime_add_ms(ktime_get(), SCOUTFS_QUORUM_HB_TIMEO_MS);
 }
 
 static int create_socket(struct super_block *sb)
 {
 	DECLARE_QUORUM_INFO(sb, qinf);
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	struct socket *sock = NULL;
 	struct sockaddr_in sin;
 	int addrlen;
@@ -189,10 +173,9 @@ static int create_socket(struct super_block *sb)
 		goto out;
 	}
 
-	/* rather fail and retry than block waiting for free */
-	sock->sk->sk_allocation = GFP_ATOMIC;
+	sock->sk->sk_allocation = GFP_NOFS;
 
-	quorum_slot_sin(&qinf->qconf, qinf->our_quorum_slot_nr, &sin);
+	scoutfs_quorum_slot_sin(super, qinf->our_quorum_slot_nr, &sin);
 
 	addrlen = sizeof(sin);
 	ret = kernel_bind(sock, (struct sockaddr *)&sin, addrlen);
@@ -219,20 +202,16 @@ static __le32 quorum_message_crc(struct scoutfs_quorum_message *qmes)
 	return cpu_to_le32(crc32c(~0, qmes, len));
 }
 
-/*
- * Returns the number of failures from sendmsg.
- */
-static int send_msg_members(struct super_block *sb, int type, u64 term, int only)
+static void send_msg_members(struct super_block *sb, int type, u64 term,
+			     int only)
 {
-	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
 	DECLARE_QUORUM_INFO(sb, qinf);
-	int failed = 0;
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	ktime_t now;
-	int ret;
 	int i;
 
 	struct scoutfs_quorum_message qmes = {
-		.fsid = cpu_to_le64(sbi->fsid),
+		.fsid = super->hdr.fsid,
 		.term = cpu_to_le64(term),
 		.type = type,
 		.from = qinf->our_quorum_slot_nr,
@@ -254,21 +233,15 @@ static int send_msg_members(struct super_block *sb, int type, u64 term, int only
 
 	qmes.crc = quorum_message_crc(&qmes);
 
+
 	for (i = 0; i < SCOUTFS_QUORUM_MAX_SLOTS; i++) {
-		if (!quorum_slot_present(&qinf->qconf, i) ||
+		if (!quorum_slot_present(super, i) ||
 		    (only >= 0 && i != only) || i == qinf->our_quorum_slot_nr)
 			continue;
 
-		if (scoutfs_forcing_unmount(sb)) {
-			failed = 0;
-			break;
-		}
-
-		scoutfs_quorum_slot_sin(&qinf->qconf, i, &sin);
+		scoutfs_quorum_slot_sin(super, i, &sin);
 		now = ktime_get();
-		ret = kernel_sendmsg(qinf->sock, &mh, &kv, 1, kv.iov_len);
-		if (ret != kv.iov_len)
-			failed++;
+		kernel_sendmsg(qinf->sock, &mh, &kv, 1, kv.iov_len);
 
 		spin_lock(&qinf->show_lock);
 		qinf->last_send[i].msg.term = term;
@@ -279,8 +252,6 @@ static int send_msg_members(struct super_block *sb, int type, u64 term, int only
 		if (i == only)
 			break;
 	}
-
-	return failed;
 }
 
 #define send_msg_to(sb, type, term, nr)  send_msg_members(sb, type, term, nr)
@@ -296,7 +267,7 @@ static int recv_msg(struct super_block *sb, struct quorum_host_msg *msg,
 		    ktime_t abs_to)
 {
 	DECLARE_QUORUM_INFO(sb, qinf);
-	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	struct scoutfs_quorum_message qmes;
 	struct timeval tv;
 	ktime_t rel_to;
@@ -335,17 +306,14 @@ static int recv_msg(struct super_block *sb, struct quorum_host_msg *msg,
 	if (ret < 0)
 		return ret;
 
-	if (scoutfs_forcing_unmount(sb))
-		return 0;
-
 	now = ktime_get();
 
 	if (ret != sizeof(qmes) ||
 	    qmes.crc != quorum_message_crc(&qmes) ||
-	    qmes.fsid != cpu_to_le64(sbi->fsid) ||
+	    qmes.fsid != super->hdr.fsid ||
 	    qmes.type >= SCOUTFS_QUORUM_MSG_INVALID ||
 	    qmes.from >= SCOUTFS_QUORUM_MAX_SLOTS ||
-	    !quorum_slot_present(&qinf->qconf, qmes.from)) {
+	    !quorum_slot_present(super, qmes.from)) {
 		/* should we be trying to open a new socket? */
 		scoutfs_inc_counter(sb, quorum_recv_invalid);
 		return -EAGAIN;
@@ -375,7 +343,7 @@ static int read_quorum_block(struct super_block *sb, u64 blkno, struct scoutfs_q
 			     bool check_rid)
 {
 	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
-	const u64 fsid = sbi->fsid;
+	struct scoutfs_super_block *super = &sbi->super;
 	const u64 rid = sbi->rid;
 	char msg[150];
 	__le32 crc;
@@ -400,9 +368,9 @@ static int read_quorum_block(struct super_block *sb, u64 blkno, struct scoutfs_q
 	else if (le32_to_cpu(blk->hdr.magic) != SCOUTFS_BLOCK_MAGIC_QUORUM) 
 		snprintf(msg, sizeof(msg), "blk magic %08x != %08x",
 			 le32_to_cpu(blk->hdr.magic), SCOUTFS_BLOCK_MAGIC_QUORUM);
-	else if (blk->hdr.fsid != cpu_to_le64(fsid))
+	else if (blk->hdr.fsid != super->hdr.fsid)
 		snprintf(msg, sizeof(msg), "blk fsid %016llx != %016llx",
-			 le64_to_cpu(blk->hdr.fsid), fsid);
+			 le64_to_cpu(blk->hdr.fsid), le64_to_cpu(super->hdr.fsid));
 	else if (le64_to_cpu(blk->hdr.blkno) != blkno)
 		snprintf(msg, sizeof(msg), "blk blkno %llu != %llu",
 			 le64_to_cpu(blk->hdr.blkno), blkno);
@@ -443,7 +411,8 @@ out:
  */
 static void read_greatest_term(struct super_block *sb, u64 *term)
 {
-	DECLARE_QUORUM_INFO(sb, qinf);
+	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &sbi->super;
 	struct scoutfs_quorum_block blk;
 	int ret;
 	int e;
@@ -452,7 +421,7 @@ static void read_greatest_term(struct super_block *sb, u64 *term)
 	*term = 0;
 
 	for (s = 0; s < SCOUTFS_QUORUM_MAX_SLOTS; s++) {
-		if (!quorum_slot_present(&qinf->qconf, s))
+		if (!quorum_slot_present(super, s))
 			continue;
 
 		ret = read_quorum_block(sb, SCOUTFS_QUORUM_BLKNO + s, &blk, false);
@@ -525,6 +494,16 @@ static int update_quorum_block(struct super_block *sb, int event, u64 term, bool
 	return ret;
 }
 
+/*
+ * The calling server has fenced previous leaders and reclaimed their
+ * resources.  We can now update our fence event with a greater term to
+ * stop future leaders from doing the same.
+ */
+int scoutfs_quorum_fence_complete(struct super_block *sb, u64 term)
+{
+	return update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_FENCE, term, true);
+}
+
 /*
  * The calling server has been elected and has started running but can't
  * yet assume that it has exclusive access to the metadata device.  We
@@ -546,15 +525,14 @@ static int update_quorum_block(struct super_block *sb, int event, u64 term, bool
  * keeps us from being fenced while we allow userspace fencing to take a
  * reasonably long time.  We still want to timeout eventually.
  */
-int scoutfs_quorum_fence_leaders(struct super_block *sb, struct scoutfs_quorum_config *qconf,
-				 u64 term)
+int scoutfs_quorum_fence_leaders(struct super_block *sb, u64 term)
 {
 #define NR_OLD 2
 	struct scoutfs_quorum_block_event old[SCOUTFS_QUORUM_MAX_SLOTS][NR_OLD] = {{{0,}}};
 	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &sbi->super;
 	struct scoutfs_quorum_block blk;
 	struct sockaddr_in sin;
-	const __le64 lefsid = cpu_to_le64(sbi->fsid);
 	const u64 rid = sbi->rid;
 	bool fence_started = false;
 	u64 fenced = 0;
@@ -567,7 +545,7 @@ int scoutfs_quorum_fence_leaders(struct super_block *sb, struct scoutfs_quorum_c
 	BUILD_BUG_ON(SCOUTFS_QUORUM_BLOCKS < SCOUTFS_QUORUM_MAX_SLOTS);
 
 	for (i = 0; i < SCOUTFS_QUORUM_MAX_SLOTS; i++) {
-		if (!quorum_slot_present(qconf, i))
+		if (!quorum_slot_present(super, i))
 			continue;
 
 		ret = read_quorum_block(sb, SCOUTFS_QUORUM_BLKNO + i, &blk, false);
@@ -600,11 +578,11 @@ int scoutfs_quorum_fence_leaders(struct super_block *sb, struct scoutfs_quorum_c
 				continue;
 
 			scoutfs_inc_counter(sb, quorum_fence_leader);
-			quorum_slot_sin(qconf, i, &sin);
+			scoutfs_quorum_slot_sin(super, i, &sin);
 			fence_rid = old[i][j].rid;
 
 			scoutfs_info(sb, "fencing previous leader "SCSBF" at term %llu in slot %u with address "SIN_FMT,
-				     SCSB_LEFR_ARGS(lefsid, fence_rid),
+				     SCSB_LEFR_ARGS(super->hdr.fsid, fence_rid),
 				     le64_to_cpu(old[i][j].term), i, SIN_ARG(&sin));
 			ret = scoutfs_fence_start(sb, le64_to_cpu(fence_rid), sin.sin_addr.s_addr,
 						  SCOUTFS_FENCE_QUORUM_BLOCK_LEADER);
@@ -615,9 +593,15 @@ int scoutfs_quorum_fence_leaders(struct super_block *sb, struct scoutfs_quorum_c
 	}
 
 out:
-	err = scoutfs_fence_wait_fenced(sb, msecs_to_jiffies(SCOUTFS_QUORUM_FENCE_TO_MS));
-	if (ret == 0)
-		ret = err;
+	if (fence_started) {
+		err = scoutfs_fence_wait_fenced(sb, msecs_to_jiffies(SCOUTFS_QUORUM_FENCE_TO_MS));
+		if (ret == 0)
+			ret = err;
+	} else {
+		err = scoutfs_quorum_fence_complete(sb, term);
+		if (ret == 0)
+			ret = err;
+	}
 
 	if (ret < 0)
 		scoutfs_inc_counter(sb, quorum_fence_error);
@@ -625,71 +609,6 @@ out:
 	return ret;
 }
 
-static void clear_hb_delay(struct quorum_info *qinf)
-{
-	int i;
-
-	spin_lock(&qinf->show_lock);
-	qinf->max_hb_delay = 0;
-	for (i = 0; i < HB_DELAY_NR; i++) {
-		qinf->hb_delay[i].recent = ns_to_ktime(0);
-		qinf->hb_delay[i].count = 0;
-	}
-	spin_unlock(&qinf->show_lock);
-}
-
-struct hb_recording {
-	ktime_t prev;
-	int count;
-};
-
-/*
- * Record long heartbeat delays.  We only record the delay between back
- * to back send attempts in the leader or back to back recv messages in
- * the followers.  The worker caller sets record_hb when their iteration
- * sent or received a heartbeat.  An iteration that does anything else
- * resets the tracking.
- */
-static void record_hb_delay(struct super_block *sb, struct quorum_info *qinf,
-			    struct hb_recording *hbr, bool record_hb, int role)
-{
-	bool log = false;
-	ktime_t now;
-	s64 s;
-
-	if (!record_hb) {
-		hbr->count = 0;
-		return;
-	}
-
-	now = ktime_get();
-
-	if (hbr->count < 2 && ++hbr->count < 2) {
-		hbr->prev = now;
-		return;
-	}
-
-	s = ktime_ms_delta(now, hbr->prev) / MSEC_PER_SEC;
-	hbr->prev = now;
-
-	if (s <= 0 || s >= HB_DELAY_NR)
-		return;
-
-	spin_lock(&qinf->show_lock);
-	if (qinf->max_hb_delay < s) {
-		qinf->max_hb_delay = s;
-		if (s >= 3)
-			log = true;
-	}
-	qinf->hb_delay[s].recent = now;
-	qinf->hb_delay[s].count++;
-	spin_unlock(&qinf->show_lock);
-
-	if (log)
-		scoutfs_info(sb, "longest quorum heartbeat %s delay of %lld sec",
-			     role == LEADER ? "send" : "recv", s);
-}
-
 /*
  * The main quorum task maintains its private status.  It seemed cleaner
  * to occasionally copy the status for showing in sysfs/debugfs files
@@ -708,37 +627,35 @@ static void update_show_status(struct quorum_info *qinf, struct quorum_status *q
 /*
  * The quorum work always runs in the background of quorum member
  * mounts.  It's responsible for starting and stopping the server if
- * it's elected leader.  While it's leader it sends heartbeats to
- * suppress other quorum work from standing for election.
+ * it's elected leader, and the server can call back into it to let it
+ * know that it has shut itself down (perhaps due to error) so that the
+ * work should stop sending heartbeats.
  */
 static void scoutfs_quorum_worker(struct work_struct *work)
 {
 	struct quorum_info *qinf = container_of(work, struct quorum_info, work);
-	struct scoutfs_mount_options opts;
 	struct super_block *sb = qinf->sb;
 	struct sockaddr_in unused;
 	struct quorum_host_msg msg;
-	struct quorum_status qst = {0,};
-	struct hb_recording hbr = {{0,},};
-	bool record_hb;
+	struct quorum_status qst;
 	int ret;
 	int err;
 
 	/* recording votes from slots as native single word bitmap */
 	BUILD_BUG_ON(SCOUTFS_QUORUM_MAX_SLOTS > BITS_PER_LONG);
 
-	scoutfs_options_read(sb, &opts);
-
 	/* start out as a follower */
 	qst.role = FOLLOWER;
+	qst.term = 0;
 	qst.vote_for = -1;
+	qst.vote_bits = 0;
 
 	/* read our starting term from greatest in all events in all slots */
 	read_greatest_term(sb, &qst.term);
 
 	/* see if there's a server to chose heartbeat or election timeout */
 	if (scoutfs_quorum_server_sin(sb, &unused) == 0)
-		qst.timeout = heartbeat_timeout(&opts);
+		qst.timeout = heartbeat_timeout();
 	else
 		qst.timeout = election_timeout();
 
@@ -762,14 +679,25 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 			ret = 0;
 		}
 
-		scoutfs_options_read(sb, &opts);
-		record_hb = false;
-
 		/* ignore messages from older terms */
 		if (msg.type != SCOUTFS_QUORUM_MSG_INVALID &&
 		    msg.term < qst.term)
 			msg.type = SCOUTFS_QUORUM_MSG_INVALID;
 
+		/* if the server has shutdown we become follower */
+		if (!test_bit(QINF_FLAG_SERVER, &qinf->flags) &&
+		    qst.role == LEADER) {
+			qst.role = FOLLOWER;
+			qst.vote_for = -1;
+			qst.vote_bits = 0;
+			qst.timeout = election_timeout();
+			scoutfs_inc_counter(sb, quorum_server_shutdown);
+
+			send_msg_others(sb, SCOUTFS_QUORUM_MSG_RESIGNATION,
+					qst.term);
+			scoutfs_inc_counter(sb, quorum_send_resignation);
+		}
+
 		trace_scoutfs_quorum_loop(sb, qst.role, qst.term, qst.vote_for,
 					  qst.vote_bits,
 					  ktime_to_timespec64(qst.timeout));
@@ -780,7 +708,8 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 			if (qst.role == LEADER) {
 				scoutfs_warn(sb, "saw msg type %u from %u for term %llu while leader in term %llu, shutting down server.",
 					     msg.type, msg.from, msg.term, qst.term);
-				clear_hb_delay(qinf);
+				update_show_status(qinf, &qst);
+				scoutfs_server_stop(sb);
 			}
 			qst.role = FOLLOWER;
 			qst.term = msg.term;
@@ -789,7 +718,7 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 			scoutfs_inc_counter(sb, quorum_term_follower);
 
 			if (msg.type == SCOUTFS_QUORUM_MSG_HEARTBEAT)
-				qst.timeout = heartbeat_timeout(&opts);
+				qst.timeout = heartbeat_timeout();
 			else
 				qst.timeout = election_timeout();
 
@@ -799,31 +728,9 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 				goto out;
 		}
 
-		/* receiving heartbeats extends timeout, delaying elections */
-		if (msg.type == SCOUTFS_QUORUM_MSG_HEARTBEAT) {
-			qst.timeout = heartbeat_timeout(&opts);
-			scoutfs_inc_counter(sb, quorum_recv_heartbeat);
-			record_hb = true;
-		}
-
-		/* receiving a resignation from server starts election */
-		if (msg.type == SCOUTFS_QUORUM_MSG_RESIGNATION &&
-		    qst.role == FOLLOWER &&
-		    msg.term == qst.term) {
-			qst.timeout = election_timeout();
-			scoutfs_inc_counter(sb, quorum_recv_resignation);
-		}
-
 		/* followers and candidates start new election on timeout */
 		if (qst.role != LEADER &&
 		    ktime_after(ktime_get(), qst.timeout)) {
-			/* .. but only if their server has stopped */
-			if (!scoutfs_server_is_down(sb)) {
-				qst.timeout = election_timeout();
-				scoutfs_inc_counter(sb, quorum_candidate_server_stopping);
-				continue;
-			}
-
 			qst.role = CANDIDATE;
 			qst.term++;
 			qst.vote_for = -1;
@@ -866,85 +773,53 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 			qst.timeout = heartbeat_interval();
 
 			update_show_status(qinf, &qst);
-			clear_hb_delay(qinf);
 
 			/* record that we've been elected before starting up server */
 			ret = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_ELECT, qst.term, true);
 			if (ret < 0)
 				goto out;
 
-			qst.server_start_term = qst.term;
-			qst.server_event = SCOUTFS_QUORUM_EVENT_ELECT;
-			scoutfs_server_start(sb, &qinf->qconf, qst.term);
-		}
-
-		/*
-		 * This leader's server is up, having finished fencing
-		 * previous leaders.  We update the fence event with the
-		 * current term to let future leaders know that previous
-		 * servers have been fenced.
-		 */
-		if (qst.role == LEADER && qst.server_event != SCOUTFS_QUORUM_EVENT_FENCE &&
-		    scoutfs_server_is_up(sb)) {
-			ret = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_FENCE, qst.term, true);
-			if (ret < 0)
-				goto out;
-			qst.server_event = SCOUTFS_QUORUM_EVENT_FENCE;
-		}
-
-		/*
-		 * Stop a running server if we're no longer leader in
-		 * its term.
-		 */
-		if (!(qst.role == LEADER && qst.term == qst.server_start_term) &&
-		    scoutfs_server_is_running(sb)) {
+			/* make very sure server is fully shut down */
 			scoutfs_server_stop(sb);
-		}
+			/* set server bit before server shutdown could clear */
+			set_bit(QINF_FLAG_SERVER, &qinf->flags);
 
-		/*
-		 * A previously running server has stopped.  The quorum
-		 * protocol might have shut it down by changing roles or
-		 * it might have stopped on its own, perhaps on errors.
-		 * If we're still a leader then we become a follower and
-		 * send resignations to encourage the next election.
-		 * Always update the _STOP event to stop connections and
-		 * fencing.
-		 */
-		if (qst.server_start_term > 0 && scoutfs_server_is_down(sb)) {
-			if (qst.role == LEADER) {
-				qst.role = FOLLOWER;
-				qst.vote_for = -1;
-				qst.vote_bits = 0;
-				qst.timeout = election_timeout();
-				scoutfs_inc_counter(sb, quorum_server_shutdown);
-
-				send_msg_others(sb, SCOUTFS_QUORUM_MSG_RESIGNATION,
-						qst.server_start_term);
-				scoutfs_inc_counter(sb, quorum_send_resignation);
-				clear_hb_delay(qinf);
+			ret = scoutfs_server_start(sb, qst.term);
+			if (ret < 0) {
+				clear_bit(QINF_FLAG_SERVER, &qinf->flags);
+				/* store our increased term */
+				err = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_STOP, qst.term,
+							  true);
+				if (err < 0) {
+					ret = err;
+					goto out;
+				}
+				ret = 0;
+				continue;
 			}
-
-			ret = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_STOP,
-						  qst.server_start_term, true);
-			if (ret < 0)
-				goto out;
-
-			qst.server_start_term = 0;
 		}
 
 		/* leaders regularly send heartbeats to delay elections */
 		if (qst.role == LEADER &&
 		    ktime_after(ktime_get(), qst.timeout)) {
-			ret = send_msg_others(sb, SCOUTFS_QUORUM_MSG_HEARTBEAT, qst.term);
-			if (ret > 0) {
-				scoutfs_add_counter(sb, quorum_send_heartbeat_dropped, ret);
-				ret = 0;
-			}
-
+			send_msg_others(sb, SCOUTFS_QUORUM_MSG_HEARTBEAT,
+					qst.term);
 			qst.timeout = heartbeat_interval();
 			scoutfs_inc_counter(sb, quorum_send_heartbeat);
-			record_hb = true;
+		}
 
+		/* receiving heartbeats extends timeout, delaying elections */
+		if (msg.type == SCOUTFS_QUORUM_MSG_HEARTBEAT) {
+			qst.timeout = heartbeat_timeout();
+			scoutfs_inc_counter(sb, quorum_recv_heartbeat);
+		}
+
+		/* receiving a resignation from server starts election */
+		if (msg.type == SCOUTFS_QUORUM_MSG_RESIGNATION &&
+		    qst.role == FOLLOWER &&
+		    msg.term == qst.term) {
+			qst.timeout = election_timeout();
+			scoutfs_inc_counter(sb, quorum_recv_resignation);
 		}
 
 		/* followers vote once per term */
@@ -956,23 +831,16 @@ static void scoutfs_quorum_worker(struct work_struct *work)
 				    msg.from);
 			scoutfs_inc_counter(sb, quorum_send_vote);
 		}
-
-		record_hb_delay(sb, qinf, &hbr, record_hb, qst.role);
 	}
 
 	update_show_status(qinf, &qst);
 
 	/* always try to stop a running server as we stop */
-	if (scoutfs_server_is_running(sb)) {
-		scoutfs_server_stop_wait(sb);
-		send_msg_others(sb, SCOUTFS_QUORUM_MSG_RESIGNATION, qst.term);
-
-		if (qst.server_start_term > 0) {
-			err = update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_STOP,
-						  qst.server_start_term, true);
-			if (err < 0 && ret == 0)
-				ret = err;
-		}
+	if (test_bit(QINF_FLAG_SERVER, &qinf->flags)) {
+		scoutfs_server_stop(sb);
+		scoutfs_fence_stop(sb);
+		send_msg_others(sb, SCOUTFS_QUORUM_MSG_RESIGNATION,
+				qst.term);
 	}
 
 	/* record that this slot no longer has an active quorum */
@@ -984,6 +852,21 @@ out:
 	}
 }
 
+/*
+ * The calling server has shutdown and is no longer using shared
+ * resources.  Clear the bit so that we stop sending heartbeats and
+ * allow the next server to be elected.  Update the stop event so that
+ * it won't be considered available by clients or fenced by the next
+ * leader.
+ */
+void scoutfs_quorum_server_shutdown(struct super_block *sb, u64 term)
+{
+	DECLARE_QUORUM_INFO(sb, qinf);
+
+	clear_bit(QINF_FLAG_SERVER, &qinf->flags);
+	update_quorum_block(sb, SCOUTFS_QUORUM_EVENT_STOP, term, true);
+}
+
 /*
  * Clients read quorum blocks looking for the leader with a server whose
  * address it can try and connect to.
@@ -995,25 +878,16 @@ out:
  */
 int scoutfs_quorum_server_sin(struct super_block *sb, struct sockaddr_in *sin)
 {
-	struct scoutfs_super_block *super = NULL;
+	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &sbi->super;
 	struct scoutfs_quorum_block blk;
 	u64 elect_term;
 	u64 term = 0;
 	int ret = 0;
 	int i;
 
-	super = kmalloc(sizeof(struct scoutfs_super_block), GFP_NOFS);
-	if (!super) {
-		ret = -ENOMEM;
-		goto out;
-	}
-
-	ret = scoutfs_read_super(sb, super);
-	if (ret)
-		goto out;
-
 	for (i = 0; i < SCOUTFS_QUORUM_MAX_SLOTS; i++) {
-		if (!quorum_slot_present(&super->qconf, i))
+		if (!quorum_slot_present(super, i))
 			continue;
 
 		ret = read_quorum_block(sb, SCOUTFS_QUORUM_BLKNO + i, &blk, false);
@@ -1027,7 +901,7 @@ int scoutfs_quorum_server_sin(struct super_block *sb, struct sockaddr_in *sin)
 		if (elect_term > term &&
 		    elect_term > le64_to_cpu(blk.events[SCOUTFS_QUORUM_EVENT_STOP].term)) {
 			term = elect_term;
-			scoutfs_quorum_slot_sin(&super->qconf, i, sin);
+			scoutfs_quorum_slot_sin(super, i, sin);
 			continue;
 		}
 	}
@@ -1036,7 +910,6 @@ int scoutfs_quorum_server_sin(struct super_block *sb, struct sockaddr_in *sin)
 		ret = -ENOENT;
 
 out:
-	kfree(super);
 	return ret;
 }
 
@@ -1052,9 +925,12 @@ u8 scoutfs_quorum_votes_needed(struct super_block *sb)
 	return qinf->votes_needed;
 }
 
-void scoutfs_quorum_slot_sin(struct scoutfs_quorum_config *qconf, int i, struct sockaddr_in *sin)
+void scoutfs_quorum_slot_sin(struct scoutfs_super_block *super, int i,
+			     struct sockaddr_in *sin)
 {
-	return quorum_slot_sin(qconf, i, sin);
+	BUG_ON(i < 0 || i >= SCOUTFS_QUORUM_MAX_SLOTS);
+
+	scoutfs_addr_to_sin(sin, &super->qconf.slots[i].addr);
 }
 
 static char *role_str(int role)
@@ -1094,11 +970,9 @@ static ssize_t status_show(struct kobject *kobj, struct kobj_attribute *attr,
 {
 	DECLARE_QUORUM_INFO_KOBJ(kobj, qinf);
 	struct quorum_status qst;
-	struct count_recent cr;
 	struct last_msg last;
 	struct timespec64 ts;
 	const ktime_t now = ktime_get();
-	unsigned long ul;
 	size_t size;
 	int ret;
 	int i;
@@ -1114,8 +988,6 @@ static ssize_t status_show(struct kobject *kobj, struct kobj_attribute *attr,
 		     qinf->our_quorum_slot_nr);
 	snprintf_ret(buf, size, &ret, "term %llu\n",
 		     qst.term);
-	snprintf_ret(buf, size, &ret, "server_start_term %llu\n", qst.server_start_term);
-	snprintf_ret(buf, size, &ret, "server_event %d\n", qst.server_event);
 	snprintf_ret(buf, size, &ret, "role %d (%s)\n",
 		     qst.role, role_str(qst.role));
 	snprintf_ret(buf, size, &ret, "vote_for %d\n",
@@ -1156,26 +1028,6 @@ static ssize_t status_show(struct kobject *kobj, struct kobj_attribute *attr,
 			     (s64)ts.tv_sec, (int)ts.tv_nsec);
 	}
 
-	spin_lock(&qinf->show_lock);
-	ul = qinf->max_hb_delay;
-	spin_unlock(&qinf->show_lock);
-	if (ul)
-		snprintf_ret(buf, size, &ret, "HB Delay(s)      Count  Secs Since\n");
-
-	for (i = 1; i <= ul && i < HB_DELAY_NR; i++) {
-		spin_lock(&qinf->show_lock);
-		cr = qinf->hb_delay[i];
-		spin_unlock(&qinf->show_lock);
-
-		if (cr.count == 0)
-			continue;
-
-		ts = ktime_to_timespec64(ktime_sub(now, cr.recent));
-		snprintf_ret(buf, size, &ret,
-			     "%11u  %9llu  %lld.%09u\n",
-			     i, cr.count, (s64)ts.tv_sec, (int)ts.tv_nsec);
-	}
-
 	return ret;
 }
 SCOUTFS_ATTR_RO(status);
@@ -1207,10 +1059,11 @@ static inline bool valid_ipv4_port(__be16 port)
 	return port != 0 && be16_to_cpu(port) != U16_MAX;
 }
 
-static int verify_quorum_slots(struct super_block *sb, struct quorum_info *qinf,
-			       struct scoutfs_quorum_config *qconf)
+static int verify_quorum_slots(struct super_block *sb)
 {
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 	char slots[(SCOUTFS_QUORUM_MAX_SLOTS * 3) + 1];
+	DECLARE_QUORUM_INFO(sb, qinf);
 	struct sockaddr_in other;
 	struct sockaddr_in sin;
 	int found = 0;
@@ -1220,10 +1073,10 @@ static int verify_quorum_slots(struct super_block *sb, struct quorum_info *qinf,
 
 
 	for (i = 0; i < SCOUTFS_QUORUM_MAX_SLOTS; i++) {
-		if (!quorum_slot_present(qconf, i))
+		if (!quorum_slot_present(super, i))
 			continue;
 
-		scoutfs_quorum_slot_sin(qconf, i, &sin);
+		scoutfs_quorum_slot_sin(super, i, &sin);
 
 		if (!valid_ipv4_unicast(sin.sin_addr.s_addr)) {
 			scoutfs_err(sb, "quorum slot #%d has invalid ipv4 unicast address: "SIN_FMT,
@@ -1238,10 +1091,10 @@ static int verify_quorum_slots(struct super_block *sb, struct quorum_info *qinf,
 		}
 
 		for (j = i + 1; j < SCOUTFS_QUORUM_MAX_SLOTS; j++) {
-			if (!quorum_slot_present(qconf, j))
+			if (!quorum_slot_present(super, j))
 				continue;
 
-			scoutfs_quorum_slot_sin(qconf, j, &other);
+			scoutfs_quorum_slot_sin(super, j, &other);
 
 			if (sin.sin_addr.s_addr == other.sin_addr.s_addr &&
 			    sin.sin_port == other.sin_port) {
@@ -1259,11 +1112,11 @@ static int verify_quorum_slots(struct super_block *sb, struct quorum_info *qinf,
 		return -EINVAL;
 	}
 
-	if (!quorum_slot_present(qconf, qinf->our_quorum_slot_nr)) {
+	if (!quorum_slot_present(super, qinf->our_quorum_slot_nr)) {
 		char *str = slots;
 		*str = '\0';
 		for (i = 0; i < SCOUTFS_QUORUM_MAX_SLOTS; i++) {
-			if (quorum_slot_present(qconf, i)) {
+			if (quorum_slot_present(super, i)) {
 				ret = snprintf(str, &slots[ARRAY_SIZE(slots)] - str, "%c%u",
 					       str == slots ? ' ' : ',', i);
 				if (ret < 2 || ret > 3) {
@@ -1287,22 +1140,16 @@ static int verify_quorum_slots(struct super_block *sb, struct quorum_info *qinf,
 	else
 		qinf->votes_needed = (found / 2) + 1;
 
-	qinf->qconf = *qconf;
-
 	return 0;
 }
 
 /*
  * Once this schedules the quorum worker it can be elected leader and
- * start the server, possibly before this returns.  The quorum agent
- * would be responsible for tracking the quorum config in the super
- * block if it changes.  Until then uses a static config that it reads
- * during setup.
+ * start the server, possibly before this returns.
  */
 int scoutfs_quorum_setup(struct super_block *sb)
 {
 	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
-	struct scoutfs_super_block *super = NULL;
 	struct scoutfs_mount_options opts;
 	struct quorum_info *qinf;
 	int ret;
@@ -1312,14 +1159,7 @@ int scoutfs_quorum_setup(struct super_block *sb)
 		return 0;
 
 	qinf = kzalloc(sizeof(struct quorum_info), GFP_KERNEL);
-	super = kmalloc(sizeof(struct scoutfs_super_block), GFP_KERNEL);
-	if (qinf)
-		qinf->hb_delay = __vmalloc(HB_DELAY_NR * sizeof(struct count_recent),
-					   GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL);
-	if (!qinf || !super || !qinf->hb_delay) {
-		if (qinf)
-			vfree(qinf->hb_delay);
-		kfree(qinf);
+	if (!qinf) {
 		ret = -ENOMEM;
 		goto out;
 	}
@@ -1333,20 +1173,7 @@ int scoutfs_quorum_setup(struct super_block *sb)
 	sbi->quorum_info = qinf;
 	qinf->sb = sb;
 
-	/* a high priority single threaded context without mem reclaim */
-	qinf->workq = alloc_workqueue("scoutfs_quorum_work",
-				       WQ_NON_REENTRANT | WQ_UNBOUND |
-				       WQ_HIGHPRI, 1);
-	if (!qinf->workq) {
-		ret = -ENOMEM;
-		goto out;
-	}
-
-	ret = scoutfs_read_super(sb, super);
-	if (ret < 0)
-		goto out;
-
-	ret = verify_quorum_slots(sb, qinf, &super->qconf);
+	ret = verify_quorum_slots(sb);
 	if (ret < 0)
 		goto out;
 
@@ -1360,13 +1187,12 @@ int scoutfs_quorum_setup(struct super_block *sb)
 	if (ret < 0)
 		goto out;
 
-	queue_work(qinf->workq, &qinf->work);
+	schedule_work(&qinf->work);
 
 out:
 	if (ret)
 		scoutfs_quorum_destroy(sb);
 
-	kfree(super);
 	return ret;
 }
 
@@ -1390,14 +1216,10 @@ void scoutfs_quorum_destroy(struct super_block *sb)
 		qinf->shutdown = true;
 		flush_work(&qinf->work);
 
-		if (qinf->workq)
-			destroy_workqueue(qinf->workq);
-
 		scoutfs_sysfs_destroy_attrs(sb, &qinf->ssa);
 		if (qinf->sock)
 			sock_release(qinf->sock);
 
-		vfree(qinf->hb_delay);
 		kfree(qinf);
 		sbi->quorum_info = NULL;
 	}

kmod/src/quorum.h

@@ -2,13 +2,14 @@
 #define _SCOUTFS_QUORUM_H_
 
 int scoutfs_quorum_server_sin(struct super_block *sb, struct sockaddr_in *sin);
+void scoutfs_quorum_server_shutdown(struct super_block *sb, u64 term);
 
 u8 scoutfs_quorum_votes_needed(struct super_block *sb);
-void scoutfs_quorum_slot_sin(struct scoutfs_quorum_config *qconf, int i,
+void scoutfs_quorum_slot_sin(struct scoutfs_super_block *super, int i,
 			     struct sockaddr_in *sin);
 
-int scoutfs_quorum_fence_leaders(struct super_block *sb, struct scoutfs_quorum_config *qconf,
-				 u64 term);
+int scoutfs_quorum_fence_leaders(struct super_block *sb, u64 term);
+int scoutfs_quorum_fence_complete(struct super_block *sb, u64 term);
 
 int scoutfs_quorum_setup(struct super_block *sb);
 void scoutfs_quorum_shutdown(struct super_block *sb);

kmod/src/scoutfs_trace.h

@@ -691,16 +691,16 @@ TRACE_EVENT(scoutfs_evict_inode,
 
 TRACE_EVENT(scoutfs_drop_inode,
 	TP_PROTO(struct super_block *sb, __u64 ino, unsigned int nlink,
-		 unsigned int unhashed, bool lock_covered),
+		 unsigned int unhashed, bool drop_invalidated),
 
-	TP_ARGS(sb, ino, nlink, unhashed, lock_covered),
+	TP_ARGS(sb, ino, nlink, unhashed, drop_invalidated),
 
 	TP_STRUCT__entry(
 		SCSB_TRACE_FIELDS
 		__field(__u64, ino)
 		__field(unsigned int, nlink)
 		__field(unsigned int, unhashed)
-		__field(unsigned int, lock_covered)
+		__field(unsigned int, drop_invalidated)
 	),
 
 	TP_fast_assign(
@@ -708,12 +708,12 @@ TRACE_EVENT(scoutfs_drop_inode,
 		__entry->ino = ino;
 		__entry->nlink = nlink;
 		__entry->unhashed = unhashed;
-		__entry->lock_covered = !!lock_covered;
+		__entry->drop_invalidated = !!drop_invalidated;
 	),
 
-	TP_printk(SCSBF" ino %llu nlink %u unhashed %d lock_covered %u", SCSB_TRACE_ARGS,
+	TP_printk(SCSBF" ino %llu nlink %u unhashed %d drop_invalidated %u", SCSB_TRACE_ARGS,
 		  __entry->ino, __entry->nlink, __entry->unhashed,
-		  __entry->lock_covered)
+		  __entry->drop_invalidated)
 );
 
 TRACE_EVENT(scoutfs_inode_walk_writeback,
@@ -1417,71 +1417,42 @@ TRACE_EVENT(scoutfs_rename,
 );
 
 TRACE_EVENT(scoutfs_d_revalidate,
-	TP_PROTO(struct super_block *sb, struct dentry *dentry, int flags, u64 dir_ino, int ret),
+	TP_PROTO(struct super_block *sb,
+		 struct dentry *dentry, int flags, struct dentry *parent,
+		 bool is_covered, int ret),
 
-	TP_ARGS(sb, dentry, flags, dir_ino, ret),
+	TP_ARGS(sb, dentry, flags, parent, is_covered, ret),
 
 	TP_STRUCT__entry(
 		SCSB_TRACE_FIELDS
-		__field(void *, dentry)
 		__string(name, dentry->d_name.name)
 		__field(__u64, ino)
-		__field(__u64, dir_ino)
+		__field(__u64, parent_ino)
 		__field(int, flags)
 		__field(int, is_root)
+		__field(int, is_covered)
 		__field(int, ret)
 	),
 
 	TP_fast_assign(
 		SCSB_TRACE_ASSIGN(sb);
-		__entry->dentry = dentry;
 		__assign_str(name, dentry->d_name.name)
-		__entry->ino = dentry->d_inode ? scoutfs_ino(dentry->d_inode) : 0;
-		__entry->dir_ino = dir_ino;
+		__entry->ino = dentry->d_inode ?
+			       scoutfs_ino(dentry->d_inode) : 0;
+		__entry->parent_ino = parent->d_inode ?
+			       scoutfs_ino(parent->d_inode) : 0;
 		__entry->flags = flags;
 		__entry->is_root = IS_ROOT(dentry);
+		__entry->is_covered = is_covered;
 		__entry->ret = ret;
 	),
 
-	TP_printk(SCSBF" dentry %p name %s ino %llu dir_ino %llu flags 0x%x s_root %u ret %d",
-		  SCSB_TRACE_ARGS, __entry->dentry, __get_str(name), __entry->ino, __entry->dir_ino,
-		  __entry->flags, __entry->is_root, __entry->ret)
-);
-
-TRACE_EVENT(scoutfs_validate_dentry,
-	TP_PROTO(struct super_block *sb, struct dentry *dentry, u64 dir_ino, u64 dentry_ino,
-		 u64 dent_ino, u64 refresh_gen, int ret),
-
-	TP_ARGS(sb, dentry, dir_ino, dentry_ino, dent_ino, refresh_gen, ret),
-
-	TP_STRUCT__entry(
-		SCSB_TRACE_FIELDS
-		__field(void *, dentry)
-		__field(__u64, dir_ino)
-		__string(name, dentry->d_name.name)
-		__field(__u64, dentry_ino)
-		__field(__u64, dent_ino)
-		__field(__u64, fsdata_gen)
-		__field(__u64, refresh_gen)
-		__field(int, ret)
-	),
-
-	TP_fast_assign(
-		SCSB_TRACE_ASSIGN(sb);
-		__entry->dentry = dentry;
-		__entry->dir_ino = dir_ino;
-		__assign_str(name, dentry->d_name.name)
-		__entry->dentry_ino = dentry_ino;
-		__entry->dent_ino = dent_ino;
-		__entry->fsdata_gen = (unsigned long long)dentry->d_fsdata;
-		__entry->refresh_gen = refresh_gen;
-		__entry->ret = ret;
-	),
-
-	TP_printk(SCSBF" dentry %p dir %llu name %s dentry_ino %llu dent_ino %llu fsdata_gen %llu refresh_gen %llu ret %d",
-		  SCSB_TRACE_ARGS, __entry->dentry, __entry->dir_ino, __get_str(name),
-		  __entry->dentry_ino, __entry->dent_ino, __entry->fsdata_gen,
-		  __entry->refresh_gen, __entry->ret)
+	TP_printk(SCSBF" name %s ino %llu parent_ino %llu flags 0x%x s_root %u is_covered %u ret %d",
+		  SCSB_TRACE_ARGS, __get_str(name), __entry->ino,
+		  __entry->parent_ino, __entry->flags,
+		  __entry->is_root,
+		  __entry->is_covered,
+		  __entry->ret)
 );
 
 DECLARE_EVENT_CLASS(scoutfs_super_lifecycle_class,
@@ -1872,53 +1843,6 @@ DEFINE_EVENT(scoutfs_server_client_count_class, scoutfs_server_client_down,
 	TP_ARGS(sb, rid, nr_clients)
 );
 
-DECLARE_EVENT_CLASS(scoutfs_server_commit_users_class,
-        TP_PROTO(struct super_block *sb, int holding, int applying, int nr_holders,
-		 u32 avail_before, u32 freed_before, int exceeded),
-        TP_ARGS(sb, holding, applying, nr_holders, avail_before, freed_before, exceeded),
-        TP_STRUCT__entry(
-		SCSB_TRACE_FIELDS
-		__field(int, holding)
-		__field(int, applying)
-		__field(int, nr_holders)
-		__field(__u32, avail_before)
-		__field(__u32, freed_before)
-		__field(int, exceeded)
-        ),
-        TP_fast_assign(
-		SCSB_TRACE_ASSIGN(sb);
-		__entry->holding = !!holding;
-		__entry->applying = !!applying;
-		__entry->nr_holders = nr_holders;
-		__entry->avail_before = avail_before;
-		__entry->freed_before = freed_before;
-		__entry->exceeded = !!exceeded;
-        ),
-	TP_printk(SCSBF" holding %u applying %u nr %u avail_before %u freed_before %u exceeded %u",
-		  SCSB_TRACE_ARGS, __entry->holding, __entry->applying, __entry->nr_holders,
-		  __entry->avail_before, __entry->freed_before, __entry->exceeded)
-);
-DEFINE_EVENT(scoutfs_server_commit_users_class, scoutfs_server_commit_hold,
-        TP_PROTO(struct super_block *sb, int holding, int applying, int nr_holders,
-		 u32 avail_before, u32 freed_before, int exceeded),
-        TP_ARGS(sb, holding, applying, nr_holders, avail_before, freed_before, exceeded)
-);
-DEFINE_EVENT(scoutfs_server_commit_users_class, scoutfs_server_commit_apply,
-        TP_PROTO(struct super_block *sb, int holding, int applying, int nr_holders,
-		 u32 avail_before, u32 freed_before, int exceeded),
-        TP_ARGS(sb, holding, applying, nr_holders, avail_before, freed_before, exceeded)
-);
-DEFINE_EVENT(scoutfs_server_commit_users_class, scoutfs_server_commit_start,
-        TP_PROTO(struct super_block *sb, int holding, int applying, int nr_holders,
-		 u32 avail_before, u32 freed_before, int exceeded),
-        TP_ARGS(sb, holding, applying, nr_holders, avail_before, freed_before, exceeded)
-);
-DEFINE_EVENT(scoutfs_server_commit_users_class, scoutfs_server_commit_end,
-        TP_PROTO(struct super_block *sb, int holding, int applying, int nr_holders,
-		 u32 avail_before, u32 freed_before, int exceeded),
-        TP_ARGS(sb, holding, applying, nr_holders, avail_before, freed_before, exceeded)
-);
-
 #define slt_symbolic(mode)						\
 	__print_symbolic(mode,					\
 		{ SLT_CLIENT,		"client" },	\

kmod/src/server.h

@@ -64,6 +64,8 @@ int scoutfs_server_lock_response(struct super_block *sb, u64 rid, u64 id,
 				 struct scoutfs_net_lock *nl);
 int scoutfs_server_lock_recover_request(struct super_block *sb, u64 rid,
 					struct scoutfs_key *key);
+void scoutfs_server_hold_commit(struct super_block *sb);
+int scoutfs_server_apply_commit(struct super_block *sb, int err);
 void scoutfs_server_recov_finish(struct super_block *sb, u64 rid, int which);
 
 int scoutfs_server_send_omap_request(struct super_block *sb, u64 rid,
@@ -75,12 +77,9 @@ u64 scoutfs_server_seq(struct super_block *sb);
 u64 scoutfs_server_next_seq(struct super_block *sb);
 void scoutfs_server_set_seq_if_greater(struct super_block *sb, u64 seq);
 
-void scoutfs_server_start(struct super_block *sb, struct scoutfs_quorum_config *qconf, u64 term);
+int scoutfs_server_start(struct super_block *sb, u64 term);
+void scoutfs_server_abort(struct super_block *sb);
 void scoutfs_server_stop(struct super_block *sb);
-void scoutfs_server_stop_wait(struct super_block *sb);
-bool scoutfs_server_is_running(struct super_block *sb);
-bool scoutfs_server_is_up(struct super_block *sb);
-bool scoutfs_server_is_down(struct super_block *sb);
 
 int scoutfs_server_setup(struct super_block *sb);
 void scoutfs_server_destroy(struct super_block *sb);

kmod/src/srch.c

@@ -861,6 +861,7 @@ int scoutfs_srch_search_xattrs(struct super_block *sb,
 			       struct scoutfs_srch_rb_root *sroot,
 			       u64 hash, u64 ino, u64 last_ino, bool *done)
 {
+	struct scoutfs_net_roots prev_roots;
 	struct scoutfs_net_roots roots;
 	struct scoutfs_srch_entry start;
 	struct scoutfs_srch_entry end;
@@ -868,7 +869,6 @@ int scoutfs_srch_search_xattrs(struct super_block *sb,
 	struct scoutfs_log_trees lt;
 	struct scoutfs_srch_file sfl;
 	SCOUTFS_BTREE_ITEM_REF(iref);
-	DECLARE_SAVED_REFS(saved);
 	struct scoutfs_key key;
 	unsigned long limit = SRCH_LIMIT;
 	int ret;
@@ -877,6 +877,7 @@ int scoutfs_srch_search_xattrs(struct super_block *sb,
 
 	*done = false;
 	srch_init_rb_root(sroot);
+	memset(&prev_roots, 0, sizeof(prev_roots));
 
 	start.hash = cpu_to_le64(hash);
 	start.ino = cpu_to_le64(ino);
@@ -891,6 +892,7 @@ retry:
 	ret = scoutfs_client_get_roots(sb, &roots);
 	if (ret)
 		goto out;
+	memset(&roots.fs_root, 0, sizeof(roots.fs_root));
 
 	end = final;
 
@@ -966,10 +968,16 @@ retry:
 	*done = sre_cmp(&end, &final) == 0;
 	ret = 0;
 out:
-	ret = scoutfs_block_check_stale(sb, ret, &saved, &roots.srch_root.ref,
-					&roots.logs_root.ref);
-	if (ret == -ESTALE)
-		goto retry;
+	if (ret == -ESTALE) {
+		if (memcmp(&prev_roots, &roots, sizeof(roots)) == 0) {
+			scoutfs_inc_counter(sb, srch_search_stale_eio);
+			ret = -EIO;
+		} else {
+			scoutfs_inc_counter(sb, srch_search_stale_retry);
+			prev_roots = roots;
+			goto retry;
+		}
+	}
 
 	return ret;
 }
@@ -995,14 +1003,6 @@ int scoutfs_srch_rotate_log(struct super_block *sb,
 		      le64_to_cpu(sfl->ref.blkno), 0);
 	ret = scoutfs_btree_insert(sb, alloc, wri, root, &key,
 				   sfl, sizeof(*sfl));
-	/*
-	 * While it's fine to replay moving the client's logging srch
-	 * file to the core btree item, server commits should keep it
-	 * from happening.  So we'll warn if we see it happen.  This can
-	 * be removed eventually.
-	 */
-	if (WARN_ON_ONCE(ret == -EEXIST))
-		ret = 0;
 	if (ret == 0) {
 		memset(sfl, 0, sizeof(*sfl));
 		scoutfs_inc_counter(sb, srch_rotate_log);

kmod/src/super.c

@@ -47,7 +47,6 @@
 #include "omap.h"
 #include "volopt.h"
 #include "fence.h"
-#include "xattr.h"
 #include "scoutfs_trace.h"
 
 static struct dentry *scoutfs_debugfs_root;
@@ -461,8 +460,9 @@ static int scoutfs_read_supers(struct super_block *sb)
 		goto out;
 	}
 
-	sbi->fsid = le64_to_cpu(meta_super->hdr.fsid);
+
 	sbi->fmt_vers = le64_to_cpu(meta_super->fmt_vers);
+	sbi->super = *meta_super;
 out:
 	kfree(meta_super);
 	kfree(data_super);
@@ -482,11 +482,8 @@ static int scoutfs_fill_super(struct super_block *sb, void *data, int silent)
 	sb->s_magic = SCOUTFS_SUPER_MAGIC;
 	sb->s_maxbytes = MAX_LFS_FILESIZE;
 	sb->s_op = &scoutfs_super_ops;
-	sb->s_d_op = &scoutfs_dentry_ops;
 	sb->s_export_op = &scoutfs_export_ops;
-	sb->s_xattr = scoutfs_xattr_handlers;
-	sb->s_flags |= MS_I_VERSION | MS_POSIXACL;
-	sb->s_time_gran = 1;
+	sb->s_flags |= MS_I_VERSION;
 
 	/* btree blocks use long lived bh->b_data refs */
 	mapping_set_gfp_mask(sb->s_bdev->bd_inode->i_mapping, GFP_NOFS);
@@ -499,7 +496,7 @@ static int scoutfs_fill_super(struct super_block *sb, void *data, int silent)
 
 	ret = assign_random_id(sbi);
 	if (ret < 0)
-		goto out;
+		return ret;
 
 	spin_lock_init(&sbi->next_ino_lock);
 	spin_lock_init(&sbi->data_wait_root.lock);
@@ -508,7 +505,7 @@ static int scoutfs_fill_super(struct super_block *sb, void *data, int silent)
 	/* parse options early for use during setup */
 	ret = scoutfs_options_early_setup(sb, data);
 	if (ret < 0)
-		goto out;
+		return ret;
 	scoutfs_options_read(sb, &opts);
 
 	ret = sb_set_blocksize(sb, SCOUTFS_BLOCK_SM_SIZE);
@@ -631,6 +628,7 @@ MODULE_ALIAS_FS("scoutfs");
 static void teardown_module(void)
 {
 	debugfs_remove(scoutfs_debugfs_root);
+	scoutfs_dir_exit();
 	scoutfs_inode_exit();
 	scoutfs_sysfs_exit();
 }
@@ -668,6 +666,7 @@ static int __init scoutfs_module_init(void)
 		goto out;
 	}
 	ret = scoutfs_inode_init() ?:
+	      scoutfs_dir_init() ?:
 	      register_filesystem(&scoutfs_fs_type);
 out:
 	if (ret)

kmod/src/super.h

@@ -35,10 +35,11 @@ struct scoutfs_sb_info {
 	struct super_block *sb;
 
 	/* assigned once at the start of each mount, read-only */
-	u64 fsid;
 	u64 rid;
 	u64 fmt_vers;
 
+	struct scoutfs_super_block super;
+
 	struct block_device *meta_bdev;
 
 	spinlock_t next_ino_lock;
@@ -134,14 +135,14 @@ static inline bool scoutfs_unmounting(struct super_block *sb)
 	(int)(le64_to_cpu(fsid) >> SCSB_SHIFT),				    \
 	(int)(le64_to_cpu(rid) >> SCSB_SHIFT)
 #define SCSB_ARGS(sb)							    \
-	(int)(SCOUTFS_SB(sb)->fsid >> SCSB_SHIFT),			    \
+	(int)(le64_to_cpu(SCOUTFS_SB(sb)->super.hdr.fsid) >> SCSB_SHIFT),   \
 	(int)(SCOUTFS_SB(sb)->rid >> SCSB_SHIFT)
 #define SCSB_TRACE_FIELDS	\
 	__field(__u64, fsid)	\
 	__field(__u64, rid)
 #define SCSB_TRACE_ASSIGN(sb)						\
 	__entry->fsid = SCOUTFS_HAS_SBI(sb) ?				\
-		        SCOUTFS_SB(sb)->fsid : 0;			\
+			le64_to_cpu(SCOUTFS_SB(sb)->super.hdr.fsid) : 0;\
 	__entry->rid = SCOUTFS_HAS_SBI(sb) ?				\
 		       SCOUTFS_SB(sb)->rid : 0;
 #define SCSB_TRACE_ARGS				\

kmod/src/sysfs.c

@@ -37,15 +37,6 @@ struct attr_funcs {
 #define ATTR_FUNCS_RO(_name) \
 	static struct attr_funcs _name##_attr_funcs = __ATTR_RO(_name)
 
-static ssize_t data_device_maj_min_show(struct kobject *kobj, struct attribute *attr, char *buf)
-{
-	struct super_block *sb = KOBJ_TO_SB(kobj, sb_id_kobj);
-
-	return snprintf(buf, PAGE_SIZE, "%u:%u\n",
-			MAJOR(sb->s_bdev->bd_dev), MINOR(sb->s_bdev->bd_dev));
-}
-ATTR_FUNCS_RO(data_device_maj_min);
-
 static ssize_t format_version_show(struct kobject *kobj, struct attribute *attr,
 			 char *buf)
 {
@@ -60,9 +51,10 @@ static ssize_t fsid_show(struct kobject *kobj, struct attribute *attr,
 			 char *buf)
 {
 	struct super_block *sb = KOBJ_TO_SB(kobj, sb_id_kobj);
-	struct scoutfs_sb_info *sbi = SCOUTFS_SB(sb);
+	struct scoutfs_super_block *super = &SCOUTFS_SB(sb)->super;
 
-	return snprintf(buf, PAGE_SIZE, "%016llx\n", sbi->fsid);
+	return snprintf(buf, PAGE_SIZE, "%016llx\n",
+			le64_to_cpu(super->hdr.fsid));
 }
 ATTR_FUNCS_RO(fsid);
 
@@ -109,7 +101,6 @@ static ssize_t attr_funcs_show(struct kobject *kobj, struct attribute *attr,
 
 
 static struct attribute *sb_id_attrs[] = {
-	&data_device_maj_min_attr_funcs.attr,
 	&format_version_attr_funcs.attr,
 	&fsid_attr_funcs.attr,
 	&rid_attr_funcs.attr,
@@ -267,7 +258,7 @@ int __init scoutfs_sysfs_init(void)
 	return 0;
 }
 
-void scoutfs_sysfs_exit(void)
+void __exit scoutfs_sysfs_exit(void)
 {
 	if (scoutfs_kset)
 		kset_unregister(scoutfs_kset);

kmod/src/sysfs.h

@@ -53,6 +53,6 @@ int scoutfs_setup_sysfs(struct super_block *sb);
 void scoutfs_destroy_sysfs(struct super_block *sb);
 
 int __init scoutfs_sysfs_init(void);
-void scoutfs_sysfs_exit(void);
+void __exit scoutfs_sysfs_exit(void);
 
 #endif

kmod/src/xattr.c

@@ -15,7 +15,6 @@
 #include <linux/dcache.h>
 #include <linux/xattr.h>
 #include <linux/crc32c.h>
-#include <linux/posix_acl.h>
 
 #include "format.h"
 #include "inode.h"
@@ -27,7 +26,6 @@
 #include "xattr.h"
 #include "lock.h"
 #include "hash.h"
-#include "acl.h"
 #include "scoutfs_trace.h"
 
 /*
@@ -59,6 +57,12 @@ static u32 xattr_names_equal(const char *a_name, unsigned int a_len,
 	return a_len == b_len && memcmp(a_name, b_name, a_len) == 0;
 }
 
+static unsigned int xattr_full_bytes(struct scoutfs_xattr *xat)
+{
+	return offsetof(struct scoutfs_xattr,
+		        name[xat->name_len + le16_to_cpu(xat->val_len)]);
+}
+
 static unsigned int xattr_nr_parts(struct scoutfs_xattr *xat)
 {
 	return SCOUTFS_XATTR_NR_PARTS(xat->name_len,
@@ -81,6 +85,16 @@ static void init_xattr_key(struct scoutfs_key *key, u64 ino, u32 name_hash,
 #define SCOUTFS_XATTR_PREFIX		"scoutfs."
 #define SCOUTFS_XATTR_PREFIX_LEN	(sizeof(SCOUTFS_XATTR_PREFIX) - 1)
 
+static int unknown_prefix(const char *name)
+{
+	return strncmp(name, XATTR_USER_PREFIX, XATTR_USER_PREFIX_LEN) &&
+	       strncmp(name, XATTR_TRUSTED_PREFIX, XATTR_TRUSTED_PREFIX_LEN) &&
+	       strncmp(name, XATTR_SYSTEM_PREFIX, XATTR_SYSTEM_PREFIX_LEN) &&
+	       strncmp(name, XATTR_SECURITY_PREFIX, XATTR_SECURITY_PREFIX_LEN)&&
+	       strncmp(name, SCOUTFS_XATTR_PREFIX, SCOUTFS_XATTR_PREFIX_LEN);
+}
+
+
 #define HIDE_TAG	"hide."
 #define SRCH_TAG	"srch."
 #define TOTL_TAG	"totl."
@@ -123,29 +137,12 @@ int scoutfs_xattr_parse_tags(const char *name, unsigned int name_len,
 }
 
 /*
- * xattrs are stored in multiple items.   The first item is a
- * concatenation of an initial header, the name, and then as much of the
- * value as fits in the remainder of the first item.  This return the
- * size of the first item that'd store an xattr with the given name
- * length and value payload size.
- */
-static int first_item_bytes(int name_len, size_t size)
-{
-	if (WARN_ON_ONCE(name_len <= 0) ||
-	    WARN_ON_ONCE(name_len > SCOUTFS_XATTR_MAX_NAME_LEN))
-		return 0;
-
-	return min_t(int, sizeof(struct scoutfs_xattr) + name_len + size,
-			  SCOUTFS_XATTR_MAX_PART_SIZE);
-}
-
-/*
- * Find the next xattr, set the caller's key, and copy as much of the
- * first item into the callers buffer as we can.  Returns the number of
- * bytes copied which can include the header, name, and start of the
- * value from the first item.  The caller is responsible for comparing
- * their lengths, the header, and the returned length before safely
- * using the buffer.
+ * Find the next xattr and copy the key, xattr header, and as much of
+ * the name and value into the callers buffer as we can.  Returns the
+ * number of bytes copied which include the header, name, and value and
+ * can be limited by the xattr length or the callers buffer.  The caller
+ * is responsible for comparing their lengths, the header, and the
+ * returned length before safely using the xattr.
  *
  * If a name is provided then we'll iterate over items with a matching
  * name_hash until we find a matching name.  If we don't find a matching
@@ -157,17 +154,20 @@ static int first_item_bytes(int name_len, size_t size)
  * Returns -ENOENT if it didn't find a next item.
  */
 static int get_next_xattr(struct inode *inode, struct scoutfs_key *key,
-			  struct scoutfs_xattr *xat, unsigned int xat_bytes,
+			  struct scoutfs_xattr *xat, unsigned int bytes,
 			  const char *name, unsigned int name_len,
 			  u64 name_hash, u64 id, struct scoutfs_lock *lock)
 {
 	struct super_block *sb = inode->i_sb;
 	struct scoutfs_key last;
+	u8 last_part;
+	int total;
+	u8 part;
 	int ret;
 
 	/* need to be able to see the name we're looking for */
-	if (WARN_ON_ONCE(name_len > 0 &&
-			 xat_bytes < offsetof(struct scoutfs_xattr, name[name_len])))
+	if (WARN_ON_ONCE(name_len > 0 && bytes < offsetof(struct scoutfs_xattr,
+							  name[name_len])))
 		return -EINVAL;
 
 	if (name_len)
@@ -176,15 +176,26 @@ static int get_next_xattr(struct inode *inode, struct scoutfs_key *key,
 	init_xattr_key(key, scoutfs_ino(inode), name_hash, id);
 	init_xattr_key(&last, scoutfs_ino(inode), U32_MAX, U64_MAX);
 
+	last_part = 0;
+	part = 0;
+	total = 0;
+
 	for (;;) {
-		ret = scoutfs_item_next(sb, key, &last, xat, xat_bytes, lock);
-		if (ret < 0)
+		key->skx_part = part;
+		ret = scoutfs_item_next(sb, key, &last,
+					(void *)xat + total, bytes - total,
+					lock);
+		if (ret < 0) {
+			/* XXX corruption, ran out of parts */
+			if (ret == -ENOENT && part > 0)
+				ret = -EIO;
 			break;
+		}
 
 		trace_scoutfs_xattr_get_next_key(sb, key);
 
 		/* XXX corruption */
-		if (key->skx_part != 0) {
+		if (key->skx_part != part) {
 			ret = -EIO;
 			break;
 		}
@@ -194,7 +205,8 @@ static int get_next_xattr(struct inode *inode, struct scoutfs_key *key,
 		 * the first part and if the next xattr name fits in our
 		 * buffer then the item must have included it.
 		 */
-		if ((ret < sizeof(struct scoutfs_xattr) ||
+		if (part == 0 &&
+		    (ret < sizeof(struct scoutfs_xattr) ||
 		     (xat->name_len <= name_len &&
 		      ret < offsetof(struct scoutfs_xattr,
 				     name[xat->name_len])) ||
@@ -204,7 +216,7 @@ static int get_next_xattr(struct inode *inode, struct scoutfs_key *key,
 			break;
 		}
 
-		if (name_len > 0) {
+		if (part == 0 && name_len) {
 			/* ran out of names that could match */
 			if (le64_to_cpu(key->skx_name_hash) != name_hash) {
 				ret = -ENOENT;
@@ -212,126 +224,64 @@ static int get_next_xattr(struct inode *inode, struct scoutfs_key *key,
 			}
 
 			/* keep looking for our name */
-			if (!xattr_names_equal(name, name_len, xat->name, xat->name_len)) {
+			if (!xattr_names_equal(name, name_len,
+					       xat->name, xat->name_len)) {
+				part = 0;
 				le64_add_cpu(&key->skx_id, 1);
 				continue;
 			}
+
+			/* use the matching name we found */
+			last_part = xattr_nr_parts(xat) - 1;
 		}
 
-		/* found next name */
-		break;
+		total += ret;
+		if (total == bytes || part == last_part) {
+			/* copied as much as we could */
+			ret = total;
+			break;
+		}
+		part++;
 	}
 
 	return ret;
 }
 
-/*
- * The caller has already read and verified the xattr's first item.
- * Copy the value from the tail of the first item and from any future
- * items into the destination buffer.
- */
-static int copy_xattr_value(struct super_block *sb, struct scoutfs_key *xat_key,
-			    struct scoutfs_xattr *xat, int xat_bytes,
-			    char *buffer, size_t size,
-			    struct scoutfs_lock *lock)
-{
-	struct scoutfs_key key;
-	size_t copied = 0;
-	int val_tail;
-	int bytes;
-	int ret;
-	int i;
-
-	/* must have first item up to value */
-	if (WARN_ON_ONCE(xat_bytes < sizeof(struct scoutfs_xattr)) ||
-	    WARN_ON_ONCE(xat_bytes < offsetof(struct scoutfs_xattr, name[xat->name_len])))
-		return -EINVAL;
-
-	/* only ever copy up to the full value */
-	size = min_t(size_t, size, le16_to_cpu(xat->val_len));
-
-	/* must have full first item if caller needs value from second item */
-	val_tail = SCOUTFS_XATTR_MAX_PART_SIZE -
-		   offsetof(struct scoutfs_xattr, name[xat->name_len]);
-	if (WARN_ON_ONCE(size > val_tail && xat_bytes != SCOUTFS_XATTR_MAX_PART_SIZE))
-		return -EINVAL;
-
-	/* copy from tail of first item */
-	bytes = min_t(unsigned int, size, val_tail);
-	if (bytes > 0) {
-		memcpy(buffer, &xat->name[xat->name_len], bytes);
-		copied += bytes;
-	}
-
-	key = *xat_key;
-	for (i = 1; copied < size; i++) {
-		key.skx_part = i;
-		bytes = min_t(unsigned int, size - copied, SCOUTFS_XATTR_MAX_PART_SIZE);
-
-		ret = scoutfs_item_lookup(sb, &key, buffer + copied, bytes, lock);
-		if (ret >= 0 && ret != bytes)
-			ret = -EIO;
-		if (ret < 0)
-			return ret;
-
-		copied += ret;
-	}
-
-	return copied;
-}
-
-/*
- * The caller is working with items that are either in the allocated
- * first compound item or further items that are offsets into a value
- * buffer.  Give them a pointer and length of the start of the item.
- */
-static void xattr_item_part_buffer(void **buf, int *len, int part,
-				   struct scoutfs_xattr *xat, unsigned int xat_bytes,
-				   const char *value, size_t size)
-{
-	int off;
-
-	if (part == 0) {
-		*buf = xat;
-		*len = xat_bytes;
-	} else {
-		off = (part * SCOUTFS_XATTR_MAX_PART_SIZE) -
-		      offsetof(struct scoutfs_xattr, name[xat->name_len]);
-		BUG_ON(off >= size); /* calls limited by number of parts */
-		*buf = (void *)value + off;
-		*len = min_t(size_t, size - off, SCOUTFS_XATTR_MAX_PART_SIZE);
-	}
-}
-
 /*
  * Create all the items associated with the given xattr.  If this
  * returns an error it will have already cleaned up any items it created
  * before seeing the error.
  */
-static int create_xattr_items(struct inode *inode, u64 id, struct scoutfs_xattr *xat,
-			      int xat_bytes, const char *value, size_t size, u8 new_parts,
+static int create_xattr_items(struct inode *inode, u64 id,
+			      struct scoutfs_xattr *xat, unsigned int bytes,
 			      struct scoutfs_lock *lock)
 {
 	struct super_block *sb = inode->i_sb;
 	struct scoutfs_key key;
-	int ret = 0;
-	void *buf;
-	int len;
-	int i;
+	unsigned int part_bytes;
+	unsigned int total;
+	int ret;
 
 	init_xattr_key(&key, scoutfs_ino(inode),
 		       xattr_name_hash(xat->name, xat->name_len), id);
 
-	for (i = 0; i < new_parts; i++) {
-		key.skx_part = i;
-		xattr_item_part_buffer(&buf, &len, i, xat, xat_bytes, value, size);
+	total = 0;
+	ret = 0;
+	while (total < bytes) {
+		part_bytes = min_t(unsigned int, bytes - total,
+				   SCOUTFS_XATTR_MAX_PART_SIZE);
 
-		ret = scoutfs_item_create(sb, &key, buf, len, lock);
-		if (ret < 0) {
+		ret = scoutfs_item_create(sb, &key,
+					  (void *)xat + total, part_bytes,
+					  lock);
+		if (ret) {
 			while (key.skx_part-- > 0)
 				scoutfs_item_delete(sb, &key, lock);
 			break;
 		}
+
+		total += part_bytes;
+		key.skx_part++;
 	}
 
 	return ret;
@@ -379,20 +329,20 @@ out:
  * deleted items.
  */
 static int change_xattr_items(struct inode *inode, u64 id,
-			      struct scoutfs_xattr *xat, int xat_bytes,
-			      const char *value, size_t size,
-			      u8 new_parts, u8 old_parts, struct scoutfs_lock *lock)
+			      struct scoutfs_xattr *new_xat,
+			      unsigned int new_bytes, u8 new_parts,
+			      u8 old_parts, struct scoutfs_lock *lock)
 {
 	struct super_block *sb = inode->i_sb;
 	struct scoutfs_key key;
 	int last_created = -1;
-	void *buf;
-	int len;
+	int bytes;
+	int off;
 	int i;
 	int ret;
 
 	init_xattr_key(&key, scoutfs_ino(inode),
-		       xattr_name_hash(xat->name, xat->name_len), id);
+		       xattr_name_hash(new_xat->name, new_xat->name_len), id);
 
 	/* dirty existing old items */
 	for (i = 0; i < old_parts; i++) {
@@ -404,10 +354,13 @@ static int change_xattr_items(struct inode *inode, u64 id,
 
 	/* create any new items past the old */
 	for (i = old_parts; i < new_parts; i++) {
-		key.skx_part = i;
-		xattr_item_part_buffer(&buf, &len, i, xat, xat_bytes, value, size);
+		off = i * SCOUTFS_XATTR_MAX_PART_SIZE;
+		bytes = min_t(unsigned int, new_bytes - off,
+			      SCOUTFS_XATTR_MAX_PART_SIZE);
 
-		ret = scoutfs_item_create(sb, &key, buf, len, lock);
+		key.skx_part = i;
+		ret = scoutfs_item_create(sb, &key, (void *)new_xat + off,
+					  bytes, lock);
 		if (ret)
 			goto out;
 
@@ -416,10 +369,13 @@ static int change_xattr_items(struct inode *inode, u64 id,
 
 	/* update dirtied overlapping existing items, last partial first */
 	for (i = min(old_parts, new_parts) - 1; i >= 0; i--) {
-		key.skx_part = i;
-		xattr_item_part_buffer(&buf, &len, i, xat, xat_bytes, value, size);
+		off = i * SCOUTFS_XATTR_MAX_PART_SIZE;
+		bytes = min_t(unsigned int, new_bytes - off,
+			      SCOUTFS_XATTR_MAX_PART_SIZE);
 
-		ret = scoutfs_item_update(sb, &key, buf, len, lock);
+		key.skx_part = i;
+		ret = scoutfs_item_update(sb, &key, (void *)new_xat + off,
+					  bytes, lock);
 		/* only last partial can fail, then we unwind created */
 		if (ret < 0)
 			goto out;
@@ -447,69 +403,72 @@ out:
  * Copy the value for the given xattr name into the caller's buffer, if it
  * fits.  Return the bytes copied or -ERANGE if it doesn't fit.
  */
-int scoutfs_xattr_get_locked(struct inode *inode, const char *name, void *buffer, size_t size,
-			     struct scoutfs_lock *lck)
+ssize_t scoutfs_getxattr(struct dentry *dentry, const char *name, void *buffer,
+			 size_t size)
 {
+	struct inode *inode = dentry->d_inode;
 	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
 	struct super_block *sb = inode->i_sb;
 	struct scoutfs_xattr *xat = NULL;
+	struct scoutfs_lock *lck = NULL;
 	struct scoutfs_key key;
-	unsigned int xat_bytes;
+	unsigned int bytes;
 	size_t name_len;
 	int ret;
 
+	if (unknown_prefix(name))
+		return -EOPNOTSUPP;
+
 	name_len = strlen(name);
 	if (name_len > SCOUTFS_XATTR_MAX_NAME_LEN)
 		return -ENODATA;
 
-	xat_bytes = first_item_bytes(name_len, size);
-	xat = kmalloc(xat_bytes, GFP_NOFS);
+	/* only need enough for caller's name and value sizes */
+	bytes = sizeof(struct scoutfs_xattr) + name_len + size;
+	xat = __vmalloc(bytes, GFP_NOFS, PAGE_KERNEL);
 	if (!xat)
 		return -ENOMEM;
 
+	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_READ, 0, inode, &lck);
+	if (ret)
+		goto out;
+
 	down_read(&si->xattr_rwsem);
 
-	ret = get_next_xattr(inode, &key, xat, xat_bytes, name, name_len, 0, 0, lck);
+	ret = get_next_xattr(inode, &key, xat, bytes,
+			     name, name_len, 0, 0, lck);
+
+	up_read(&si->xattr_rwsem);
+	scoutfs_unlock(sb, lck, SCOUTFS_LOCK_READ);
 
 	if (ret < 0) {
 		if (ret == -ENOENT)
 			ret = -ENODATA;
-		goto unlock;
+		goto out;
 	}
 
 	/* the caller just wants to know the size */
 	if (size == 0) {
 		ret = le16_to_cpu(xat->val_len);
-		goto unlock;
+		goto out;
 	}
 
 	/* the caller's buffer wasn't big enough */
 	if (size < le16_to_cpu(xat->val_len)) {
 		ret = -ERANGE;
-		goto unlock;
+		goto out;
 	}
 
-	ret = copy_xattr_value(sb, &key, xat, xat_bytes, buffer, size, lck);
-unlock:
-	up_read(&si->xattr_rwsem);
-
-	kfree(xat);
-	return ret;
-}
-
-static int scoutfs_xattr_get(struct dentry *dentry, const char *name, void *buffer, size_t size)
-{
-	struct inode *inode = dentry->d_inode;
-	struct super_block *sb = inode->i_sb;
-	struct scoutfs_lock *lock = NULL;
-	int ret;
-
-	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_READ, 0, inode, &lock);
-	if (ret == 0) {
-		ret = scoutfs_xattr_get_locked(inode, name, buffer, size, lock);
-		scoutfs_unlock(sb, lock, SCOUTFS_LOCK_READ);
+	/* XXX corruption, the items didn't match the header */
+	if (ret < xattr_full_bytes(xat)) {
+		ret = -EIO;
+		goto out;
 	}
 
+	ret = le16_to_cpu(xat->val_len);
+	memcpy(buffer, &xat->name[xat->name_len], ret);
+out:
+	vfree(xat);
 	return ret;
 }
 
@@ -617,32 +576,29 @@ int scoutfs_xattr_combine_totl(void *dst, int dst_len, void *src, int src_len)
  * cause creation to fail if the xattr already exists (_CREATE) or
  * doesn't already exist (_REPLACE).  xattrs can have a zero length
  * value.
- *
- * The caller has acquired cluster locks, holds a transaction, and has
- * dirtied the inode item so that they can update it after we modify it.
- * The caller has to know the tags to acquire cluster locks before
- * holding the transaction so they pass in the parsed tags, or all 0s for
- * non scoutfs. prefixes.
  */
-int scoutfs_xattr_set_locked(struct inode *inode, const char *name, size_t name_len,
-			     const void *value, size_t size, int flags,
-			     const struct scoutfs_xattr_prefix_tags *tgs,
-			     struct scoutfs_lock *lck, struct scoutfs_lock *totl_lock,
-			     struct list_head *ind_locks)
+static int scoutfs_xattr_set(struct dentry *dentry, const char *name,
+			     const void *value, size_t size, int flags)
 {
+	struct inode *inode = dentry->d_inode;
 	struct scoutfs_inode_info *si = SCOUTFS_I(inode);
 	struct super_block *sb = inode->i_sb;
 	const u64 ino = scoutfs_ino(inode);
 	struct scoutfs_xattr_totl_val tval = {0,};
+	struct scoutfs_xattr_prefix_tags tgs;
 	struct scoutfs_xattr *xat = NULL;
+	struct scoutfs_lock *lck = NULL;
+	struct scoutfs_lock *totl_lock = NULL;
+	size_t name_len = strlen(name);
 	struct scoutfs_key totl_key;
 	struct scoutfs_key key;
 	bool undo_srch = false;
 	bool undo_totl = false;
+	LIST_HEAD(ind_locks);
 	u8 found_parts;
-	unsigned int xat_bytes_totl;
-	unsigned int xat_bytes;
+	unsigned int bytes;
 	unsigned int val_len;
+	u64 ind_seq;
 	u64 total;
 	u64 hash = 0;
 	u64 id = 0;
@@ -651,9 +607,6 @@ int scoutfs_xattr_set_locked(struct inode *inode, const char *name, size_t name_
 
 	trace_scoutfs_xattr_set(sb, name_len, value, size, flags);
 
-	if (WARN_ON_ONCE(tgs->totl && !totl_lock))
-		return -EINVAL;
-
 	/* mirror the syscall's errors for large names and values */
 	if (name_len > SCOUTFS_XATTR_MAX_NAME_LEN)
 		return -ERANGE;
@@ -664,61 +617,73 @@ int scoutfs_xattr_set_locked(struct inode *inode, const char *name, size_t name_
 	    (flags & ~(XATTR_CREATE | XATTR_REPLACE)))
 		return -EINVAL;
 
-	if ((tgs->hide | tgs->srch | tgs->totl) && !capable(CAP_SYS_ADMIN))
+	if (unknown_prefix(name))
+		return -EOPNOTSUPP;
+
+	if (scoutfs_xattr_parse_tags(name, name_len, &tgs) != 0)
+		return -EINVAL;
+
+	if ((tgs.hide | tgs.srch | tgs.totl) && !capable(CAP_SYS_ADMIN))
 		return -EPERM;
 
-	if (tgs->totl && ((ret = parse_totl_key(&totl_key, name, name_len)) != 0))
+	if (tgs.totl && ((ret = parse_totl_key(&totl_key, name, name_len)) != 0))
 		return ret;
 
-	/* allocate enough to always read an existing xattr's totl */
-	xat_bytes_totl = first_item_bytes(name_len,
-					  max_t(size_t, size, SCOUTFS_XATTR_MAX_TOTL_U64));
-	/* but store partial first item that only includes the new xattr's value */
-	xat_bytes = first_item_bytes(name_len, size);
-	xat = kmalloc(xat_bytes_totl, GFP_NOFS);
-	if (!xat)
-		return -ENOMEM;
+	bytes = sizeof(struct scoutfs_xattr) + name_len + size;
+	/* alloc enough to read old totl value */
+	xat = __vmalloc(bytes + SCOUTFS_XATTR_MAX_TOTL_U64, GFP_NOFS, PAGE_KERNEL);
+	if (!xat) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_WRITE,
+				 SCOUTFS_LKF_REFRESH_INODE, inode, &lck);
+	if (ret)
+		goto out;
 
 	down_write(&si->xattr_rwsem);
 
 	/* find an existing xattr to delete, including possible totl value */
-	ret = get_next_xattr(inode, &key, xat, xat_bytes_totl, name, name_len, 0, 0, lck);
+	ret = get_next_xattr(inode, &key, xat,
+			     sizeof(struct scoutfs_xattr) + name_len + SCOUTFS_XATTR_MAX_TOTL_U64,
+			     name, name_len, 0, 0, lck);
 	if (ret < 0 && ret != -ENOENT)
-		goto out;
+		goto unlock;
 
 	/* check existence constraint flags */
 	if (ret == -ENOENT && (flags & XATTR_REPLACE)) {
 		ret = -ENODATA;
-		goto out;
+		goto unlock;
 	} else if (ret >= 0 && (flags & XATTR_CREATE)) {
 		ret = -EEXIST;
-		goto out;
+		goto unlock;
 	}
 
 	/* not an error to delete something that doesn't exist */
 	if (ret == -ENOENT && !value) {
 		ret = 0;
-		goto out;
+		goto unlock;
 	}
 
 	/* s64 count delta if we create or delete */
-	if (tgs->totl)
+	if (tgs.totl)
 		tval.count = cpu_to_le64((u64)!!(value) - (u64)!!(ret != -ENOENT));
 
 	/* found fields in key will also be used */
 	found_parts = ret >= 0 ? xattr_nr_parts(xat) : 0;
 
-	if (found_parts && tgs->totl) {
+	if (found_parts && tgs.totl) {
 		/* parse old totl value before we clobber xat buf */
 		val_len = ret - offsetof(struct scoutfs_xattr, name[xat->name_len]);
 		ret = parse_totl_u64(&xat->name[xat->name_len], val_len, &total);
 		if (ret < 0)
-			goto out;
+			goto unlock;
 
 		le64_add_cpu(&tval.total, -total);
 	}
 
-	/* prepare the xattr header, name, and start of value in first item */
+	/* prepare our xattr */
 	if (value) {
 		if (found_parts)
 			id = le64_to_cpu(key.skx_id);
@@ -728,94 +693,17 @@ int scoutfs_xattr_set_locked(struct inode *inode, const char *name, size_t name_
 		xat->val_len = cpu_to_le16(size);
 		memset(xat->__pad, 0, sizeof(xat->__pad));
 		memcpy(xat->name, name, name_len);
-		memcpy(&xat->name[name_len], value,
-		       min(size, SCOUTFS_XATTR_MAX_PART_SIZE -
-			         offsetof(struct scoutfs_xattr, name[name_len])));
+		memcpy(&xat->name[xat->name_len], value, size);
 
-		if (tgs->totl) {
+		if (tgs.totl) {
 			ret = parse_totl_u64(value, size, &total);
 			if (ret < 0)
-				goto out;
+				goto unlock;
 		}
 
 		le64_add_cpu(&tval.total, total);
 	}
 
-	if (tgs->srch && !(found_parts && value)) {
-		if (found_parts)
-			id = le64_to_cpu(key.skx_id);
-		hash = scoutfs_hash64(name, name_len);
-		ret = scoutfs_forest_srch_add(sb, hash, ino, id);
-		if (ret < 0)
-			goto out;
-		undo_srch = true;
-	}
-
-	if (tgs->totl) {
-		ret = apply_totl_delta(sb, &totl_key, &tval, totl_lock);
-		if (ret < 0)
-			goto out;
-		undo_totl = true;
-	}
-
-	if (found_parts && value)
-		ret = change_xattr_items(inode, id, xat, xat_bytes, value, size,
-					 xattr_nr_parts(xat), found_parts, lck);
-	else if (found_parts)
-		ret = delete_xattr_items(inode, le64_to_cpu(key.skx_name_hash),
-					 le64_to_cpu(key.skx_id), found_parts,
-					 lck);
-	else
-		ret = create_xattr_items(inode, id, xat, xat_bytes, value, size,
-					 xattr_nr_parts(xat), lck);
-	if (ret < 0)
-		goto out;
-
-	/* XXX do these want i_mutex or anything? */
-	inode_inc_iversion(inode);
-	inode->i_ctime = CURRENT_TIME;
-	ret = 0;
-
-out:
-	if (ret < 0 && undo_srch) {
-		err = scoutfs_forest_srch_add(sb, hash, ino, id);
-		BUG_ON(err);
-	}
-	if (ret < 0 && undo_totl) {
-		/* _delta() on dirty items shouldn't fail */
-		tval.total = cpu_to_le64(-le64_to_cpu(tval.total));
-		tval.count = cpu_to_le64(-le64_to_cpu(tval.count));
-		err = apply_totl_delta(sb, &totl_key, &tval, totl_lock);
-		BUG_ON(err);
-	}
-
-	up_write(&si->xattr_rwsem);
-	kfree(xat);
-
-	return ret;
-}
-
-static int scoutfs_xattr_set(struct dentry *dentry, const char *name, const void *value,
-			     size_t size, int flags)
-{
-	struct inode *inode = dentry->d_inode;
-	struct super_block *sb = inode->i_sb;
-	struct scoutfs_xattr_prefix_tags tgs;
-	struct scoutfs_lock *totl_lock = NULL;
-	struct scoutfs_lock *lck = NULL;
-	size_t name_len = strlen(name);
-	LIST_HEAD(ind_locks);
-	u64 ind_seq;
-	int ret;
-
-	if (scoutfs_xattr_parse_tags(name, name_len, &tgs) != 0)
-		return -EINVAL;
-
-	ret = scoutfs_lock_inode(sb, SCOUTFS_LOCK_WRITE,
-				 SCOUTFS_LKF_REFRESH_INODE, inode, &lck);
-	if (ret)
-		goto unlock;
-
 	if (tgs.totl) {
 		ret = scoutfs_lock_xattr_totl(sb, SCOUTFS_LOCK_WRITE_ONLY, 0, &totl_lock);
 		if (ret)
@@ -835,98 +723,79 @@ retry:
 	if (ret < 0)
 		goto release;
 
-	ret = scoutfs_xattr_set_locked(dentry->d_inode, name, name_len, value, size, flags, &tgs,
-				       lck, totl_lock, &ind_locks);
-	if (ret == 0)
-		scoutfs_update_inode_item(inode, lck, &ind_locks);
+	if (tgs.srch && !(found_parts && value)) {
+		if (found_parts)
+			id = le64_to_cpu(key.skx_id);
+		hash = scoutfs_hash64(name, name_len);
+		ret = scoutfs_forest_srch_add(sb, hash, ino, id);
+		if (ret < 0)
+			goto release;
+		undo_srch = true;
+	}
+
+	if (tgs.totl) {
+		ret = apply_totl_delta(sb, &totl_key, &tval, totl_lock);
+		if (ret < 0)
+			goto release;
+		undo_totl = true;
+	}
+
+	if (found_parts && value)
+		ret = change_xattr_items(inode, id, xat, bytes,
+					 xattr_nr_parts(xat), found_parts, lck);
+	else if (found_parts)
+		ret = delete_xattr_items(inode, le64_to_cpu(key.skx_name_hash),
+					 le64_to_cpu(key.skx_id), found_parts,
+					 lck);
+	else
+		ret = create_xattr_items(inode, id, xat, bytes, lck);
+	if (ret < 0)
+		goto release;
+
+	/* XXX do these want i_mutex or anything? */
+	inode_inc_iversion(inode);
+	inode->i_ctime = CURRENT_TIME;
+	scoutfs_update_inode_item(inode, lck, &ind_locks);
+	ret = 0;
 
 release:
+	if (ret < 0 && undo_srch) {
+		err = scoutfs_forest_srch_add(sb, hash, ino, id);
+		BUG_ON(err);
+	}
+	if (ret < 0 && undo_totl) {
+		/* _delta() on dirty items shouldn't fail */
+		tval.total = cpu_to_le64(-le64_to_cpu(tval.total));
+		tval.count = cpu_to_le64(-le64_to_cpu(tval.count));
+		err = apply_totl_delta(sb, &totl_key, &tval, totl_lock);
+		BUG_ON(err);
+	}
+
 	scoutfs_release_trans(sb);
 	scoutfs_inode_index_unlock(sb, &ind_locks);
 unlock:
+	up_write(&si->xattr_rwsem);
 	scoutfs_unlock(sb, lck, SCOUTFS_LOCK_WRITE);
 	scoutfs_unlock(sb, totl_lock, SCOUTFS_LOCK_WRITE_ONLY);
+out:
+	vfree(xat);
 
 	return ret;
 }
 
-/*
- * Future kernels have this amazing hack to rewind the name to get the
- * skipped prefix.  We're back in the stone ages without the handler
- * arg, so we Just Know that this is possible.  This will become a
- * compat hook to either call the kernel's xattr_full_name(handler), or
- * our hack to use the flags as the prefix length.
- */
-static const char *full_name_hack(void *handler, const char *name, int len)
+int scoutfs_setxattr(struct dentry *dentry, const char *name,
+		     const void *value, size_t size, int flags)
 {
-	return name - len;
-}
+	if (size == 0)
+		value = ""; /* set empty value */
 
-static int scoutfs_xattr_get_handler(struct dentry *dentry, const char *name,
-				     void *value, size_t size, int handler_flags)
-{
-	name = full_name_hack(NULL, name, handler_flags);
-	return scoutfs_xattr_get(dentry, name, value, size);
-}
-
-static int scoutfs_xattr_set_handler(struct dentry *dentry, const char *name,
-				     const void *value, size_t size, int flags, int handler_flags)
-{
-	name = full_name_hack(NULL, name, handler_flags);
 	return scoutfs_xattr_set(dentry, name, value, size, flags);
 }
 
-static const struct xattr_handler scoutfs_xattr_user_handler = {
-	.prefix = XATTR_USER_PREFIX,
-	.flags = XATTR_USER_PREFIX_LEN,
-	.get = scoutfs_xattr_get_handler,
-	.set = scoutfs_xattr_set_handler,
-};
-
-static const struct xattr_handler scoutfs_xattr_scoutfs_handler = {
-	.prefix = SCOUTFS_XATTR_PREFIX,
-	.flags = SCOUTFS_XATTR_PREFIX_LEN,
-	.get = scoutfs_xattr_get_handler,
-	.set = scoutfs_xattr_set_handler,
-};
-
-static const struct xattr_handler scoutfs_xattr_trusted_handler = {
-	.prefix = XATTR_TRUSTED_PREFIX,
-	.flags = XATTR_TRUSTED_PREFIX_LEN,
-	.get = scoutfs_xattr_get_handler,
-	.set = scoutfs_xattr_set_handler,
-};
-
-static const struct xattr_handler scoutfs_xattr_security_handler = {
-	.prefix = XATTR_SECURITY_PREFIX,
-	.flags = XATTR_SECURITY_PREFIX_LEN,
-	.get = scoutfs_xattr_get_handler,
-	.set = scoutfs_xattr_set_handler,
-};
-
-static const struct xattr_handler scoutfs_xattr_acl_access_handler = {
-	.prefix = XATTR_NAME_POSIX_ACL_ACCESS,
-	.flags  = ACL_TYPE_ACCESS,
-	.get    = scoutfs_acl_get_xattr,
-	.set    = scoutfs_acl_set_xattr,
-};
-
-static const struct xattr_handler scoutfs_xattr_acl_default_handler = {
-	.prefix = XATTR_NAME_POSIX_ACL_DEFAULT,
-	.flags  = ACL_TYPE_DEFAULT,
-	.get    = scoutfs_acl_get_xattr,
-	.set    = scoutfs_acl_set_xattr,
-};
-
-const struct xattr_handler *scoutfs_xattr_handlers[] = {
-	&scoutfs_xattr_user_handler,
-	&scoutfs_xattr_scoutfs_handler,
-	&scoutfs_xattr_trusted_handler,
-	&scoutfs_xattr_security_handler,
-	&scoutfs_xattr_acl_access_handler,
-	&scoutfs_xattr_acl_default_handler,
-	NULL
-};
+int scoutfs_removexattr(struct dentry *dentry, const char *name)
+{
+	return scoutfs_xattr_set(dentry, name, NULL, 0, XATTR_REPLACE);
+}
 
 ssize_t scoutfs_list_xattrs(struct inode *inode, char *buffer,
 			    size_t size, __u32 *hash_pos, __u64 *id_pos,
@@ -938,7 +807,7 @@ ssize_t scoutfs_list_xattrs(struct inode *inode, char *buffer,
 	struct scoutfs_xattr *xat = NULL;
 	struct scoutfs_lock *lck = NULL;
 	struct scoutfs_key key;
-	unsigned int xat_bytes;
+	unsigned int bytes;
 	ssize_t total = 0;
 	u32 name_hash = 0;
 	bool is_hidden;
@@ -951,8 +820,8 @@ ssize_t scoutfs_list_xattrs(struct inode *inode, char *buffer,
 		id = *id_pos;
 
 	/* need a buffer large enough for all possible names */
-	xat_bytes = first_item_bytes(SCOUTFS_XATTR_MAX_NAME_LEN, 0);
-	xat = kmalloc(xat_bytes, GFP_NOFS);
+	bytes = sizeof(struct scoutfs_xattr) + SCOUTFS_XATTR_MAX_NAME_LEN;
+	xat = kmalloc(bytes, GFP_NOFS);
 	if (!xat) {
 		ret = -ENOMEM;
 		goto out;
@@ -965,7 +834,8 @@ ssize_t scoutfs_list_xattrs(struct inode *inode, char *buffer,
 	down_read(&si->xattr_rwsem);
 
 	for (;;) {
-		ret = get_next_xattr(inode, &key, xat, xat_bytes, NULL, 0, name_hash, id, lck);
+		ret = get_next_xattr(inode, &key, xat, bytes,
+				     NULL, 0, name_hash, id, lck);
 		if (ret < 0) {
 			if (ret == -ENOENT)
 				ret = total;

kmod/src/xattr.h

@@ -1,29 +1,25 @@
 #ifndef _SCOUTFS_XATTR_H_
 #define _SCOUTFS_XATTR_H_
 
+ssize_t scoutfs_getxattr(struct dentry *dentry, const char *name, void *buffer,
+			 size_t size);
+int scoutfs_setxattr(struct dentry *dentry, const char *name,
+		     const void *value, size_t size, int flags);
+int scoutfs_removexattr(struct dentry *dentry, const char *name);
+ssize_t scoutfs_listxattr(struct dentry *dentry, char *buffer, size_t size);
+ssize_t scoutfs_list_xattrs(struct inode *inode, char *buffer,
+			    size_t size, __u32 *hash_pos, __u64 *id_pos,
+			    bool e_range, bool show_hidden);
+
+int scoutfs_xattr_drop(struct super_block *sb, u64 ino,
+		       struct scoutfs_lock *lock);
+
 struct scoutfs_xattr_prefix_tags {
 	unsigned long hide:1,
 		      srch:1,
 		      totl:1;
 };
 
-extern const struct xattr_handler *scoutfs_xattr_handlers[];
-
-int scoutfs_xattr_get_locked(struct inode *inode, const char *name, void *buffer, size_t size,
-			     struct scoutfs_lock *lck);
-int scoutfs_xattr_set_locked(struct inode *inode, const char *name, size_t name_len,
-			     const void *value, size_t size, int flags,
-			     const struct scoutfs_xattr_prefix_tags *tgs,
-			     struct scoutfs_lock *lck, struct scoutfs_lock *totl_lock,
-			     struct list_head *ind_locks);
-
-ssize_t scoutfs_listxattr(struct dentry *dentry, char *buffer, size_t size);
-ssize_t scoutfs_list_xattrs(struct inode *inode, char *buffer,
-			    size_t size, __u32 *hash_pos, __u64 *id_pos,
-			    bool e_range, bool show_hidden);
-int scoutfs_xattr_drop(struct super_block *sb, u64 ino,
-		       struct scoutfs_lock *lock);
-
 int scoutfs_xattr_parse_tags(const char *name, unsigned int name_len,
 			     struct scoutfs_xattr_prefix_tags *tgs);
 

tests/.gitignore

@@ -8,4 +8,3 @@ src/bulk_create_paths
 src/find_xattrs
 src/stage_tmpfile
 src/create_xattr_loop
-src/o_tmpfile_umask

tests/Makefile

@@ -10,9 +10,7 @@ BIN := src/createmany			\
 	src/bulk_create_paths		\
 	src/stage_tmpfile		\
 	src/find_xattrs			\
-	src/create_xattr_loop		\
-	src/fragmented_data_extents	\
-	src/o_tmpfile_umask
+	src/create_xattr_loop
 
 DEPS := $(wildcard src/*.d)
 

tests/fenced-local-force-unmount.sh

@@ -1,18 +1,5 @@
 #!/usr/bin/bash
 
-#
-# This fencing script is used for testing clusters of multiple mounts on
-# a single host.  It finds mounts to fence by looking for their rids and
-# only knows how to "fence" by using forced unmount.
-#
-
-echo "$0 running rid '$SCOUTFS_FENCED_REQ_RID' ip '$SCOUTFS_FENCED_REQ_IP' args '$@'"
-
-log() {
-	echo "$@" > /dev/stderr
-	exit 1
-}
-
 echo_fail() {
 	echo "$@" > /dev/stderr
 	exit 1
@@ -20,24 +7,29 @@ echo_fail() {
 
 rid="$SCOUTFS_FENCED_REQ_RID"
 
-for fs in /sys/fs/scoutfs/*; do
-	[ ! -d "$fs" ] && continue
+#
+# Look for a local mount with the rid to fence.  Typically we'll at
+# least find the mount with the server that requested the fence that
+# we're processing.   But it's possible that mounts are unmounted
+# before, or while, we're running.
+#
+mnts=$(findmnt -l -n -t scoutfs -o TARGET) || \
+	echo_fail "findmnt -t scoutfs failed" > /dev/stderr
 
-	fs_rid="$(cat $fs/rid)" || \
-		echo_fail "failed to get rid in $fs"
-	if [ "$fs_rid" != "$rid" ]; then
-		continue
-	fi
+for mnt in $mnts; do
+	mnt_rid=$(scoutfs statfs -p "$mnt" -s rid) || \
+		echo_fail "scoutfs statfs $mnt failed"
 
-	nr="$(cat $fs/data_device_maj_min)" || \
-		echo_fail "failed to get data device major:minor in $fs"
-
-	mnts=$(findmnt -l -n -t scoutfs -o TARGET -S $nr) || \
-		echo_fail "findmnt -t scoutfs -S $nr failed"
-	for mnt in $mnts; do
+	if [ "$mnt_rid" == "$rid" ]; then
 		umount -f "$mnt" || \
-			echo_fail "umout -f $mnt failed"
-	done
+			echo_fail "umout -f $mnt"
+
+		exit 0
+	fi
 done
 
+#
+# If the mount doesn't exist on this host then it can't access the
+# devices by definition and can be considered fenced.
+#
 exit 0

tests/funcs/exec.sh

@@ -39,18 +39,6 @@ t_quiet()
 		t_fail "quiet command failed"
 }
 
-#
-# Quietly run a command during a test.  The output is logged but only
-# the return code is printed, presumably because the output contains
-# a lot of invocation specific text that is difficult to filter.
-#
-t_rc()
-{
-	echo "# $*" >> "$T_TMP.rc.log"
-	"$@" >> "$T_TMP.rc.log" 2>&1
-	echo "rc: $?"
-}
-
 #
 # redirect test output back to the output of the invoking script intead
 # of the compared output.

tests/funcs/filter.sh

@@ -61,7 +61,6 @@ t_filter_dmesg()
 	re="$re|scoutfs .* error: meta_super META flag not set"
 	re="$re|scoutfs .* error: could not open metadev:.*"
 	re="$re|scoutfs .* error: Unknown or malformed option,.*"
-	re="$re|scoutfs .* error: invalid quorum_heartbeat_timeout_ms value"
 
 	# in debugging kernels we can slow things down a bit
 	re="$re|hrtimer: interrupt took .*"
@@ -82,7 +81,6 @@ t_filter_dmesg()
 	re="$re|scoutfs .* error .* freeing merged btree blocks.*.final commit del.upd freeing item"
 	re="$re|scoutfs .* error .*reading quorum block.*to update event.*"
 	re="$re|scoutfs .* error.*server failed to bind to.*"
-	re="$re|scoutfs .* critical transaction commit failure.*"
 
 	egrep -v "($re)" 
 }

tests/funcs/fs.sh

@@ -75,15 +75,6 @@ t_fs_nrs()
 	seq 0 $((T_NR_MOUNTS - 1))
 }
 
-#
-# output the fs nrs of quorum nodes, we "know" that
-# the quorum nrs are the first consequtive nrs
-#
-t_quorum_nrs()
-{
-	seq 0 $((T_QUORUM - 1))
-}
-
 #
 # outputs "1" if the fs number has "1" in its quorum/is_leader file.
 # All other cases output 0, including the fs nr being a client which
@@ -386,21 +377,13 @@ t_wait_for_leader() {
 	done
 }
 
-t_get_sysfs_mount_option() {
-	local nr="$1"
-	local name="$2"
-	local opt="$(t_sysfs_path $nr)/mount_options/$name"
-
-	cat "$opt"
-}
-
 t_set_sysfs_mount_option() {
 	local nr="$1"
 	local name="$2"
 	local val="$3"
 	local opt="$(t_sysfs_path $nr)/mount_options/$name"
 
-	echo "$val" > "$opt" 2>/dev/null
+	echo "$val" > "$opt"
 }
 
 t_set_all_sysfs_mount_options() {
@@ -422,7 +405,7 @@ t_save_all_sysfs_mount_options() {
 
 	for i in $(t_fs_nrs); do
 		opt="$(t_sysfs_path $i)/mount_options/$name"
-		ind="${name}_${i}"
+		ind="$name_$i"
 
 		_saved_opts[$ind]="$(cat $opt)"
 	done
@@ -434,7 +417,7 @@ t_restore_all_sysfs_mount_options() {
 	local i
 
 	for i in $(t_fs_nrs); do
-		ind="${name}_${i}"
+		ind="$name_$i"
 
 		t_set_sysfs_mount_option $i $name "${_saved_opts[$ind]}"
 	done

tests/golden/basic-posix-consistency

@@ -49,7 +49,6 @@ four
 --- can't overwrite non-empty dir
 mv: cannot move ‘/mnt/test/test/basic-posix-consistency/dir/c/clobber’ to ‘/mnt/test/test/basic-posix-consistency/dir/a/dir’: Directory not empty
 --- can overwrite empty dir
---- can rename into root
 == path resoluion
 == inode indexes match after syncing existing
 == inode indexes match after copying and syncing

tests/golden/basic-truncate

@@ -1,6 +0,0 @@
-== truncate writes zeroed partial end of file block
-0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79
-*
-0006144 0000 0000 0000 0000 0000 0000 0000 0000
-*
-0012288

tests/golden/change-devices

@@ -1,27 +0,0 @@
-== make tmp sparse data dev files
-== make scratch fs
-== small new data device fails
-rc: 1
-== check sees data device errors
-rc: 1
-rc: 0
-== preparing while mounted fails
-rc: 1
-== preparing without recovery fails
-rc: 1
-== check sees metadata errors
-rc: 1
-rc: 1
-== preparing with file data fails
-rc: 1
-== preparing after emptied
-rc: 0
-== checks pass
-rc: 0
-rc: 0
-== using prepared
-== preparing larger and resizing
-rc: 0
-equal_prepared
-large_prepared
-resized larger test rc: 0

tests/golden/data-prealloc

@@ -1,26 +0,0 @@
-== initial writes smaller than prealloc grow to prealloc size
-/mnt/test/test/data-prealloc/file-1: 7 extents found
-/mnt/test/test/data-prealloc/file-2: 7 extents found
-== larger files get full prealloc extents
-/mnt/test/test/data-prealloc/file-1: 9 extents found
-/mnt/test/test/data-prealloc/file-2: 9 extents found
-== non-streaming writes with contig have per-block extents
-/mnt/test/test/data-prealloc/file-1: 32 extents found
-/mnt/test/test/data-prealloc/file-2: 32 extents found
-== any writes to region prealloc get full extents
-/mnt/test/test/data-prealloc/file-1: 4 extents found
-/mnt/test/test/data-prealloc/file-2: 4 extents found
-/mnt/test/test/data-prealloc/file-1: 4 extents found
-/mnt/test/test/data-prealloc/file-2: 4 extents found
-== streaming offline writes get full extents either way
-/mnt/test/test/data-prealloc/file-1: 4 extents found
-/mnt/test/test/data-prealloc/file-2: 4 extents found
-/mnt/test/test/data-prealloc/file-1: 4 extents found
-/mnt/test/test/data-prealloc/file-2: 4 extents found
-== goofy preallocation amounts work
-/mnt/test/test/data-prealloc/file-1: 5 extents found
-/mnt/test/test/data-prealloc/file-2: 5 extents found
-/mnt/test/test/data-prealloc/file-1: 5 extents found
-/mnt/test/test/data-prealloc/file-2: 5 extents found
-/mnt/test/test/data-prealloc/file-1: 3 extents found
-/mnt/test/test/data-prealloc/file-2: 3 extents found

tests/golden/large-fragmented-free

@@ -1,3 +0,0 @@
-== creating fragmented extents
-== unlink file with moved extents to free extents per block
-== cleanup

tests/golden/lock-recover-invalidate

@@ -1,3 +0,0 @@
-== starting background invalidating read/write load
-== 60s of lock recovery during invalidating load
-== stopping background load

tests/golden/quorum-heartbeat-timeout

@@ -1,2 +0,0 @@
-== bad timeout values fail
-== test different timeouts

tests/golden/simple-inode-index

@@ -7,4 +7,3 @@ found second
 == changing metadata must increase meta seq
 == changing contents must increase data seq
 == make sure dirtying doesn't livelock walk
-== concurrent update attempts maintain single entries

tests/golden/stage-tmpfile

@@ -1,11 +1,3 @@
-== non-acl O_TMPFILE creation honors umask
-umask 022
-fstat after open(0777): 0100755
-stat after linkat: 0100755
-umask 077
-fstat after open(0777): 0100700
-stat after linkat: 0100700
-== stage from tmpfile
 total file size 33669120
 00000000  41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41  |AAAAAAAAAAAAAAAA|
 *

tests/golden/xfstests

@@ -40,19 +40,22 @@ generic/092
 generic/098
 generic/101
 generic/104
-generic/105
 generic/106
 generic/107
 generic/117
 generic/124
 generic/129
+generic/130
 generic/131
+generic/135
 generic/169
 generic/184
+generic/211
+generic/212
+generic/214
 generic/221
 generic/228
 generic/236
-generic/237
 generic/245
 generic/249
 generic/257
@@ -65,7 +68,6 @@ generic/308
 generic/309
 generic/313
 generic/315
-generic/319
 generic/322
 generic/335
 generic/336
@@ -75,7 +77,6 @@ generic/342
 generic/343
 generic/348
 generic/360
-generic/375
 generic/376
 generic/377
 Not
@@ -104,13 +105,9 @@ generic/078
 generic/079
 generic/081
 generic/082
-generic/091
-generic/094
 generic/096
 generic/110
 generic/111
-generic/113
-generic/114
 generic/115
 generic/116
 generic/118
@@ -119,9 +116,7 @@ generic/121
 generic/122
 generic/123
 generic/128
-generic/130
 generic/134
-generic/135
 generic/136
 generic/138
 generic/139
@@ -169,7 +164,6 @@ generic/194
 generic/195
 generic/196
 generic/197
-generic/198
 generic/199
 generic/200
 generic/201
@@ -177,11 +171,6 @@ generic/202
 generic/203
 generic/205
 generic/206
-generic/207
-generic/210
-generic/211
-generic/212
-generic/214
 generic/216
 generic/217
 generic/218
@@ -189,13 +178,11 @@ generic/219
 generic/220
 generic/222
 generic/223
-generic/225
 generic/227
 generic/229
 generic/230
 generic/235
 generic/238
-generic/240
 generic/244
 generic/250
 generic/252
@@ -207,7 +194,6 @@ generic/259
 generic/260
 generic/261
 generic/262
-generic/263
 generic/264
 generic/265
 generic/266
@@ -286,4 +272,4 @@ shared/004
 shared/032
 shared/051
 shared/289
-Passed all 79 tests
+Passed all 80 tests

tests/run-tests.sh

@@ -58,7 +58,6 @@ $(basename $0) options:
     -m        | Run mkfs on the device before mounting and running
               | tests.  Implies unmounting existing mounts first.
     -n <nr>   | The number of devices and mounts to test.
-    -o <opts> | Add option string to all mounts during all tests.
     -P        | Enable trace_printk.
     -p        | Exit script after preparing mounts only, don't run tests.
     -q <nr>   | The first <nr> mounts will be quorum members.  Must be
@@ -69,7 +68,6 @@ $(basename $0) options:
     -s        | Skip git repo checkouts.
     -t        | Enabled trace events that match the given glob argument.
               | Multiple options enable multiple globbed events.
-    -T <nr>   | Multiply the original trace buffer size by nr during the run.
     -X        | xfstests git repo. Used by tests/xfstests.sh.
     -x        | xfstests git branch to checkout and track.
     -y        | xfstests ./check additional args
@@ -138,12 +136,6 @@ while true; do
 		T_NR_MOUNTS="$2"
 		shift
 		;;
-	-o)
-		test -n "$2" || die "-o must have option string argument"
-		# always appending to existing options
-		T_MNT_OPTIONS+=",$2"
-		shift
-		;;
 	-P)
 		T_TRACE_PRINTK="1"
 		;;
@@ -168,11 +160,6 @@ while true; do
 		T_TRACE_GLOB+=("$2")
 		shift
 		;;
-	-T)
-		test -n "$2" || die "-T must have trace buffer size multiplier argument"
-		T_TRACE_MULT="$2"
-		shift
-		;;
 	-X)
 		test -n "$2" || die "-X requires xfstests git repo dir argument"
 		T_XFSTESTS_REPO="$2"
@@ -358,13 +345,6 @@ if [ -n "$T_INSMOD" ]; then
 	cmd insmod "$T_KMOD/src/scoutfs.ko"
 fi
 
-if [ -n "$T_TRACE_MULT" ]; then
-	orig_trace_size=$(cat /sys/kernel/debug/tracing/buffer_size_kb)
-	mult_trace_size=$((orig_trace_size * T_TRACE_MULT))
-	msg "increasing trace buffer size from $orig_trace_size KiB to $mult_trace_size KiB"
-	echo $mult_trace_size > /sys/kernel/debug/tracing/buffer_size_kb
-fi
-
 nr_globs=${#T_TRACE_GLOB[@]}
 if [ $nr_globs -gt 0 ]; then
 	echo 0 > /sys/kernel/debug/tracing/events/scoutfs/enable
@@ -394,21 +374,19 @@ fi
 # always describe tracing in the logs
 cmd cat /sys/kernel/debug/tracing/set_event
 cmd grep .  /sys/kernel/debug/tracing/options/trace_printk \
-	    /sys/kernel/debug/tracing/buffer_size_kb \
 	    /proc/sys/kernel/ftrace_dump_on_oops
 
 #
 # Build a fenced config that runs scripts out of the repository rather
 # than the default system directory
 #
-conf="$T_RESULTS/scoutfs-fenced.conf"
+conf="$T_RESULTS/scoutfs-fencd.conf"
 cat > $conf << EOF
 SCOUTFS_FENCED_DELAY=1
 SCOUTFS_FENCED_RUN=$T_TESTS/fenced-local-force-unmount.sh
-SCOUTFS_FENCED_RUN_ARGS="ignored run args"
+SCOUTFS_FENCED_RUN_ARGS=""
 EOF
 export SCOUTFS_FENCED_CONFIG_FILE="$conf"
-T_FENCED_LOG="$T_RESULTS/fenced.log"
 
 #
 # Run the agent in the background, log its output, an kill it if we
@@ -416,7 +394,7 @@ T_FENCED_LOG="$T_RESULTS/fenced.log"
 #
 fenced_log()
 {
-	echo "[$(timestamp)] $*" >> "$T_FENCED_LOG"
+	echo "[$(timestamp)] $*" >> "$T_RESULTS/fenced.stdout.log"
 }
 fenced_pid=""
 kill_fenced()
@@ -427,7 +405,7 @@ kill_fenced()
 	fi
 }
 trap kill_fenced EXIT
-$T_UTILS/fenced/scoutfs-fenced > "$T_FENCED_LOG" 2>&1 &
+$T_UTILS/fenced/scoutfs-fenced > "$T_RESULTS/fenced.stdout.log" 2> "$T_RESULTS/fenced.stderr.log" &
 fenced_pid=$!
 fenced_log "started fenced pid $fenced_pid in the background"
 
@@ -451,7 +429,6 @@ for i in $(seq 0 $((T_NR_MOUNTS - 1))); do
 	if [ "$i" -lt "$T_QUORUM" ]; then
 		opts="$opts,quorum_slot_nr=$i"
 	fi
-	opts="${opts}${T_MNT_OPTIONS}"
 
 	msg "mounting $meta_dev|$data_dev on $dir"
 	cmd mount -t scoutfs $opts "$data_dev" "$dir" &
@@ -626,9 +603,6 @@ if [ -n "$T_TRACE_GLOB" -o -n "$T_TRACE_PRINTK" ]; then
 	echo 0 > /sys/kernel/debug/tracing/events/scoutfs/enable
 	echo 0 > /sys/kernel/debug/tracing/options/trace_printk
 	cat /sys/kernel/debug/tracing/trace > "$T_RESULTS/traces"
-	if [ -n "$orig_trace_size" ]; then
-		echo $orig_trace_size > /sys/kernel/debug/tracing/buffer_size_kb
-	fi
 fi
 
 if [ "$skipped" == 0 -a "$failed" == 0 ]; then

tests/sequence

@@ -6,12 +6,9 @@ simple-inode-index.sh
 simple-staging.sh
 simple-release-extents.sh
 fallocate.sh
-basic-truncate.sh
-data-prealloc.sh
 setattr_more.sh
 offline-extent-waiting.sh
 move-blocks.sh
-large-fragmented-free.sh
 enospc.sh
 srch-basic-functionality.sh
 simple-xattr-unit.sh
@@ -20,14 +17,13 @@ lock-refleak.sh
 lock-shrink-consistency.sh
 lock-pr-cw-conflict.sh
 lock-revoke-getcwd.sh
-lock-recover-invalidate.sh
 export-lookup-evict-race.sh
 createmany-parallel.sh
 createmany-large-names.sh
 createmany-rename-large-dir.sh
 stage-release-race-alloc.sh
 stage-multi-part.sh
-o_tmpfile.sh
+stage-tmpfile.sh
 basic-posix-consistency.sh
 dirent-consistency.sh
 mkdir-rename-rmdir.sh
@@ -36,9 +32,7 @@ cross-mount-data-free.sh
 persistent-item-vers.sh
 setup-error-teardown.sh
 resize-devices.sh
-change-devices.sh
 fence-and-reclaim.sh
-quorum-heartbeat-timeout.sh
 orphan-inodes.sh
 mount-unmount-race.sh
 client-unmount-recovery.sh

tests/src/fragmented_data_extents.c

@@ -1,113 +0,0 @@
-/*
- * Copyright (C) 2021 Versity Software, Inc.  All rights reserved.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public
- * License v2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- */
-
-/*
- * This creates fragmented data extents.
- *
- * A file is created that has alternating free and allocated extents.
- * This also results in the global allocator having the matching
- * fragmented free extent pattern.  While that file is being created,
- * occasionally an allocated extent is moved to another file.   This
- * results in a file that has fragmented extents at a given stride that
- * can be deleted to create free data extents with a given stride.
- *
- * We don't have hole punching so to do this quickly we use a goofy
- * combination of fallocate, truncate, and our move_blocks ioctl.
- */
-
-#ifndef _GNU_SOURCE
-#define _GNU_SOURCE
-#endif
-#include <unistd.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/ioctl.h>
-#include <fcntl.h>
-#include <errno.h>
-#include <linux/types.h>
-#include <assert.h>
-
-#include "ioctl.h"
-
-#define BLOCK_SIZE 4096
-
-int main(int argc, char **argv)
-{
-	struct scoutfs_ioctl_move_blocks mb = {0,};
-	unsigned long long freed_extents;
-	unsigned long long move_stride;
-	unsigned long long i;
-	int alloc_fd;
-	int trunc_fd;
-	off_t off;
-	int ret;
-
-	if (argc != 5) {
-		printf("%s <freed_extents> <move_stride> <alloc_file> <trunc_file>\n", argv[0]);
-		return 1;
-	}
-
-	freed_extents = strtoull(argv[1], NULL, 0);
-	move_stride = strtoull(argv[2], NULL, 0);
-
-	alloc_fd = open(argv[3], O_RDWR | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR);
-	if (alloc_fd == -1) {
-		fprintf(stderr, "error opening %s: %d (%s)\n", argv[3], errno, strerror(errno));
-		exit(1);
-	}
-
-	trunc_fd = open(argv[4], O_RDWR | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR);
-	if (trunc_fd == -1) {
-		fprintf(stderr, "error opening %s: %d (%s)\n", argv[4], errno, strerror(errno));
-		exit(1);
-	}
-
-	for (i = 0, off = 0; i < freed_extents; i++, off += BLOCK_SIZE * 2) {
-
-		ret = fallocate(alloc_fd, 0, off, BLOCK_SIZE * 2);
-		if (ret < 0) {
-			fprintf(stderr, "fallocate at off %llu error: %d (%s)\n",
-				(unsigned long long)off, errno, strerror(errno));
-			exit(1);
-		}
-
-		ret = ftruncate(alloc_fd, off + BLOCK_SIZE);
-		if (ret < 0) {
-			fprintf(stderr, "truncate to off %llu error: %d (%s)\n",
-				(unsigned long long)off + BLOCK_SIZE, errno, strerror(errno));
-			exit(1);
-		}
-
-		if ((i % move_stride) == 0) {
-			mb.from_fd = alloc_fd;
-			mb.from_off = off;
-			mb.len = BLOCK_SIZE;
-			mb.to_off = i * BLOCK_SIZE;
-
-			ret = ioctl(trunc_fd, SCOUTFS_IOC_MOVE_BLOCKS, &mb);
-			if (ret < 0) {
-				fprintf(stderr, "move from off %llu error: %d (%s)\n",
-					(unsigned long long)off,
-					errno, strerror(errno));
-			}
-		}
-	}
-
-	if (alloc_fd > -1)
-		close(alloc_fd);
-	if (trunc_fd > -1)
-		close(trunc_fd);
-
-	return 0;
-}

tests/src/o_tmpfile_umask.c

@@ -1,97 +0,0 @@
-/*
- * Show the modes of files as we create them with O_TMPFILE and link
- * them into the namespace.
- *
- * Copyright (C) 2022 Versity Software, Inc.  All rights reserved.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public
- * License v2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * General Public License for more details.
- */
-
-#ifndef _GNU_SOURCE
-#define _GNU_SOURCE
-#endif
-#include <unistd.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/ioctl.h>
-#include <fcntl.h>
-#include <errno.h>
-#include <sys/stat.h>
-#include <assert.h>
-#include <limits.h>
-
-static void linkat_tmpfile_modes(char *dir, char *lpath, mode_t mode)
-{
-	char proc_self[PATH_MAX];
-	struct stat st;
-	int ret;
-	int fd;
-
-	umask(mode);
-	printf("umask 0%o\n", mode);
-
-	fd = open(dir, O_RDWR | O_TMPFILE, 0777);
-	if (fd < 0) {
-		perror("open(O_TMPFILE)");
-		exit(1);
-	}
-
-	ret = fstat(fd, &st);
-	if (ret < 0) {
-		perror("fstat");
-		exit(1);
-	}
-
-	printf("fstat after open(0777): 0%o\n", st.st_mode);
-
-	snprintf(proc_self, sizeof(proc_self), "/proc/self/fd/%d", fd);
-
-	ret = linkat(AT_FDCWD, proc_self, AT_FDCWD, lpath, AT_SYMLINK_FOLLOW);
-	if (ret < 0) {
-		perror("linkat");
-		exit(1);
-	}
-
-	close(fd);
-
-	ret = stat(lpath, &st);
-	if (ret < 0) {
-		perror("fstat");
-		exit(1);
-	}
-
-	printf("stat after linkat: 0%o\n", st.st_mode);
-
-	ret = unlink(lpath);
-	if (ret < 0) {
-		perror("unlink");
-		exit(1);
-	}
-}
-
-int main(int argc, char **argv)
-{
-	char *lpath;
-	char *dir;
-
-	if (argc < 3) {
-		printf("%s <open_dir> <linkat_path>\n", argv[0]);
-		return 1;
-	}
-
-	dir = argv[1];
-	lpath = argv[2];
-
-	linkat_tmpfile_modes(dir, lpath, 022);
-	linkat_tmpfile_modes(dir, lpath, 077);
-
-	return 0;
-}

tests/tests/basic-bad-mounts.sh

@@ -12,7 +12,7 @@ mount_fail()
 }
 
 echo "== prepare devices, mount point, and logs"
-SCR="$T_TMPDIR/mnt.scratch"
+SCR="/mnt/scoutfs.extra"
 mkdir -p "$SCR"
 > $T_TMP.mount.out
 scoutfs mkfs -f -Q 0,127.0.0.1,53000 "$T_EX_META_DEV" "$T_EX_DATA_DEV" > $T_TMP.mkfs.out 2>&1 \

tests/tests/basic-posix-consistency.sh

@@ -149,10 +149,6 @@ find "$T_D0/dir" -ls 2>&1 | t_filter_fs > "$T_TMP.0"
 find "$T_D1/dir" -ls 2>&1 | t_filter_fs > "$T_TMP.1"
 diff -u "$T_TMP.0" "$T_TMP.1"
 rm -rf "$T_D0/dir"
-echo "--- can rename into root"
-touch "$T_D0/rename-into-root"
-mv "$T_D0/rename-into-root" "$T_M0/"
-rm -f "$T_M0/rename-into-root"
 
 echo "== path resoluion"
 touch "$T_D0/file"

tests/tests/basic-truncate.sh

@@ -1,21 +0,0 @@
-#
-# Test basic correctness of truncate.
-#
-
-t_require_commands yes dd od truncate
-
-FILE="$T_D0/file"
-
-#
-# We forgot to write a dirty block that zeroed the tail of a partial
-# final block as we truncated past it.
-#
-echo "== truncate writes zeroed partial end of file block"
-yes | dd of="$FILE" bs=8K count=1 status=none
-sync
-truncate -s 6K "$FILE"
-truncate -s 12K "$FILE"
-echo 3 > /proc/sys/vm/drop_caches
-od -Ad -x "$FILE"
-
-t_pass

tests/tests/change-devices.sh

@@ -1,76 +0,0 @@
-#
-# test changing devices
-#
-
-echo "== make tmp sparse data dev files"
-sz=$(blockdev --getsize64 "$T_EX_DATA_DEV")
-large_sz=$((sz * 2))
-touch "$T_TMP."{small,equal,large}
-truncate -s 1MB "$T_TMP.small"
-truncate -s $sz "$T_TMP.equal"
-truncate -s $large_sz "$T_TMP.large"
-
-echo "== make scratch fs"
-t_quiet scoutfs mkfs -f -Q 0,127.0.0.1,53000 "$T_EX_META_DEV" "$T_EX_DATA_DEV"
-SCR="$T_TMPDIR/mnt.scratch"
-mkdir -p "$SCR"
-
-echo "== small new data device fails"
-t_rc scoutfs prepare-empty-data-device "$T_EX_META_DEV" "$T_TMP.small"
-
-echo "== check sees data device errors"
-t_rc scoutfs prepare-empty-data-device --check "$T_EX_META_DEV" "$T_TMP.small"
-t_rc scoutfs prepare-empty-data-device --check "$T_EX_META_DEV"
-
-echo "== preparing while mounted fails"
-mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 "$T_EX_DATA_DEV" "$SCR"
-t_rc scoutfs prepare-empty-data-device "$T_EX_META_DEV" "$T_TMP.equal"
-umount "$SCR"
-
-echo "== preparing without recovery fails"
-mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 "$T_EX_DATA_DEV" "$SCR"
-umount -f "$SCR"
-t_rc scoutfs prepare-empty-data-device "$T_EX_META_DEV" "$T_TMP.equal"
-
-echo "== check sees metadata errors"
-t_rc scoutfs prepare-empty-data-device --check "$T_EX_META_DEV"
-t_rc scoutfs prepare-empty-data-device --check "$T_EX_META_DEV" "$T_TMP.equal"
-
-echo "== preparing with file data fails"
-mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 "$T_EX_DATA_DEV" "$SCR"
-echo hi > "$SCR"/file
-umount "$SCR"
-scoutfs print "$T_EX_META_DEV" > "$T_TMP.print"
-t_rc scoutfs prepare-empty-data-device "$T_EX_META_DEV" "$T_TMP.equal"
-
-echo "== preparing after emptied"
-mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 "$T_EX_DATA_DEV" "$SCR"
-rm -f "$SCR"/file
-umount "$SCR"
-t_rc scoutfs prepare-empty-data-device "$T_EX_META_DEV" "$T_TMP.equal"
-
-echo "== checks pass"
-t_rc scoutfs prepare-empty-data-device --check "$T_EX_META_DEV"
-t_rc scoutfs prepare-empty-data-device --check "$T_EX_META_DEV" "$T_TMP.equal"
-
-echo "== using prepared"
-scr_loop=$(losetup --find --show "$T_TMP.equal")
-mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 "$scr_loop" "$SCR"
-touch "$SCR"/equal_prepared
-equal_tot=$(scoutfs statfs -s total_data_blocks -p "$SCR")
-umount "$SCR"
-losetup -d "$scr_loop"
-
-echo "== preparing larger and resizing"
-t_rc scoutfs prepare-empty-data-device "$T_EX_META_DEV" "$T_TMP.large"
-scr_loop=$(losetup --find --show "$T_TMP.large")
-mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 "$scr_loop" "$SCR"
-touch "$SCR"/large_prepared
-ls "$SCR"
-scoutfs resize-devices -p "$SCR" -d $large_sz
-large_tot=$(scoutfs statfs -s total_data_blocks -p "$SCR")
-test "$large_tot" -gt "$equal_tot" ; echo "resized larger test rc: $?"
-umount "$SCR"
-losetup -d "$scr_loop"
-
-t_pass

tests/tests/data-prealloc.sh

@@ -1,136 +0,0 @@
-#
-# test that the data prealloc options behave as expected.  We write to
-# two files a block at a time so that a single file doesn't naturally
-# merge adjacent consecutive allocations.  (we don't have multiple
-# allocation cursors)
-#
-t_require_commands scoutfs stat filefrag dd touch truncate
-
-write_forwards()
-{
-	local prefix="$1"
-	local nr="$2"
-	local blk
-
-	touch "$prefix"-{1,2}
-	truncate -s 0 "$prefix"-{1,2}
-
-	for blk in $(seq 0 1 $((nr - 1))); do
-		dd if=/dev/zero of="$prefix"-1 bs=4096 seek=$blk count=1 conv=notrunc status=none
-		dd if=/dev/zero of="$prefix"-2 bs=4096 seek=$blk count=1 conv=notrunc status=none
-	done
-}
-
-write_backwards()
-{
-	local prefix="$1"
-	local nr="$2"
-	local blk
-
-	touch "$prefix"-{1,2}
-	truncate -s 0 "$prefix"-{1,2}
-
-	for blk in $(seq $((nr - 1)) -1 0); do
-		dd if=/dev/zero of="$prefix"-1 bs=4096 seek=$blk count=1 conv=notrunc status=none
-		dd if=/dev/zero of="$prefix"-2 bs=4096 seek=$blk count=1 conv=notrunc status=none
-	done
-}
-
-release_files() {
-	local prefix="$1"
-	local size=$(($2 * 4096))
-	local vers
-	local f
-
-	for f in "$prefix"*; do
-		size=$(stat -c "%s" "$f")
-		vers=$(scoutfs stat -s data_version "$f")
-		scoutfs release "$f" -V "$vers" -o 0 -l $size
-	done
-}
-
-stage_files() {
-	local prefix="$1"
-	local nr="$2"
-	local vers
-	local f
-
-	for blk in $(seq 0 1 $((nr - 1))); do
-		for f in "$prefix"*; do
-			vers=$(scoutfs stat -s data_version "$f")
-			scoutfs stage /dev/zero "$f" -V "$vers" -o $((blk * 4096)) -l 4096
-		done
-	done
-}
-
-print_extents_found()
-{
-	local prefix="$1"
-
-	filefrag "$prefix"* 2>&1 | grep "extent.*found" | t_filter_fs
-}
-
-t_save_all_sysfs_mount_options data_prealloc_blocks
-t_save_all_sysfs_mount_options data_prealloc_contig_only
-restore_options()
-{
-	t_restore_all_sysfs_mount_options data_prealloc_blocks
-	t_restore_all_sysfs_mount_options data_prealloc_contig_only
-}
-trap restore_options EXIT
-
-prefix="$T_D0/file"
-
-echo "== initial writes smaller than prealloc grow to prealloc size"
-t_set_sysfs_mount_option 0 data_prealloc_blocks 32
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 1
-write_forwards $prefix 64
-print_extents_found $prefix
-
-echo "== larger files get full prealloc extents"
-t_set_sysfs_mount_option 0 data_prealloc_blocks 32
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 1
-write_forwards $prefix 128
-print_extents_found $prefix
-
-echo "== non-streaming writes with contig have per-block extents"
-t_set_sysfs_mount_option 0 data_prealloc_blocks 32
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 1
-write_backwards $prefix 32
-print_extents_found $prefix
-
-echo "== any writes to region prealloc get full extents"
-t_set_sysfs_mount_option 0 data_prealloc_blocks 16
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 0
-write_forwards $prefix 64
-print_extents_found $prefix
-write_backwards $prefix 64
-print_extents_found $prefix
-
-echo "== streaming offline writes get full extents either way"
-t_set_sysfs_mount_option 0 data_prealloc_blocks 16
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 1
-write_forwards $prefix 64
-release_files $prefix 64
-stage_files $prefix 64
-print_extents_found $prefix
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 0
-release_files $prefix 64
-stage_files $prefix 64
-print_extents_found $prefix
-
-echo "== goofy preallocation amounts work"
-t_set_sysfs_mount_option 0 data_prealloc_blocks 7
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 1
-write_forwards $prefix 14
-print_extents_found $prefix
-t_set_sysfs_mount_option 0 data_prealloc_blocks 13
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 0
-write_forwards $prefix 53
-print_extents_found $prefix
-t_set_sysfs_mount_option 0 data_prealloc_blocks 1
-t_set_sysfs_mount_option 0 data_prealloc_contig_only 0
-write_forwards $prefix 3
-print_extents_found $prefix
-
-t_pass

tests/tests/enospc.sh

@@ -59,7 +59,7 @@ echo "== make small meta fs"
 # meta device just big enough for reserves and the metadata we'll fill
 scoutfs mkfs -A -f -Q 0,127.0.0.1,53000 -m 10G "$T_EX_META_DEV" "$T_EX_DATA_DEV" > $T_TMP.mkfs.out 2>&1 || \
 	t_fail "mkfs failed"
-SCR="$T_TMPDIR/mnt.scratch"
+SCR="/mnt/scoutfs.enospc"
 mkdir -p "$SCR"
 mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 \
 	"$T_EX_DATA_DEV" "$SCR"

tests/tests/fence-and-reclaim.sh

@@ -45,18 +45,6 @@ check_read_write()
 	fi
 }
 
-# verify that fenced ran our testing fence script
-verify_fenced_run()
-{
-	local rids="$@"
-	local rid
-
-	for rid in $rids; do
-		grep -q ".* running rid '$rid'.* args 'ignored run args'" "$T_FENCED_LOG" || \
-			t_fail "fenced didn't execute RUN script for rid $rid"
-	done
-}
-
 echo "== make sure all mounts can see each other"
 check_read_write
 
@@ -74,14 +62,12 @@ done
 while t_rid_is_fencing $rid; do
 	sleep .5
 done
-verify_fenced_run $rid
 t_mount $cl
 check_read_write
 
 echo "== force unmount all non-server, connection timeout, fence nop, mount"
 sv=$(t_server_nr)
 pattern="nonsense"
-rids=""
 sync
 for cl in $(t_fs_nrs); do
 	if [ $cl == $sv ]; then
@@ -89,7 +75,6 @@ for cl in $(t_fs_nrs); do
 	fi
 
 	rid=$(t_mount_rid $cl)
-	rids="$rids $rid"
 	pattern="$pattern|$rid"
 	echo "cl $cl sv $sv rid $rid" >> "$T_TMP.log"
 
@@ -104,7 +89,6 @@ done
 while test -d $(echo /sys/fs/scoutfs/*/fence/* | cut -d " " -f 1); do
 	sleep .5
 done
-verify_fenced_run $rids
 # remount all the clients
 for cl in $(t_fs_nrs); do
 	if [ $cl == $sv ]; then
@@ -125,17 +109,11 @@ t_wait_for_leader
 while t_rid_is_fencing $rid; do
 	sleep .5
 done
-verify_fenced_run $rid
 t_mount $sv
 check_read_write
 
 echo "== force unmount everything, new server fences all previous"
 sync
-rids=""
-# get rids before forced unmount breaks scoutfs statfs
-for nr in $(t_fs_nrs); do
-	rids="$rids $(t_mount_rid $nr)"
-done
 for nr in $(t_fs_nrs); do
 	t_force_umount $nr
 done
@@ -144,7 +122,6 @@ t_mount_all
 while test -d $(echo /sys/fs/scoutfs/*/fence/* | cut -d " " -f 1); do
 	sleep .5
 done
-verify_fenced_run $rids
 check_read_write
 
 t_pass

tests/tests/large-fragmented-free.sh

@@ -1,22 +0,0 @@
-#
-# Make sure the server can handle a transaction with a data_freed whose
-# blocks all hit different btree blocks in the main free list.  It
-# probably has to be merged in multiple commits.
-#
-
-t_require_commands fragmented_data_extents
-
-EXTENTS_PER_BTREE_BLOCK=600
-EXTENTS_PER_LIST_BLOCK=8192
-FREED_EXTENTS=$((EXTENTS_PER_BTREE_BLOCK * EXTENTS_PER_LIST_BLOCK))
-
-echo "== creating fragmented extents"
-fragmented_data_extents $FREED_EXTENTS $EXTENTS_PER_BTREE_BLOCK "$T_D0/alloc" "$T_D0/move"
-
-echo "== unlink file with moved extents to free extents per block"
-rm -f "$T_D0/move"
-
-echo "== cleanup"
-rm -f "$T_D0/alloc"
-
-t_pass

tests/tests/lock-recover-invalidate.sh

@@ -1,43 +0,0 @@
-#
-# trigger server failover and lock recovery during heavy invalidating
-# load on multiple mounts
-#
-
-majority_nr=$(t_majority_count)
-quorum_nr=$T_QUORUM
-
-test "$quorum_nr" == "$majority_nr" && \
-        t_skip "need remaining majority when leader unmounted"
-
-test "$T_NR_MOUNTS" -lt "$((quorum_nr + 2))" && \
-        t_skip "need at least 2 non-quorum load mounts"
-
-echo "== starting background invalidating read/write load"
-touch "$T_D0/file"
-load_pids=""
-for i in $(t_fs_nrs); do
-	if [ "$i" -ge "$quorum_nr" ]; then
-		eval path="\$T_D${i}/file"
-
-		(while true; do touch $path > /dev/null 2>&1; done) &
-		load_pids="$load_pids $!"
-		(while true; do stat $path > /dev/null 2>&1; done) &
-		load_pids="$load_pids $!"
-	fi
-done
-
-# had it reproduce in ~40s on wimpy debug kernel guests
-LENGTH=60
-echo "== ${LENGTH}s of lock recovery during invalidating load"
-END=$((SECONDS + LENGTH))
-while [ "$SECONDS" -lt "$END" ]; do
-        sv=$(t_server_nr)
-        t_umount $sv
-        t_mount $sv
-	# new server had to process greeting for mount to finish
-done
-
-echo "== stopping background load"
-kill $load_pids
-
-t_pass

tests/tests/o_tmpfile.sh

@@ -1,16 +0,0 @@
-#
-# basic tests of O_TMPFILE
-#
-
-t_require_commands stage_tmpfile hexdump
-
-echo "== non-acl O_TMPFILE creation honors umask"
-o_tmpfile_umask "$T_D0" "$T_D0/umask-file" 
-
-echo "== stage from tmpfile"
-DEST_FILE="$T_D0/dest_file"
-stage_tmpfile $T_D0 $DEST_FILE
-hexdump -C "$DEST_FILE"
-rm -f "$DEST_FILE"
-
-t_pass

tests/tests/quorum-heartbeat-timeout.sh

@@ -1,89 +0,0 @@
-#
-# test that the quorum_heartbeat_time_ms option affects how long it
-# takes to recover from a failed mount.
-#
-
-t_require_mounts 2
-
-time_ms()
-{
-	# time_t in seconds, then trunate nanoseconds to 3 most dig digits
-	date +%s%3N
-}
-
-set_bad_timeout() {
-	local to="$1"
-	t_set_sysfs_mount_option 0 quorum_heartbeat_timeout_ms $to && \
-		t_fail "set bad q hb to $to"
-}
-
-set_quorum_timeouts()
-{
-	local to="$1"
-	local was
-	local is
-
-	for nr in $(t_quorum_nrs); do
-		local mnt="$(eval echo \$T_M$nr)"
-
-		was=$(t_get_sysfs_mount_option $nr quorum_heartbeat_timeout_ms)
-		t_set_sysfs_mount_option $nr quorum_heartbeat_timeout_ms $to
-		is=$(t_get_sysfs_mount_option $nr quorum_heartbeat_timeout_ms)
-
-		if [ "$is" != "$to" ]; then
-			t_fail "tried to set qhbto on $nr to $to but got $is"
-		fi
-	done
-}
-
-test_timeout()
-{
-	local to="$1"
-	local orig_to
-	local start
-	local nr
-	local delay
-
-	# set new timeouts, saving original
-	orig_to=$(t_get_sysfs_mount_option 0 quorum_heartbeat_timeout_ms)
-	set_quorum_timeouts $to
-
-	# give followers time to recv heartbeats and reset timeouts
-	sleep 1
-
-	# tear down the current server/leader
-	nr=$(t_server_nr)
-	t_force_umount $nr
-
-	# see how long it takes for the next leader to start
-	start=$(time_ms)
-	t_wait_for_leader
-	delay=$(($(time_ms) - start))
-
-	# kind of fun to have these logged
-	echo "to $to delay $delay" >> $T_TMP.delay
-
-	# restore the mount that we tore down
-	t_mount $nr
-
-	# reset the original timeouts
-	set_quorum_timeouts $orig_to
-
-	# make sure the new leader delay was reasonable
-	test "$delay" -gt "$to" || t_fail "delay $delay < to $to"
-	# allow 5 seconds of slop
-	test "$delay" -lt $(($to + 5000)) || t_fail "delay $delay > to $to + 5sec"
-}
-
-echo "== bad timeout values fail"
-set_bad_timeout 0
-set_bad_timeout -1
-set_bad_timeout 1000000
-
-echo "== test different timeouts"
-def=$(t_get_sysfs_mount_option 0 quorum_heartbeat_timeout_ms)
-test_timeout $def
-test_timeout 3000
-test_timeout $((def + 19000))
-
-t_pass

tests/tests/resize-devices.sh

@@ -73,7 +73,7 @@ echo "== make initial small fs"
 scoutfs mkfs -A -f -Q 0,127.0.0.1,53000 -m $quarter_meta -d $quarter_data \
 	"$T_EX_META_DEV" "$T_EX_DATA_DEV" > $T_TMP.mkfs.out 2>&1 || \
 		t_fail "mkfs failed"
-SCR="$T_TMPDIR/mnt.scratch"
+SCR="/mnt/scoutfs.enospc"
 mkdir -p "$SCR"
 mount -t scoutfs -o metadev_path=$T_EX_META_DEV,quorum_slot_nr=0 \
 	"$T_EX_DATA_DEV" "$SCR"

tests/tests/simple-inode-index.sh

@@ -103,34 +103,4 @@ while [ "$nr" -lt 100 ]; do
 	((nr++))
 done
 
-#
-# make sure rapid concurrent metadata updates don't create multiple
-# meta_seq entries
-#
-# we had a bug where deletion items created under concurrent_write locks
-# could get versions older than the items they're deleting which were
-# protected by read/write locks.
-#
-echo "== concurrent update attempts maintain single entries"
-FILES=4
-nr=1
-while [ "$nr" -lt 10 ]; do
-	# touch a bunch of files in parallel from all mounts
-	for i in $(t_fs_nrs); do
-		eval path="\$T_D${i}"
-		seq -f "$path/file-%.0f" 1 $FILES | xargs touch &
-	done
-	wait || t_fail "concurrent file updates failed"
-
-	# make sure no inodes have duplicate entries
-	sync
-	scoutfs walk-inodes -p "$T_D0" meta_seq -- 0 -1 | \
-		grep -v "minor" | \
-		awk '{print $4}' | \
-		sort -n | uniq -c | \
-		awk '($1 != 1)' | \
-		sort -n
-	((nr++))
-done
-
 t_pass

tests/tests/simple-xattr-unit.sh

@@ -36,8 +36,7 @@ test_xattr_lengths() {
 	else
 		echo "$name=\"$val\"" > "$T_TMP.good"
 	fi
-	cmp "$T_TMP.good" "$T_TMP.got" || \
-		t_fail "cmp failed name len $name_len val len $val_len"
+	cmp "$T_TMP.good" "$T_TMP.got" || exit 1
 
 	setfattr -x $name "$FILE"
 }

tests/tests/stage-tmpfile.sh

@@ -0,0 +1,15 @@
+#
+# Run tmpfile_stage and check the output with hexdump.
+#
+
+t_require_commands stage_tmpfile hexdump
+
+DEST_FILE="$T_D0/dest_file"
+
+stage_tmpfile $T_D0 $DEST_FILE
+
+hexdump -C "$DEST_FILE"
+
+rm -fr "$DEST_FILE"
+
+t_pass

tests/tests/xfstests.sh

@@ -64,22 +64,36 @@ generic/029	# mmap missing
 generic/030	# mmap missing
 generic/075	# file content mismatch failures (fds, etc)
 generic/080	# mmap missing
+generic/091	# skip fsx tests
+generic/094	# odirect streaming pre-alloc treated as failure in xfstests
 generic/103	# enospc causes trans commit failures
+generic/105	# needs trigage: something about acls
 generic/108	# mount fails on failing device?
 generic/112	# file content mismatch failures (fds, etc)
+generic/113	# block aio dio runs
+generic/114	# block aio dio runs
 generic/120	# (can't exec 'cause no mmap)
 generic/126	# (can't exec 'cause no mmap)
 generic/141	# mmap missing
+generic/198	# block aio dio runs
+generic/207	# block aio dio runs
+generic/210	# block aio dio runs
 generic/213	# enospc causes trans commit failures
 generic/215	# mmap missing
+generic/225	# odirect streaming pre-alloc treated as failure in xfstests
+generic/237	# wrong error return from failing setfacl?
+generic/240	# block aio dio runs
 generic/246	# mmap missing
 generic/247	# mmap missing
 generic/248	# mmap missing
+generic/263	# do not support allocate mode FALLOC_FL_PUNCH_HOLE, FALLOC_FL_KEEP_SIZE, FALLOC_FL_ZERO_RANGE...
+generic/319	# utils output change?  update branch?
 generic/321	# requires selinux enabled for '+' in ls?
 generic/325	# mmap missing
 generic/338	# BUG_ON update inode error handling
 generic/346	# mmap missing
 generic/347	# _dmthin_mount doesn't work?
+generic/375	# utils output change?  update branch?
 EOF
 
 t_restore_output

utils/fenced/scoutfs-fenced

@@ -55,21 +55,9 @@ test -x "$SCOUTFS_FENCED_RUN" || \
 	error_exit "SCOUTFS_FENCED_RUN '$SCOUTFS_FENCED_RUN' isn't executable"
 
 #
-# Main loop watching for fence request across all filesystems.   The
-# server can shut down without waiting for pending fence requests to
-# finish.  All of the interaction with the fence directory and files can
-# fail at any moment.  We will generate log messages when the dir or
-# files disappear.
+# main loop watching for fence request across all filesystems 
 #
 
-# generate failure messages to stderr while still echoing 0 for the caller
-careful_cat()
-{
-	local path="$@"
-
-	cat "$@" || echo 0
-}
-
 while sleep $SCOUTFS_FENCED_DELAY; do
 	for fence in /sys/fs/scoutfs/*/fence/*; do
 		# catches unmatched regex when no dirs
@@ -78,8 +66,7 @@ while sleep $SCOUTFS_FENCED_DELAY; do
 		fi
 
 		# skip requests that have been handled
-		if [ "$(careful_cat $fence/fenced)" == 1 -o \
-		     "$(careful_cat $fence/error)" == 1 ]; then
+		if [ $(cat "$fence/fenced") == 1 -o $(cat "$fence/error") == 1 ]; then
 			continue
 		fi
 
@@ -94,10 +81,10 @@ while sleep $SCOUTFS_FENCED_DELAY; do
 		export SCOUTFS_FENCED_REQ_RID="$rid"
 		export SCOUTFS_FENCED_REQ_IP="$ip"
 
-		$SCOUTFS_FENCED_RUN $SCOUTFS_FENCED_RUN_ARGS
+		$run $SCOUTFS_FENCED_RUN_ARGS
 		rc=$?
 		if [ "$rc" != 0 ]; then
-			log_message "server $srv fencing rid $rid saw error status $rc"
+			log_message "server $srv fencing rid $rid saw error status $rc from $run"
 			echo 1 > "$fence/error"
 			continue
 		fi

utils/man/scoutfs.5

@@ -15,61 +15,12 @@ general mount options described in the
 .BR mount (8)
 manual page.
 .TP
-.B acl
-The acl mount option enables support for POSIX Access Control Lists
-as detailed in
-.BR acl (5) .
-Support for POSIX ACLs is the default.
-.TP
-.B data_prealloc_blocks=<blocks>
-Set the size of preallocation regions of data files, in 4KiB blocks.
-Writes to these regions that contain no extents will attempt to
-preallocate the size of the full region.  This can waste a lot of space
-with small files, files with sparse regions, and files whose final
-length isn't a multiple of the preallocation size.  The following
-data_prealloc_contig_only option, which is the default, restricts this
-behaviour to waste less space.
-.sp
-All the preallocation options can be changed in an active mount by
-writing to their respective files in the options directory in the
-mount's sysfs directory.
-.sp
-It is worth noting that it is always more efficient in every way to use 
-.BR fallocate (2)
-to precisely allocate large extents for the resulting size of the file.
-Always attempt to enable it in software that supports it.
-.TP
-.B data_prealloc_contig_only=<0|1>
-This option, currently the default, limits file data preallocation in
-two ways.  First, it will only preallocate when extending a fully
-allocated file.  Second, it will limit the size of preallocation to the
-existing length of the file.  These limits reduce the amount of
-preallocation wasted per file at the cost of multiple initial extents in
-all files.  It only supports simple streaming writes, any other write
-pattern will not be recognized and could result in many fragmented
-extent allocations.
-.sp
-This option can be disabled to encourage large allocated extents
-regardless of write patterns.  This can be helpful if files are written
-with initial sparse regions (perhaps by multiple threads writing to
-different regions) and wasted space isn't an issue (perhaps because the
-file population contains few small files).
-.TP
 .B metadev_path=<device>
 The metadev_path option specifies the path to the block device that
 contains the filesystem's metadata.
 .sp
 This option is required.
 .TP
-.B noacl
-The noacl mount option disables the default support for POSIX Access
-Control Lists.  Any existing system.posix_acl_default and
-system.posix_acl_access extended attributes remain in inodes.   They
-will appear in listings from
-.BR listxattr (5)
-but specific retrieval or reomval operations will fail.  They will be
-used for enforcement again if ACL support is later enabled.
-.TP
 .B orphan_scan_delay_ms=<number>
 This option sets the average expected delay, in milliseconds, between
 each mount's scan of the global orphaned inode list.  Jitter is added to
@@ -85,25 +36,6 @@ the options directory in the mount's sysfs directory.  Writing a new
 value will cause the next pending orphan scan to be rescheduled
 with the newly written delay time.
 .TP
-.B quorum_heartbeat_timeout_ms=<number>
-This option sets the amount of time, in milliseconds, that a quorum
-member will wait without receiving heartbeat messages from the current
-leader before trying to take over as leader.  This setting is per-mount
-and only changes the behavior of that mount.
-.sp
-This determines how long it may take before a failed leader is replaced
-by a waiting quorum member.  Setting it too low may lead to spurious
-fencing as active leaders are prematurely replaced due to task or
-network delays that prevent the quorum members from promptly sending and
-receiving messages.  The ideal setting is the longest acceptable
-downtime during server failover.  The default is 10000 (10s) and it can
-not be less than 2000 greater than 60000.
-.sp
-This option can be changed in an active mount by writing to its file in
-the options directory in the mount's sysfs directory.  Writing a new
-value will take effect the next time the quorum agent receives a
-heartbeat message and sets the next timeout.
-.TP
 .B quorum_slot_nr=<number>
 The quorum_slot_nr option assigns a quorum member slot to the mount.
 The mount will use the slot assignment to claim exclusive ownership of

utils/man/scoutfs.8

@@ -76,97 +76,6 @@ run when the file system will not be mounted.
 .RE
 .PD
 
-.TP
-.BI "counters [-t|--table] SYSFS-DIR"
-.sp
-Display the counters and their values for a mounted ScoutFS filesystem.
-.RS 1.0i
-.PD 0
-.sp
-.TP
-.B SYSFS-DIR
-The mount's sysfs directory in which to find the
-.B counters/
-directory when then contains files for each counter.
-The sysfs directory is
-of the form
-.I /sys/fs/scoutfs/f.<fsid>.r.<rid>/
-\&.
-.TP
-.B "-t, --table"
-Format the counters into a columnar table that fills the width of the display
-instead of printing one counter per line.
-.RE
-.PD
-
-.TP
-.BI "data-waiting {-I|--inode} INODE-NUM {-B|--block} BLOCK-NUM [-p|--path PATH]"
-.sp
-Display all the files and blocks for which there is a task blocked waiting on
-offline data.
-.sp
-The results are sorted by the file's inode number and the
-logical block offset that is being waited on.
-.sp
-Each line of output describes a block in a file that has a task waiting
-and is formatted as:
-.I "ino <nr> iblock <nr> ops [str]"
-\&. The ops string indicates blocked operations seperated by commas and can
-include
-.B read
-for a read operation,
-.B write
-for a write operation, and
-.B change_size
-for a truncate or extending write.
-.RS 1.0i
-.PD 0
-.sp
-.TP
-.B "-I, --inode INODE-NUM"
-Start iterating over waiting tasks from the given inode number.
-Value of 0 will show all waiting tasks.
-.TP
-.B "-B, --block BLOCK-NUM"
-Start iterating over waiting tasks from the given logical block number
-in the starting inode.  Value of 0 will show blocks in the first inode
-and then continue to show all blocks with tasks waiting in all the
-remaining inodes.
-.TP
-.B "-p, --path PATH"
-A path within a ScoutFS filesystem.
-.RE
-.PD
-
-.TP
-.BI "data-wait-err {-I|--inode} INODE-NUM {-V|--version} VER-NUM {-F|--offset} OFF-NUM {-C|--count} COUNT {-O|--op} OP {-E|--err} ERR [-p|--path PATH]"
-.sp
-Return error from matching waiters.
-.RS 1.0i
-.PD 0
-.sp
-.TP
-.B "-C, --count COUNT"
-Count.
-.TP
-.B "-E, --err ERR"
-Error.
-.TP
-.B "-F, --offset OFF-NUM"
-Offset. May be expressed in bytes, or with KMGTP (Kibi, Mibi, etc.) size
-suffixes.
-.TP
-.B "-I, --inode INODE-NUM"
-Inode number.
-.TP
-.B "-O, --op OP"
-Operation. One of: "read", "write", "change_size".
-.TP
-.B "-p, --path PATH"
-A path within a ScoutFS filesystem.
-.RE
-.PD
-
 .TP
 .BI "df [-h|--human-readable] [-p|--path PATH]"
 .sp
@@ -184,72 +93,6 @@ A path within a ScoutFS filesystem.
 .RE
 .PD
 
-.TP
-.BI "get-allocated-inos [-i|--ino INO] [-s|--single] [-p|--path PATH]"
-.sp
-This debugging command prints allocated inode numbers.  It only prints
-inodes
-found in the group that contains the starting inode.  The printed inode
-numbers aren't necessarily reachable.  They could be anywhere in the
-process from being unlinked to finally deleted when their items
-were found.
-.RS 1.0i
-.PD 0
-.TP
-.sp
-.B "-i, --ino INO"
-The first 64bit inode number which could be printed.
-.TP
-.B "-s, --single"
-Only print the single starting inode when it is allocated, all other allocated
-inode numbers will be ignored.
-.TP
-.B "-p, --path PATH"
-A path within a ScoutFS filesystem.
-.RE
-.PD
-
-.TP
-.BI "ino-path INODE-NUM [-p|--path PATH]"
-.sp
-Display all paths that reference an inode number.
-.sp
-Ongoing filesystem changes, such as renaming a common parent of multiple paths,
-can cause displayed paths to be inconsistent.
-.RS 1.0i
-.PD 0
-.sp
-.TP
-.B "INODE-NUM"
-The inode number of the target inode.
-.TP
-.B "-p|--path PATH"
-A path within a ScoutFS filesystem.
-.RE
-.PD
-
-.TP
-.BI "list-hidden-xattrs FILE"
-.sp
-Display extended attributes starting with the
-.BR scoutfs.
-prefix and containing the
-.BR hide.
-tag
-which makes them invisible to
-.BR listxattr (2) .
-The names of each attribute are output, one per line.  Their order
-is not specified.
-.RS 1.0i
-.PD 0
-.TP
-.sp
-.B "FILE"
-The path to a file within a ScoutFS filesystem.  File permissions must allow
-reading.
-.RE
-.PD
-
 .TP
 .BI "mkfs META-DEVICE DATA-DEVICE {-Q|--quorum-slot} NR,ADDR,PORT [-m|--max-meta-size SIZE] [-d|--max-data-size SIZE] [-z|--data-alloc-zone-blocks BLOCKS] [-f|--force] [-A|--allow-small-size] [-V|--format-version VERS]"
 .sp
@@ -328,79 +171,6 @@ The range of supported versions is visible in the output of
 .RE
 .PD
 
-.TP
-.BI "prepare-empty-data-device {-c|--check} META-DEVICE DATA-DEVICE"
-.sp
-Prepare an unused device for use as the data device for an existing file
-system.  This will write an initialized super block to the specified
-data device, destroying any existing contents.  The specified metadata
-device will not be modified.  The file system must be fully unmounted
-and any client mount recovery must be complete.
-.sp
-The existing metadata device is read to ensure that it's safe to stop
-using the old data device.  The data block allocators must indicate that
-all data blocks are free.  If there are still data blocks referenced by
-files then the command will fail.  The contents of these files must be
-freed for the command to proceed.
-.sp
-A new super block is written to the new data device.  The device can
-then be used as the data device to mount the file system.  As this
-switch is made all client mounts must refer to the new device.  The old
-device is not modified and still contains a valid data super block that
-could be mounted, creating data device writes that wouldn't be read by
-mounts using the new device.
-.sp
-The number of data blocks available to the file system will not change
-as the new data device is used.  The new device must be large enough to
-store all the data blocks that were available on the old device.  If the
-new device is larger then its added capacity can be used by growing the
-new data device with the resize-devices command once it is mounted.
-.RS 1.0i
-.PD 0
-.TP
-.sp
-.B "-c, --check"
-Only check for errors that would prevent a new empty data device from
-being used.  No changes will be made to the data device.  If the data
-device is provided then its size will be checked to make sure that it is
-large enough.  This can be used to test the metadata for data references
-before destroying an old empty data device.
-.RE
-.PD
-
-.TP
-.BI "print {-S|--skip-likely-huge} META-DEVICE"
-.sp
-Prints out all of the metadata in the file system.  This makes no effort
-to ensure that the structures are consistent as they're traversed and
-can present structures that seem corrupt as they change as they're
-output.
-.RS 1.0i
-.PD 0
-.TP
-.sp
-.B "-S, --skip-likely-huge"
-Skip printing structures that are likely to be very large.  The
-structures that are skipped tend to be global and whose size tends to be
-related to the size of the volume.   Examples of skipped structures include
-the global fs items, srch files, and metadata and data
-allocators.  Similar structures that are not skipped are related to the
-number of mounts and are maintained at a relatively reasonable size.
-These include per-mount log trees, srch files, allocators, and the
-metadata allocators used by server commits.
-.sp
-Skipping the larger structures limits the print output to a relatively
-constant size rather than being a large multiple of the used metadata
-space of the volume making the output much more useful for inspection.
-.TP
-.B "META-DEVICE"
-The path to the metadata device for the filesystem whose metadata will be
-printed.  An attempt will be made to flush the host's buffer cache for
-this device with the BLKFLSBUF ioctl, or with posix_fadvise() if
-the path refers to a regular file.
-.RE
-.PD
-
 .TP
 .BI "resize-devices [-p|--path PATH] [-m|--meta-size SIZE] [-d|--data-size SIZE]"
 .sp
@@ -459,92 +229,6 @@ kibibytes, mebibytes, etc.
 .RE
 .PD
 
-.TP
-.BI "search-xattrs XATTR-NAME [-p|--path PATH]"
-.sp
-Display the inode numbers of inodes in the filesystem which may have
-an extended attribute with the given name.
-.sp
-The results may contain false positives.  The returned inode numbers
-should be checked to verify that the extended attribute is in fact
-present on the inode.
-.RS 1.0i
-.PD 0
-.TP
-.sp
-.B XATTR-NAME
-The full name of the extended attribute to search for as
-described in the
-.BR xattr (7)
-manual page.
-.TP
-.B "-p|--path PATH"
-A path within a ScoutFS filesystem.
-.RE
-.PD
-
-.TP
-.BI "setattr FILE [-d, --data-version=VERSION [-s, --size=SIZE [-o, --offline]]] [-t, --ctime=TIMESPEC]"
-.sp
-Set ScoutFS-specific attributes on a newly created zero-length file.
-.RS 1.0i
-.PD 0
-.sp
-.TP
-.B "-V, --data-version=VERSION"
-Set data version.
-.TP
-.B "-o, --offline"
-Set file contents as offline, not sparse. Requires
-.I --size
-option also be present.
-.TP
-.B "-s, --size=SIZE"
-Set file size. May be expressed in bytes, or with
-KMGTP (Kibi, Mibi, etc.) size suffixes. Requires
-.I --data-version
-option also be present.
-.TP
-.B "-t, --ctime=TIMESPEC"
-Set creation time using
-.I "<seconds-since-epoch>.<nanoseconds>"
-format.
-.RE
-.PD
-
-.TP
-.BI "stage ARCHIVE-FILE FILE {-V|--version} VERSION [-o, --offset OFF-NUM] [-l, --length LENGTH]"
-.sp
-.B Stage
-(i.e. return to online) the previously-offline contents of a file by copying a
-region from another file, the archive, and without updating regular inode
-metadata.  Any operations that are blocked by the existence of an offline
-region will proceed once the region has been staged.
-.RS 1.0i
-.PD 0
-.TP
-.sp
-.B "ARCHIVE-FILE"
-The source file for the file contents being staged.
-.TP
-.B "FILE"
-The regular file whose contents will be staged.
-.TP
-.B "-V, --version VERSION"
-The data_version of the contents to be staged.  It must match the
-current data_version of the file.
-.TP
-.B "-o, --offset OFF-NUM"
-The starting byte offset of the region to write.  May be expressed in bytes, or with
-KMGTP (Kibi, Mibi, etc.) size suffixes. Default is 0.
-.TP
-.B "-l, --length LENGTH"
-Length of range (bytes or KMGTP units) of file to stage. Default is the file's
-total size.
-.RE
-.PD
-
-.TP
 .BI "stat FILE [-s|--single-field FIELD-NAME]"
 .sp
 Display ScoutFS-specific metadata fields for the given file.
@@ -630,6 +314,221 @@ The total number of 4K data blocks in the filesystem.
 .RE
 .PD
 
+.TP
+.BI "counters [-t|--table] SYSFS-DIR"
+.sp
+Display the counters and their values for a mounted ScoutFS filesystem.
+.RS 1.0i
+.PD 0
+.sp
+.TP
+.B SYSFS-DIR
+The mount's sysfs directory in which to find the
+.B counters/
+directory when then contains files for each counter.
+The sysfs directory is
+of the form
+.I /sys/fs/scoutfs/f.<fsid>.r.<rid>/
+\&.
+.TP
+.B "-t, --table"
+Format the counters into a columnar table that fills the width of the display
+instead of printing one counter per line.
+.RE
+.PD
+
+.TP
+.BI "search-xattrs XATTR-NAME [-p|--path PATH]"
+.sp
+Display the inode numbers of inodes in the filesystem which may have
+an extended attribute with the given name.
+.sp
+The results may contain false positives.  The returned inode numbers
+should be checked to verify that the extended attribute is in fact
+present on the inode.
+.RS 1.0i
+.PD 0
+.TP
+.sp
+.B XATTR-NAME
+The full name of the extended attribute to search for as
+described in the
+.BR xattr (7)
+manual page.
+.TP
+.B "-p|--path PATH"
+A path within a ScoutFS filesystem.
+.RE
+.PD
+
+.TP
+.BI "list-hidden-xattrs FILE"
+.sp
+Display extended attributes starting with the
+.BR scoutfs.
+prefix and containing the
+.BR hide.
+tag
+which makes them invisible to
+.BR listxattr (2) .
+The names of each attribute are output, one per line.  Their order
+is not specified.
+.RS 1.0i
+.PD 0
+.TP
+.sp
+.B "FILE"
+The path to a file within a ScoutFS filesystem.  File permissions must allow
+reading.
+.RE
+.PD
+
+.TP
+.BI "walk-inodes {meta_seq|data_seq} FIRST-INODE LAST-INODE [-p|--path PATH]"
+.sp
+Walk an inode index in the file system and output the inode numbers
+that are found between the first and last positions in the index.
+.RS 1.0i
+.PD 0
+.sp
+.TP
+.BR meta_seq , data_seq
+Which index to walk.
+.TP
+.B "FIRST-INODE"
+An integer index value giving starting position of the index walk.
+.I 0
+is the first possible position.
+.TP
+.B "LAST-INODE"
+An integer index value giving the last position to include in the index walk.
+.I \-1
+can be given to indicate the last possible position.
+.TP
+.B "-p|--path PATH"
+A path within a ScoutFS filesystem.
+.RE
+.PD
+
+.TP
+.BI "ino-path INODE-NUM [-p|--path PATH]"
+.sp
+Display all paths that reference an inode number.
+.sp
+Ongoing filesystem changes, such as renaming a common parent of multiple paths,
+can cause displayed paths to be inconsistent.
+.RS 1.0i
+.PD 0
+.sp
+.TP
+.B "INODE-NUM"
+The inode number of the target inode.
+.TP
+.B "-p|--path PATH"
+A path within a ScoutFS filesystem.
+.RE
+.PD
+
+.TP
+.BI "data-waiting {-I|--inode} INODE-NUM {-B|--block} BLOCK-NUM [-p|--path PATH]"
+.sp
+Display all the files and blocks for which there is a task blocked waiting on
+offline data.
+.sp
+The results are sorted by the file's inode number and the
+logical block offset that is being waited on.
+.sp
+Each line of output describes a block in a file that has a task waiting
+and is formatted as:
+.I "ino <nr> iblock <nr> ops [str]"
+\&. The ops string indicates blocked operations seperated by commas and can
+include
+.B read
+for a read operation,
+.B write
+for a write operation, and
+.B change_size
+for a truncate or extending write.
+.RS 1.0i
+.PD 0
+.sp
+.TP
+.B "-I, --inode INODE-NUM"
+Start iterating over waiting tasks from the given inode number.
+Value of 0 will show all waiting tasks.
+.TP
+.B "-B, --block BLOCK-NUM"
+Start iterating over waiting tasks from the given logical block number
+in the starting inode.  Value of 0 will show blocks in the first inode
+and then continue to show all blocks with tasks waiting in all the
+remaining inodes.
+.TP
+.B "-p, --path PATH"
+A path within a ScoutFS filesystem.
+.RE
+.PD
+
+.TP
+.BI "data-wait-err {-I|--inode} INODE-NUM {-V|--version} VER-NUM {-F|--offset} OFF-NUM {-C|--count} COUNT {-O|--op} OP {-E|--err} ERR [-p|--path PATH]"
+.sp
+Return error from matching waiters.
+.RS 1.0i
+.PD 0
+.sp
+.TP
+.B "-C, --count COUNT"
+Count.
+.TP
+.B "-E, --err ERR"
+Error.
+.TP
+.B "-F, --offset OFF-NUM"
+Offset. May be expressed in bytes, or with KMGTP (Kibi, Mibi, etc.) size
+suffixes.
+.TP
+.B "-I, --inode INODE-NUM"
+Inode number.
+.TP
+.B "-O, --op OP"
+Operation. One of: "read", "write", "change_size".
+.TP
+.B "-p, --path PATH"
+A path within a ScoutFS filesystem.
+.RE
+.PD
+
+.TP
+.BI "stage ARCHIVE-FILE FILE {-V|--version} VERSION [-o, --offset OFF-NUM] [-l, --length LENGTH]"
+.sp
+.B Stage
+(i.e. return to online) the previously-offline contents of a file by copying a
+region from another file, the archive, and without updating regular inode
+metadata.  Any operations that are blocked by the existence of an offline
+region will proceed once the region has been staged.
+.RS 1.0i
+.PD 0
+.TP
+.sp
+.B "ARCHIVE-FILE"
+The source file for the file contents being staged.
+.TP
+.B "FILE"
+The regular file whose contents will be staged.
+.TP
+.B "-V, --version VERSION"
+The data_version of the contents to be staged.  It must match the
+current data_version of the file.
+.TP
+.B "-o, --offset OFF-NUM"
+The starting byte offset of the region to write.  May be expressed in bytes, or with
+KMGTP (Kibi, Mibi, etc.) size suffixes. Default is 0.
+.TP
+.B "-l, --length LENGTH"
+Length of range (bytes or KMGTP units) of file to stage. Default is the file's
+total size.
+.RE
+.PD
+
 .TP
 .BI "release FILE {-V|--version} VERSION [-o, --offset OFF-NUM] [-l, --length LENGTH]"
 .sp
@@ -669,28 +568,76 @@ total size.
 .PD
 
 .TP
-.BI "walk-inodes {meta_seq|data_seq} FIRST-INODE LAST-INODE [-p|--path PATH]"
+.BI "setattr FILE [-d, --data-version=VERSION [-s, --size=SIZE [-o, --offline]]] [-t, --ctime=TIMESPEC]"
 .sp
-Walk an inode index in the file system and output the inode numbers
-that are found between the first and last positions in the index.
+Set ScoutFS-specific attributes on a newly created zero-length file.
 .RS 1.0i
 .PD 0
 .sp
 .TP
-.BR meta_seq , data_seq
-Which index to walk.
+.B "-V, --data-version=VERSION"
+Set data version.
 .TP
-.B "FIRST-INODE"
-An integer index value giving starting position of the index walk.
-.I 0
-is the first possible position.
+.B "-o, --offline"
+Set file contents as offline, not sparse. Requires
+.I --size
+option also be present.
 .TP
-.B "LAST-INODE"
-An integer index value giving the last position to include in the index walk.
-.I \-1
-can be given to indicate the last possible position.
+.B "-s, --size=SIZE"
+Set file size. May be expressed in bytes, or with
+KMGTP (Kibi, Mibi, etc.) size suffixes. Requires
+.I --data-version
+option also be present.
 .TP
-.B "-p|--path PATH"
+.B "-t, --ctime=TIMESPEC"
+Set creation time using
+.I "<seconds-since-epoch>.<nanoseconds>"
+format.
+.RE
+.PD
+
+.TP
+.BI "print META-DEVICE"
+.sp
+Prints out all of the metadata in the file system.  This makes no effort
+to ensure that the structures are consistent as they're traversed and
+can present structures that seem corrupt as they change as they're
+output.
+.RS 1.0i
+.PD 0
+.TP
+.sp
+.B "META-DEVICE"
+The path to the metadata device for the filesystem whose metadata will be
+printed.  Since this command reads via the host's buffer cache, it may not
+reflect the current blocks in the filesystem possibly written to the shared
+block devices from another host, unless
+.B blockdev \--flushbufs
+command is used first.
+.RE
+.PD
+
+.TP
+.BI "get-allocated-inos [-i|--ino INO] [-s|--single] [-p|--path PATH]"
+.sp
+This debugging command prints allocated inode numbers.  It only prints
+inodes
+found in the group that contains the starting inode.  The printed inode
+numbers aren't necessarily reachable.  They could be anywhere in the
+process from being unlinked to finally deleted when their items
+were found.
+.RS 1.0i
+.PD 0
+.TP
+.sp
+.B "-i, --ino INO"
+The first 64bit inode number which could be printed.
+.TP
+.B "-s, --single"
+Only print the single starting inode when it is allocated, all other allocated
+inode numbers will be ignored.
+.TP
+.B "-p, --path PATH"
 A path within a ScoutFS filesystem.
 .RE
 .PD

utils/src/dev.c

@@ -3,7 +3,6 @@
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <unistd.h>
-#include <fcntl.h>
 #include <sys/ioctl.h>
 #include <linux/fs.h>
 #include <errno.h>
@@ -12,10 +11,13 @@
 #include "sparse.h"
 #include "dev.h"
 
-int get_device_size(char *path, int fd, u64 *size_ret)
+int device_size(char *path, int fd,
+		u64 min_size, u64 max_size, bool allow_small_size,
+		char *use_type, u64 *size_ret)
 {
 	struct stat st;
 	u64 size;
+	char *target_type;
 	int ret;
 
 	if (fstat(fd, &st)) {
@@ -27,6 +29,7 @@ int get_device_size(char *path, int fd, u64 *size_ret)
 
 	if (S_ISREG(st.st_mode)) {
 		size = st.st_size;
+		target_type = "file";
 	} else if (S_ISBLK(st.st_mode)) {
 		if (ioctl(fd, BLKGETSIZE64, &size)) {
 			ret = -errno;
@@ -34,26 +37,13 @@ int get_device_size(char *path, int fd, u64 *size_ret)
 				path, strerror(errno), errno);
 			return ret;
 		}
+		target_type = "device";
 	} else {
 		fprintf(stderr, "path isn't regular or device file '%s'\n",
 			path);
 		return -EINVAL;
 	}
 
-	*size_ret = size;
-	return 0;
-}
-
-int limit_device_size(char *path, int fd, u64 min_size, u64 max_size, bool allow_small_size,
-		      char *use_type, u64 *size_ret)
-{
-	u64 size;
-	int ret;
-
-	ret = get_device_size(path, fd, &size);
-	if (ret < 0)
-		return ret;
-
 	if (max_size) {
 		if (size > max_size) {
 			printf("Limiting use of "BASE_SIZE_FMT
@@ -73,9 +63,9 @@ int limit_device_size(char *path, int fd, u64 min_size, u64 max_size, bool allow
 
 	if (size < min_size) {
 		fprintf(stderr,
-			BASE_SIZE_FMT" too small for min "
+			BASE_SIZE_FMT" %s too small for min "
 			BASE_SIZE_FMT" %s device%s\n",
-			BASE_SIZE_ARGS(size),
+			BASE_SIZE_ARGS(size), target_type,
 			BASE_SIZE_ARGS(min_size), use_type,
 			allow_small_size ? ", allowing with -A" : "");
 
@@ -113,44 +103,3 @@ char *size_str(u64 nr, unsigned size)
 
 	return suffixes[i];
 }
-
-/*
- * Try to flush the local read cache for a device.  This is only a best
- * effort as these interfaces don't block waiting to fully purge the
- * cache.  This is OK because it's used by cached readers that are known
- * to be racy anyway.
- */
-int flush_device(int fd)
-{
-	struct stat st;
-	int ret;
-
-	ret = fstat(fd, &st);
-	if (ret < 0) {
-		ret = -errno;
-		fprintf(stderr, "fstat failed: %s (%d)\n", strerror(errno), errno);
-		goto out;
-	}
-
-	if (S_ISREG(st.st_mode)) {
-		ret = posix_fadvise(fd, 0, st.st_size, POSIX_FADV_DONTNEED);
-		if (ret < 0) {
-			ret = -errno;
-			fprintf(stderr, "POSIX_FADV_DONTNEED failed: %s (%d)\n",
-				strerror(errno), errno);
-			goto out;
-		}
-
-	} else if (S_ISBLK(st.st_mode)) {
-		ret = ioctl(fd, BLKFLSBUF, 0);
-		if (ret < 0) {
-			ret = -errno;
-			fprintf(stderr, "BLKFLSBUF, failed: %s (%d)\n", strerror(errno), errno);
-			goto out;
-		}
-	}
-
-	ret = 0;
-out:
-	return ret;
-}

utils/src/dev.h

@@ -9,11 +9,10 @@
 #define SIZE_FMT "%llu (%.2f %s)"
 #define SIZE_ARGS(nr, sz) (nr), size_flt(nr, sz), size_str(nr, sz)
 
-int get_device_size(char *path, int fd, u64 *size_ret);
-int limit_device_size(char *path, int fd, u64 min_size, u64 max_size, bool allow_small_size,
-		      char *use_type, u64 *size_ret);
+int device_size(char *path, int fd,
+		u64 min_size, u64 max_size, bool allow_small_size,
+		char *use_type, u64 *size_ret);
 float size_flt(u64 nr, unsigned size);
 char *size_str(u64 nr, unsigned size);
-int flush_device(int fd);
 
 #endif

utils/src/mkfs.c

@@ -118,33 +118,6 @@ struct mkfs_args {
 	struct scoutfs_quorum_slot slots[SCOUTFS_QUORUM_MAX_SLOTS];
 };
 
-static int open_mkfs_dev(struct mkfs_args *args, char *path, mode_t mode, char *which)
-{
-	int ret;
-	int fd = -1;
-
-	fd = open(path, mode);
-	if (fd < 0) {
-		ret = -errno;
-		fprintf(stderr, "failed to open %s dev '%s': %s (%d)\n",
-			which, path, strerror(errno), errno);
-		goto out;
-	}
-
-	ret = flush_device(fd);
-	if (ret < 0)
-		goto out;
-
-	if (!args->force)
-		ret = check_bdev(fd, path, which);
-
-out:
-	if (ret < 0 && fd >= 0)
-		close(fd);
-
-	return ret ?: fd;
-}
-
 /*
  * Make a new file system by writing:
  *  - super blocks
@@ -183,17 +156,32 @@ static int do_mkfs(struct mkfs_args *args)
 	gettimeofday(&tv, NULL);
 	pseudo_random_bytes(&fsid, sizeof(fsid));
 
-	meta_fd = open_mkfs_dev(args, args->meta_device, O_RDWR | O_EXCL, "meta");
+	meta_fd = open(args->meta_device, O_RDWR | O_EXCL);
 	if (meta_fd < 0) {
-		ret = meta_fd;
+		ret = -errno;
+		fprintf(stderr, "failed to open '%s': %s (%d)\n",
+			args->meta_device, strerror(errno), errno);
 		goto out;
 	}
+	if (!args->force) {
+		ret = check_bdev(meta_fd, args->meta_device, "meta");
+		if (ret)
+			return ret;
+	}
 
-	data_fd = open_mkfs_dev(args, args->data_device, O_RDWR | O_EXCL, "data");
+	data_fd = open(args->data_device, O_RDWR | O_EXCL);
 	if (data_fd < 0) {
-		ret = data_fd;
+		ret = -errno;
+		fprintf(stderr, "failed to open '%s': %s (%d)\n",
+			args->data_device, strerror(errno), errno);
 		goto out;
 	}
+	if (!args->force) {
+		ret = check_bdev(data_fd, args->data_device, "data");
+		if (ret)
+			return ret;
+	}
+
 
 	super = calloc(1, SCOUTFS_BLOCK_SM_SIZE);
 	bt = calloc(1, SCOUTFS_BLOCK_LG_SIZE);
@@ -206,14 +194,14 @@ static int do_mkfs(struct mkfs_args *args)
 	}
 
 	/* minumum meta device size to make reserved blocks reasonably large */
-	ret = limit_device_size(args->meta_device, meta_fd, 64ULL * (1024 * 1024 * 1024),
-				args->max_meta_size, args->allow_small_size, "meta", &meta_size);
+	ret = device_size(args->meta_device, meta_fd, 64ULL * (1024 * 1024 * 1024),
+			  args->max_meta_size, args->allow_small_size, "meta", &meta_size);
 	if (ret)
 		goto out;
 
 	/* .. then arbitrarily the same minimum data device size */
-	ret = limit_device_size(args->data_device, data_fd, 64ULL * (1024 * 1024 * 1024),
-				args->max_data_size, args->allow_small_size, "data", &data_size);
+	ret = device_size(args->data_device, data_fd, 64ULL * (1024 * 1024 * 1024),
+			  args->max_data_size, args->allow_small_size, "data", &data_size);
 	if (ret)
 		goto out;
 

utils/src/prepare_empty_data_device.c

@@ -1,247 +0,0 @@
-#define _GNU_SOURCE /* O_DIRECT */
-#include <unistd.h>
-#include <stdbool.h>
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include <errno.h>
-#include <sys/time.h>
-#include <uuid/uuid.h>
-#include <fcntl.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <unistd.h>
-#include <assert.h>
-#include <sys/socket.h>
-#include <netinet/in.h>
-#include <arpa/inet.h>
-#include <ctype.h>
-#include <inttypes.h>
-#include <argp.h>
-
-#include "sparse.h"
-#include "cmd.h"
-#include "util.h"
-#include "format.h"
-#include "parse.h"
-#include "crc.h"
-#include "rand.h"
-#include "dev.h"
-#include "key.h"
-#include "bitops.h"
-#include "btree.h"
-#include "leaf_item_hash.h"
-#include "blkid.h"
-#include "quorum.h"
-
-struct prepare_empty_data_dev_args {
-	char *meta_device;
-	char *data_device;
-	bool check;
-};
-
-static int do_prepare_empty_data_dev(struct prepare_empty_data_dev_args *args)
-{
-	struct scoutfs_super_block *meta_super = NULL;
-	struct scoutfs_super_block *data_super = NULL;
-	char uuid_str[37];
-	int meta_fd = -1;
-	int data_fd = -1;
-	u64 data_blocks;
-	u64 data_size;
-	u64 in_use;
-	int ret;
-
-	ret = posix_memalign((void **)&data_super, SCOUTFS_BLOCK_SM_SIZE, SCOUTFS_BLOCK_SM_SIZE);
-	if (ret < 0) {
-		ret = -errno;
-		fprintf(stderr, "failed to allocate data super block: %s (%d)\n",
-			strerror(errno), errno);
-		goto out;
-	}
-
-	meta_fd = open(args->meta_device, O_DIRECT | O_SYNC | O_RDONLY | O_EXCL);
-	if (meta_fd < 0) {
-		ret = -errno;
-		fprintf(stderr, "failed to open meta device '%s': %s (%d)\n",
-			args->meta_device, strerror(errno), errno);
-		goto out;
-	}
-
-	ret = read_block_verify(meta_fd, SCOUTFS_BLOCK_MAGIC_SUPER, 0, SCOUTFS_SUPER_BLKNO,
-				SCOUTFS_BLOCK_SM_SHIFT, (void **)&meta_super);
-	if (ret) {
-		ret = -errno;
-		fprintf(stderr, "failed to read meta super block: %s (%d)\n",
-			strerror(errno), errno);
-		goto out;
-	}
-
-	ret = meta_super_in_use(meta_fd, meta_super);
-	if (ret < 0) {
-		if (ret == -EBUSY)
-			fprintf(stderr, "The filesystem must be fully recovered and cleanly unmounted to determine if the data device is empty.\n");
-		goto out;
-	}
-
-	in_use = (le64_to_cpu(meta_super->total_data_blocks) - SCOUTFS_DATA_DEV_START_BLKNO) -
-		 le64_to_cpu(meta_super->data_alloc.total_len);
-	if (in_use) {
-		fprintf(stderr, "Data block allocator metadata shows "SIZE_FMT" data blocks used by files.  They must be removed, truncated, or released before a new empty data device can be used.\n",
-		       SIZE_ARGS(in_use, SCOUTFS_BLOCK_SM_SIZE));
-		ret = -EINVAL;
-		goto out;
-	}
-
-	if (args->data_device) {
-		data_fd = open(args->data_device, O_DIRECT | O_EXCL |
-						  (args->check ? O_RDONLY : O_RDWR | O_SYNC));
-		if (data_fd < 0) {
-			ret = -errno;
-			fprintf(stderr, "failed to open data device '%s': %s (%d)\n",
-				args->data_device, strerror(errno), errno);
-			goto out;
-		}
-
-		ret = get_device_size(args->data_device, data_fd, &data_size);
-		if (ret < 0)
-			goto out;
-
-		data_blocks = data_size >> SCOUTFS_BLOCK_SM_SHIFT;
-
-		if (data_blocks < le64_to_cpu(meta_super->total_data_blocks)) {
-			fprintf(stderr, "new data device %s of size "BASE_SIZE_FMT" has %llu 4KiB blocks, it needs at least "SIZE_FMT" blocks.\n",
-			       args->data_device,
-			       BASE_SIZE_ARGS(data_size),
-			       data_blocks,
-			       SIZE_ARGS(le64_to_cpu(meta_super->total_data_blocks),
-					  SCOUTFS_BLOCK_SM_SIZE));
-			ret = -EINVAL;
-			goto out;
-		}
-	}
-
-	if (args->check) {
-		ret = 0;
-		goto out;
-	}
-
-	/* the data device superblock only needs fs identifying fields */
-	memset(data_super, 0, sizeof(struct scoutfs_super_block));
-	data_super->id = meta_super->id;
-	data_super->fmt_vers = meta_super->fmt_vers;
-	data_super->flags = meta_super->flags &~ cpu_to_le64(SCOUTFS_FLAG_IS_META_BDEV);
-	memcpy(data_super->uuid, meta_super->uuid,sizeof(data_super->uuid));
-	data_super->seq = meta_super->seq;
-	data_super->total_meta_blocks = meta_super->total_meta_blocks;
-	data_super->total_data_blocks = meta_super->total_data_blocks;
-
-	ret = write_block(data_fd, SCOUTFS_BLOCK_MAGIC_SUPER, meta_super->hdr.fsid, 1,
-			  SCOUTFS_SUPER_BLKNO, SCOUTFS_BLOCK_SM_SHIFT, &data_super->hdr);
-	if (ret < 0) {
-		ret = -errno;
-		fprintf(stderr, "Error writing super block to new data device '%s': %s (%d)\n",
-			args->data_device, strerror(errno), errno);
-		goto out;
-	}
-
-	uuid_unparse(meta_super->uuid, uuid_str);
-
-	printf("Successfully initialized empty data device for scoutfs filesystem:\n"
-	       "  meta device path:       %s\n"
-	       "  data device path:       %s\n"
-	       "  fsid:                   %llx\n"
-	       "  uuid:                   %s\n"
-	       "  format version:         %llu\n"
-	       "  64KB metadata blocks:   "SIZE_FMT"\n"
-	       "  4KB data blocks:        "SIZE_FMT"\n",
-		args->meta_device,
-	        args->data_device,
-		le64_to_cpu(meta_super->hdr.fsid),
-		uuid_str,
-		le64_to_cpu(meta_super->fmt_vers),
-		SIZE_ARGS(le64_to_cpu(meta_super->total_meta_blocks),
-			  SCOUTFS_BLOCK_LG_SIZE),
-		SIZE_ARGS(le64_to_cpu(meta_super->total_data_blocks),
-			  SCOUTFS_BLOCK_SM_SIZE));
-
-	ret = 0;
-out:
-	if (args->check) {
-		if (ret == 0)
-			printf("All checks passed.\n");
-		else
-			printf("Errors were found that must be addressed before a new empty data device could be prepared and used.\n");
-	}
-
-	if (meta_super)
-		free(meta_super);
-	if (data_super)
-		free(data_super);
-	if (meta_fd != -1)
-		close(meta_fd);
-	if (data_fd != -1)
-		close(data_fd);
-	return ret;
-}
-
-static int parse_opt(int key, char *arg, struct argp_state *state)
-{
-	struct prepare_empty_data_dev_args *args = state->input;
-
-	switch (key) {
-	case 'c':
-		args->check = true;
-		break;
-	case ARGP_KEY_ARG:
-		if (!args->meta_device)
-			args->meta_device = strdup_or_error(state, arg);
-		else if (!args->data_device)
-			args->data_device = strdup_or_error(state, arg);
-		else
-			argp_error(state, "more than two device arguments given");
-		break;
-	case ARGP_KEY_FINI:
-		if (!args->meta_device)
-			argp_error(state, "no metadata device argument given");
-		if (!args->data_device && !args->check)
-			argp_error(state, "no data device argument given");
-		break;
-	default:
-		break;
-	}
-
-	return 0;
-}
-
-static struct argp_option options[] = {
-	{ "check", 'c', NULL, 0, "Only check for errors and do not write", },
-	{ NULL }
-};
-
-static struct argp argp = {
-	options,
-	parse_opt,
-	"META-DEVICE DATA-DEVICE",
-	"Prepare empty data device for use with an existing ScoutFS filesystem"
-};
-
-static int prepare_empty_data_dev_cmd(int argc, char *argv[])
-{
-	struct prepare_empty_data_dev_args prepare_empty_data_dev_args = { 
-		.check = false,
-	};
-	int ret;
-
-	ret = argp_parse(&argp, argc, argv, 0, NULL, &prepare_empty_data_dev_args);
-	if (ret)
-		return ret;
-
-	return do_prepare_empty_data_dev(&prepare_empty_data_dev_args);
-}
-
-static void __attribute__((constructor)) prepare_empty_data_dev_ctor(void)
-{
-	cmd_register_argp("prepare-empty-data-device", &argp, GROUP_CORE,
-			  prepare_empty_data_dev_cmd);
-}

utils/src/print.c

@@ -8,7 +8,6 @@
 #include <errno.h>
 #include <string.h>
 #include <stdarg.h>
-#include <stdbool.h>
 #include <ctype.h>
 #include <uuid/uuid.h>
 #include <sys/socket.h>
@@ -27,7 +26,6 @@
 #include "avl.h"
 #include "srch.h"
 #include "leaf_item_hash.h"
-#include "dev.h"
 
 static void print_block_header(struct scoutfs_block_header *hdr, int size)
 {
@@ -991,10 +989,9 @@ static void print_super_block(struct scoutfs_super_block *super, u64 blkno)
 
 struct print_args {
 	char *meta_device;
-	bool skip_likely_huge;
 };
 
-static int print_volume(int fd, struct print_args *args)
+static int print_volume(int fd)
 {
 	struct scoutfs_super_block *super = NULL;
 	struct print_recursion_args pa;
@@ -1044,26 +1041,23 @@ static int print_volume(int fd, struct print_args *args)
 			ret = err;
 	}
 
-	if (!args->skip_likely_huge) {
-		for (i = 0; i < array_size(super->meta_alloc); i++) {
-			snprintf(str, sizeof(str), "meta_alloc[%u]", i);
-			err = print_btree(fd, super, str, &super->meta_alloc[i].root,
-					  print_alloc_item, NULL);
-			if (err && !ret)
-				ret = err;
-		}
-
-		err = print_btree(fd, super, "data_alloc", &super->data_alloc.root,
+	for (i = 0; i < array_size(super->meta_alloc); i++) {
+		snprintf(str, sizeof(str), "meta_alloc[%u]", i);
+		err = print_btree(fd, super, str, &super->meta_alloc[i].root,
 				  print_alloc_item, NULL);
 		if (err && !ret)
 			ret = err;
 	}
 
+	err = print_btree(fd, super, "data_alloc", &super->data_alloc.root,
+			  print_alloc_item, NULL);
+	if (err && !ret)
+		ret = err;
+
 	err = print_btree(fd, super, "srch_root", &super->srch_root,
 			  print_srch_root_item, NULL);
 	if (err && !ret)
 		ret = err;
-
 	err = print_btree(fd, super, "logs_root", &super->logs_root,
 			  print_log_trees_item, NULL);
 	if (err && !ret)
@@ -1071,23 +1065,19 @@ static int print_volume(int fd, struct print_args *args)
 
 	pa.super = super;
 	pa.fd = fd;
-	if (!args->skip_likely_huge) {
-		err = print_btree_leaf_items(fd, super, &super->srch_root.ref,
-					     print_srch_root_files, &pa);
-		if (err && !ret)
-			ret = err;
-	}
+	err = print_btree_leaf_items(fd, super, &super->srch_root.ref,
+				     print_srch_root_files, &pa);
+	if (err && !ret)
+		ret = err;
 	err = print_btree_leaf_items(fd, super, &super->logs_root.ref,
 				     print_log_trees_roots, &pa);
 	if (err && !ret)
 		ret = err;
 
-	if (!args->skip_likely_huge) {
-		err = print_btree(fd, super, "fs_root", &super->fs_root,
-				  print_fs_item, NULL);
-		if (err && !ret)
-			ret = err;
-	}
+	err = print_btree(fd, super, "fs_root", &super->fs_root,
+			  print_fs_item, NULL);
+	if (err && !ret)
+		ret = err;
 
 out:
 	free(super);
@@ -1108,12 +1098,7 @@ static int do_print(struct print_args *args)
 		return ret;
 	}
 
-	ret = flush_device(fd);
-	if (ret < 0)
-		goto out;
-
-	ret = print_volume(fd, args);
-out:
+	ret = print_volume(fd);
 	close(fd);
 	return ret;
 };
@@ -1123,9 +1108,6 @@ static int parse_opt(int key, char *arg, struct argp_state *state)
 	struct print_args *args = state->input;
 
 	switch (key) {
-	case 'S':
-		args->skip_likely_huge = true;
-		break;
 	case ARGP_KEY_ARG:
 		if (!args->meta_device)
 			args->meta_device = strdup_or_error(state, arg);
@@ -1143,13 +1125,8 @@ static int parse_opt(int key, char *arg, struct argp_state *state)
 	return 0;
 }
 
-static struct argp_option options[] = {
-	{ "skip-likely-huge", 'S', NULL, 0, "Skip large structures to minimize output size"},
-	{ NULL }
-};
-
 static struct argp argp = {
-	options,
+	NULL,
 	parse_opt,
 	"META-DEV",
 	"Print metadata structures"