Abort pending net requests when a mount fails

A failed mount tears down with scoutfs_put_super(), which calls
scoutfs_srch_destroy() first.  srch_destroy() does cancel_work_sync() on
the srch compact worker, but that worker can be parked in an
uninterruptible scoutfs_net_sync_request() to a server that will never
respond (e.g. the server is stuck in recovery).  Nothing completes the
request: the forced-unmount drain in the net shutdown path only runs for
umount -f, and a failed mount never calls ->umount_begin, so the request
sits on the resend queue and cancel_work_sync() waits forever.

The result is an unkillable D-state mount that survives the SIGKILL a
mount timeout sends, and only clears on reboot:

  systemd[1]: data-archive.mount: Killing process 15740 (mount) with signal SIGKILL.
  systemd[1]: data-archive.mount: Mount process still around after SIGKILL. Ignoring.

  cat /proc/26717/stack
  [<0>] __flush_work+0x16f/0x240
  [<0>] __cancel_work_sync+0x135/0x1a0
  [<0>] scoutfs_srch_destroy+0x33/0x70 [scoutfs]
  [<0>] scoutfs_put_super+0x4f/0x1a0 [scoutfs]
  [<0>] scoutfs_fill_super+0x260/0x520 [scoutfs]
  [<0>] mount_bdev+0xf9/0x150
  [<0>] do_new_mount+0x17a/0x310
  [<0>] __x64_sys_mount+0x107/0x140

The worker it waits on, blocked in the sync request that never returns:

  task:kworker/u269:1  state:D
  Workqueue: scoutfs_srch_compact scoutfs_srch_compact_worker [scoutfs]
  Call Trace:
   __wait_for_common+0x90/0x1d0
   scoutfs_net_sync_request+0xdb/0xf0 [scoutfs]
   scoutfs_client_srch_get_compact+0x2e/0x40 [scoutfs]
   scoutfs_srch_compact_worker+0x64/0x3d0 [scoutfs]

On the fill_super error path, mark forced_unmount and shut the client
connection down before teardown.  That drains pending requests with
-ECONNABORTED so the worker returns and srch_destroy()'s cancel_work_sync
completes.  It is done for any failure, before the direct put_super call
and before returning to generic_shutdown_super (which calls put_super
when s_root was set), so both teardown paths are covered.  sbi is
allocated before any goto out, and scoutfs_client_net_shutdown() is a
no-op when the client or connection was never set up, so early failures
are safe.

Signed-off-by: Auke Kok <auke.kok@versity.com>

kmod/src/super.c

@@ -627,6 +627,19 @@ static int scoutfs_fill_super(struct super_block *sb, void *data, int silent)
 	scoutfs_trans_restart_sync_deadline(sb);
 	ret = 0;
 out:
+	if (ret) {
+		/*
+		 * The mount failed and we're about to tear down, either here
+		 * or via generic_shutdown_super if s_root was set. Any worker
+		 * started during fill_super can be blocked in an uninterruptible
+		 * net request to a server that will never respond. Force the
+		 * client connection down so those pending requests abort with
+		 * -ECONNABORTED before teardown.
+		 */
+		SCOUTFS_SB(sb)->forced_unmount = true;
+		scoutfs_client_net_shutdown(sb);
+	}
+
 	/* on error, generic_shutdown_super calls put_super if s_root */
 	if (ret && !sb->s_root)
 		scoutfs_put_super(sb);