Give the WebDav chunk reader a bounded, invalidatable location cache (#10801)

* mount: re-resolve volume locations after a failed chunk read

NewChunkGroup passed nil as the ReaderCache's CacheInvalidator, so
retryFetchAfterCacheInvalidation was dead code on the FUSE read path. A
mount that cached a volume's locations while one server was down kept
retrying that server after it died, then returned EIO, even though the
master and filer both resolved the live replica. The S3 gateway already
passes its filerClient; do the same for the mount.

* test: FUSE integration tests for volume server failover

One mount appends while a second tails, and a volume server is killed,
started or restarted mid-stream against a 001-replicated cluster of three
volume servers. Automates the scenario matrix reported for Docker Swarm
mounts, including the large-file variant and a no-chaos control.

* test: report the filer's own view when append content mismatches

A mismatch between what the writer wrote and what the reader sees can come
from either side's cache. Read the file back through the filer's HTTP
handler as well, and let the mount verbosity be raised from the
environment, so a failing run says which layer lost the data.

* test: wait for the reader mount to converge before comparing

A mount caches metadata for about a second, so reading the file the instant
the writer's last close returned can legitimately come back short. Poll the
reader until it matches or the timeout expires; content that is wrong rather
than merely late never converges and still fails, now with the writer's
mount and the filer's own view alongside it.

* test: detect a failover cluster child that exited at startup

Signal(0) succeeds for a zombie and nothing reaped these children until
shutdown, so a process that died on startup looked alive until the readiness
timeout expired. Reap each child as it is started and consult the result.

* test: read a file the killed volume server actually holds

Placement decides which two of three servers back each volume, so killing
volume N and reading readfile-N could pass without the victim ever holding a
replica of it. Resolve each file's volumes through the filer and the master,
and pick one the victim backs, preferring a file the reader has not cached.

* ci: stop persisting checkout credentials in the failover workflow

The job does not use the token after cloning. Also tag the README's command
block as bash and match the timeout the workflow actually uses.

* test: discard the ignored errors errcheck flags in the failover harness

* test: resolve manifests when mapping a file to its volumes

A manifest chunk's own fid names the volume holding the manifest, not the
volumes holding the data, so a large enough file would point the failover
victim at the wrong server.

* test: pin the stale-location recovery path with a primed reader

Reading a file for the first time after a server dies proves nothing: the
lookup is fresh and returns the survivor. Kill one holder and wait for the
master to drop it, read a file on that volume so the reader caches the lone
survivor, restart the first server, then kill the survivor. The reader's only
cached location is now dead while the data is live elsewhere, which is the
case the invalidator exists for: EIO without it, recovery with it.

* filer: re-look-up a chunk's locations as soon as they all fail

A read that fails against every location it was given is far more likely to be
holding a stale list than to be hitting a cluster that is briefly slow, but the
retry loops spent the whole backoff ladder, about 13 s, before the caller got a
chance to invalidate and look the chunk up again. Give the loops a refresh hook
and let the reader cache invalidate on the first fully failed pass, so recovery
starts in milliseconds. Clients without an invalidator keep the old behavior.

The filer's streaming read path has its own fetch loop and is not covered.

* webdav: give the chunk reader a bounded, invalidatable location cache

WebDav resolved chunk locations through filer.LookupFn, whose own doc asks
long-running processes to prefer wdclient.FilerClient: its cache is unbounded,
and it has no way to invalidate an entry, so the reader cache was constructed
with a nil invalidator and a WebDav server that had cached a location kept
reading from it after the volume moved or died. Use FilerClient, as the mount
and the S3 gateway already do.

* filer: refresh locations on the random-read path too

readChunkSliceAt bypasses the chunk cacher in random-access mode and fetches
the range directly, which left it without the invalidation the cacher does:
a random reader parked on a stale location had no way back at all. Hoist the
refresh hook onto the reader cache so both paths share it.

* filer: compare chunk locations as a set, not in order

Lookups shuffle the locations they return, so comparing positionally reads a
reshuffle of the very same replicas as a fresh set and spends an immediate
retry on locations that just failed. weed/filer already had an
order-independent comparison for this; move it next to the retry loops so
both callers share one helper.

weed/server/webdav_server.go

@@ -20,6 +20,7 @@ import (
 	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
 	"github.com/seaweedfs/seaweedfs/weed/util"
 	"github.com/seaweedfs/seaweedfs/weed/util/chunk_cache"
+	"github.com/seaweedfs/seaweedfs/weed/wdclient"
 
 	"github.com/seaweedfs/seaweedfs/weed/filer"
 	"github.com/seaweedfs/seaweedfs/weed/glog"
@@ -89,6 +90,7 @@ type WebDavFileSystem struct {
 	option      *WebDavOption
 	chunkCache  *chunk_cache.TieredChunkCache
 	readerCache *filer.ReaderCache
+	filerClient *wdclient.FilerClient
 	signature   int32
 }
 
@@ -140,7 +142,13 @@ func NewWebDavFileSystem(option *WebDavOption) (webdav.FileSystem, error) {
 		chunkCache: chunkCache,
 		signature:  util.RandomInt32(),
 	}
-	t.readerCache = filer.NewReaderCache(32, chunkCache, filer.LookupFn(t), nil)
+	// FilerClient rather than filer.LookupFn: its cache is bounded, which the
+	// LookupFn doc asks long-running processes to prefer, and it can invalidate
+	// a volume's locations, so a read that failed against every cached location
+	// looks the volume up again instead of retrying a server that has moved or
+	// died.
+	t.filerClient = wdclient.NewFilerClient([]pb.ServerAddress{option.Filer}, option.GrpcDialOption, "")
+	t.readerCache = filer.NewReaderCache(32, chunkCache, t.filerClient.GetLookupFileIdFunction(), t.filerClient)
 	return t, nil
 }
 
@@ -555,7 +563,7 @@ func (f *WebDavFile) Read(p []byte) (readSize int, err error) {
 		return 0, io.EOF
 	}
 	if f.visibleIntervals == nil {
-		f.visibleIntervals, _ = filer.NonOverlappingVisibleIntervals(f.ctx, filer.LookupFn(f.fs), f.entry.GetChunks(), 0, fileSize)
+		f.visibleIntervals, _ = filer.NonOverlappingVisibleIntervals(f.ctx, f.fs.filerClient.GetLookupFileIdFunction(), f.entry.GetChunks(), 0, fileSize)
 		f.reader = nil
 	}
 	if f.reader == nil {