mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-07-30 03:53:18 +00:00
Chris LuandGitHub
9a7cfaf371
* admin: let waiting shell clients win the admin lock The admin lock manager re-acquired the cluster shell lock immediately after releasing it, while a waiting weed shell client only polls the master once a second, so an operator could lose the race indefinitely. Leave the lock free for slightly more than one poll interval before re-acquiring, and once overlapping reference-counted holds have kept the lock continuously held for over a minute, make new admin-side acquires wait for a full release before piggybacking. * admin/plugin: take the shell lock per detection and per job, not per batch The default-lane scheduler acquired the cluster shell lock once and held it across the whole pass: every due job type's detection plus all of its dispatched jobs, each drained to completion. A manual weed shell operation sharing that lock could stall behind the batch for up to the extended execution window. Hold the lock only while it protects something: around each detection scan and around each dispatched job. Manual shell operations now wait for at most one in-flight job. The admin UI detect+execute path stops wrapping dispatch in an outer hold, since a nested acquire would deadlock against the lock manager's fairness window; detection and per-job dispatch take the lock themselves. * admin/plugin: stop extending the dispatch window to the largest job estimate When any proposal's estimated runtime exceeded the remaining JobTypeMaxRuntime, the whole dispatch context was replaced with a fresh one capped at eight hours, so a single balance backlog could hold the default lane (and with it erasure_coding and vacuum detection) for that long. Keep the dispatch window at JobTypeMaxRuntime and instead detach each started attempt onto its own estimated-runtime deadline. Large jobs still get their full time once started; jobs not yet started when the window closes are canceled and re-proposed by a later detection, so sibling job types get a turn every window. * worker/balance: re-check each planned move against the master before executing A balance plan is computed at detection time, but the admin lock is released between detection and execution, so a manual shell operation can rearrange the volume in the gap. The task's own guards catch a vanished source, but a target that gained a replica in the meantime would be silently overwritten by VolumeCopy and the source delete would then reduce the volume to a single copy. Before executing each move, ask the master for the volume's current locations (uncached) and skip the move if the volume has left the source or the target already holds a replica. Skipped moves fail with a stale-move error and the next detection replans them. Without master addresses in the cluster context the check is skipped, preserving the old behavior with older admins. * admin: block re-acquire while the final lock release is in flight Release dropped the manager mutex before calling ReleaseLock, so a concurrent Acquire could see hold count zero and call RequestLock while the locker still considered itself locked. That request no-ops, leaving a hold with no live master lease. Track the in-flight release and make Acquire wait for it. Also normalize a nil release function from lock manager implementations, and make the yield and fairness windows per-instance fields so tests stop mutating globals.
76 lines
2.2 KiB
Go
76 lines
2.2 KiB
Go
package pluginworker
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"strconv"
|
|
"time"
|
|
|
|
"github.com/seaweedfs/seaweedfs/weed/pb"
|
|
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
|
|
"google.golang.org/grpc"
|
|
)
|
|
|
|
// LookupVolumeLocations asks the masters for the current registered locations
|
|
// of a volume, bypassing any client-side caches. Execution handlers use it to
|
|
// re-check a proposal against the live topology, since a proposal can go
|
|
// stale between detection and execution. Returns the location URLs
|
|
// (host:port) the master reports; an empty slice means the volume is not
|
|
// registered anywhere.
|
|
func LookupVolumeLocations(ctx context.Context, masterAddresses []string, grpcDialOption grpc.DialOption, volumeID uint32) ([]string, error) {
|
|
if grpcDialOption == nil {
|
|
return nil, fmt.Errorf("grpc dial option is not configured")
|
|
}
|
|
if len(masterAddresses) == 0 {
|
|
return nil, fmt.Errorf("no master addresses provided in cluster context")
|
|
}
|
|
|
|
vid := strconv.FormatUint(uint64(volumeID), 10)
|
|
var lastErr error
|
|
for _, address := range masterAddresses {
|
|
for _, candidate := range MasterAddressCandidates(address) {
|
|
if ctx.Err() != nil {
|
|
return nil, ctx.Err()
|
|
}
|
|
|
|
dialCtx, cancelDial := context.WithTimeout(ctx, 5*time.Second)
|
|
conn, err := pb.GrpcDial(dialCtx, candidate, false, grpcDialOption)
|
|
cancelDial()
|
|
if err != nil {
|
|
lastErr = err
|
|
continue
|
|
}
|
|
|
|
client := master_pb.NewSeaweedClient(conn)
|
|
callCtx, cancelCall := context.WithTimeout(ctx, 10*time.Second)
|
|
resp, callErr := client.LookupVolume(callCtx, &master_pb.LookupVolumeRequest{
|
|
VolumeOrFileIds: []string{vid},
|
|
})
|
|
cancelCall()
|
|
_ = conn.Close()
|
|
if callErr != nil {
|
|
lastErr = callErr
|
|
continue
|
|
}
|
|
|
|
for _, vidLocation := range resp.VolumeIdLocations {
|
|
if vidLocation.VolumeOrFileId != vid {
|
|
continue
|
|
}
|
|
locations := make([]string, 0, len(vidLocation.Locations))
|
|
for _, location := range vidLocation.Locations {
|
|
locations = append(locations, location.Url)
|
|
}
|
|
return locations, nil
|
|
}
|
|
// The master answered but does not know the volume.
|
|
return nil, nil
|
|
}
|
|
}
|
|
|
|
if lastErr == nil {
|
|
lastErr = fmt.Errorf("no valid master address candidate")
|
|
}
|
|
return nil, lastErr
|
|
}
|