From d605feb40394c710825d4287cd442931f7183586 Mon Sep 17 00:00:00 2001 From: Chris Lu Date: Sun, 3 May 2026 21:46:21 -0700 Subject: [PATCH] refactor(command): expand "~" in all path-style CLI flags (#9306) * refactor(command): expand "~" in all path-style CLI flags Many of weed's path-bearing flags (-s3.config, -s3.iam.config, -admin.dataDir, -webdav.cacheDir, -volume.dir.idx, TLS cert/key files, profile output paths, mount cache dirs, sftp key files, ...) were never run through util.ResolvePath, so a value like "~/iam.json" was used literally. Tilde only worked when the shell expanded it, which silently fails for the common -flag=~/path form (bash leaves the tilde literal in --opt=~/path). - Extend util.ResolvePath to also handle "~user" / "~user/rest", matching shell tilde expansion. Add unit tests. - Apply util.ResolvePath at the top of each shared start* function (s3, webdav, sftp) so mini/server/filer/standalone callers all inherit it; resolve at the few one-off use sites (mount cache dirs, volume idx folder, mini admin.dataDir, profile paths). - Drop the duplicate expandHomeDir helper from admin.go in favor of the now-equivalent util.ResolvePath. * fixup: handle comma-separated -dir flags for tilde expansion `weed mini -dir`, `weed server -dir`, and `weed volume -dir` accept comma-separated paths (`dir[,dir]...`). Calling util.ResolvePath on the whole string mishandled multi-folder values with tilde, e.g. "~/d1,~/d2" would resolve as if "d1,~/d2" were a single subpath. - Add util.ResolveCommaSeparatedPaths: split on ",", run each entry through ResolvePath, rejoin. Short-circuits when no "~" present. - Use it for *miniDataFolders (mini.go), *volumeDataFolders (server.go), and resolve each entry of v.folders in-place (volume.go) so all downstream consumers see resolved paths. - Add 7-case TestResolveCommaSeparatedPaths covering empty, single, multiple, and mixed inputs. * address PR review: metaFolder + Windows backslash - master.go: resolve *m.metaFolder at the top of runMaster so util.FullPath(*m.metaFolder) on the next line sees an expanded path. Drop the now-redundant ResolvePath in TestFolderWritable. - server.go: same treatment for *masterOptions.metaFolder, paired with the existing cpu/mem profile resolves. Drop the redundant inner ResolvePath at TestFolderWritable. - file_util.go: ResolvePath now accepts filepath.Separator as a separator after the tilde, so "~\\data" works on Windows. Other platforms keep current behaviour (backslash stays literal because it is a valid filename character in usernames and paths). - file_util_test.go: add two cases using filepath.Separator that exercise the new code path on Windows and remain a no-op on Unix. * address PR review: resolve "~" in remaining command path flags Comprehensive sweep of path-bearing flags across every weed subcommand, applying util.ResolvePath in-place at the top of each run* function so all downstream consumers see expanded paths. - webdav.go: resolve *wo.cacheDir at the top of startWebDav so mini/server/filer/standalone callers all inherit it. - mount_std.go: cpu/mem profile paths. - filer_sync.go: cpu/mem profile paths. - mq_broker.go: cpu/mem profile paths. - benchmark.go: cpuprofile output path. - backup.go: -dir resolved once at runBackup; drop the duplicated inline ResolvePath in NewVolume calls. - compact.go: -dir resolved at runCompact; drop inline ResolvePath. - export.go: -dir and -o resolved at runExport; drop inline ResolvePath in LoadFromIdx and ScanVolumeFile. - download.go: -dir resolved at runDownload; drop inline. - update.go: -dir resolved at runUpdate so filepath.Join uses the expanded path; drop inline ResolvePath in TestFolderWritable. - scaffold.go: -output expanded before filepath.Join. - worker.go: -workingDir expanded before being passed to runtime. * address PR review: resolve option-struct paths at run* entry points server.go:381 propagates s3Options.config to filerOptions.s3ConfigFile *before* startS3Server runs, which meant the filer-side code saw the unresolved tilde-prefixed pointer. Same pattern for webdavOptions and sftpOptions (and equivalent in mini.go / filer.go). The fix: hoist resolution from the shared start* functions up to the run* entry points, where every shared pointer is set up before any propagation happens. - s3.go, webdav.go, sftp.go: extract a resolvePaths() method on each Options struct that runs every path field through util.ResolvePath in-place. Idempotent. - runS3, runWebDav, runSftp: call the standalone struct's resolvePaths before starting metrics / loading security config. - runServer, runMini, runFiler: call resolvePaths on every embedded options struct, plus resolve loose flags (serverIamConfig, miniS3Config, miniIamConfig, miniMasterOptions.metaFolder, and filer's defaultLevelDbDirectory) so they're expanded before any pointer copy or use. - Drop the now-redundant inline ResolvePath at filer's defaultLevelDbDirectory composition. * address PR review: re-resolve mini -dir post-config, cover misc paths - mini.go: applyConfigFileOptions can overwrite -dir with a literal ~/data from mini.options. Re-resolve *miniDataFolders after the config-file apply, alongside the other path resolves, so the mini filer no longer ends up with a literal ~/data/filerldb2. - benchmark.go: resolve *b.idListFile (-list). - filer_sync.go: resolve *syncOptions.aSecurity / .bSecurity (-a.security / -b.security) before LoadClientTLSFromFile. - filer_cat.go: resolve *filerCat.output (-o) before os.OpenFile. - admin.go: drop trailing blank line at EOF (git diff --check). * address PR review: resolve -a.security/-b.security/-config before use Three follow-up fixes: - filer_sync.go: the -a.security / -b.security resolves were placed *after* LoadClientTLSFromFile / LoadHTTPClientFromFile were called, so weed filer.sync -a.security=~/a.toml still passed the literal tilde path. Hoist the resolves above the security-loading block so TLS clients see expanded paths. - filer_sync_verify.go: same flag pair was never resolved at all in the verify command; resolve at the top of runFilerSyncVerify. - filer_meta_backup.go: -config (the backup_filer.toml path) was passed directly to viper. Resolve at the top of runFilerMetaBackup. - mini.go: master.dir defaulted to the entire comma-joined miniDataFolders. With weed mini -dir=~/d1,~/d2 (or any multi-dir setup), TestFolderWritable then stat'd the joined string instead of a single directory. Default to the first entry via StringSplit to mirror the disk-space calculation a few lines below, and drop the now-redundant ResolvePath in TestFolderWritable. --- weed/command/admin.go | 53 ++--------------------- weed/command/backup.go | 5 ++- weed/command/benchmark.go | 2 + weed/command/compact.go | 3 +- weed/command/download.go | 3 +- weed/command/export.go | 7 ++- weed/command/filer.go | 6 ++- weed/command/filer_cat.go | 1 + weed/command/filer_meta_backup.go | 2 + weed/command/filer_sync.go | 5 +++ weed/command/filer_sync_verify.go | 2 + weed/command/master.go | 5 ++- weed/command/mini.go | 27 +++++++++--- weed/command/mount_std.go | 9 ++-- weed/command/mq_broker.go | 2 + weed/command/s3.go | 14 ++++++ weed/command/scaffold.go | 2 +- weed/command/server.go | 10 ++++- weed/command/sftp.go | 9 ++++ weed/command/update.go | 3 +- weed/command/volume.go | 11 +++-- weed/command/webdav.go | 11 ++++- weed/command/worker.go | 2 + weed/util/file_util.go | 71 ++++++++++++++++++++++++++----- weed/util/file_util_test.go | 66 ++++++++++++++++++++++++++++ 25 files changed, 245 insertions(+), 86 deletions(-) diff --git a/weed/command/admin.go b/weed/command/admin.go index e29e83983..41ca40cde 100644 --- a/weed/command/admin.go +++ b/weed/command/admin.go @@ -11,7 +11,6 @@ import ( "net/http" "os" "os/signal" - "os/user" "path/filepath" "runtime/debug" "strings" @@ -173,6 +172,8 @@ func runAdmin(cmd *Command, args []string) bool { grace.StartDebugServer(*a.debugPort) } + *a.cpuProfile = util.ResolvePath(*a.cpuProfile) + *a.memProfile = util.ResolvePath(*a.memProfile) grace.SetupProfiling(*a.cpuProfile, *a.memProfile) // Load security configuration @@ -309,18 +310,10 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, // Create data directory first if specified (needed for session key storage) var dataDir string if *options.dataDir != "" { - // Expand tilde (~) to home directory - expandedDir, err := expandHomeDir(*options.dataDir) - if err != nil { - return fmt.Errorf("failed to expand dataDir path %s: %v", *options.dataDir, err) - } - dataDir = expandedDir - - // Show path expansion if it occurred + dataDir = util.ResolvePath(*options.dataDir) if dataDir != *options.dataDir { fmt.Printf("Expanded dataDir: %s -> %s\n", *options.dataDir, dataDir) } - if err := os.MkdirAll(dataDir, 0755); err != nil { return fmt.Errorf("failed to create data directory %s: %v", dataDir, err) } @@ -645,43 +638,3 @@ func applyViperFallback(cmd *Command, flagPtr *string, flagName, viperKey string } } } - -// expandHomeDir expands the tilde (~) in a path to the user's home directory -func expandHomeDir(path string) (string, error) { - if path == "" { - return path, nil - } - - if !strings.HasPrefix(path, "~") { - return path, nil - } - - // Get current user - currentUser, err := user.Current() - if err != nil { - return "", fmt.Errorf("failed to get current user: %w", err) - } - - // Handle different tilde patterns - if path == "~" { - return currentUser.HomeDir, nil - } - - if strings.HasPrefix(path, "~/") { - return filepath.Join(currentUser.HomeDir, path[2:]), nil - } - - // Handle ~username/ patterns - parts := strings.SplitN(path[1:], "/", 2) - username := parts[0] - - targetUser, err := user.Lookup(username) - if err != nil { - return "", fmt.Errorf("user %s not found: %v", username, err) - } - - if len(parts) == 1 { - return targetUser.HomeDir, nil - } - return filepath.Join(targetUser.HomeDir, parts[1]), nil -} diff --git a/weed/command/backup.go b/weed/command/backup.go index df5b106d2..1ede2530d 100644 --- a/weed/command/backup.go +++ b/weed/command/backup.go @@ -130,7 +130,7 @@ func backupFromLocation(volumeServer pb.ServerAddress, grpcDialOption grpc.DialO ver := needle.Version(stats.Version) // Create or load the volume - v, err := storage.NewVolume(util.ResolvePath(*s.dir), util.ResolvePath(*s.dir), *s.collection, vid, storage.NeedleMapInMemory, replication, ttl, 0, ver, 0, 0) + v, err := storage.NewVolume(*s.dir, *s.dir, *s.collection, vid, storage.NeedleMapInMemory, replication, ttl, 0, ver, 0, 0) if err != nil { return fmt.Errorf("creating or reading volume: %w", err), false } @@ -162,7 +162,7 @@ func backupFromLocation(volumeServer pb.ServerAddress, grpcDialOption grpc.DialO } v.Close() // Close the destroyed volume // recreate an empty volume - v, err = storage.NewVolume(util.ResolvePath(*s.dir), util.ResolvePath(*s.dir), *s.collection, vid, storage.NeedleMapInMemory, replication, ttl, 0, ver, 0, 0) + v, err = storage.NewVolume(*s.dir, *s.dir, *s.collection, vid, storage.NeedleMapInMemory, replication, ttl, 0, ver, 0, 0) if err != nil { return fmt.Errorf("recreating volume: %w", err), false } @@ -180,6 +180,7 @@ func backupFromLocation(volumeServer pb.ServerAddress, grpcDialOption grpc.DialO func runBackup(cmd *Command, args []string) bool { + *s.dir = util.ResolvePath(*s.dir) util.LoadSecurityConfiguration() grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.client") diff --git a/weed/command/benchmark.go b/weed/command/benchmark.go index 1833ad165..de631585c 100644 --- a/weed/command/benchmark.go +++ b/weed/command/benchmark.go @@ -123,6 +123,8 @@ func runBenchmark(cmd *Command, args []string) bool { *b.maxCpu = runtime.NumCPU() } runtime.GOMAXPROCS(*b.maxCpu) + *b.cpuprofile = util.ResolvePath(*b.cpuprofile) + *b.idListFile = util.ResolvePath(*b.idListFile) if *b.cpuprofile != "" { f, err := os.Create(*b.cpuprofile) if err != nil { diff --git a/weed/command/compact.go b/weed/command/compact.go index f6117e237..891967956 100644 --- a/weed/command/compact.go +++ b/weed/command/compact.go @@ -41,10 +41,11 @@ func runCompact(cmd *Command, args []string) bool { return false } + *compactVolumePath = util.ResolvePath(*compactVolumePath) preallocateBytes := *compactVolumePreallocate * (1 << 20) vid := needle.VolumeId(*compactVolumeId) - v, err := storage.NewVolume(util.ResolvePath(*compactVolumePath), util.ResolvePath(*compactVolumePath), *compactVolumeCollection, vid, storage.NeedleMapInMemory, nil, nil, preallocateBytes, needle.GetCurrentVersion(), 0, 0) + v, err := storage.NewVolume(*compactVolumePath, *compactVolumePath, *compactVolumeCollection, vid, storage.NeedleMapInMemory, nil, nil, preallocateBytes, needle.GetCurrentVersion(), 0, 0) if err != nil { glog.Fatalf("Load Volume [ERROR] %s\n", err) } diff --git a/weed/command/download.go b/weed/command/download.go index a155ad74a..9c91badbd 100644 --- a/weed/command/download.go +++ b/weed/command/download.go @@ -59,8 +59,9 @@ func runDownload(cmd *Command, args []string) bool { masterServer = *d.server } + *d.dir = util.ResolvePath(*d.dir) for _, fid := range args { - if e := downloadToFile(func(_ context.Context) pb.ServerAddress { return pb.ServerAddress(masterServer) }, grpcDialOption, fid, util.ResolvePath(*d.dir)); e != nil { + if e := downloadToFile(func(_ context.Context) pb.ServerAddress { return pb.ServerAddress(masterServer) }, grpcDialOption, fid, *d.dir); e != nil { fmt.Println("Download Error: ", fid, e) } } diff --git a/weed/command/export.go b/weed/command/export.go index e09d57056..c1cd052eb 100644 --- a/weed/command/export.go +++ b/weed/command/export.go @@ -148,6 +148,9 @@ func (scanner *VolumeFileScanner4Export) VisitNeedle(n *needle.Needle, offset in func runExport(cmd *Command, args []string) bool { + *export.dir = util.ResolvePath(*export.dir) + *output = util.ResolvePath(*output) + var err error if *newer != "" { @@ -200,7 +203,7 @@ func runExport(cmd *Command, args []string) bool { needleMap := needle_map.NewMemDb() defer needleMap.Close() - if err := needleMap.LoadFromIdx(path.Join(util.ResolvePath(*export.dir), fileName+".idx")); err != nil { + if err := needleMap.LoadFromIdx(path.Join(*export.dir, fileName+".idx")); err != nil { glog.Fatalf("cannot load needle map from %s.idx: %s", fileName, err) } @@ -213,7 +216,7 @@ func runExport(cmd *Command, args []string) bool { fmt.Printf("key\tname\tsize\tgzip\tmime\tmodified\tttl\tdeleted\tstart\tstop\n") } - err = storage.ScanVolumeFile(util.ResolvePath(*export.dir), *export.collection, vid, storage.NeedleMapInMemory, volumeFileScanner) + err = storage.ScanVolumeFile(*export.dir, *export.collection, vid, storage.NeedleMapInMemory, volumeFileScanner) if err != nil && err != io.EOF { glog.Errorf("Export Volume File [ERROR] %s\n", err) } diff --git a/weed/command/filer.go b/weed/command/filer.go index e056fbcf6..736ac9898 100644 --- a/weed/command/filer.go +++ b/weed/command/filer.go @@ -230,6 +230,10 @@ func runFiler(cmd *Command, args []string) bool { go http.ListenAndServe(fmt.Sprintf(":%d", *f.debugPort), nil) } + *f.defaultLevelDbDirectory = util.ResolvePath(*f.defaultLevelDbDirectory) + filerS3Options.resolvePaths() + filerWebDavOptions.resolvePaths() + filerSftpOptions.resolvePaths() util.LoadSecurityConfiguration() switch { @@ -328,7 +332,7 @@ func (fo *FilerOptions) startFiler() { *fo.allowedOrigins = "*" } - defaultLevelDbDirectory := util.ResolvePath(*fo.defaultLevelDbDirectory + "/filerldb2") + defaultLevelDbDirectory := *fo.defaultLevelDbDirectory + "/filerldb2" filerAddress := pb.NewServerAddress(*fo.ip, *fo.port, *fo.portGrpc) diff --git a/weed/command/filer_cat.go b/weed/command/filer_cat.go index 7f2ac12d6..73ae320d2 100644 --- a/weed/command/filer_cat.go +++ b/weed/command/filer_cat.go @@ -60,6 +60,7 @@ var cmdFilerCat = &Command{ func runFilerCat(cmd *Command, args []string) bool { + *filerCat.output = util.ResolvePath(*filerCat.output) util.LoadSecurityConfiguration() if len(args) == 0 { diff --git a/weed/command/filer_meta_backup.go b/weed/command/filer_meta_backup.go index a76c92d80..9276f98e2 100644 --- a/weed/command/filer_meta_backup.go +++ b/weed/command/filer_meta_backup.go @@ -65,6 +65,8 @@ When both match, the deeper prefix wins. func runFilerMetaBackup(cmd *Command, args []string) bool { + *metaBackup.backupFilerConfig = util.ResolvePath(*metaBackup.backupFilerConfig) + util.LoadSecurityConfiguration() metaBackup.grpcDialOption = security.LoadClientTLS(util.GetViper(), "grpc.client") diff --git a/weed/command/filer_sync.go b/weed/command/filer_sync.go index 8463ce0ac..06547cd02 100644 --- a/weed/command/filer_sync.go +++ b/weed/command/filer_sync.go @@ -147,6 +147,11 @@ func runFilerSynchronize(cmd *Command, args []string) bool { grace.StartDebugServer(*syncOptions.debugPort) } + *syncCpuProfile = util.ResolvePath(*syncCpuProfile) + *syncMemProfile = util.ResolvePath(*syncMemProfile) + *syncOptions.aSecurity = util.ResolvePath(*syncOptions.aSecurity) + *syncOptions.bSecurity = util.ResolvePath(*syncOptions.bSecurity) + util.LoadSecurityConfiguration() grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.client") diff --git a/weed/command/filer_sync_verify.go b/weed/command/filer_sync_verify.go index e46a5b9ca..b60e5f60a 100644 --- a/weed/command/filer_sync_verify.go +++ b/weed/command/filer_sync_verify.go @@ -63,6 +63,8 @@ var cmdFilerSyncVerify = &Command{ } func runFilerSyncVerify(cmd *Command, args []string) bool { + *syncVerifyOptions.aSecurity = util.ResolvePath(*syncVerifyOptions.aSecurity) + *syncVerifyOptions.bSecurity = util.ResolvePath(*syncVerifyOptions.bSecurity) util.LoadSecurityConfiguration() grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.client") diff --git a/weed/command/master.go b/weed/command/master.go index 80bba5fca..bf2fe26ce 100644 --- a/weed/command/master.go +++ b/weed/command/master.go @@ -144,6 +144,9 @@ func runMaster(cmd *Command, args []string) bool { *m.metaFolder = v } + *m.metaFolder = util.ResolvePath(*m.metaFolder) + *masterCpuProfile = util.ResolvePath(*masterCpuProfile) + *masterMemProfile = util.ResolvePath(*masterMemProfile) grace.SetupProfiling(*masterCpuProfile, *masterMemProfile) parent, _ := util.FullPath(*m.metaFolder).DirAndName() @@ -152,7 +155,7 @@ func runMaster(cmd *Command, args []string) bool { glog.Fatalf("Could not create Meta Folder %s: %v", *m.metaFolder, err) } } - if err := util.TestFolderWritable(util.ResolvePath(*m.metaFolder)); err != nil { + if err := util.TestFolderWritable(*m.metaFolder); err != nil { glog.Fatalf("Check Meta Folder (-mdir) Writable %s : %s", *m.metaFolder, err) } diff --git a/weed/command/mini.go b/weed/command/mini.go index 10c57c444..9e7c96b29 100644 --- a/weed/command/mini.go +++ b/weed/command/mini.go @@ -875,7 +875,7 @@ func saveMiniConfiguration(dataFolder string) error { } func runMini(cmd *Command, args []string) bool { - *miniDataFolders = util.ResolvePath(*miniDataFolders) + *miniDataFolders = util.ResolveCommaSeparatedPaths(*miniDataFolders) // Capture which port flags were explicitly passed on CLI BEFORE config file is applied // This is necessary to distinguish user-specified ports from defaults or config file options @@ -900,6 +900,17 @@ func runMini(cmd *Command, args []string) bool { util.LoadSecurityConfiguration() util.LoadConfiguration("master", false) + // applyConfigFileOptions above may have overwritten -dir from the + // mini.options file, so re-resolve it here alongside the other paths. + *miniDataFolders = util.ResolveCommaSeparatedPaths(*miniDataFolders) + *miniOptions.cpuprofile = util.ResolvePath(*miniOptions.cpuprofile) + *miniOptions.memprofile = util.ResolvePath(*miniOptions.memprofile) + *miniS3Config = util.ResolvePath(*miniS3Config) + *miniIamConfig = util.ResolvePath(*miniIamConfig) + *miniMasterOptions.metaFolder = util.ResolvePath(*miniMasterOptions.metaFolder) + *miniAdminOptions.dataDir = util.ResolvePath(*miniAdminOptions.dataDir) + miniS3Options.resolvePaths() + miniWebDavOptions.resolvePaths() grace.SetupProfiling(*miniOptions.cpuprofile, *miniOptions.memprofile) // Determine bind IP @@ -976,9 +987,13 @@ func runMini(cmd *Command, args []string) bool { } if *miniMasterOptions.metaFolder == "" { - *miniMasterOptions.metaFolder = *miniDataFolders + // -dir may be comma-separated (dir[,dir]...); the master expects a + // single directory, so default to the first entry. Both miniDataFolders + // and miniMasterOptions.metaFolder were already tilde-resolved at the + // top of runMini. + *miniMasterOptions.metaFolder = util.StringSplit(*miniDataFolders, ",")[0] } - if err := util.TestFolderWritable(util.ResolvePath(*miniMasterOptions.metaFolder)); err != nil { + if err := util.TestFolderWritable(*miniMasterOptions.metaFolder); err != nil { glog.Fatalf("Check Meta Folder (-dir=\"%s\") Writable: %s", *miniMasterOptions.metaFolder, err) } miniFilerOptions.defaultLevelDbDirectory = miniMasterOptions.metaFolder @@ -987,9 +1002,9 @@ func runMini(cmd *Command, args []string) bool { // Only auto-calculate if user didn't explicitly specify a value via -master.volumeSizeLimitMB if !isFlagPassed("master.volumeSizeLimitMB") { // User didn't override, use auto-calculated value - // The -dir flag can accept comma-separated directories; use the first one for disk space calculation - resolvedDataFolder := util.ResolvePath(util.StringSplit(*miniDataFolders, ",")[0]) - optimalVolumeSizeMB := calculateOptimalVolumeSizeMB(resolvedDataFolder) + // The -dir flag can accept comma-separated directories; use the first one for disk space calculation. + // miniDataFolders was already tilde-resolved at the top of runMini. + optimalVolumeSizeMB := calculateOptimalVolumeSizeMB(util.StringSplit(*miniDataFolders, ",")[0]) miniMasterOptions.volumeSizeLimitMB = &optimalVolumeSizeMB glog.Infof("Mini started with auto-calculated optimal volume size limit: %dMB", optimalVolumeSizeMB) } else { diff --git a/weed/command/mount_std.go b/weed/command/mount_std.go index f74964acd..c133d82e7 100644 --- a/weed/command/mount_std.go +++ b/weed/command/mount_std.go @@ -41,6 +41,8 @@ func runMount(cmd *Command, args []string) bool { go http.ListenAndServe(fmt.Sprintf(":%d", *mountOptions.debugPort), nil) } + *mountCpuProfile = util.ResolvePath(*mountCpuProfile) + *mountMemProfile = util.ResolvePath(*mountMemProfile) grace.SetupProfiling(*mountCpuProfile, *mountMemProfile) if *mountReadRetryTime < time.Second { *mountReadRetryTime = time.Second @@ -320,9 +322,10 @@ func RunMount(option *MountOptions, umask os.FileMode) bool { mountRoot = mountRoot[0 : len(mountRoot)-1] } - cacheDirForWrite := *option.cacheDirForWrite + cacheDirForRead := util.ResolvePath(*option.cacheDirForRead) + cacheDirForWrite := util.ResolvePath(*option.cacheDirForWrite) if cacheDirForWrite == "" { - cacheDirForWrite = *option.cacheDirForRead + cacheDirForWrite = cacheDirForRead } seaweedFileSystem := mount.NewSeaweedFileSystem(&mount.Option{ @@ -339,7 +342,7 @@ func RunMount(option *MountOptions, umask os.FileMode) bool { ChunkSizeLimit: int64(chunkSizeLimitMB) * 1024 * 1024, ConcurrentWriters: *option.concurrentWriters, ConcurrentReaders: *option.concurrentReaders, - CacheDirForRead: *option.cacheDirForRead, + CacheDirForRead: cacheDirForRead, CacheSizeMBForRead: *option.cacheSizeMBForRead, CacheDirForWrite: cacheDirForWrite, WriteBufferSizeMB: *option.writeBufferSizeMB, diff --git a/weed/command/mq_broker.go b/weed/command/mq_broker.go index 8e3b198c5..979eca84b 100644 --- a/weed/command/mq_broker.go +++ b/weed/command/mq_broker.go @@ -73,6 +73,8 @@ func runMqBroker(cmd *Command, args []string) bool { func (mqBrokerOpt *MessageQueueBrokerOptions) startQueueServer() bool { + *mqBrokerStandaloneOptions.cpuprofile = util.ResolvePath(*mqBrokerStandaloneOptions.cpuprofile) + *mqBrokerStandaloneOptions.memprofile = util.ResolvePath(*mqBrokerStandaloneOptions.memprofile) grace.SetupProfiling(*mqBrokerStandaloneOptions.cpuprofile, *mqBrokerStandaloneOptions.memprofile) grpcDialOption := security.LoadClientTLS(util.GetViper(), "grpc.msg_broker") diff --git a/weed/command/s3.go b/weed/command/s3.go index 06d3700ea..8f1175d2d 100644 --- a/weed/command/s3.go +++ b/weed/command/s3.go @@ -211,6 +211,7 @@ func runS3(cmd *Command, args []string) bool { grace.StartDebugServer(*s3StandaloneOptions.debugPort) } + s3StandaloneOptions.resolvePaths() util.LoadSecurityConfiguration() switch { @@ -244,6 +245,19 @@ func (s3opt *S3Options) parseDefaultFileMode() (uint32, error) { return uint32(mode), nil } +// resolvePaths expands "~" in every user-supplied path flag so callers +// that share these pointers (e.g. server.go propagating s3Options.config +// to filerOptions.s3ConfigFile before startS3Server runs) see resolved +// values. Idempotent — safe to call from any entry point. +func (s3opt *S3Options) resolvePaths() { + *s3opt.config = util.ResolvePath(*s3opt.config) + *s3opt.iamConfig = util.ResolvePath(*s3opt.iamConfig) + *s3opt.tlsCertificate = util.ResolvePath(*s3opt.tlsCertificate) + *s3opt.tlsPrivateKey = util.ResolvePath(*s3opt.tlsPrivateKey) + *s3opt.tlsCACertificate = util.ResolvePath(*s3opt.tlsCACertificate) + *s3opt.auditLogConfig = util.ResolvePath(*s3opt.auditLogConfig) +} + func (s3opt *S3Options) startS3Server() bool { filerAddresses := pb.ServerAddresses(*s3opt.filer).ToAddresses() diff --git a/weed/command/scaffold.go b/weed/command/scaffold.go index 26de2e1fd..b667dcb4b 100644 --- a/weed/command/scaffold.go +++ b/weed/command/scaffold.go @@ -59,7 +59,7 @@ func runScaffold(cmd *Command, args []string) bool { } if *outputPath != "" { - util.WriteFile(filepath.Join(*outputPath, *config+".toml"), []byte(content), 0644) + util.WriteFile(filepath.Join(util.ResolvePath(*outputPath), *config+".toml"), []byte(content), 0644) } else { fmt.Println(content) } diff --git a/weed/command/server.go b/weed/command/server.go index 227748006..96e0c1f11 100644 --- a/weed/command/server.go +++ b/weed/command/server.go @@ -224,6 +224,13 @@ func runServer(cmd *Command, args []string) bool { util.LoadSecurityConfiguration() util.LoadConfiguration("master", false) + *serverOptions.cpuprofile = util.ResolvePath(*serverOptions.cpuprofile) + *serverOptions.memprofile = util.ResolvePath(*serverOptions.memprofile) + *serverIamConfig = util.ResolvePath(*serverIamConfig) + *masterOptions.metaFolder = util.ResolvePath(*masterOptions.metaFolder) + s3Options.resolvePaths() + webdavOptions.resolvePaths() + sftpOptions.resolvePaths() grace.SetupProfiling(*serverOptions.cpuprofile, *serverOptions.memprofile) if *isStartingS3 { @@ -318,6 +325,7 @@ func runServer(cmd *Command, args []string) bool { go stats_collect.StartMetricsServer(*serverMetricsHttpIp, *serverMetricsHttpPort) + *volumeDataFolders = util.ResolveCommaSeparatedPaths(*volumeDataFolders) folders := strings.Split(*volumeDataFolders, ",") if *masterOptions.volumeSizeLimitMB > util.VolumeSizeLimitGB*1000 { @@ -327,7 +335,7 @@ func runServer(cmd *Command, args []string) bool { if *masterOptions.metaFolder == "" { *masterOptions.metaFolder = folders[0] } - if err := util.TestFolderWritable(util.ResolvePath(*masterOptions.metaFolder)); err != nil { + if err := util.TestFolderWritable(*masterOptions.metaFolder); err != nil { glog.Fatalf("Check Meta Folder (-mdir=\"%s\") Writable: %s", *masterOptions.metaFolder, err) } filerOptions.defaultLevelDbDirectory = masterOptions.metaFolder diff --git a/weed/command/sftp.go b/weed/command/sftp.go index 389624355..142b1927b 100644 --- a/weed/command/sftp.go +++ b/weed/command/sftp.go @@ -79,6 +79,7 @@ func init() { // runSftp is the command entry point. func runSftp(cmd *Command, args []string) bool { + sftpOptionsStandalone.resolvePaths() // Load security configuration as done in other SeaweedFS services. util.LoadSecurityConfiguration() @@ -94,6 +95,14 @@ func runSftp(cmd *Command, args []string) bool { return sftpOptionsStandalone.startSftpServer() } +// resolvePaths expands "~" in every user-supplied path flag. +// Idempotent — safe to call from any entry point. +func (sftpOpt *SftpOptions) resolvePaths() { + *sftpOpt.sshPrivateKey = util.ResolvePath(*sftpOpt.sshPrivateKey) + *sftpOpt.hostKeysFolder = util.ResolvePath(*sftpOpt.hostKeysFolder) + *sftpOpt.userStoreFile = util.ResolvePath(*sftpOpt.userStoreFile) +} + func (sftpOpt *SftpOptions) startSftpServer() bool { if *sftpOpt.bindIp == "" { *sftpOpt.bindIp = "0.0.0.0" diff --git a/weed/command/update.go b/weed/command/update.go index 57953c565..53474e69e 100644 --- a/weed/command/update.go +++ b/weed/command/update.go @@ -85,8 +85,9 @@ func runUpdate(cmd *Command, args []string) bool { path, _ := os.Executable() _, name := filepath.Split(path) + *updateOpt.dir = util.ResolvePath(*updateOpt.dir) if *updateOpt.dir != "" { - if err := util.TestFolderWritable(util.ResolvePath(*updateOpt.dir)); err != nil { + if err := util.TestFolderWritable(*updateOpt.dir); err != nil { glog.Fatalf("Check Folder(-dir) Writable %s : %s", *updateOpt.dir, err) return false } diff --git a/weed/command/volume.go b/weed/command/volume.go index ccb9a55a1..2b86f16c6 100644 --- a/weed/command/volume.go +++ b/weed/command/volume.go @@ -150,6 +150,8 @@ func runVolume(cmd *Command, args []string) bool { // If --pprof is set we assume the caller wants to be able to collect // cpu and memory profiles via go tool pprof if !*v.pprof { + *v.cpuProfile = util.ResolvePath(*v.cpuProfile) + *v.memProfile = util.ResolvePath(*v.memProfile) grace.SetupProfiling(*v.cpuProfile, *v.memProfile) } @@ -179,9 +181,10 @@ func (v VolumeServerOptions) startVolumeServer(volumeFolders, maxVolumeCounts, v // Set multiple folders and each folder's max volume count limit' v.folders = strings.Split(volumeFolders, ",") - for _, folder := range v.folders { - if err := util.TestFolderWritable(util.ResolvePath(folder)); err != nil { - glog.Fatalf("Check Data Folder(-dir) Writable %s : %s", folder, err) + for i, folder := range v.folders { + v.folders[i] = util.ResolvePath(folder) + if err := util.TestFolderWritable(v.folders[i]); err != nil { + glog.Fatalf("Check Data Folder(-dir) Writable %s : %s", v.folders[i], err) } } @@ -284,7 +287,7 @@ func (v VolumeServerOptions) startVolumeServer(volumeFolders, maxVolumeCounts, v volumeServer := weed_server.NewVolumeServer(volumeMux, publicVolumeMux, *v.ip, *v.port, *v.portGrpc, *v.publicUrl, volumeServerId, v.folders, v.folderMaxLimits, minFreeSpaces, diskTypes, folderTags, - *v.idxFolder, + util.ResolvePath(*v.idxFolder), volumeNeedleMapKind, v.masters, constants.VolumePulsePeriod, *v.dataCenter, *v.rack, v.whiteList, diff --git a/weed/command/webdav.go b/weed/command/webdav.go index 050e194ed..71141b304 100644 --- a/weed/command/webdav.go +++ b/weed/command/webdav.go @@ -69,6 +69,7 @@ var cmdWebDav = &Command{ func runWebDav(cmd *Command, args []string) bool { + webDavStandaloneOptions.resolvePaths() util.LoadSecurityConfiguration() listenAddress := fmt.Sprintf("%s:%d", *webDavStandaloneOptions.ipBind, *webDavStandaloneOptions.port) @@ -78,6 +79,14 @@ func runWebDav(cmd *Command, args []string) bool { } +// resolvePaths expands "~" in every user-supplied path flag. +// Idempotent — safe to call from any entry point. +func (wo *WebDavOption) resolvePaths() { + *wo.cacheDir = util.ResolvePath(*wo.cacheDir) + *wo.tlsCertificate = util.ResolvePath(*wo.tlsCertificate) + *wo.tlsPrivateKey = util.ResolvePath(*wo.tlsPrivateKey) +} + func (wo *WebDavOption) startWebDav() bool { // detect current user @@ -126,7 +135,7 @@ func (wo *WebDavOption) startWebDav() bool { Uid: uid, Gid: gid, Cipher: cipher, - CacheDir: util.ResolvePath(*wo.cacheDir), + CacheDir: *wo.cacheDir, CacheSizeMB: *wo.cacheSizeMB, MaxMB: *wo.maxMB, }) diff --git a/weed/command/worker.go b/weed/command/worker.go index 4db89b121..bddb1c424 100644 --- a/weed/command/worker.go +++ b/weed/command/worker.go @@ -3,6 +3,7 @@ package command import ( "time" + "github.com/seaweedfs/seaweedfs/weed/util" "github.com/seaweedfs/seaweedfs/weed/util/grace" ) @@ -62,6 +63,7 @@ func runWorker(cmd *Command, args []string) bool { grace.StartDebugServer(*workerDebugPort) } + *workerWorkingDir = util.ResolvePath(*workerWorkingDir) return runPluginWorkerWithOptions(pluginWorkerRunOptions{ AdminServer: *workerAdminServer, WorkerID: *workerID, diff --git a/weed/util/file_util.go b/weed/util/file_util.go index 6c5f33a24..ea28d7d9c 100644 --- a/weed/util/file_util.go +++ b/weed/util/file_util.go @@ -82,25 +82,74 @@ func CheckFile(filename string) (exists, canRead, canWrite bool, modTime time.Ti return } +// ResolvePath expands a leading "~", "~/", or "~username/" in path to the +// corresponding home directory, mirroring shell tilde expansion. A path +// without a leading "~" or whose tilde cannot be resolved is returned +// unchanged so callers can pass any user-supplied path through this helper +// without worrying about non-tilde inputs or lookup failures. +// +// Forward slashes are always recognised as separators after the tilde; +// the platform-native separator is also accepted so "~\\data" works on +// Windows. Backslashes are deliberately not treated as separators on +// other platforms because they are legal characters in usernames and +// path segments there. func ResolvePath(path string) string { - if !strings.Contains(path, "~") { + if !strings.HasPrefix(path, "~") { return path } - usr, _ := user.Current() - dir := usr.HomeDir - if path == "~" { - // In case of "~", which won't be caught by the "else if" - path = dir - } else if strings.HasPrefix(path, "~/") { - // Use strings.HasPrefix so we don't match paths like - // "/something/~/something/" - path = filepath.Join(dir, path[2:]) + if usr, err := user.Current(); err == nil { + return usr.HomeDir + } + return path } - return path + isSep := func(b byte) bool { + return b == '/' || b == byte(filepath.Separator) + } + + if isSep(path[1]) { + if usr, err := user.Current(); err == nil { + return filepath.Join(usr.HomeDir, path[2:]) + } + return path + } + + // "~username" or "~usernamerest" + name := path[1:] + rest := "" + for i := 0; i < len(name); i++ { + if isSep(name[i]) { + rest = name[i+1:] + name = name[:i] + break + } + } + usr, err := user.Lookup(name) + if err != nil { + return path + } + if rest == "" { + return usr.HomeDir + } + return filepath.Join(usr.HomeDir, rest) +} + +// ResolveCommaSeparatedPaths splits paths on "," and runs each entry through +// ResolvePath, then rejoins them. This lets flags like `weed mini -dir` or +// `weed volume -dir` accept tilde-prefixed entries (e.g. "~/d1,~/d2") even +// in the comma-separated form the help text advertises. +func ResolveCommaSeparatedPaths(paths string) string { + if !strings.Contains(paths, "~") { + return paths + } + parts := strings.Split(paths, ",") + for i, p := range parts { + parts[i] = ResolvePath(p) + } + return strings.Join(parts, ",") } func FileNameBase(filename string) string { diff --git a/weed/util/file_util_test.go b/weed/util/file_util_test.go index a1f924fed..0c8a02ead 100644 --- a/weed/util/file_util_test.go +++ b/weed/util/file_util_test.go @@ -1,6 +1,8 @@ package util import ( + "os/user" + "path/filepath" "testing" ) @@ -20,3 +22,67 @@ func TestToShortFileName(t *testing.T) { } } } + +func TestResolvePath(t *testing.T) { + usr, err := user.Current() + if err != nil { + t.Fatalf("user.Current: %v", err) + } + home := usr.HomeDir + + cases := []struct { + name string + in string + want string + }{ + {"empty", "", ""}, + {"absolute", "/var/data", "/var/data"}, + {"relative", "data", "data"}, + {"tilde mid path is literal", "/foo/~/bar", "/foo/~/bar"}, + {"bare tilde", "~", home}, + {"tilde slash", "~/", home}, + {"tilde with subpath", "~/data", filepath.Join(home, "data")}, + {"tilde with current user", "~" + usr.Username, home}, + {"tilde with current user and subpath", "~" + usr.Username + "/data", filepath.Join(home, "data")}, + {"tilde unknown user falls back to literal", "~no-such-user-xyz/data", "~no-such-user-xyz/data"}, + // Native separator: identical to "~/data" on Unix; exercises backslash on Windows. + {"tilde with native separator", "~" + string(filepath.Separator) + "data", filepath.Join(home, "data")}, + {"tilde user with native separator", "~" + usr.Username + string(filepath.Separator) + "data", filepath.Join(home, "data")}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := ResolvePath(c.in); got != c.want { + t.Errorf("ResolvePath(%q) = %q; want %q", c.in, got, c.want) + } + }) + } +} + +func TestResolveCommaSeparatedPaths(t *testing.T) { + usr, err := user.Current() + if err != nil { + t.Fatalf("user.Current: %v", err) + } + home := usr.HomeDir + + cases := []struct { + name string + in string + want string + }{ + {"empty", "", ""}, + {"single absolute", "/a", "/a"}, + {"single tilde", "~/a", filepath.Join(home, "a")}, + {"two absolutes", "/a,/b", "/a,/b"}, + {"two tildes", "~/a,~/b", filepath.Join(home, "a") + "," + filepath.Join(home, "b")}, + {"mixed", "/a,~/b,/c", "/a," + filepath.Join(home, "b") + ",/c"}, + {"no tilde fast path", "/a,/b,/c", "/a,/b,/c"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + if got := ResolveCommaSeparatedPaths(c.in); got != c.want { + t.Errorf("ResolveCommaSeparatedPaths(%q) = %q; want %q", c.in, got, c.want) + } + }) + } +}