1.05

fix invalid point when delete collection

.travis.yml

@@ -2,6 +2,7 @@ sudo: false
 language: go
 go:
 - 1.10.x
+- 1.11.x
 - tip
 
 before_install:

README.md

@@ -78,10 +78,10 @@ SeaweedFS can work very well with just the object store. [[Filer]] is added late
 * Support Etag, Accept-Range, Last-Modified, etc.
 * Support in-memory/leveldb/boltdb/btree mode tuning for memory/performance balance.
 ## Filer Features
-* [filer server][Filer] provides "normal" directories and files via http.
+* [filer server][Filer] provide "normal" directories and files via http.
 * [mount filer][Mount] to read and write files directly as a local directory via FUSE.
 * [Amazon S3 compatible API][AmazonS3API] to access files with S3 tooling.
-* [Async Backup To Cloud][BackupToCloud] can enjoy extreme fast local access and backup to Amazon S3, Google Cloud Storage, Azure.
+* [Async Backup To Cloud][BackupToCloud] can enjoy extreme fast local access and backup to Amazon S3, Google Cloud Storage, Azure, BackBlaze.
 
 [Filer]: https://github.com/chrislusf/seaweedfs/wiki/Directories-and-Files
 [Mount]: https://github.com/chrislusf/seaweedfs/wiki/Mount
@@ -280,16 +280,8 @@ All file meta information on volume server is readable from memory without disk
 
 Most other distributed file systems seem more complicated than necessary.
 
-### Compared to Ceph ###
-
-Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. [Here is a more detailed comparison](https://github.com/chrislusf/seaweedfs/issues/120)
-
 SeaweedFS is meant to be fast and simple, both during usage and during setup. If you do not understand how it works when you reach here, we failed! Please raise an issue with any questions or update this file with clarifications.
 
-SeaweedFS has a centralized master to look up free volumes, while Ceph uses hashing to locate its objects. Having a centralized master makes it easy to code and manage. HDFS/GFS has the single name node for years. SeaweedFS now support multiple master nodes.
-
-Ceph hashing avoids SPOF, but makes it complicated when moving or adding servers.
-
 ### Compared to HDFS ###
 
 HDFS uses the chunk approach for each file, and is ideal for storing large files.
@@ -304,9 +296,9 @@ SeaweedFS can also store extra large files by splitting them into manageable dat
 The architectures are mostly the same. SeaweedFS aims to store and read files fast, with a simple and flat architecture. The main differences are
 
 * SeaweedFS optimizes for small files, ensuring O(1) disk seek operation, and can also handle large files.
-* SeaweedFS statically assign a volume id for a file. Locating file content becomes just a lookup of the volume id, which can be easily cached.
-* SeaweedFS Filer metadata store can be any well-known and proven data stores, e.g., Cassandra, Redis, MySql, PostGres, etc, and is easy to customized.
-* SeaweedFS Volume server also communicate directly with clients via HTTP, supporting range queries, direct uploads, etc.
+* SeaweedFS statically assigns a volume id for a file. Locating file content becomes just a lookup of the volume id, which can be easily cached.
+* SeaweedFS Filer metadata store can be any well-known and proven data stores, e.g., Cassandra, Redis, MySql, Postgres, etc, and is easy to customized.
+* SeaweedFS Volume server also communicates directly with clients via HTTP, supporting range queries, direct uploads, etc.
 
 | System         | File Meta                       | File Content Read| POSIX  | REST API | Optimized for small files |
 | -------------  | ------------------------------- | ---------------- | ------ | -------- | ------------------------- |
@@ -323,10 +315,18 @@ GlusterFS hashes the path and filename into ids, and assigned to virtual volumes
 
 ### Compared to Ceph ###
 
+Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. [Here is a more detailed comparison](https://github.com/chrislusf/seaweedfs/issues/120)
+
+SeaweedFS has a centralized master group to look up free volumes, while Ceph uses hashing and metadata servers to locate its objects. Having a centralized master makes it easy to code and manage. 
+
 Same as SeaweedFS, Ceph is also based on a object store RADOS. Ceph is rather complicated with mixed reviews.
 
 Ceph uses CRUSH hashing to automatically manage the data placement. SeaweedFS places data by assigned volumes.
 
+SeaweedFS is optimized for small files. Small files are stored as one continuous block of content, with at most 8 unused bytes between files. Small file access is O(1) disk read.
+
+SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Redis, Cassandra, to manage file directories. There are proven, scalable, and easier to manage.
+
 | SeaweedFS         | comparable to Ceph | advantage |
 | -------------  | ------------- | ---------------- |
 | Master  | MDS | simpler |
@@ -337,7 +337,7 @@ Ceph uses CRUSH hashing to automatically manage the data placement. SeaweedFS pl
 ## Dev plan ##
 
 More tools and documentation, on how to maintain and scale the system. For example, how to move volumes, automatically balancing data, how to grow volumes, how to check system status, etc.
-Other key features include: Erasure Encoding, support S3 API, JWT security.
+Other key features include: Erasure Encoding, JWT security.
 
 This is a super exciting project! And I need helpers and [support](https://www.patreon.com/seaweedfs)!
 
@@ -357,11 +357,17 @@ http://mercurial.selenic.com/downloads
 
 step 3: download, compile, and install the project by executing the following command
 
+```bash
 go get github.com/chrislusf/seaweedfs/weed
+```
 
-once this is done, you should see the executable "weed" under $GOPATH/bin
+once this is done, you should see the executable "weed" under `$GOPATH/bin`
 
-step 4: after you modify your code locally, you could start a local build by calling "go install" under $GOPATH/src/github.com/chrislusf/seaweedfs/weed
+step 4: after you modify your code locally, you could start a local build by calling `go install` under 
+
+```
+$GOPATH/src/github.com/chrislusf/seaweedfs/weed
+```
 
 ## Disk Related topics ##
 

other/java/s3copy/copier/src/main/java/com/seaweedfs/s3/PutObject.java

@@ -12,13 +12,21 @@ import com.amazonaws.services.s3.AmazonS3;
 import com.amazonaws.services.s3.AmazonS3ClientBuilder;
 import com.amazonaws.services.s3.model.ObjectMetadata;
 import com.amazonaws.services.s3.model.PutObjectRequest;
+import com.amazonaws.services.s3.model.S3Object;
+import com.amazonaws.util.IOUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
 
 import java.io.File;
+import java.io.IOException;
 
 /**
  * Hello world!
  */
 public class PutObject {
+
+    private static Log log = LogFactory.getLog(PutObject.class);
+
     public static void main(String[] args) {
 
         AWSCredentials credentials = new BasicAWSCredentials("ANY-ACCESSKEYID", "ANY-SECRETACCESSKEY");
@@ -39,10 +47,11 @@ public class PutObject {
         String fileObjKeyName = "fileObject2";
         String fileName = args[0];
 
+        String stringContent = "Uploaded String Object v3";
         try {
 
             // Upload a text string as a new object.
-            s3Client.putObject(bucketName, stringObjKeyName, "Uploaded String Object v3");
+            s3Client.putObject(bucketName, stringObjKeyName, stringContent);
 
             // Upload a file as a new object with ContentType and title specified.
             PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, new File(fileName));
@@ -51,6 +60,32 @@ public class PutObject {
             metadata.addUserMetadata("x-amz-meta-title", "someTitle");
             request.setMetadata(metadata);
             s3Client.putObject(request);
+
+            // test reads
+            S3Object written = s3Client.getObject(bucketName, stringObjKeyName);
+            try {
+                String expected = IOUtils.toString(written.getObjectContent());
+
+                if (!stringContent.equals(expected)){
+                    System.out.println("Failed to put and get back the content!");
+                }
+
+            } catch (IOException e) {
+                throw new SdkClientException("Error streaming content from S3 during download");
+            } finally {
+                IOUtils.closeQuietly(written, log);
+            }
+
+            // test deletes
+            s3Client.deleteObject(bucketName, stringObjKeyName);
+
+
+            // delete bucket
+            String tmpBucket = "tmpbucket";
+            s3Client.createBucket(tmpBucket);
+            s3Client.putObject(tmpBucket, stringObjKeyName, stringContent);
+            s3Client.deleteBucket(tmpBucket);
+
         } catch (AmazonServiceException e) {
             // The call was transmitted successfully, but Amazon S3 couldn't process
             // it, so it returned an error response.

unmaintained/repeated_vacuum/repeated_vacuum.go

@@ -0,0 +1,45 @@
+package main
+
+import (
+	"bytes"
+	"flag"
+	"fmt"
+	"log"
+	"math/rand"
+
+	"github.com/chrislusf/seaweedfs/weed/operation"
+	"github.com/chrislusf/seaweedfs/weed/util"
+)
+
+var (
+	master = flag.String("master", "127.0.0.1:9333", "the master server")
+	repeat = flag.Int("n", 5, "repeat how many times")
+)
+
+func main() {
+	flag.Parse()
+
+	for i := 0; i < *repeat; i++ {
+		assignResult, err := operation.Assign(*master, &operation.VolumeAssignRequest{Count: 1})
+		if err != nil {
+			log.Fatalf("assign: %v", err)
+		}
+
+		data := make([]byte, 1024)
+		rand.Read(data)
+		reader := bytes.NewReader(data)
+
+		targetUrl := fmt.Sprintf("http://%s/%s", assignResult.Url,assignResult.Fid)
+
+		_, err = operation.Upload(targetUrl, fmt.Sprintf("test%d", i), reader, false, "", nil, "")
+		if err != nil {
+			log.Fatalf("upload: %v", err)
+		}
+
+		util.Delete(targetUrl, "")
+
+		util.Get(fmt.Sprintf("http://%s/vol/vacuum", *master))
+
+	}
+
+}

unmaintained/see_idx/see_idx.go

@@ -2,13 +2,13 @@ package main
 
 import (
 	"flag"
+	"fmt"
 	"os"
 	"path"
 	"strconv"
 
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/storage"
-	"fmt"
 	"github.com/chrislusf/seaweedfs/weed/storage/types"
 )
 

weed/command/backup.go

@@ -59,7 +59,7 @@ func runBackup(cmd *Command, args []string) bool {
 	}
 	volumeServer := lookup.Locations[0].Url
 
-	stats, err := operation.GetVolumeSyncStatus(volumeServer, vid.String())
+	stats, err := operation.GetVolumeSyncStatus(volumeServer, uint32(vid))
 	if err != nil {
 		fmt.Printf("Error get volume %d status: %v\n", vid, err)
 		return true

weed/command/filer_export.go

@@ -3,6 +3,8 @@ package command
 import (
 	"github.com/chrislusf/seaweedfs/weed/filer2"
 	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/notification"
+	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/server"
 	"github.com/spf13/viper"
 )
@@ -22,14 +24,19 @@ var cmdFilerExport = &Command{
 
 	If target store is empty, only the directory tree will be listed.
 
+	If target store is "notification", the list of entries will be sent to notification.
+	This is usually used to bootstrap filer replication to a remote system.
+
   `,
 }
 
 var (
 	// filerExportOutputFile  = cmdFilerExport.Flag.String("output", "", "the output file. If empty, only list out the directory tree")
-	filerExportSourceStore = cmdFilerExport.Flag.String("sourceStore", "", "the source store name in filer.toml")
-	filerExportTargetStore = cmdFilerExport.Flag.String("targetStore", "", "the target store name in filer.toml")
+	filerExportSourceStore = cmdFilerExport.Flag.String("sourceStore", "", "the source store name in filer.toml, default to currently enabled store")
+	filerExportTargetStore = cmdFilerExport.Flag.String("targetStore", "", "the target store name in filer.toml, or \"notification\" to export all files to message queue")
+	dir                    = cmdFilerExport.Flag.String("dir", "/", "only process files under this directory")
 	dirListLimit           = cmdFilerExport.Flag.Int("dirListLimit", 100000, "limit directory list size")
+	dryRun                 = cmdFilerExport.Flag.Bool("dryRun", false, "not actually moving data")
 )
 
 type statistics struct {
@@ -45,10 +52,10 @@ func runFilerExport(cmd *Command, args []string) bool {
 	var sourceStore, targetStore filer2.FilerStore
 
 	for _, store := range filer2.Stores {
-		if store.GetName() == *filerExportSourceStore {
+		if store.GetName() == *filerExportSourceStore || *filerExportSourceStore == "" && config.GetBool(store.GetName()+".enabled") {
 			viperSub := config.Sub(store.GetName())
 			if err := store.Initialize(viperSub); err != nil {
-				glog.Fatalf("Failed to initialize store for %s: %+v",
+				glog.Fatalf("Failed to initialize source store for %s: %+v",
 					store.GetName(), err)
 			} else {
 				sourceStore = store
@@ -61,7 +68,7 @@ func runFilerExport(cmd *Command, args []string) bool {
 		if store.GetName() == *filerExportTargetStore {
 			viperSub := config.Sub(store.GetName())
 			if err := store.Initialize(viperSub); err != nil {
-				glog.Fatalf("Failed to initialize store for %s: %+v",
+				glog.Fatalf("Failed to initialize target store for %s: %+v",
 					store.GetName(), err)
 			} else {
 				targetStore = store
@@ -79,19 +86,49 @@ func runFilerExport(cmd *Command, args []string) bool {
 		return false
 	}
 
+	if targetStore == nil && *filerExportTargetStore != "" && *filerExportTargetStore != "notification" {
+		glog.Errorf("Failed to find target store %s", *filerExportTargetStore)
+		println("existing data sources are:")
+		for _, store := range filer2.Stores {
+			println("    " + store.GetName())
+		}
+		return false
+	}
+
 	stat := statistics{}
 
 	var fn func(level int, entry *filer2.Entry) error
 
-	if targetStore == nil {
+	if *filerExportTargetStore == "notification" {
+		weed_server.LoadConfiguration("notification", false)
+		v := viper.GetViper()
+		notification.LoadConfiguration(v.Sub("notification"))
+
+		fn = func(level int, entry *filer2.Entry) error {
+			printout(level, entry)
+			if *dryRun {
+				return nil
+			}
+			return notification.Queue.SendMessage(
+				string(entry.FullPath),
+				&filer_pb.EventNotification{
+					NewEntry: entry.ToProtoEntry(),
+				},
+			)
+		}
+	} else if targetStore == nil {
 		fn = printout
 	} else {
 		fn = func(level int, entry *filer2.Entry) error {
+			printout(level, entry)
+			if *dryRun {
+				return nil
+			}
 			return targetStore.InsertEntry(entry)
 		}
 	}
 
-	doTraverse(&stat, sourceStore, filer2.FullPath("/"), 0, fn)
+	doTraverse(&stat, sourceStore, filer2.FullPath(*dir), 0, fn)
 
 	glog.Infof("processed %d directories, %d files", stat.directoryCount, stat.fileCount)
 
@@ -126,7 +163,11 @@ func doTraverse(stat *statistics, filerStore filer2.FilerStore, parentPath filer
 
 func printout(level int, entry *filer2.Entry) error {
 	for i := 0; i < level; i++ {
-		print("  ")
+		if i == level-1 {
+			print("+-")
+		} else {
+			print("| ")
+		}
 	}
 	println(entry.FullPath.Name())
 	return nil

weed/command/filer_replication.go

@@ -6,9 +6,12 @@ import (
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/replication"
 	"github.com/chrislusf/seaweedfs/weed/replication/sink"
+	_ "github.com/chrislusf/seaweedfs/weed/replication/sink/azuresink"
+	_ "github.com/chrislusf/seaweedfs/weed/replication/sink/b2sink"
 	_ "github.com/chrislusf/seaweedfs/weed/replication/sink/filersink"
 	_ "github.com/chrislusf/seaweedfs/weed/replication/sink/gcssink"
 	_ "github.com/chrislusf/seaweedfs/weed/replication/sink/s3sink"
+	"github.com/chrislusf/seaweedfs/weed/replication/sub"
 	"github.com/chrislusf/seaweedfs/weed/server"
 	"github.com/spf13/viper"
 )
@@ -33,11 +36,12 @@ var cmdFilerReplicate = &Command{
 func runFilerReplicate(cmd *Command, args []string) bool {
 
 	weed_server.LoadConfiguration("replication", true)
+	weed_server.LoadConfiguration("notification", true)
 	config := viper.GetViper()
 
-	var notificationInput replication.NotificationInput
+	var notificationInput sub.NotificationInput
 
-	for _, input := range replication.NotificationInputs {
+	for _, input := range sub.NotificationInputs {
 		if config.GetBool("notification." + input.GetName() + ".enabled") {
 			viperSub := config.Sub("notification." + input.GetName())
 			if err := input.Initialize(viperSub); err != nil {
@@ -50,6 +54,12 @@ func runFilerReplicate(cmd *Command, args []string) bool {
 		}
 	}
 
+	if notificationInput == nil {
+		println("No notification is defined in notification.toml file.")
+		println("Please follow 'weed scaffold -config=notification' to see example notification configurations.")
+		return true
+	}
+
 	// avoid recursive replication
 	if config.GetBool("notification.source.filer.enabled") && config.GetBool("notification.sink.filer.enabled") {
 		sourceConfig, sinkConfig := config.Sub("source.filer"), config.Sub("sink.filer")
@@ -77,7 +87,7 @@ func runFilerReplicate(cmd *Command, args []string) bool {
 	}
 
 	if dataSink == nil {
-		println("no data sink configured:")
+		println("no data sink configured in replication.toml:")
 		for _, sk := range sink.Sinks {
 			println("    " + sk.GetName())
 		}
@@ -92,6 +102,10 @@ func runFilerReplicate(cmd *Command, args []string) bool {
 			glog.Errorf("receive %s: %+v", key, err)
 			continue
 		}
+		if key == "" {
+			// long poll received no messages
+			continue
+		}
 		if m.OldEntry != nil && m.NewEntry == nil {
 			glog.V(1).Infof("delete: %s", key)
 		} else if m.OldEntry == nil && m.NewEntry != nil {
@@ -101,6 +115,8 @@ func runFilerReplicate(cmd *Command, args []string) bool {
 		}
 		if err = replicator.Replicate(key, m); err != nil {
 			glog.Errorf("replicate %s: %+v", key, err)
+		} else {
+			glog.V(1).Infof("replicated %s", key)
 		}
 	}
 

weed/command/master.go

@@ -42,7 +42,7 @@ var (
 	defaultReplicaPlacement = cmdMaster.Flag.String("defaultReplication", "000", "Default replication type if not specified.")
 	// mTimeout                = cmdMaster.Flag.Int("idleTimeout", 30, "connection idle seconds")
 	mMaxCpu               = cmdMaster.Flag.Int("maxCpu", 0, "maximum number of CPUs. 0 means all available CPUs")
-	garbageThreshold      = cmdMaster.Flag.String("garbageThreshold", "0.3", "threshold to vacuum and reclaim spaces")
+	garbageThreshold      = cmdMaster.Flag.Float64("garbageThreshold", 0.3, "threshold to vacuum and reclaim spaces")
 	masterWhiteListOption = cmdMaster.Flag.String("whiteList", "", "comma separated Ip addresses having write permission. No limit if empty.")
 	masterSecureKey       = cmdMaster.Flag.String("secure.secret", "", "secret to encrypt Json Web Token(JWT)")
 	masterCpuProfile      = cmdMaster.Flag.String("cpuprofile", "", "cpu profile output file")

weed/command/mount_std.go

@@ -5,13 +5,14 @@ package command
 import (
 	"fmt"
 	"runtime"
+	"strings"
+	"time"
 
 	"bazil.org/fuse"
 	"bazil.org/fuse/fs"
 	"github.com/chrislusf/seaweedfs/weed/filesys"
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/util"
-	"strings"
 )
 
 func runMount(cmd *Command, args []string) bool {
@@ -38,7 +39,7 @@ func runMount(cmd *Command, args []string) bool {
 		fuse.AllowOther(),
 		fuse.AllowSUID(),
 		fuse.DefaultPermissions(),
-		// fuse.MaxReadahead(1024*128), // TODO: not tested yet, possibly improving read performance
+		fuse.MaxReadahead(1024*128),
 		fuse.AsyncRead(),
 		fuse.WritebackCache(),
 	)
@@ -72,6 +73,7 @@ func runMount(cmd *Command, args []string) bool {
 		ChunkSizeLimit:     int64(*mountOptions.chunkSizeLimitMB) * 1024 * 1024,
 		DataCenter:         *mountOptions.dataCenter,
 		DirListingLimit:    *mountOptions.dirListingLimit,
+		EntryCacheTtl:      3 * time.Second,
 	}))
 	if err != nil {
 		fuse.Unmount(*mountOptions.dir)

weed/command/scaffold.go

@@ -19,7 +19,7 @@ var cmdScaffold = &Command{
 
 var (
 	outputPath = cmdScaffold.Flag.String("output", "", "if not empty, save the configuration file to this directory")
-	config     = cmdScaffold.Flag.String("config", "filer", "[filer|replication] the configuration file to generate")
+	config     = cmdScaffold.Flag.String("config", "filer", "[filer|notification|replication] the configuration file to generate")
 )
 
 func runScaffold(cmd *Command, args []string) bool {
@@ -28,6 +28,8 @@ func runScaffold(cmd *Command, args []string) bool {
 	switch *config {
 	case "filer":
 		content = FILER_TOML_EXAMPLE
+	case "notification":
+		content = NOTIFICATION_TOML_EXAMPLE
 	case "replication":
 		content = REPLICATION_TOML_EXAMPLE
 	}
@@ -37,7 +39,7 @@ func runScaffold(cmd *Command, args []string) bool {
 	}
 
 	if *outputPath != "" {
-		ioutil.WriteFile(filepath.Join(*outputPath, *config+".toml"), []byte(content), 0x755)
+		ioutil.WriteFile(filepath.Join(*outputPath, *config+".toml"), []byte(content), 0644)
 	} else {
 		println(content)
 	}
@@ -131,22 +133,53 @@ addresses = [
     "localhost:30006",
 ]
 
+`
+
+	NOTIFICATION_TOML_EXAMPLE = `
+# A sample TOML config file for SeaweedFS filer store
+# Used by both "weed filer" or "weed server -filer" and "weed filer.replicate"
+# Put this file to one of the location, with descending priority
+#    ./notification.toml
+#    $HOME/.seaweedfs/notification.toml
+#    /etc/seaweedfs/notification.toml
 
 ####################################################
 # notification
-# sends filer updates for each file to an external message queue
+# send and receive filer updates for each file to an external message queue
 ####################################################
 [notification.log]
+# this is only for debugging perpose and does not work with "weed filer.replicate"
 enabled = false
 
+
 [notification.kafka]
 enabled = false
 hosts = [
   "localhost:9092"
 ]
 topic = "seaweedfs_filer"
+offsetFile = "./last.offset"
+offsetSaveIntervalSeconds = 10
+
+
+[notification.aws_sqs]
+# experimental, let me know if it works
+enabled = false
+aws_access_key_id     = ""        # if empty, loads from the shared credentials file (~/.aws/credentials).
+aws_secret_access_key = ""        # if empty, loads from the shared credentials file (~/.aws/credentials).
+region = "us-east-2"
+sqs_queue_name = "my_filer_queue" # an existing queue name
+
+
+[notification.google_pub_sub]
+# read credentials doc at https://cloud.google.com/docs/authentication/getting-started
+enabled = false
+google_application_credentials = "/path/to/x.json" # path to json credential file
+project_id = ""                       # an existing project id
+topic = "seaweedfs_filer_topic"       # a topic, auto created if does not exists
 
 `
+
 	REPLICATION_TOML_EXAMPLE = `
 # A sample TOML config file for replicating SeaweedFS filer
 # Used with "weed filer.replicate"
@@ -160,15 +193,6 @@ enabled = true
 grpcAddress = "localhost:18888"
 directory = "/buckets"    # all files under this directory tree are replicated
 
-[notification.kafka]
-enabled = false
-hosts = [
-  "localhost:9092"
-]
-topic = "seaweedfs_filer1_to_filer2"
-offsetFile = "./last.offset"
-offsetSaveIntervalSeconds = 10
-
 [sink.filer]
 enabled = false
 grpcAddress = "localhost:18888"
@@ -185,7 +209,7 @@ aws_access_key_id     = ""     # if empty, loads from the shared credentials fil
 aws_secret_access_key = ""     # if empty, loads from the shared credentials file (~/.aws/credentials).
 region = "us-east-2"
 bucket = "your_bucket_name"    # an existing bucket
-directory = ""                 # destination directory (do not prefix or suffix with "/")
+directory = "/"                # destination directory
 
 [sink.google_cloud_storage]
 # read credentials doc at https://cloud.google.com/docs/authentication/getting-started
@@ -195,12 +219,19 @@ bucket = "your_bucket_seaweedfs"    # an existing bucket
 directory = "/"                     # destination directory
 
 [sink.azure]
-# experimental
+# experimental, let me know if it works
 enabled = false
 account_name = ""
 account_key  = ""
 container = "mycontainer"      # an existing container
-directory = ""                 # destination directory (do not prefix or suffix with "/")
+directory = "/"                # destination directory
+
+[sink.backblaze]
+enabled = false
+b2_account_id = ""
+b2_master_application_key  = ""
+bucket = "mybucket"            # an existing bucket
+directory = "/"                # destination directory
 
 `
 )

weed/command/server.go

@@ -60,7 +60,7 @@ var (
 	serverWhiteListOption         = cmdServer.Flag.String("whiteList", "", "comma separated Ip addresses having write permission. No limit if empty.")
 	serverPeers                   = cmdServer.Flag.String("master.peers", "", "all master nodes in comma separated ip:masterPort list")
 	serverSecureKey               = cmdServer.Flag.String("secure.secret", "", "secret to encrypt Json Web Token(JWT)")
-	serverGarbageThreshold        = cmdServer.Flag.String("garbageThreshold", "0.3", "threshold to vacuum and reclaim spaces")
+	serverGarbageThreshold        = cmdServer.Flag.Float64("garbageThreshold", 0.3, "threshold to vacuum and reclaim spaces")
 	masterPort                    = cmdServer.Flag.Int("master.port", 9333, "master server http listen port")
 	masterMetaFolder              = cmdServer.Flag.String("master.dir", "", "data directory to store meta data, default to same as -dir specified")
 	masterVolumeSizeLimitMB       = cmdServer.Flag.Uint("master.volumeSizeLimitMB", 30*1000, "Master stops directing writes to oversized volumes.")

weed/filer2/entry.go

@@ -43,3 +43,15 @@ func (entry *Entry) Timestamp() time.Time {
 		return entry.Mtime
 	}
 }
+
+func (entry *Entry) ToProtoEntry() *filer_pb.Entry {
+	if entry == nil {
+		return nil
+	}
+	return &filer_pb.Entry{
+		Name:        string(entry.FullPath),
+		IsDirectory: entry.IsDirectory(),
+		Attributes:  EntryAttributeToPb(entry),
+		Chunks:      entry.Chunks,
+	}
+}

weed/filer2/filer.go

@@ -176,7 +176,7 @@ func (f *Filer) DeleteEntryMetaAndData(p FullPath, isRecursive bool, shouldDelet
 	}
 
 	if shouldDeleteChunks {
-		f.deleteChunks(entry.Chunks)
+		f.DeleteChunks(entry.Chunks)
 	}
 
 	if p == "/" {
@@ -229,18 +229,18 @@ func (f *Filer) cacheSetDirectory(dirpath string, dirEntry *Entry, level int) {
 	f.directoryCache.Set(dirpath, dirEntry, time.Duration(minutes)*time.Minute)
 }
 
-func (f *Filer) deleteChunks(chunks []*filer_pb.FileChunk) {
+func (f *Filer) DeleteChunks(chunks []*filer_pb.FileChunk) {
 	for _, chunk := range chunks {
 		f.DeleteFileByFileId(chunk.FileId)
 	}
 }
 
 func (f *Filer) DeleteFileByFileId(fileId string) {
-	fileUrlOnVolume, err := f.MasterClient.LookupFileId(fileId)
+	volumeServer, err := f.MasterClient.LookupVolumeServer(fileId)
 	if err != nil {
 		glog.V(0).Infof("can not find file %s: %v", fileId, err)
 	}
-	if err := operation.DeleteFromVolumeServer(fileUrlOnVolume, ""); err != nil {
+	if _, err := operation.DeleteFilesAtOneVolumeServer(volumeServer, []string{fileId}); err != nil {
 		glog.V(0).Infof("deleting file %s: %v", fileId, err)
 	}
 }
@@ -251,7 +251,7 @@ func (f *Filer) deleteChunksIfNotNew(oldEntry, newEntry *Entry) {
 		return
 	}
 	if newEntry == nil {
-		f.deleteChunks(oldEntry.Chunks)
+		f.DeleteChunks(oldEntry.Chunks)
 	}
 
 	var toDelete []*filer_pb.FileChunk
@@ -268,5 +268,5 @@ func (f *Filer) deleteChunksIfNotNew(oldEntry, newEntry *Entry) {
 			toDelete = append(toDelete, oldChunk)
 		}
 	}
-	f.deleteChunks(toDelete)
+	f.DeleteChunks(toDelete)
 }

weed/filer2/filer_notify.go

@@ -23,23 +23,11 @@ func (f *Filer) NotifyUpdateEvent(oldEntry, newEntry *Entry, deleteChunks bool)
 		notification.Queue.SendMessage(
 			key,
 			&filer_pb.EventNotification{
-				OldEntry:     toProtoEntry(oldEntry),
-				NewEntry:     toProtoEntry(newEntry),
+				OldEntry:     oldEntry.ToProtoEntry(),
+				NewEntry:     newEntry.ToProtoEntry(),
 				DeleteChunks: deleteChunks,
 			},
 		)
 
 	}
 }
-
-func toProtoEntry(entry *Entry) *filer_pb.Entry {
-	if entry == nil {
-		return nil
-	}
-	return &filer_pb.Entry{
-		Name:        string(entry.FullPath),
-		IsDirectory: entry.IsDirectory(),
-		Attributes:  EntryAttributeToPb(entry),
-		Chunks:      entry.Chunks,
-	}
-}

weed/filer2/filer_notify_test.go

@@ -0,0 +1,51 @@
+package filer2
+
+import (
+	"testing"
+	"time"
+
+	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
+	"github.com/golang/protobuf/proto"
+)
+
+func TestProtoMarshalText(t *testing.T) {
+
+	oldEntry := &Entry{
+		FullPath: FullPath("/this/path/to"),
+		Attr: Attr{
+			Mtime:  time.Now(),
+			Mode:   0644,
+			Uid:    1,
+			Mime:   "text/json",
+			TtlSec: 25,
+		},
+		Chunks: []*filer_pb.FileChunk{
+			&filer_pb.FileChunk{
+				FileId:       "234,2423423422",
+				Offset:       234234,
+				Size:         234,
+				Mtime:        12312423,
+				ETag:         "2342342354",
+				SourceFileId: "23234,2342342342",
+			},
+		},
+	}
+
+	notification := &filer_pb.EventNotification{
+		OldEntry:     oldEntry.ToProtoEntry(),
+		NewEntry:     nil,
+		DeleteChunks: true,
+	}
+
+	text := proto.MarshalTextString(notification)
+
+	notification2 := &filer_pb.EventNotification{}
+	proto.UnmarshalText(text, notification2)
+
+	if notification2.OldEntry.Chunks[0].SourceFileId != notification.OldEntry.Chunks[0].SourceFileId {
+		t.Fatalf("marshal/unmarshal error: %s", text)
+	}
+
+	println(text)
+
+}

weed/filesys/dir.go

@@ -70,6 +70,8 @@ func (dir *Dir) Attr(context context.Context, attr *fuse.Attr) error {
 			dir.attributes = resp.Entry.Attributes
 		}
 
+		dir.wfs.listDirectoryEntriesCache.Set(dir.Path, resp.Entry, dir.wfs.option.EntryCacheTtl)
+
 		return nil
 	})
 
@@ -185,24 +187,34 @@ func (dir *Dir) Mkdir(ctx context.Context, req *fuse.MkdirRequest) (fs.Node, err
 func (dir *Dir) Lookup(ctx context.Context, req *fuse.LookupRequest, resp *fuse.LookupResponse) (node fs.Node, err error) {
 
 	var entry *filer_pb.Entry
-	err = dir.wfs.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
 
-		request := &filer_pb.LookupDirectoryEntryRequest{
-			Directory: dir.Path,
-			Name:      req.Name,
-		}
+	item := dir.wfs.listDirectoryEntriesCache.Get(path.Join(dir.Path, req.Name))
+	if item != nil && !item.Expired() {
+		entry = item.Value().(*filer_pb.Entry)
+	}
 
-		glog.V(4).Infof("lookup directory entry: %v", request)
-		resp, err := client.LookupDirectoryEntry(ctx, request)
-		if err != nil {
-			// glog.V(0).Infof("lookup %s/%s: %v", dir.Path, name, err)
-			return fuse.ENOENT
-		}
+	if entry == nil {
+		err = dir.wfs.withFilerClient(func(client filer_pb.SeaweedFilerClient) error {
 
-		entry = resp.Entry
+			request := &filer_pb.LookupDirectoryEntryRequest{
+				Directory: dir.Path,
+				Name:      req.Name,
+			}
 
-		return nil
-	})
+			glog.V(4).Infof("lookup directory entry: %v", request)
+			resp, err := client.LookupDirectoryEntry(ctx, request)
+			if err != nil {
+				// glog.V(0).Infof("lookup %s/%s: %v", dir.Path, name, err)
+				return fuse.ENOENT
+			}
+
+			entry = resp.Entry
+
+			dir.wfs.listDirectoryEntriesCache.Set(path.Join(dir.Path, entry.Name), entry, dir.wfs.option.EntryCacheTtl)
+
+			return nil
+		})
+	}
 
 	if entry != nil {
 		if entry.IsDirectory {
@@ -240,6 +252,8 @@ func (dir *Dir) ReadDirAll(ctx context.Context) (ret []fuse.Dirent, err error) {
 			return fuse.EIO
 		}
 
+		cacheTtl := estimatedCacheTtl(len(resp.Entries))
+
 		for _, entry := range resp.Entries {
 			if entry.IsDirectory {
 				dirent := fuse.Dirent{Name: entry.Name, Type: fuse.DT_Dir}
@@ -248,7 +262,7 @@ func (dir *Dir) ReadDirAll(ctx context.Context) (ret []fuse.Dirent, err error) {
 				dirent := fuse.Dirent{Name: entry.Name, Type: fuse.DT_File}
 				ret = append(ret, dirent)
 			}
-			dir.wfs.listDirectoryEntriesCache.Set(dir.Path+"/"+entry.Name, entry, 300*time.Millisecond)
+			dir.wfs.listDirectoryEntriesCache.Set(path.Join(dir.Path, entry.Name), entry, cacheTtl)
 		}
 
 		return nil
@@ -275,6 +289,8 @@ func (dir *Dir) Remove(ctx context.Context, req *fuse.RemoveRequest) error {
 			return fuse.EIO
 		}
 
+		dir.wfs.listDirectoryEntriesCache.Delete(path.Join(dir.Path, req.Name))
+
 		return nil
 	})
 
@@ -317,7 +333,30 @@ func (dir *Dir) Setattr(ctx context.Context, req *fuse.SetattrRequest, resp *fus
 			return fuse.EIO
 		}
 
+		dir.wfs.listDirectoryEntriesCache.Delete(dir.Path)
+
 		return nil
 	})
 
 }
+
+func estimatedCacheTtl(numEntries int) time.Duration {
+	if numEntries < 100 {
+		// 30 ms per entry
+		return 3 * time.Second
+	}
+	if numEntries < 1000 {
+		// 10 ms per entry
+		return 10 * time.Second
+	}
+	if numEntries < 10000 {
+		// 10 ms per entry
+		return 100 * time.Second
+	}
+	if numEntries < 100000 {
+		// 10 ms per entry
+		return 1000 * time.Second
+	}
+	// 2 ms per entry
+	return time.Duration(numEntries*2) * time.Millisecond
+}

weed/filesys/dirty_page.go

@@ -71,7 +71,7 @@ func (pages *ContinuousDirtyPages) AddPage(ctx context.Context, offset int64, da
 			copy(pages.Data[pages.Size:], data[pages.Size:])
 		} else {
 			if pages.Size != 0 {
-				glog.V(0).Infof("%s/%s add page: pages [%d, %d) write [%d, %d)", pages.f.dir.Path, pages.f.Name, pages.Offset, pages.Offset+pages.Size, offset, offset+int64(len(data)))
+				glog.V(1).Infof("%s/%s add page: pages [%d, %d) write [%d, %d)", pages.f.dir.Path, pages.f.Name, pages.Offset, pages.Offset+pages.Size, offset, offset+int64(len(data)))
 			}
 			return pages.flushAndSave(ctx, offset, data)
 		}

weed/filesys/file.go

@@ -43,7 +43,7 @@ func (file *File) Attr(ctx context.Context, attr *fuse.Attr) error {
 	attr.Gid = file.entry.Attributes.Gid
 	attr.Uid = file.entry.Attributes.Uid
 	attr.Blocks = attr.Size/blockSize + 1
-	attr.BlockSize = blockSize
+	attr.BlockSize = uint32(file.wfs.option.ChunkSizeLimit)
 
 	return nil
 
@@ -153,6 +153,8 @@ func (file *File) maybeLoadAttributes(ctx context.Context) error {
 
 				glog.V(1).Infof("file attr %v %+v: %d", file.fullpath(), file.entry.Attributes, filer2.TotalSize(file.entry.Chunks))
 
+				file.wfs.listDirectoryEntriesCache.Set(file.fullpath(), file.entry, file.wfs.option.EntryCacheTtl)
+
 				return nil
 			})
 

weed/filesys/filehandle.go

@@ -163,7 +163,7 @@ func (fh *FileHandle) Release(ctx context.Context, req *fuse.ReleaseRequest) err
 
 	glog.V(4).Infof("%v release fh %d", fh.f.fullpath(), fh.handle)
 
-	fh.f.wfs.ReleaseHandle(fuse.HandleID(fh.handle))
+	fh.f.wfs.ReleaseHandle(fh.f.fullpath(), fuse.HandleID(fh.handle))
 
 	fh.f.isOpen = false
 

weed/filesys/wfs.go

@@ -1,14 +1,16 @@
 package filesys
 
 import (
+	"fmt"
+	"sync"
+	"time"
+
 	"bazil.org/fuse"
 	"bazil.org/fuse/fs"
-	"fmt"
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/util"
 	"github.com/karlseguin/ccache"
-	"sync"
 )
 
 type Option struct {
@@ -20,6 +22,7 @@ type Option struct {
 	ChunkSizeLimit     int64
 	DataCenter         string
 	DirListingLimit    int
+	EntryCacheTtl      time.Duration
 }
 
 type WFS struct {
@@ -34,8 +37,8 @@ type WFS struct {
 
 func NewSeaweedFileSystem(option *Option) *WFS {
 	return &WFS{
-		option: option,
-		listDirectoryEntriesCache: ccache.New(ccache.Configure().MaxSize(6000).ItemsToPrune(100)),
+		option:                    option,
+		listDirectoryEntriesCache: ccache.New(ccache.Configure().MaxSize(int64(option.DirListingLimit) + 200).ItemsToPrune(100)),
 		pathToHandleIndex:         make(map[string]int),
 	}
 }
@@ -96,11 +99,12 @@ func (wfs *WFS) AcquireHandle(file *File, uid, gid uint32) (fileHandle *FileHand
 	return
 }
 
-func (wfs *WFS) ReleaseHandle(handleId fuse.HandleID) {
+func (wfs *WFS) ReleaseHandle(fullpath string, handleId fuse.HandleID) {
 	wfs.pathToHandleLock.Lock()
 	defer wfs.pathToHandleLock.Unlock()
 
-	glog.V(4).Infoln("releasing handle id", handleId, "current handles lengh", len(wfs.handles))
+	glog.V(4).Infof("%s releasing handle id %dcurrent handles lengh %d", fullpath, handleId, len(wfs.handles))
+	delete(wfs.pathToHandleIndex, fullpath)
 	if int(handleId) < len(wfs.handles) {
 		wfs.handles[int(handleId)] = nil
 	}

weed/notification/aws_sqs/aws_sqs_pub.go

@@ -0,0 +1,92 @@
+package aws_sqs
+
+import (
+	"fmt"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/awserr"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/sqs"
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/notification"
+	"github.com/chrislusf/seaweedfs/weed/util"
+	"github.com/golang/protobuf/proto"
+)
+
+func init() {
+	notification.MessageQueues = append(notification.MessageQueues, &AwsSqsPub{})
+}
+
+type AwsSqsPub struct {
+	svc      *sqs.SQS
+	queueUrl string
+}
+
+func (k *AwsSqsPub) GetName() string {
+	return "aws_sqs"
+}
+
+func (k *AwsSqsPub) Initialize(configuration util.Configuration) (err error) {
+	glog.V(0).Infof("filer.notification.aws_sqs.region: %v", configuration.GetString("region"))
+	glog.V(0).Infof("filer.notification.aws_sqs.sqs_queue_name: %v", configuration.GetString("sqs_queue_name"))
+	return k.initialize(
+		configuration.GetString("aws_access_key_id"),
+		configuration.GetString("aws_secret_access_key"),
+		configuration.GetString("region"),
+		configuration.GetString("sqs_queue_name"),
+	)
+}
+
+func (k *AwsSqsPub) initialize(awsAccessKeyId, aswSecretAccessKey, region, queueName string) (err error) {
+
+	config := &aws.Config{
+		Region: aws.String(region),
+	}
+	if awsAccessKeyId != "" && aswSecretAccessKey != "" {
+		config.Credentials = credentials.NewStaticCredentials(awsAccessKeyId, aswSecretAccessKey, "")
+	}
+
+	sess, err := session.NewSession(config)
+	if err != nil {
+		return fmt.Errorf("create aws session: %v", err)
+	}
+	k.svc = sqs.New(sess)
+
+	result, err := k.svc.GetQueueUrl(&sqs.GetQueueUrlInput{
+		QueueName: aws.String(queueName),
+	})
+	if err != nil {
+		if aerr, ok := err.(awserr.Error); ok && aerr.Code() == sqs.ErrCodeQueueDoesNotExist {
+			return fmt.Errorf("unable to find queue %s", queueName)
+		}
+		return fmt.Errorf("get queue %s url: %v", queueName, err)
+	}
+
+	k.queueUrl = *result.QueueUrl
+
+	return nil
+}
+
+func (k *AwsSqsPub) SendMessage(key string, message proto.Message) (err error) {
+
+	text := proto.MarshalTextString(message)
+
+	_, err = k.svc.SendMessage(&sqs.SendMessageInput{
+		DelaySeconds: aws.Int64(10),
+		MessageAttributes: map[string]*sqs.MessageAttributeValue{
+			"key": &sqs.MessageAttributeValue{
+				DataType:    aws.String("String"),
+				StringValue: aws.String(key),
+			},
+		},
+		MessageBody: aws.String(text),
+		QueueUrl:    &k.queueUrl,
+	})
+
+	if err != nil {
+		return fmt.Errorf("send message to sqs %s: %v", k.queueUrl, err)
+	}
+
+	return nil
+}

weed/notification/configuration.go

@@ -35,7 +35,7 @@ func LoadConfiguration(config *viper.Viper) {
 					store.GetName(), err)
 			}
 			Queue = store
-			glog.V(0).Infof("Configure message queue for %s", store.GetName())
+			glog.V(0).Infof("Configure notification message queue for %s", store.GetName())
 			return
 		}
 	}

weed/notification/google_pub_sub/google_pub_sub.go

@@ -0,0 +1,89 @@
+package google_pub_sub
+
+import (
+	"context"
+	"fmt"
+	"os"
+
+	"cloud.google.com/go/pubsub"
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/notification"
+	"github.com/chrislusf/seaweedfs/weed/util"
+	"github.com/golang/protobuf/proto"
+	"google.golang.org/api/option"
+)
+
+func init() {
+	notification.MessageQueues = append(notification.MessageQueues, &GooglePubSub{})
+}
+
+type GooglePubSub struct {
+	topic *pubsub.Topic
+}
+
+func (k *GooglePubSub) GetName() string {
+	return "google_pub_sub"
+}
+
+func (k *GooglePubSub) Initialize(configuration util.Configuration) (err error) {
+	glog.V(0).Infof("notification.google_pub_sub.project_id: %v", configuration.GetString("project_id"))
+	glog.V(0).Infof("notification.google_pub_sub.topic: %v", configuration.GetString("topic"))
+	return k.initialize(
+		configuration.GetString("google_application_credentials"),
+		configuration.GetString("project_id"),
+		configuration.GetString("topic"),
+	)
+}
+
+func (k *GooglePubSub) initialize(google_application_credentials, projectId, topicName string) (err error) {
+
+	ctx := context.Background()
+	// Creates a client.
+	if google_application_credentials == "" {
+		var found bool
+		google_application_credentials, found = os.LookupEnv("GOOGLE_APPLICATION_CREDENTIALS")
+		if !found {
+			glog.Fatalf("need to specific GOOGLE_APPLICATION_CREDENTIALS env variable or google_application_credentials in filer.toml")
+		}
+	}
+
+	client, err := pubsub.NewClient(ctx, projectId, option.WithCredentialsFile(google_application_credentials))
+	if err != nil {
+		glog.Fatalf("Failed to create client: %v", err)
+	}
+
+	k.topic = client.Topic(topicName)
+	if exists, err := k.topic.Exists(ctx); err == nil {
+		if !exists {
+			k.topic, err = client.CreateTopic(ctx, topicName)
+			if err != nil {
+				glog.Fatalf("Failed to create topic %s: %v", topicName, err)
+			}
+		}
+	} else {
+		glog.Fatalf("Failed to check topic %s: %v", topicName, err)
+	}
+
+	return nil
+}
+
+func (k *GooglePubSub) SendMessage(key string, message proto.Message) (err error) {
+
+	bytes, err := proto.Marshal(message)
+	if err != nil {
+		return
+	}
+
+	ctx := context.Background()
+	result := k.topic.Publish(ctx, &pubsub.Message{
+		Data:       bytes,
+		Attributes: map[string]string{"key": key},
+	})
+
+	_, err = result.Get(ctx)
+	if err != nil {
+		return fmt.Errorf("send message to google pub sub %s: %v", k.topic.String(), err)
+	}
+
+	return nil
+}

weed/notification/kafka/kafka_queue.go

@@ -37,6 +37,9 @@ func (k *KafkaQueue) initialize(hosts []string, topic string) (err error) {
 	config.Producer.Return.Successes = true
 	config.Producer.Return.Errors = true
 	k.producer, err = sarama.NewAsyncProducer(hosts, config)
+	if err != nil {
+		return err
+	}
 	k.topic = topic
 	go k.handleSuccess()
 	go k.handleError()

weed/operation/chunked_file.go

@@ -70,16 +70,22 @@ func (cm *ChunkManifest) Marshal() ([]byte, error) {
 }
 
 func (cm *ChunkManifest) DeleteChunks(master string) error {
-	deleteError := 0
+	var fileIds []string
 	for _, ci := range cm.Chunks {
-		if e := DeleteFile(master, ci.Fid, ""); e != nil {
-			deleteError++
-			glog.V(0).Infof("Delete %s error: %v, master: %s", ci.Fid, e, master)
+		fileIds = append(fileIds, ci.Fid)
+	}
+	results, err := DeleteFiles(master, fileIds)
+	if err != nil {
+		glog.V(0).Infof("delete %+v: %v", fileIds, err)
+		return fmt.Errorf("chunk delete: %v", err)
+	}
+	for _, result := range results {
+		if result.Error != "" {
+			glog.V(0).Infof("delete file %+v: %v", result.FileId, result.Error)
+			return fmt.Errorf("chunk delete %v: %v", result.FileId, result.Error)
 		}
 	}
-	if deleteError > 0 {
-		return errors.New("Not all chunks deleted.")
-	}
+
 	return nil
 }
 

weed/operation/delete_content.go

@@ -1,18 +1,14 @@
 package operation
 
 import (
-	"encoding/json"
+	"context"
 	"errors"
 	"fmt"
-	"net/url"
+	"net/http"
 	"strings"
 	"sync"
 
-	"net/http"
-
-	"github.com/chrislusf/seaweedfs/weed/glog"
-	"github.com/chrislusf/seaweedfs/weed/security"
-	"github.com/chrislusf/seaweedfs/weed/util"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 )
 
 type DeleteResult struct {
@@ -22,27 +18,6 @@ type DeleteResult struct {
 	Error  string `json:"error,omitempty"`
 }
 
-func DeleteFromVolumeServer(fileUrlOnVolume string, jwt security.EncodedJwt) error {
-	err := util.Delete(fileUrlOnVolume, jwt)
-	if err != nil {
-		return fmt.Errorf("Failed to delete %s:%v", fileUrlOnVolume, err)
-	}
-	return nil
-}
-
-func DeleteFile(master string, fileId string, jwt security.EncodedJwt) error {
-	fileUrl, err := LookupFileId(master, fileId)
-	if err != nil {
-		glog.V(0).Infof("Delete %s lookup: %v, master: %s", fileId, err, master)
-		return nil
-	}
-	err = util.Delete(fileUrl, jwt)
-	if err != nil {
-		return fmt.Errorf("Failed to delete %s:%v", fileUrl, err)
-	}
-	return nil
-}
-
 func ParseFileId(fid string) (vid string, key_cookie string, err error) {
 	commaIndex := strings.Index(fid, ",")
 	if commaIndex <= 0 {
@@ -51,20 +26,18 @@ func ParseFileId(fid string) (vid string, key_cookie string, err error) {
 	return fid[:commaIndex], fid[commaIndex+1:], nil
 }
 
-type DeleteFilesResult struct {
-	Errors  []string
-	Results []DeleteResult
-}
+// DeleteFiles batch deletes a list of fileIds
+func DeleteFiles(master string, fileIds []string) ([]*volume_server_pb.DeleteResult, error) {
+
+	var ret []*volume_server_pb.DeleteResult
 
-func DeleteFiles(master string, fileIds []string) (*DeleteFilesResult, error) {
 	vid_to_fileIds := make(map[string][]string)
-	ret := &DeleteFilesResult{}
 	var vids []string
 	for _, fileId := range fileIds {
 		vid, _, err := ParseFileId(fileId)
 		if err != nil {
-			ret.Results = append(ret.Results, DeleteResult{
-				Fid:    vid,
+			ret = append(ret, &volume_server_pb.DeleteResult{
+				FileId: vid,
 				Status: http.StatusBadRequest,
 				Error:  err.Error()},
 			)
@@ -85,7 +58,11 @@ func DeleteFiles(master string, fileIds []string) (*DeleteFilesResult, error) {
 	server_to_fileIds := make(map[string][]string)
 	for vid, result := range lookupResults {
 		if result.Error != "" {
-			ret.Errors = append(ret.Errors, result.Error)
+			ret = append(ret, &volume_server_pb.DeleteResult{
+				FileId: vid,
+				Status: http.StatusBadRequest,
+				Error:  err.Error()},
+			)
 			continue
 		}
 		for _, location := range result.Locations {
@@ -103,25 +80,52 @@ func DeleteFiles(master string, fileIds []string) (*DeleteFilesResult, error) {
 		wg.Add(1)
 		go func(server string, fidList []string) {
 			defer wg.Done()
-			values := make(url.Values)
-			for _, fid := range fidList {
-				values.Add("fid", fid)
+
+			if deleteResults, deleteErr := DeleteFilesAtOneVolumeServer(server, fidList); deleteErr != nil {
+				err = deleteErr
+			} else {
+				ret = append(ret, deleteResults...)
 			}
-			jsonBlob, err := util.Post("http://"+server+"/delete", values)
-			if err != nil {
-				ret.Errors = append(ret.Errors, err.Error()+" "+string(jsonBlob))
-				return
-			}
-			var result []DeleteResult
-			err = json.Unmarshal(jsonBlob, &result)
-			if err != nil {
-				ret.Errors = append(ret.Errors, err.Error()+" "+string(jsonBlob))
-				return
-			}
-			ret.Results = append(ret.Results, result...)
+
 		}(server, fidList)
 	}
 	wg.Wait()
 
-	return ret, nil
+	return ret, err
+}
+
+// DeleteFilesAtOneVolumeServer deletes a list of files that is on one volume server via gRpc
+func DeleteFilesAtOneVolumeServer(volumeServer string, fileIds []string) (ret []*volume_server_pb.DeleteResult, err error) {
+
+	err = WithVolumeServerClient(volumeServer, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
+
+		req := &volume_server_pb.BatchDeleteRequest{
+			FileIds: fileIds,
+		}
+
+		resp, err := volumeServerClient.BatchDelete(context.Background(), req)
+
+		fmt.Printf("deleted %v %v: %v\n", fileIds, err, resp)
+
+		if err != nil {
+			return err
+		}
+
+		ret = append(ret, resp.Results...)
+
+		return nil
+	})
+
+	if err != nil {
+		return
+	}
+
+	for _, result := range ret {
+		if result.Error != "" {
+			return nil, fmt.Errorf("delete fileId %s: %v", result.FileId, result.Error)
+		}
+	}
+
+	return
+
 }

weed/operation/grpc_client.go

@@ -0,0 +1,53 @@
+package operation
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/master_pb"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/chrislusf/seaweedfs/weed/util"
+)
+
+func WithVolumeServerClient(volumeServer string, fn func(volume_server_pb.VolumeServerClient) error) error {
+
+	grpcAddress, err := toVolumeServerGrpcAddress(volumeServer)
+	if err != nil {
+		return err
+	}
+
+	grpcConnection, err := util.GrpcDial(grpcAddress)
+	if err != nil {
+		return fmt.Errorf("fail to dial %s: %v", grpcAddress, err)
+	}
+	defer grpcConnection.Close()
+
+	client := volume_server_pb.NewVolumeServerClient(grpcConnection)
+
+	return fn(client)
+}
+
+func toVolumeServerGrpcAddress(volumeServer string) (grpcAddress string, err error) {
+	sepIndex := strings.LastIndex(volumeServer, ":")
+	port, err := strconv.Atoi(volumeServer[sepIndex+1:])
+	if err != nil {
+		glog.Errorf("failed to parse volume server address: %v", volumeServer)
+		return "", err
+	}
+	return fmt.Sprintf("%s:%d", volumeServer[0:sepIndex], port+10000), nil
+}
+
+func withMasterServerClient(masterServer string, fn func(masterClient master_pb.SeaweedClient) error) error {
+
+	grpcConnection, err := util.GrpcDial(masterServer)
+	if err != nil {
+		return fmt.Errorf("fail to dial %s: %v", masterServer, err)
+	}
+	defer grpcConnection.Close()
+
+	client := master_pb.NewSeaweedClient(grpcConnection)
+
+	return fn(client)
+}

weed/operation/lookup.go

@@ -1,6 +1,7 @@
 package operation
 
 import (
+	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -9,6 +10,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/chrislusf/seaweedfs/weed/pb/master_pb"
 	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
@@ -95,24 +97,41 @@ func LookupVolumeIds(server string, vids []string) (map[string]LookupResult, err
 	}
 
 	//only query unknown_vids
-	values := make(url.Values)
-	for _, vid := range unknown_vids {
-		values.Add("volumeId", vid)
-	}
-	jsonBlob, err := util.Post("http://"+server+"/vol/lookup", values)
+
+	err := withMasterServerClient(server, func(masterClient master_pb.SeaweedClient) error {
+		req := &master_pb.LookupVolumeRequest{
+			VolumeIds: unknown_vids,
+		}
+		resp, grpcErr := masterClient.LookupVolume(context.Background(), req)
+		if grpcErr != nil {
+			return grpcErr
+		}
+
+		//set newly checked vids to cache
+		for _, vidLocations := range resp.VolumeIdLocations {
+			var locations []Location
+			for _, loc := range vidLocations.Locations {
+				locations = append(locations, Location{
+					Url:       loc.Url,
+					PublicUrl: loc.PublicUrl,
+				})
+			}
+			if vidLocations.Error != "" {
+				vc.Set(vidLocations.VolumeId, locations, 10*time.Minute)
+			}
+			ret[vidLocations.VolumeId] = LookupResult{
+				VolumeId:  vidLocations.VolumeId,
+				Locations: locations,
+				Error:     vidLocations.Error,
+			}
+		}
+
+		return nil
+	})
+
 	if err != nil {
 		return nil, err
 	}
-	err = json.Unmarshal(jsonBlob, &ret)
-	if err != nil {
-		return nil, errors.New(err.Error() + " " + string(jsonBlob))
-	}
-
-	//set newly checked vids to cache
-	for _, vid := range unknown_vids {
-		locations := ret[vid].Locations
-		vc.Set(vid, locations, 10*time.Minute)
-	}
 
 	return ret, nil
 }

weed/operation/sync_volume.go

@@ -1,55 +1,45 @@
 package operation
 
 import (
-	"encoding/json"
-	"fmt"
-	"net/url"
+	"context"
 
-	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	. "github.com/chrislusf/seaweedfs/weed/storage/types"
 	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
-type SyncVolumeResponse struct {
-	Replication     string `json:"Replication,omitempty"`
-	Ttl             string `json:"Ttl,omitempty"`
-	TailOffset      uint64 `json:"TailOffset,omitempty"`
-	CompactRevision uint16 `json:"CompactRevision,omitempty"`
-	IdxFileSize     uint64 `json:"IdxFileSize,omitempty"`
-	Error           string `json:"error,omitempty"`
-}
+func GetVolumeSyncStatus(server string, vid uint32) (resp *volume_server_pb.VolumeSyncStatusResponse, err error) {
 
-func GetVolumeSyncStatus(server string, vid string) (*SyncVolumeResponse, error) {
-	values := make(url.Values)
-	values.Add("volume", vid)
-	jsonBlob, err := util.Post("http://"+server+"/admin/sync/status", values)
-	glog.V(2).Info("sync volume result :", string(jsonBlob))
-	if err != nil {
-		return nil, err
-	}
-	var ret SyncVolumeResponse
-	err = json.Unmarshal(jsonBlob, &ret)
-	if err != nil {
-		return nil, err
-	}
-	if ret.Error != "" {
-		return nil, fmt.Errorf("Volume %s get sync status error: %s", vid, ret.Error)
-	}
-	return &ret, nil
+	WithVolumeServerClient(server, func(client volume_server_pb.VolumeServerClient) error {
+		resp, err = client.VolumeSyncStatus(context.Background(), &volume_server_pb.VolumeSyncStatusRequest{
+			VolumdId: vid,
+		})
+		return nil
+	})
+
+	return
 }
+
+func GetVolumeIdxEntries(server string, vid uint32, eachEntryFn func(key NeedleId, offset Offset, size uint32)) error {
+
+	return WithVolumeServerClient(server, func(client volume_server_pb.VolumeServerClient) error {
+		resp, err := client.VolumeSyncIndex(context.Background(), &volume_server_pb.VolumeSyncIndexRequest{
+			VolumdId: vid,
+		})
+		if err != nil {
+			return err
+		}
+
+		dataSize := len(resp.IndexFileContent)
+
+		for idx := 0; idx+NeedleEntrySize <= dataSize; idx += NeedleEntrySize {
+			line := resp.IndexFileContent[idx : idx+NeedleEntrySize]
+			key := BytesToNeedleId(line[:NeedleIdSize])
+			offset := BytesToOffset(line[NeedleIdSize : NeedleIdSize+OffsetSize])
+			size := util.BytesToUint32(line[NeedleIdSize+OffsetSize : NeedleIdSize+OffsetSize+SizeSize])
+			eachEntryFn(key, offset, size)
+		}
 
-func GetVolumeIdxEntries(server string, vid string, eachEntryFn func(key NeedleId, offset Offset, size uint32)) error {
-	values := make(url.Values)
-	values.Add("volume", vid)
-	line := make([]byte, NeedleEntrySize)
-	err := util.GetBufferStream("http://"+server+"/admin/sync/index", values, line, func(bytes []byte) {
-		key := BytesToNeedleId(line[:NeedleIdSize])
-		offset := BytesToOffset(line[NeedleIdSize : NeedleIdSize+OffsetSize])
-		size := util.BytesToUint32(line[NeedleIdSize+OffsetSize : NeedleIdSize+OffsetSize+SizeSize])
-		eachEntryFn(key, offset, size)
+		return nil
 	})
-	if err != nil {
-		return err
-	}
-	return nil
 }

weed/operation/upload_content.go

@@ -37,6 +37,7 @@ func init() {
 
 var fileNameEscaper = strings.NewReplacer("\\", "\\\\", "\"", "\\\"")
 
+// Upload sends a POST request to a volume server to upload the content
 func Upload(uploadUrl string, filename string, reader io.Reader, isGzipped bool, mtype string, pairMap map[string]string, jwt security.EncodedJwt) (*UploadResult, error) {
 	return upload_content(uploadUrl, func(w io.Writer) (err error) {
 		_, err = io.Copy(w, reader)

weed/pb/master.proto

@@ -9,6 +9,8 @@ service Seaweed {
     }
     rpc KeepConnected (stream ClientListenRequest) returns (stream VolumeLocation) {
     }
+    rpc LookupVolume (LookupVolumeRequest) returns (LookupVolumeResponse) {
+    }
 }
 
 //////////////////////////////////////////////////
@@ -69,3 +71,21 @@ message VolumeLocation {
     repeated uint32 new_vids = 3;
     repeated uint32 deleted_vids = 4;
 }
+
+message LookupVolumeRequest {
+    repeated string volume_ids = 1;
+    string collection = 2; // optional, a bit faster if provided.
+}
+message LookupVolumeResponse {
+    message VolumeIdLocation {
+        string volume_id = 1;
+        repeated Location locations = 2;
+        string error = 3;
+    }
+    repeated VolumeIdLocation volume_id_locations = 1;
+}
+
+message Location {
+    string url = 1;
+    string public_url = 2;
+}

weed/pb/master_pb/master.pb.go

@@ -16,6 +16,9 @@ It has these top-level messages:
 	SuperBlockExtra
 	ClientListenRequest
 	VolumeLocation
+	LookupVolumeRequest
+	LookupVolumeResponse
+	Location
 */
 package master_pb
 
@@ -370,6 +373,104 @@ func (m *VolumeLocation) GetDeletedVids() []uint32 {
 	return nil
 }
 
+type LookupVolumeRequest struct {
+	VolumeIds  []string `protobuf:"bytes,1,rep,name=volume_ids,json=volumeIds" json:"volume_ids,omitempty"`
+	Collection string   `protobuf:"bytes,2,opt,name=collection" json:"collection,omitempty"`
+}
+
+func (m *LookupVolumeRequest) Reset()                    { *m = LookupVolumeRequest{} }
+func (m *LookupVolumeRequest) String() string            { return proto.CompactTextString(m) }
+func (*LookupVolumeRequest) ProtoMessage()               {}
+func (*LookupVolumeRequest) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{7} }
+
+func (m *LookupVolumeRequest) GetVolumeIds() []string {
+	if m != nil {
+		return m.VolumeIds
+	}
+	return nil
+}
+
+func (m *LookupVolumeRequest) GetCollection() string {
+	if m != nil {
+		return m.Collection
+	}
+	return ""
+}
+
+type LookupVolumeResponse struct {
+	VolumeIdLocations []*LookupVolumeResponse_VolumeIdLocation `protobuf:"bytes,1,rep,name=volume_id_locations,json=volumeIdLocations" json:"volume_id_locations,omitempty"`
+}
+
+func (m *LookupVolumeResponse) Reset()                    { *m = LookupVolumeResponse{} }
+func (m *LookupVolumeResponse) String() string            { return proto.CompactTextString(m) }
+func (*LookupVolumeResponse) ProtoMessage()               {}
+func (*LookupVolumeResponse) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{8} }
+
+func (m *LookupVolumeResponse) GetVolumeIdLocations() []*LookupVolumeResponse_VolumeIdLocation {
+	if m != nil {
+		return m.VolumeIdLocations
+	}
+	return nil
+}
+
+type LookupVolumeResponse_VolumeIdLocation struct {
+	VolumeId  string      `protobuf:"bytes,1,opt,name=volume_id,json=volumeId" json:"volume_id,omitempty"`
+	Locations []*Location `protobuf:"bytes,2,rep,name=locations" json:"locations,omitempty"`
+	Error     string      `protobuf:"bytes,3,opt,name=error" json:"error,omitempty"`
+}
+
+func (m *LookupVolumeResponse_VolumeIdLocation) Reset()         { *m = LookupVolumeResponse_VolumeIdLocation{} }
+func (m *LookupVolumeResponse_VolumeIdLocation) String() string { return proto.CompactTextString(m) }
+func (*LookupVolumeResponse_VolumeIdLocation) ProtoMessage()    {}
+func (*LookupVolumeResponse_VolumeIdLocation) Descriptor() ([]byte, []int) {
+	return fileDescriptor0, []int{8, 0}
+}
+
+func (m *LookupVolumeResponse_VolumeIdLocation) GetVolumeId() string {
+	if m != nil {
+		return m.VolumeId
+	}
+	return ""
+}
+
+func (m *LookupVolumeResponse_VolumeIdLocation) GetLocations() []*Location {
+	if m != nil {
+		return m.Locations
+	}
+	return nil
+}
+
+func (m *LookupVolumeResponse_VolumeIdLocation) GetError() string {
+	if m != nil {
+		return m.Error
+	}
+	return ""
+}
+
+type Location struct {
+	Url       string `protobuf:"bytes,1,opt,name=url" json:"url,omitempty"`
+	PublicUrl string `protobuf:"bytes,2,opt,name=public_url,json=publicUrl" json:"public_url,omitempty"`
+}
+
+func (m *Location) Reset()                    { *m = Location{} }
+func (m *Location) String() string            { return proto.CompactTextString(m) }
+func (*Location) ProtoMessage()               {}
+func (*Location) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{9} }
+
+func (m *Location) GetUrl() string {
+	if m != nil {
+		return m.Url
+	}
+	return ""
+}
+
+func (m *Location) GetPublicUrl() string {
+	if m != nil {
+		return m.PublicUrl
+	}
+	return ""
+}
+
 func init() {
 	proto.RegisterType((*Heartbeat)(nil), "master_pb.Heartbeat")
 	proto.RegisterType((*HeartbeatResponse)(nil), "master_pb.HeartbeatResponse")
@@ -379,6 +480,10 @@ func init() {
 	proto.RegisterType((*SuperBlockExtra_ErasureCoding)(nil), "master_pb.SuperBlockExtra.ErasureCoding")
 	proto.RegisterType((*ClientListenRequest)(nil), "master_pb.ClientListenRequest")
 	proto.RegisterType((*VolumeLocation)(nil), "master_pb.VolumeLocation")
+	proto.RegisterType((*LookupVolumeRequest)(nil), "master_pb.LookupVolumeRequest")
+	proto.RegisterType((*LookupVolumeResponse)(nil), "master_pb.LookupVolumeResponse")
+	proto.RegisterType((*LookupVolumeResponse_VolumeIdLocation)(nil), "master_pb.LookupVolumeResponse.VolumeIdLocation")
+	proto.RegisterType((*Location)(nil), "master_pb.Location")
 }
 
 // Reference imports to suppress errors if they are not otherwise used.
@@ -394,6 +499,7 @@ const _ = grpc.SupportPackageIsVersion4
 type SeaweedClient interface {
 	SendHeartbeat(ctx context.Context, opts ...grpc.CallOption) (Seaweed_SendHeartbeatClient, error)
 	KeepConnected(ctx context.Context, opts ...grpc.CallOption) (Seaweed_KeepConnectedClient, error)
+	LookupVolume(ctx context.Context, in *LookupVolumeRequest, opts ...grpc.CallOption) (*LookupVolumeResponse, error)
 }
 
 type seaweedClient struct {
@@ -466,11 +572,21 @@ func (x *seaweedKeepConnectedClient) Recv() (*VolumeLocation, error) {
 	return m, nil
 }
 
+func (c *seaweedClient) LookupVolume(ctx context.Context, in *LookupVolumeRequest, opts ...grpc.CallOption) (*LookupVolumeResponse, error) {
+	out := new(LookupVolumeResponse)
+	err := grpc.Invoke(ctx, "/master_pb.Seaweed/LookupVolume", in, out, c.cc, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
 // Server API for Seaweed service
 
 type SeaweedServer interface {
 	SendHeartbeat(Seaweed_SendHeartbeatServer) error
 	KeepConnected(Seaweed_KeepConnectedServer) error
+	LookupVolume(context.Context, *LookupVolumeRequest) (*LookupVolumeResponse, error)
 }
 
 func RegisterSeaweedServer(s *grpc.Server, srv SeaweedServer) {
@@ -529,10 +645,33 @@ func (x *seaweedKeepConnectedServer) Recv() (*ClientListenRequest, error) {
 	return m, nil
 }
 
+func _Seaweed_LookupVolume_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(LookupVolumeRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(SeaweedServer).LookupVolume(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: "/master_pb.Seaweed/LookupVolume",
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(SeaweedServer).LookupVolume(ctx, req.(*LookupVolumeRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
 var _Seaweed_serviceDesc = grpc.ServiceDesc{
 	ServiceName: "master_pb.Seaweed",
 	HandlerType: (*SeaweedServer)(nil),
-	Methods:     []grpc.MethodDesc{},
+	Methods: []grpc.MethodDesc{
+		{
+			MethodName: "LookupVolume",
+			Handler:    _Seaweed_LookupVolume_Handler,
+		},
+	},
 	Streams: []grpc.StreamDesc{
 		{
 			StreamName:    "SendHeartbeat",
@@ -553,51 +692,59 @@ var _Seaweed_serviceDesc = grpc.ServiceDesc{
 func init() { proto.RegisterFile("master.proto", fileDescriptor0) }
 
 var fileDescriptor0 = []byte{
-	// 724 bytes of a gzipped FileDescriptorProto
-	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0x7c, 0x54, 0xc1, 0x6e, 0xd3, 0x4c,
-	0x10, 0xfe, 0x9d, 0xa4, 0x49, 0x3c, 0x69, 0xd2, 0x74, 0xff, 0x5f, 0xbf, 0xdc, 0x52, 0x4a, 0x30,
-	0x17, 0x23, 0x50, 0x84, 0xca, 0x99, 0x4b, 0xa3, 0x22, 0xaa, 0x16, 0xb5, 0x72, 0x44, 0x0f, 0x5c,
-	0xac, 0x8d, 0x3d, 0xad, 0x56, 0x5d, 0xaf, 0xcd, 0xee, 0xa6, 0x8d, 0x7b, 0xe1, 0x6d, 0x78, 0x0c,
-	0x4e, 0xbc, 0x0d, 0x2f, 0x81, 0x76, 0x6d, 0xa7, 0x21, 0x2d, 0xdc, 0x66, 0xbf, 0x99, 0xd9, 0x9d,
-	0xfd, 0xbe, 0x99, 0x81, 0xcd, 0x94, 0x2a, 0x8d, 0x72, 0x9c, 0xcb, 0x4c, 0x67, 0xc4, 0x2d, 0x4f,
-	0x51, 0x3e, 0xf3, 0x7f, 0x36, 0xc0, 0xfd, 0x80, 0x54, 0xea, 0x19, 0x52, 0x4d, 0x06, 0xd0, 0x60,
-	0xb9, 0xe7, 0x8c, 0x9c, 0xc0, 0x0d, 0x1b, 0x2c, 0x27, 0x04, 0x5a, 0x79, 0x26, 0xb5, 0xd7, 0x18,
-	0x39, 0x41, 0x3f, 0xb4, 0x36, 0x79, 0x0a, 0x90, 0xcf, 0x67, 0x9c, 0xc5, 0xd1, 0x5c, 0x72, 0xaf,
-	0x69, 0x63, 0xdd, 0x12, 0xf9, 0x24, 0x39, 0x09, 0x60, 0x98, 0xd2, 0x45, 0x74, 0x93, 0xf1, 0x79,
-	0x8a, 0x51, 0x9c, 0xcd, 0x85, 0xf6, 0x5a, 0x36, 0x7d, 0x90, 0xd2, 0xc5, 0x85, 0x85, 0x27, 0x06,
-	0x25, 0x23, 0x53, 0xd5, 0x22, 0xba, 0x64, 0x1c, 0xa3, 0x6b, 0x2c, 0xbc, 0x8d, 0x91, 0x13, 0xb4,
-	0x42, 0x48, 0xe9, 0xe2, 0x3d, 0xe3, 0x78, 0x82, 0x05, 0x79, 0x06, 0xbd, 0x84, 0x6a, 0x1a, 0xc5,
-	0x28, 0x34, 0x4a, 0xaf, 0x6d, 0xdf, 0x02, 0x03, 0x4d, 0x2c, 0x62, 0xea, 0x93, 0x34, 0xbe, 0xf6,
-	0x3a, 0xd6, 0x63, 0x6d, 0x53, 0x1f, 0x4d, 0x52, 0x26, 0x22, 0x5b, 0x79, 0xd7, 0x3e, 0xed, 0x5a,
-	0xe4, 0xdc, 0x94, 0xff, 0x0e, 0x3a, 0x65, 0x6d, 0xca, 0x73, 0x47, 0xcd, 0xa0, 0x77, 0xf0, 0x62,
-	0xbc, 0x64, 0x63, 0x5c, 0x96, 0x77, 0x2c, 0x2e, 0x33, 0x99, 0x52, 0xcd, 0x32, 0xf1, 0x11, 0x95,
-	0xa2, 0x57, 0x18, 0xd6, 0x39, 0x64, 0x07, 0xba, 0x02, 0x6f, 0xa3, 0x1b, 0x96, 0x28, 0x0f, 0x46,
-	0xcd, 0xa0, 0x1f, 0x76, 0x04, 0xde, 0x5e, 0xb0, 0x44, 0x91, 0xe7, 0xb0, 0x99, 0x20, 0x47, 0x8d,
-	0x49, 0xe9, 0xee, 0x59, 0x77, 0xaf, 0xc2, 0x4c, 0x88, 0xaf, 0x60, 0x7b, 0x49, 0x76, 0x88, 0x2a,
-	0xcf, 0x84, 0x42, 0x12, 0xc0, 0x56, 0x79, 0xfb, 0x94, 0xdd, 0xe1, 0x29, 0x4b, 0x99, 0xb6, 0x0a,
-	0xb4, 0xc2, 0x75, 0x98, 0xec, 0x81, 0xab, 0x30, 0x96, 0xa8, 0x4f, 0xb0, 0xb0, 0x9a, 0xb8, 0xe1,
-	0x3d, 0x40, 0xfe, 0x87, 0x36, 0x47, 0x9a, 0xa0, 0xac, 0x44, 0xa9, 0x4e, 0xfe, 0x8f, 0x06, 0x78,
-	0x7f, 0xfa, 0x98, 0x55, 0x3c, 0xb1, 0xef, 0xf5, 0xc3, 0x06, 0x4b, 0x0c, 0xa3, 0x8a, 0xdd, 0xa1,
-	0xbd, 0xbd, 0x15, 0x5a, 0x9b, 0xec, 0x03, 0xc4, 0x19, 0xe7, 0x18, 0x9b, 0xc4, 0xea, 0xf2, 0x15,
-	0xc4, 0x30, 0x6e, 0x45, 0xbc, 0x17, 0xbb, 0x15, 0xba, 0x06, 0x29, 0x75, 0x5e, 0xf2, 0x52, 0x05,
-	0x94, 0x3a, 0x57, 0xbc, 0x94, 0x21, 0xaf, 0x81, 0xd4, 0xd4, 0xcd, 0x8a, 0x65, 0x60, 0xdb, 0x06,
-	0x0e, 0x2b, 0xcf, 0x61, 0x51, 0x47, 0x3f, 0x01, 0x57, 0x22, 0x4d, 0xa2, 0x4c, 0xf0, 0xc2, 0x4a,
-	0xdf, 0x0d, 0xbb, 0x06, 0x38, 0x13, 0xbc, 0x20, 0xaf, 0x60, 0x5b, 0x62, 0xce, 0x59, 0x4c, 0xa3,
-	0x9c, 0xd3, 0x18, 0x53, 0x14, 0x75, 0x17, 0x0c, 0x2b, 0xc7, 0x79, 0x8d, 0x13, 0x0f, 0x3a, 0x37,
-	0x28, 0x95, 0xf9, 0x96, 0x6b, 0x43, 0xea, 0x23, 0x19, 0x42, 0x53, 0x6b, 0xee, 0x81, 0x45, 0x8d,
-	0xe9, 0x77, 0x60, 0xe3, 0x28, 0xcd, 0x75, 0xe1, 0x7f, 0x77, 0x60, 0x6b, 0x3a, 0xcf, 0x51, 0x1e,
-	0xf2, 0x2c, 0xbe, 0x3e, 0x5a, 0x68, 0x49, 0xc9, 0x19, 0x0c, 0x50, 0x52, 0x35, 0x97, 0xa6, 0xf6,
-	0x84, 0x89, 0x2b, 0x4b, 0x69, 0xef, 0x20, 0x58, 0x69, 0xae, 0xb5, 0x9c, 0xf1, 0x51, 0x99, 0x30,
-	0xb1, 0xf1, 0x61, 0x1f, 0x57, 0x8f, 0xbb, 0x9f, 0xa1, 0xff, 0x9b, 0xdf, 0x08, 0x63, 0x1a, 0xbf,
-	0x92, 0xca, 0xda, 0x46, 0xf1, 0x9c, 0x4a, 0xa6, 0x8b, 0x6a, 0x40, 0xab, 0x93, 0x11, 0xa4, 0x9a,
-	0x3f, 0xd3, 0x87, 0x4d, 0xdb, 0x87, 0x6e, 0x89, 0x1c, 0x27, 0xca, 0x7f, 0x09, 0xff, 0x4e, 0x38,
-	0x43, 0xa1, 0x4f, 0x99, 0xd2, 0x28, 0x42, 0xfc, 0x32, 0x47, 0xa5, 0xcd, 0x0b, 0x82, 0xa6, 0x58,
-	0x8d, 0xbf, 0xb5, 0xfd, 0xaf, 0x30, 0x28, 0x5b, 0xe7, 0x34, 0x8b, 0x6d, 0xdf, 0x18, 0x62, 0xcc,
-	0xdc, 0x97, 0x41, 0xc6, 0x5c, 0x5b, 0x08, 0x8d, 0xf5, 0x85, 0xb0, 0x3a, 0x31, 0xcd, 0xbf, 0x4f,
-	0x4c, 0xeb, 0xc1, 0xc4, 0x1c, 0x7c, 0x73, 0xa0, 0x33, 0x45, 0x7a, 0x8b, 0x98, 0x90, 0x63, 0xe8,
-	0x4f, 0x51, 0x24, 0xf7, 0xeb, 0xea, 0xbf, 0x15, 0x76, 0x97, 0xe8, 0xee, 0xde, 0x63, 0x68, 0x3d,
-	0x6d, 0xfe, 0x3f, 0x81, 0xf3, 0xc6, 0x21, 0xe7, 0xd0, 0x3f, 0x41, 0xcc, 0x27, 0x99, 0x10, 0x18,
-	0x6b, 0x4c, 0xc8, 0xfe, 0x4a, 0xd2, 0x23, 0xe4, 0xec, 0xee, 0x3c, 0xd8, 0x12, 0x35, 0x23, 0xe5,
-	0x8d, 0xb3, 0xb6, 0x5d, 0xad, 0x6f, 0x7f, 0x05, 0x00, 0x00, 0xff, 0xff, 0x19, 0xb5, 0xf3, 0x01,
-	0x6a, 0x05, 0x00, 0x00,
+	// 855 bytes of a gzipped FileDescriptorProto
+	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0x9c, 0x55, 0xcf, 0x6f, 0xdc, 0x44,
+	0x14, 0xae, 0xbd, 0x9b, 0xec, 0xfa, 0x6d, 0x36, 0xdd, 0x4c, 0x22, 0xe4, 0x2e, 0xa5, 0x5d, 0xcc,
+	0xc5, 0x08, 0x14, 0x95, 0x70, 0x44, 0x5c, 0xba, 0x0a, 0x22, 0x4a, 0x50, 0x83, 0x03, 0x3d, 0x70,
+	0x31, 0xb3, 0xf6, 0x6b, 0x35, 0xca, 0x78, 0x6c, 0x66, 0x66, 0x93, 0x75, 0x2f, 0xfc, 0x67, 0x9c,
+	0xf8, 0x6f, 0xb8, 0x71, 0xe3, 0x86, 0x66, 0x3c, 0xde, 0x1f, 0x6e, 0x00, 0x89, 0xdb, 0x9b, 0x6f,
+	0xde, 0x8f, 0x6f, 0xde, 0xf7, 0x9e, 0x0d, 0x07, 0x05, 0x55, 0x1a, 0xe5, 0x69, 0x25, 0x4b, 0x5d,
+	0x92, 0xa0, 0x39, 0xa5, 0xd5, 0x22, 0xfa, 0xc3, 0x87, 0xe0, 0x5b, 0xa4, 0x52, 0x2f, 0x90, 0x6a,
+	0x72, 0x08, 0x3e, 0xab, 0x42, 0x6f, 0xe6, 0xc5, 0x41, 0xe2, 0xb3, 0x8a, 0x10, 0xe8, 0x57, 0xa5,
+	0xd4, 0xa1, 0x3f, 0xf3, 0xe2, 0x71, 0x62, 0x6d, 0xf2, 0x11, 0x40, 0xb5, 0x5c, 0x70, 0x96, 0xa5,
+	0x4b, 0xc9, 0xc3, 0x9e, 0xf5, 0x0d, 0x1a, 0xe4, 0x47, 0xc9, 0x49, 0x0c, 0x93, 0x82, 0xae, 0xd2,
+	0xbb, 0x92, 0x2f, 0x0b, 0x4c, 0xb3, 0x72, 0x29, 0x74, 0xd8, 0xb7, 0xe1, 0x87, 0x05, 0x5d, 0xbd,
+	0xb6, 0xf0, 0xdc, 0xa0, 0x64, 0x66, 0x58, 0xad, 0xd2, 0x37, 0x8c, 0x63, 0x7a, 0x8b, 0x75, 0xb8,
+	0x37, 0xf3, 0xe2, 0x7e, 0x02, 0x05, 0x5d, 0x7d, 0xc3, 0x38, 0x5e, 0x62, 0x4d, 0x9e, 0xc3, 0x28,
+	0xa7, 0x9a, 0xa6, 0x19, 0x0a, 0x8d, 0x32, 0xdc, 0xb7, 0xb5, 0xc0, 0x40, 0x73, 0x8b, 0x18, 0x7e,
+	0x92, 0x66, 0xb7, 0xe1, 0xc0, 0xde, 0x58, 0xdb, 0xf0, 0xa3, 0x79, 0xc1, 0x44, 0x6a, 0x99, 0x0f,
+	0x6d, 0xe9, 0xc0, 0x22, 0xd7, 0x86, 0xfe, 0xd7, 0x30, 0x68, 0xb8, 0xa9, 0x30, 0x98, 0xf5, 0xe2,
+	0xd1, 0xd9, 0x27, 0xa7, 0xeb, 0x6e, 0x9c, 0x36, 0xf4, 0x2e, 0xc4, 0x9b, 0x52, 0x16, 0x54, 0xb3,
+	0x52, 0x7c, 0x87, 0x4a, 0xd1, 0xb7, 0x98, 0xb4, 0x31, 0xe4, 0x09, 0x0c, 0x05, 0xde, 0xa7, 0x77,
+	0x2c, 0x57, 0x21, 0xcc, 0x7a, 0xf1, 0x38, 0x19, 0x08, 0xbc, 0x7f, 0xcd, 0x72, 0x45, 0x3e, 0x86,
+	0x83, 0x1c, 0x39, 0x6a, 0xcc, 0x9b, 0xeb, 0x91, 0xbd, 0x1e, 0x39, 0xcc, 0xb8, 0x44, 0x0a, 0x8e,
+	0xd6, 0xcd, 0x4e, 0x50, 0x55, 0xa5, 0x50, 0x48, 0x62, 0x78, 0xdc, 0x64, 0xbf, 0x61, 0xef, 0xf0,
+	0x8a, 0x15, 0x4c, 0x5b, 0x05, 0xfa, 0x49, 0x17, 0x26, 0x4f, 0x21, 0x50, 0x98, 0x49, 0xd4, 0x97,
+	0x58, 0x5b, 0x4d, 0x82, 0x64, 0x03, 0x90, 0x0f, 0x60, 0x9f, 0x23, 0xcd, 0x51, 0x3a, 0x51, 0xdc,
+	0x29, 0xfa, 0xdd, 0x87, 0xf0, 0x9f, 0x1e, 0x66, 0x15, 0xcf, 0x6d, 0xbd, 0x71, 0xe2, 0xb3, 0xdc,
+	0x74, 0x54, 0xb1, 0x77, 0x68, 0xb3, 0xf7, 0x13, 0x6b, 0x93, 0x67, 0x00, 0x59, 0xc9, 0x39, 0x66,
+	0x26, 0xd0, 0x25, 0xdf, 0x42, 0x4c, 0xc7, 0xad, 0x88, 0x1b, 0xb1, 0xfb, 0x49, 0x60, 0x90, 0x46,
+	0xe7, 0x75, 0x5f, 0x9c, 0x43, 0xa3, 0xb3, 0xeb, 0x4b, 0xe3, 0xf2, 0x39, 0x90, 0xb6, 0x75, 0x8b,
+	0x7a, 0xed, 0xb8, 0x6f, 0x1d, 0x27, 0xee, 0xe6, 0x65, 0xdd, 0x7a, 0x7f, 0x08, 0x81, 0x44, 0x9a,
+	0xa7, 0xa5, 0xe0, 0xb5, 0x95, 0x7e, 0x98, 0x0c, 0x0d, 0xf0, 0x4a, 0xf0, 0x9a, 0x7c, 0x06, 0x47,
+	0x12, 0x2b, 0xce, 0x32, 0x9a, 0x56, 0x9c, 0x66, 0x58, 0xa0, 0x68, 0xa7, 0x60, 0xe2, 0x2e, 0xae,
+	0x5b, 0x9c, 0x84, 0x30, 0xb8, 0x43, 0xa9, 0xcc, 0xb3, 0x02, 0xeb, 0xd2, 0x1e, 0xc9, 0x04, 0x7a,
+	0x5a, 0xf3, 0x10, 0x2c, 0x6a, 0xcc, 0x68, 0x00, 0x7b, 0xe7, 0x45, 0xa5, 0xeb, 0xe8, 0x37, 0x0f,
+	0x1e, 0xdf, 0x2c, 0x2b, 0x94, 0x2f, 0x79, 0x99, 0xdd, 0x9e, 0xaf, 0xb4, 0xa4, 0xe4, 0x15, 0x1c,
+	0xa2, 0xa4, 0x6a, 0x29, 0x0d, 0xf7, 0x9c, 0x89, 0xb7, 0xb6, 0xa5, 0xa3, 0xb3, 0x78, 0x6b, 0xb8,
+	0x3a, 0x31, 0xa7, 0xe7, 0x4d, 0xc0, 0xdc, 0xfa, 0x27, 0x63, 0xdc, 0x3e, 0x4e, 0x7f, 0x82, 0xf1,
+	0xce, 0xbd, 0x11, 0xc6, 0x0c, 0xbe, 0x93, 0xca, 0xda, 0x46, 0xf1, 0x8a, 0x4a, 0xa6, 0x6b, 0xb7,
+	0xa0, 0xee, 0x64, 0x04, 0x71, 0xfb, 0x67, 0xe6, 0xb0, 0x67, 0xe7, 0x30, 0x68, 0x90, 0x8b, 0x5c,
+	0x45, 0x9f, 0xc2, 0xf1, 0x9c, 0x33, 0x14, 0xfa, 0x8a, 0x29, 0x8d, 0x22, 0xc1, 0x5f, 0x96, 0xa8,
+	0xb4, 0xa9, 0x20, 0x68, 0x81, 0x6e, 0xfd, 0xad, 0x1d, 0xfd, 0x0a, 0x87, 0xcd, 0xe8, 0x5c, 0x95,
+	0x99, 0x9d, 0x1b, 0xd3, 0x18, 0xb3, 0xf7, 0x8d, 0x93, 0x31, 0x3b, 0x1f, 0x04, 0xbf, 0xfb, 0x41,
+	0xd8, 0xde, 0x98, 0xde, 0xbf, 0x6f, 0x4c, 0xff, 0xfd, 0x8d, 0xf9, 0x01, 0x8e, 0xaf, 0xca, 0xf2,
+	0x76, 0x59, 0x35, 0x34, 0x5a, 0xae, 0xbb, 0x2f, 0xf4, 0x66, 0x3d, 0x53, 0x73, 0xfd, 0xc2, 0xce,
+	0xc4, 0xfa, 0xdd, 0x89, 0x8d, 0xfe, 0xf4, 0xe0, 0x64, 0x37, 0xad, 0xdb, 0xc5, 0x9f, 0xe1, 0x78,
+	0x9d, 0x37, 0xe5, 0xee, 0xcd, 0x4d, 0x81, 0xd1, 0xd9, 0x8b, 0x2d, 0x31, 0x1f, 0x8a, 0x6e, 0x3f,
+	0x1f, 0x79, 0xdb, 0xac, 0xe4, 0xe8, 0xae, 0x83, 0xa8, 0xe9, 0x0a, 0x26, 0x5d, 0x37, 0x33, 0xd0,
+	0xeb, 0xaa, 0xae, 0xb3, 0xc3, 0x36, 0x92, 0x7c, 0x01, 0xc1, 0x86, 0x88, 0x6f, 0x89, 0x1c, 0xef,
+	0x10, 0x71, 0xb5, 0x36, 0x5e, 0xe4, 0x04, 0xf6, 0x50, 0xca, 0xb2, 0xfd, 0x10, 0x34, 0x87, 0xe8,
+	0x2b, 0x18, 0xfe, 0x6f, 0x15, 0xcf, 0xfe, 0xf2, 0x60, 0x70, 0x83, 0xf4, 0x1e, 0x31, 0x27, 0x17,
+	0x30, 0xbe, 0x41, 0x91, 0x6f, 0x7e, 0x1b, 0x27, 0x5b, 0x7c, 0xd6, 0xe8, 0xf4, 0xe9, 0x43, 0x68,
+	0xdb, 0xab, 0xe8, 0x51, 0xec, 0xbd, 0xf0, 0xc8, 0x35, 0x8c, 0x2f, 0x11, 0xab, 0x79, 0x29, 0x04,
+	0x66, 0x1a, 0x73, 0xf2, 0x6c, 0x2b, 0xe8, 0x81, 0x21, 0x9d, 0x3e, 0x79, 0xef, 0x6b, 0xdd, 0xbe,
+	0xc9, 0x65, 0xfc, 0x1e, 0x0e, 0xb6, 0xb5, 0xd9, 0x49, 0xf8, 0xc0, 0x24, 0x4d, 0x9f, 0xff, 0x87,
+	0xa8, 0xd1, 0xa3, 0xc5, 0xbe, 0xfd, 0x6b, 0x7e, 0xf9, 0x77, 0x00, 0x00, 0x00, 0xff, 0xff, 0xa1,
+	0x9f, 0x87, 0x2e, 0x45, 0x07, 0x00, 0x00,
 }

weed/pb/volume_server.proto

@@ -8,6 +8,34 @@ service VolumeServer {
     //Experts only: takes multiple fid parameters. This function does not propagate deletes to replicas.
     rpc BatchDelete (BatchDeleteRequest) returns (BatchDeleteResponse) {
     }
+    rpc VacuumVolumeCheck (VacuumVolumeCheckRequest) returns (VacuumVolumeCheckResponse) {
+    }
+    rpc VacuumVolumeCompact (VacuumVolumeCompactRequest) returns (VacuumVolumeCompactResponse) {
+    }
+    rpc VacuumVolumeCommit (VacuumVolumeCommitRequest) returns (VacuumVolumeCommitResponse) {
+    }
+    rpc VacuumVolumeCleanup (VacuumVolumeCleanupRequest) returns (VacuumVolumeCleanupResponse) {
+    }
+
+    rpc DeleteCollection (DeleteCollectionRequest) returns (DeleteCollectionResponse) {
+    }
+    rpc AssignVolume (AssignVolumeRequest) returns (AssignVolumeResponse) {
+    }
+
+    rpc VolumeSyncStatus (VolumeSyncStatusRequest) returns (VolumeSyncStatusResponse) {
+    }
+    rpc VolumeSyncIndex (VolumeSyncIndexRequest) returns (VolumeSyncIndexResponse) {
+    }
+    rpc VolumeSyncData (VolumeSyncDataRequest) returns (VolumeSyncDataResponse) {
+    }
+
+    rpc VolumeMount (VolumeMountRequest) returns (VolumeMountResponse) {
+    }
+    rpc VolumeUnmount (VolumeUnmountRequest) returns (VolumeUnmountResponse) {
+    }
+
+    // rpc VolumeUiPage (VolumeUiPageRequest) returns (VolumeUiPageResponse) {}
+
 }
 
 //////////////////////////////////////////////////
@@ -28,3 +56,109 @@ message DeleteResult {
 
 message Empty {
 }
+
+message VacuumVolumeCheckRequest {
+    uint32 volumd_id = 1;
+}
+message VacuumVolumeCheckResponse {
+    double garbage_ratio = 1;
+}
+
+message VacuumVolumeCompactRequest {
+    uint32 volumd_id = 1;
+    int64 preallocate = 2;
+}
+message VacuumVolumeCompactResponse {
+}
+
+message VacuumVolumeCommitRequest {
+    uint32 volumd_id = 1;
+}
+message VacuumVolumeCommitResponse {
+}
+
+message VacuumVolumeCleanupRequest {
+    uint32 volumd_id = 1;
+}
+message VacuumVolumeCleanupResponse {
+}
+
+message DeleteCollectionRequest {
+    string collection = 1;
+}
+message DeleteCollectionResponse {
+}
+
+message AssignVolumeRequest {
+    uint32 volumd_id = 1;
+    string collection = 2;
+    int64 preallocate = 3;
+    string replication = 4;
+    string ttl = 5;
+}
+message AssignVolumeResponse {
+}
+
+message VolumeSyncStatusRequest {
+    uint32 volumd_id = 1;
+}
+message VolumeSyncStatusResponse {
+    uint32 volumd_id = 1;
+    string replication = 4;
+    string ttl = 5;
+    uint64 tail_offset = 6;
+    uint32 compact_revision = 7;
+    uint64 idx_file_size = 8;
+}
+
+message VolumeSyncIndexRequest {
+    uint32 volumd_id = 1;
+}
+message VolumeSyncIndexResponse {
+    bytes index_file_content = 1;
+}
+
+message VolumeSyncDataRequest {
+    uint32 volumd_id = 1;
+    uint32 revision = 2;
+    uint32 offset = 3;
+    uint32 size = 4;
+    string needle_id = 5;
+}
+message VolumeSyncDataResponse {
+    bytes file_content = 1;
+}
+
+message VolumeMountRequest {
+    uint32 volumd_id = 1;
+}
+message VolumeMountResponse {
+}
+
+message VolumeUnmountRequest {
+    uint32 volumd_id = 1;
+}
+message VolumeUnmountResponse {
+}
+
+message VolumeUiPageRequest {
+}
+message VolumeUiPageResponse {
+}
+
+message DiskStatus {
+    string dir = 1;
+    uint64 all = 2;
+    uint64 used = 3;
+    uint64 free = 4;
+}
+
+message MemStatus {
+    int32 goroutines = 1;
+    uint64 all = 2;
+    uint64 used = 3;
+    uint64 free = 4;
+    uint64 self = 5;
+    uint64 heap = 6;
+    uint64 stack = 7;
+}

weed/replication/replicator.go

@@ -34,7 +34,9 @@ func (r *Replicator) Replicate(key string, message *filer_pb.EventNotification)
 		glog.V(4).Infof("skipping %v outside of %v", key, r.source.Dir)
 		return nil
 	}
-	key = filepath.Join(r.sink.GetSinkToDirectory(), key[len(r.source.Dir):])
+	newKey := filepath.Join(r.sink.GetSinkToDirectory(), key[len(r.source.Dir):])
+	glog.V(3).Infof("replicate %s => %s", key, newKey)
+	key = newKey
 	if message.OldEntry != nil && message.NewEntry == nil {
 		glog.V(4).Infof("deleting %v", key)
 		return r.sink.DeleteEntry(key, message.OldEntry.IsDirectory, message.DeleteChunks)

weed/replication/sink/azuresink/azure_sink.go

@@ -5,9 +5,11 @@ import (
 	"context"
 	"fmt"
 	"net/url"
+	"strings"
 
-	"github.com/Azure/azure-storage-blob-go/2016-05-31/azblob"
+	"github.com/Azure/azure-storage-blob-go/azblob"
 	"github.com/chrislusf/seaweedfs/weed/filer2"
+	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/replication/sink"
 	"github.com/chrislusf/seaweedfs/weed/replication/source"
@@ -51,7 +53,10 @@ func (g *AzureSink) initialize(accountName, accountKey, container, dir string) e
 	g.dir = dir
 
 	// Use your Storage account's name and key to create a credential object.
-	credential := azblob.NewSharedKeyCredential(accountName, accountKey)
+	credential, err := azblob.NewSharedKeyCredential(accountName, accountKey)
+	if err != nil {
+		glog.Fatalf("failed to create Azure credential with account name:%s key:%s", accountName, accountKey)
+	}
 
 	// Create a request pipeline that is used to process HTTP(S) requests and responses.
 	p := azblob.NewPipeline(credential, azblob.PipelineOptions{})
@@ -67,6 +72,8 @@ func (g *AzureSink) initialize(accountName, accountKey, container, dir string) e
 
 func (g *AzureSink) DeleteEntry(key string, isDirectory, deleteIncludeChunks bool) error {
 
+	key = cleanKey(key)
+
 	if isDirectory {
 		key = key + "/"
 	}
@@ -84,6 +91,8 @@ func (g *AzureSink) DeleteEntry(key string, isDirectory, deleteIncludeChunks boo
 
 func (g *AzureSink) CreateEntry(key string, entry *filer_pb.Entry) error {
 
+	key = cleanKey(key)
+
 	if entry.IsDirectory {
 		return nil
 	}
@@ -111,7 +120,7 @@ func (g *AzureSink) CreateEntry(key string, entry *filer_pb.Entry) error {
 
 		var writeErr error
 		_, readErr := util.ReadUrlAsStream(fileUrl, chunk.Offset, int(chunk.Size), func(data []byte) {
-			_, writeErr = appendBlobURL.AppendBlock(ctx, bytes.NewReader(data), azblob.BlobAccessConditions{})
+			_, writeErr = appendBlobURL.AppendBlock(ctx, bytes.NewReader(data), azblob.AppendBlobAccessConditions{}, nil)
 		})
 
 		if readErr != nil {
@@ -128,6 +137,14 @@ func (g *AzureSink) CreateEntry(key string, entry *filer_pb.Entry) error {
 }
 
 func (g *AzureSink) UpdateEntry(key string, oldEntry, newEntry *filer_pb.Entry, deleteIncludeChunks bool) (foundExistingEntry bool, err error) {
+	key = cleanKey(key)
 	// TODO improve efficiency
 	return false, nil
 }
+
+func cleanKey(key string) string {
+	if strings.HasPrefix(key, "/") {
+		key = key[1:]
+	}
+	return key
+}

weed/replication/sink/b2sink/b2_sink.go

@@ -0,0 +1,144 @@
+package B2Sink
+
+import (
+	"context"
+	"strings"
+
+	"github.com/chrislusf/seaweedfs/weed/filer2"
+	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
+	"github.com/chrislusf/seaweedfs/weed/replication/sink"
+	"github.com/chrislusf/seaweedfs/weed/replication/source"
+	"github.com/chrislusf/seaweedfs/weed/util"
+	"github.com/kurin/blazer/b2"
+)
+
+type B2Sink struct {
+	client      *b2.Client
+	bucket      string
+	dir         string
+	filerSource *source.FilerSource
+}
+
+func init() {
+	sink.Sinks = append(sink.Sinks, &B2Sink{})
+}
+
+func (g *B2Sink) GetName() string {
+	return "backblaze"
+}
+
+func (g *B2Sink) GetSinkToDirectory() string {
+	return g.dir
+}
+
+func (g *B2Sink) Initialize(configuration util.Configuration) error {
+	return g.initialize(
+		configuration.GetString("b2_account_id"),
+		configuration.GetString("b2_master_application_key"),
+		configuration.GetString("bucket"),
+		configuration.GetString("directory"),
+	)
+}
+
+func (g *B2Sink) SetSourceFiler(s *source.FilerSource) {
+	g.filerSource = s
+}
+
+func (g *B2Sink) initialize(accountId, accountKey, bucket, dir string) error {
+	ctx := context.Background()
+	client, err := b2.NewClient(ctx, accountId, accountKey)
+	if err != nil {
+		return err
+	}
+
+	g.client = client
+	g.bucket = bucket
+	g.dir = dir
+
+	return nil
+}
+
+func (g *B2Sink) DeleteEntry(key string, isDirectory, deleteIncludeChunks bool) error {
+
+	key = cleanKey(key)
+
+	if isDirectory {
+		key = key + "/"
+	}
+
+	ctx := context.Background()
+
+	bucket, err := g.client.Bucket(ctx, g.bucket)
+	if err != nil {
+		return err
+	}
+
+	targetObject := bucket.Object(key)
+
+	return targetObject.Delete(ctx)
+
+}
+
+func (g *B2Sink) CreateEntry(key string, entry *filer_pb.Entry) error {
+
+	key = cleanKey(key)
+
+	if entry.IsDirectory {
+		return nil
+	}
+
+	totalSize := filer2.TotalSize(entry.Chunks)
+	chunkViews := filer2.ViewFromChunks(entry.Chunks, 0, int(totalSize))
+
+	ctx := context.Background()
+
+	bucket, err := g.client.Bucket(ctx, g.bucket)
+	if err != nil {
+		return err
+	}
+
+	targetObject := bucket.Object(key)
+	writer := targetObject.NewWriter(ctx)
+
+	for _, chunk := range chunkViews {
+
+		fileUrl, err := g.filerSource.LookupFileId(chunk.FileId)
+		if err != nil {
+			return err
+		}
+
+		var writeErr error
+		_, readErr := util.ReadUrlAsStream(fileUrl, chunk.Offset, int(chunk.Size), func(data []byte) {
+			_, err := writer.Write(data)
+			if err != nil {
+				writeErr = err
+			}
+		})
+
+		if readErr != nil {
+			return readErr
+		}
+		if writeErr != nil {
+			return writeErr
+		}
+
+	}
+
+	return writer.Close()
+
+}
+
+func (g *B2Sink) UpdateEntry(key string, oldEntry, newEntry *filer_pb.Entry, deleteIncludeChunks bool) (foundExistingEntry bool, err error) {
+
+	key = cleanKey(key)
+
+	// TODO improve efficiency
+	return false, nil
+}
+
+func cleanKey(key string) string {
+	if strings.HasPrefix(key, "/") {
+		key = key[1:]
+	}
+	return key
+}

weed/replication/sink/filersink/filer_sink.go

@@ -163,7 +163,11 @@ func (fs *FilerSink) UpdateEntry(key string, oldEntry, newEntry *filer_pb.Entry,
 
 	glog.V(0).Infof("oldEntry %+v, newEntry %+v, existingEntry: %+v", oldEntry, newEntry, existingEntry)
 
-	if filer2.ETag(newEntry.Chunks) == filer2.ETag(existingEntry.Chunks) {
+	if existingEntry.Attributes.Mtime > newEntry.Attributes.Mtime {
+		// skip if already changed
+		// this usually happens when the messages are not ordered
+		glog.V(0).Infof("late updates %s", key)
+	} else if filer2.ETag(newEntry.Chunks) == filer2.ETag(existingEntry.Chunks) {
 		// skip if no change
 		// this usually happens when retrying the replication
 		glog.V(0).Infof("already replicated %s", key)

weed/replication/sink/gcssink/gcs_sink.go

@@ -3,11 +3,11 @@ package gcssink
 import (
 	"context"
 	"fmt"
-	"log"
 	"os"
 
 	"cloud.google.com/go/storage"
 	"github.com/chrislusf/seaweedfs/weed/filer2"
+	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/replication/sink"
 	"github.com/chrislusf/seaweedfs/weed/replication/source"
@@ -56,12 +56,12 @@ func (g *GcsSink) initialize(google_application_credentials, bucketName, dir str
 		var found bool
 		google_application_credentials, found = os.LookupEnv("GOOGLE_APPLICATION_CREDENTIALS")
 		if !found {
-			log.Fatalf("need to specific GOOGLE_APPLICATION_CREDENTIALS env variable or google_application_credentials in replication.toml")
+			glog.Fatalf("need to specific GOOGLE_APPLICATION_CREDENTIALS env variable or google_application_credentials in replication.toml")
 		}
 	}
 	client, err := storage.NewClient(ctx, option.WithCredentialsFile(google_application_credentials))
 	if err != nil {
-		log.Fatalf("Failed to create client: %v", err)
+		glog.Fatalf("Failed to create client: %v", err)
 	}
 
 	g.client = client

weed/replication/sink/s3sink/s3_sink.go

@@ -2,6 +2,7 @@ package S3Sink
 
 import (
 	"fmt"
+	"strings"
 	"sync"
 
 	"github.com/aws/aws-sdk-go/aws"
@@ -10,6 +11,7 @@ import (
 	"github.com/aws/aws-sdk-go/service/s3"
 	"github.com/aws/aws-sdk-go/service/s3/s3iface"
 	"github.com/chrislusf/seaweedfs/weed/filer2"
+	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/replication/sink"
 	"github.com/chrislusf/seaweedfs/weed/replication/source"
@@ -37,6 +39,9 @@ func (s3sink *S3Sink) GetSinkToDirectory() string {
 }
 
 func (s3sink *S3Sink) Initialize(configuration util.Configuration) error {
+	glog.V(0).Infof("sink.s3.region: %v", configuration.GetString("region"))
+	glog.V(0).Infof("sink.s3.bucket: %v", configuration.GetString("bucket"))
+	glog.V(0).Infof("sink.s3.directory: %v", configuration.GetString("directory"))
 	return s3sink.initialize(
 		configuration.GetString("aws_access_key_id"),
 		configuration.GetString("aws_secret_access_key"),
@@ -73,6 +78,8 @@ func (s3sink *S3Sink) initialize(awsAccessKeyId, aswSecretAccessKey, region, buc
 
 func (s3sink *S3Sink) DeleteEntry(key string, isDirectory, deleteIncludeChunks bool) error {
 
+	key = cleanKey(key)
+
 	if isDirectory {
 		key = key + "/"
 	}
@@ -83,6 +90,8 @@ func (s3sink *S3Sink) DeleteEntry(key string, isDirectory, deleteIncludeChunks b
 
 func (s3sink *S3Sink) CreateEntry(key string, entry *filer_pb.Entry) error {
 
+	key = cleanKey(key)
+
 	if entry.IsDirectory {
 		return nil
 	}
@@ -121,6 +130,14 @@ func (s3sink *S3Sink) CreateEntry(key string, entry *filer_pb.Entry) error {
 }
 
 func (s3sink *S3Sink) UpdateEntry(key string, oldEntry, newEntry *filer_pb.Entry, deleteIncludeChunks bool) (foundExistingEntry bool, err error) {
+	key = cleanKey(key)
 	// TODO improve efficiency
 	return false, nil
 }
+
+func cleanKey(key string) string {
+	if strings.HasPrefix(key, "/") {
+		key = key[1:]
+	}
+	return key
+}

weed/replication/sub/notification_aws_sqs.go

@@ -0,0 +1,111 @@
+package sub
+
+import (
+	"fmt"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/awserr"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/sqs"
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
+	"github.com/chrislusf/seaweedfs/weed/util"
+	"github.com/golang/protobuf/proto"
+)
+
+func init() {
+	NotificationInputs = append(NotificationInputs, &AwsSqsInput{})
+}
+
+type AwsSqsInput struct {
+	svc      *sqs.SQS
+	queueUrl string
+}
+
+func (k *AwsSqsInput) GetName() string {
+	return "aws_sqs"
+}
+
+func (k *AwsSqsInput) Initialize(configuration util.Configuration) error {
+	glog.V(0).Infof("replication.notification.aws_sqs.region: %v", configuration.GetString("region"))
+	glog.V(0).Infof("replication.notification.aws_sqs.sqs_queue_name: %v", configuration.GetString("sqs_queue_name"))
+	return k.initialize(
+		configuration.GetString("aws_access_key_id"),
+		configuration.GetString("aws_secret_access_key"),
+		configuration.GetString("region"),
+		configuration.GetString("sqs_queue_name"),
+	)
+}
+
+func (k *AwsSqsInput) initialize(awsAccessKeyId, aswSecretAccessKey, region, queueName string) (err error) {
+
+	config := &aws.Config{
+		Region: aws.String(region),
+	}
+	if awsAccessKeyId != "" && aswSecretAccessKey != "" {
+		config.Credentials = credentials.NewStaticCredentials(awsAccessKeyId, aswSecretAccessKey, "")
+	}
+
+	sess, err := session.NewSession(config)
+	if err != nil {
+		return fmt.Errorf("create aws session: %v", err)
+	}
+	k.svc = sqs.New(sess)
+
+	result, err := k.svc.GetQueueUrl(&sqs.GetQueueUrlInput{
+		QueueName: aws.String(queueName),
+	})
+	if err != nil {
+		if aerr, ok := err.(awserr.Error); ok && aerr.Code() == sqs.ErrCodeQueueDoesNotExist {
+			return fmt.Errorf("unable to find queue %s", queueName)
+		}
+		return fmt.Errorf("get queue %s url: %v", queueName, err)
+	}
+
+	k.queueUrl = *result.QueueUrl
+
+	return nil
+}
+
+func (k *AwsSqsInput) ReceiveMessage() (key string, message *filer_pb.EventNotification, err error) {
+
+	// receive message
+	result, err := k.svc.ReceiveMessage(&sqs.ReceiveMessageInput{
+		AttributeNames: []*string{
+			aws.String(sqs.MessageSystemAttributeNameSentTimestamp),
+		},
+		MessageAttributeNames: []*string{
+			aws.String(sqs.QueueAttributeNameAll),
+		},
+		QueueUrl:            &k.queueUrl,
+		MaxNumberOfMessages: aws.Int64(1),
+		VisibilityTimeout:   aws.Int64(20), // 20 seconds
+		WaitTimeSeconds:     aws.Int64(20),
+	})
+	if err != nil {
+		err = fmt.Errorf("receive message from sqs %s: %v", k.queueUrl, err)
+		return
+	}
+	if len(result.Messages) == 0 {
+		return
+	}
+
+	// process the message
+	key = *result.Messages[0].Attributes["key"]
+	text := *result.Messages[0].Body
+	message = &filer_pb.EventNotification{}
+	err = proto.UnmarshalText(text, message)
+
+	// delete the message
+	_, err = k.svc.DeleteMessage(&sqs.DeleteMessageInput{
+		QueueUrl:      &k.queueUrl,
+		ReceiptHandle: result.Messages[0].ReceiptHandle,
+	})
+
+	if err != nil {
+		glog.V(1).Infof("delete message from sqs %s: %v", k.queueUrl, err)
+	}
+
+	return
+}

weed/replication/sub/notification_google_pub_sub.go

@@ -0,0 +1,109 @@
+package sub
+
+import (
+	"context"
+	"fmt"
+	"os"
+
+	"cloud.google.com/go/pubsub"
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
+	"github.com/chrislusf/seaweedfs/weed/util"
+	"github.com/golang/protobuf/proto"
+	"google.golang.org/api/option"
+)
+
+func init() {
+	NotificationInputs = append(NotificationInputs, &GooglePubSubInput{})
+}
+
+type GooglePubSubInput struct {
+	sub         *pubsub.Subscription
+	topicName   string
+	messageChan chan *pubsub.Message
+}
+
+func (k *GooglePubSubInput) GetName() string {
+	return "google_pub_sub"
+}
+
+func (k *GooglePubSubInput) Initialize(configuration util.Configuration) error {
+	glog.V(0).Infof("notification.google_pub_sub.project_id: %v", configuration.GetString("project_id"))
+	glog.V(0).Infof("notification.google_pub_sub.topic: %v", configuration.GetString("topic"))
+	return k.initialize(
+		configuration.GetString("google_application_credentials"),
+		configuration.GetString("project_id"),
+		configuration.GetString("topic"),
+	)
+}
+
+func (k *GooglePubSubInput) initialize(google_application_credentials, projectId, topicName string) (err error) {
+
+	ctx := context.Background()
+	// Creates a client.
+	if google_application_credentials == "" {
+		var found bool
+		google_application_credentials, found = os.LookupEnv("GOOGLE_APPLICATION_CREDENTIALS")
+		if !found {
+			glog.Fatalf("need to specific GOOGLE_APPLICATION_CREDENTIALS env variable or google_application_credentials in filer.toml")
+		}
+	}
+
+	client, err := pubsub.NewClient(ctx, projectId, option.WithCredentialsFile(google_application_credentials))
+	if err != nil {
+		glog.Fatalf("Failed to create client: %v", err)
+	}
+
+	k.topicName = topicName
+	topic := client.Topic(topicName)
+	if exists, err := topic.Exists(ctx); err == nil {
+		if !exists {
+			topic, err = client.CreateTopic(ctx, topicName)
+			if err != nil {
+				glog.Fatalf("Failed to create topic %s: %v", topicName, err)
+			}
+		}
+	} else {
+		glog.Fatalf("Failed to check topic %s: %v", topicName, err)
+	}
+
+	subscriptionName := "seaweedfs_sub"
+
+	k.sub = client.Subscription(subscriptionName)
+	if exists, err := k.sub.Exists(ctx); err == nil {
+		if !exists {
+			k.sub, err = client.CreateSubscription(ctx, subscriptionName, pubsub.SubscriptionConfig{Topic: topic})
+			if err != nil {
+				glog.Fatalf("Failed to create subscription %s: %v", subscriptionName, err)
+			}
+		}
+	} else {
+		glog.Fatalf("Failed to check subscription %s: %v", topicName, err)
+	}
+
+	k.messageChan = make(chan *pubsub.Message, 1)
+
+	go k.sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
+		k.messageChan <- m
+		m.Ack()
+	})
+
+	return err
+}
+
+func (k *GooglePubSubInput) ReceiveMessage() (key string, message *filer_pb.EventNotification, err error) {
+
+	m := <-k.messageChan
+
+	// process the message
+	key = m.Attributes["key"]
+	message = &filer_pb.EventNotification{}
+	err = proto.Unmarshal(m.Data, message)
+
+	if err != nil {
+		err = fmt.Errorf("unmarshal message from google pubsub %s: %v", k.topicName, err)
+		return
+	}
+
+	return
+}

weed/replication/sub/notification_kafka.go

@@ -1,4 +1,4 @@
-package replication
+package sub
 
 import (
 	"encoding/json"

weed/replication/sub/notifications.go

@@ -1,4 +1,4 @@
-package replication
+package sub
 
 import (
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"

weed/s3api/s3api_object_handlers.go

@@ -63,6 +63,7 @@ func (s3a *S3ApiServer) GetObjectHandler(w http.ResponseWriter, r *http.Request)
 
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
+	object := getObject(vars)
 
 	if strings.HasSuffix(r.URL.Path, "/") {
 		writeErrorResponse(w, ErrNotImplemented, r.URL)
@@ -70,7 +71,7 @@ func (s3a *S3ApiServer) GetObjectHandler(w http.ResponseWriter, r *http.Request)
 	}
 
 	destUrl := fmt.Sprintf("http://%s%s/%s%s",
-		s3a.option.Filer, s3a.option.BucketsPath, bucket, r.RequestURI)
+		s3a.option.Filer, s3a.option.BucketsPath, bucket, object)
 
 	s3a.proxyToFiler(w, r, destUrl, passThroghResponse)
 
@@ -80,9 +81,10 @@ func (s3a *S3ApiServer) HeadObjectHandler(w http.ResponseWriter, r *http.Request
 
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
+	object := getObject(vars)
 
 	destUrl := fmt.Sprintf("http://%s%s/%s%s",
-		s3a.option.Filer, s3a.option.BucketsPath, bucket, r.RequestURI)
+		s3a.option.Filer, s3a.option.BucketsPath, bucket, object)
 
 	s3a.proxyToFiler(w, r, destUrl, passThroghResponse)
 
@@ -92,9 +94,10 @@ func (s3a *S3ApiServer) DeleteObjectHandler(w http.ResponseWriter, r *http.Reque
 
 	vars := mux.Vars(r)
 	bucket := vars["bucket"]
+	object := getObject(vars)
 
 	destUrl := fmt.Sprintf("http://%s%s/%s%s",
-		s3a.option.Filer, s3a.option.BucketsPath, bucket, r.RequestURI)
+		s3a.option.Filer, s3a.option.BucketsPath, bucket, object)
 
 	s3a.proxyToFiler(w, r, destUrl, func(proxyResonse *http.Response, w http.ResponseWriter) {
 		for k, v := range proxyResonse.Header {
@@ -125,6 +128,7 @@ func (s3a *S3ApiServer) proxyToFiler(w http.ResponseWriter, r *http.Request, des
 
 	proxyReq.Header.Set("Host", s3a.option.Filer)
 	proxyReq.Header.Set("X-Forwarded-For", r.RemoteAddr)
+	proxyReq.Header.Set("Etag-MD5", "True")
 
 	for header, values := range r.Header {
 		for _, value := range values {

weed/server/common.go

@@ -19,8 +19,8 @@ import (
 	"github.com/chrislusf/seaweedfs/weed/util"
 
 	_ "github.com/chrislusf/seaweedfs/weed/statik"
-	statik "github.com/rakyll/statik/fs"
 	"github.com/gorilla/mux"
+	statik "github.com/rakyll/statik/fs"
 )
 
 var serverStats *stats.ServerStats
@@ -132,17 +132,6 @@ func submitForClientHandler(w http.ResponseWriter, r *http.Request, masterUrl st
 	return
 }
 
-func deleteForClientHandler(w http.ResponseWriter, r *http.Request, masterUrl string) {
-	r.ParseForm()
-	fids := r.Form["fid"]
-	ret, err := operation.DeleteFiles(masterUrl, fids)
-	if err != nil {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-		return
-	}
-	writeJsonQuiet(w, r, http.StatusAccepted, ret)
-}
-
 func parseURLPath(path string) (vid, fid, filename, ext string, isVolumeIdOnly bool) {
 	switch strings.Count(path, "/") {
 	case 3:

weed/server/filer_grpc_server.go

@@ -110,10 +110,7 @@ func (fs *FilerServer) CreateEntry(ctx context.Context, req *filer_pb.CreateEntr
 	fullpath := filer2.FullPath(filepath.Join(req.Directory, req.Entry.Name))
 	chunks, garbages := filer2.CompactFileChunks(req.Entry.Chunks)
 
-	for _, garbage := range garbages {
-		glog.V(0).Infof("deleting %s garbage chunk: %v, [%d, %d)", fullpath, garbage.FileId, garbage.Offset, garbage.Offset+int64(garbage.Size))
-		fs.filer.DeleteFileByFileId(garbage.FileId)
-	}
+	fs.filer.DeleteChunks(garbages)
 
 	err = fs.filer.CreateEntry(&filer2.Entry{
 		FullPath: fullpath,
@@ -168,14 +165,8 @@ func (fs *FilerServer) UpdateEntry(ctx context.Context, req *filer_pb.UpdateEntr
 	}
 
 	if err = fs.filer.UpdateEntry(newEntry); err == nil {
-		for _, garbage := range unusedChunks {
-			glog.V(0).Infof("deleting %s old chunk: %v, [%d, %d)", fullpath, garbage.FileId, garbage.Offset, garbage.Offset+int64(garbage.Size))
-			fs.filer.DeleteFileByFileId(garbage.FileId)
-		}
-		for _, garbage := range garbages {
-			glog.V(0).Infof("deleting %s garbage chunk: %v, [%d, %d)", fullpath, garbage.FileId, garbage.Offset, garbage.Offset+int64(garbage.Size))
-			fs.filer.DeleteFileByFileId(garbage.FileId)
-		}
+		fs.filer.DeleteChunks(unusedChunks)
+		fs.filer.DeleteChunks(garbages)
 	}
 
 	fs.filer.NotifyUpdateEvent(entry, newEntry, true)

weed/server/filer_server.go

@@ -12,6 +12,8 @@ import (
 	_ "github.com/chrislusf/seaweedfs/weed/filer2/redis"
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/notification"
+	_ "github.com/chrislusf/seaweedfs/weed/notification/aws_sqs"
+	_ "github.com/chrislusf/seaweedfs/weed/notification/google_pub_sub"
 	_ "github.com/chrislusf/seaweedfs/weed/notification/kafka"
 	_ "github.com/chrislusf/seaweedfs/weed/notification/log"
 	"github.com/chrislusf/seaweedfs/weed/security"
@@ -51,6 +53,7 @@ func NewFilerServer(defaultMux, readonlyMux *http.ServeMux, option *FilerOption)
 	go fs.filer.KeepConnectedToMaster()
 
 	LoadConfiguration("filer", true)
+	LoadConfiguration("notification", false)
 	v := viper.GetViper()
 
 	fs.filer.LoadConfiguration(v)
@@ -80,15 +83,15 @@ func LoadConfiguration(configFileName string, required bool) {
 
 	glog.V(0).Infof("Reading %s.toml from %s", configFileName, viper.ConfigFileUsed())
 
-	if err := viper.ReadInConfig(); err != nil { // Handle errors reading the config file
+	if err := viper.MergeInConfig(); err != nil { // Handle errors reading the config file
 		glog.V(0).Infof("Reading %s: %v", viper.ConfigFileUsed(), err)
 		if required {
-			glog.Errorf("Failed to load %s.toml file from current directory, or $HOME/.seaweedfs/, or /etc/seaweedfs/"+
+			glog.Fatalf("Failed to load %s.toml file from current directory, or $HOME/.seaweedfs/, or /etc/seaweedfs/"+
 				"\n\nPlease follow this example and add a filer.toml file to "+
 				"current directory, or $HOME/.seaweedfs/, or /etc/seaweedfs/:\n"+
 				"    https://github.com/chrislusf/seaweedfs/blob/master/weed/%s.toml\n"+
-				"\n\nOr use this command to generate the default toml file\n"+
-				"    weed scaffold -config=%s -output=.\n",
+				"\nOr use this command to generate the default toml file\n"+
+				"    weed scaffold -config=%s -output=.\n\n\n",
 				configFileName, configFileName, configFileName)
 		}
 	}

weed/server/master_grpc_server_lookup.go

@@ -0,0 +1,29 @@
+package weed_server
+
+import (
+	"context"
+
+	"github.com/chrislusf/seaweedfs/weed/pb/master_pb"
+)
+
+func (ms *MasterServer) LookupVolume(ctx context.Context, req *master_pb.LookupVolumeRequest) (*master_pb.LookupVolumeResponse, error) {
+	resp := &master_pb.LookupVolumeResponse{}
+	volumeLocations := ms.lookupVolumeId(req.VolumeIds, req.Collection)
+
+	for _, result := range volumeLocations {
+		var locations []*master_pb.Location
+		for _, loc := range result.Locations {
+			locations = append(locations, &master_pb.Location{
+				Url:       loc.Url,
+				PublicUrl: loc.PublicUrl,
+			})
+		}
+		resp.VolumeIdLocations = append(resp.VolumeIdLocations, &master_pb.LookupVolumeResponse_VolumeIdLocation{
+			VolumeId:  result.VolumeId,
+			Locations: locations,
+			Error:     result.Error,
+		})
+	}
+
+	return resp, nil
+}

weed/server/master_server.go

@@ -24,7 +24,7 @@ type MasterServer struct {
 	preallocate             int64
 	pulseSeconds            int
 	defaultReplicaPlacement string
-	garbageThreshold        string
+	garbageThreshold        float64
 	guard                   *security.Guard
 
 	Topo   *topology.Topology
@@ -43,7 +43,7 @@ func NewMasterServer(r *mux.Router, port int, metaFolder string,
 	preallocate bool,
 	pulseSeconds int,
 	defaultReplicaPlacement string,
-	garbageThreshold string,
+	garbageThreshold float64,
 	whiteList []string,
 	secureKey string,
 ) *MasterServer {
@@ -76,12 +76,10 @@ func NewMasterServer(r *mux.Router, port int, metaFolder string,
 	r.HandleFunc("/dir/lookup", ms.proxyToLeader(ms.guard.WhiteList(ms.dirLookupHandler)))
 	r.HandleFunc("/dir/status", ms.proxyToLeader(ms.guard.WhiteList(ms.dirStatusHandler)))
 	r.HandleFunc("/col/delete", ms.proxyToLeader(ms.guard.WhiteList(ms.collectionDeleteHandler)))
-	r.HandleFunc("/vol/lookup", ms.proxyToLeader(ms.guard.WhiteList(ms.volumeLookupHandler)))
 	r.HandleFunc("/vol/grow", ms.proxyToLeader(ms.guard.WhiteList(ms.volumeGrowHandler)))
 	r.HandleFunc("/vol/status", ms.proxyToLeader(ms.guard.WhiteList(ms.volumeStatusHandler)))
 	r.HandleFunc("/vol/vacuum", ms.proxyToLeader(ms.guard.WhiteList(ms.volumeVacuumHandler)))
 	r.HandleFunc("/submit", ms.guard.WhiteList(ms.submitFromMasterServerHandler))
-	r.HandleFunc("/delete", ms.guard.WhiteList(ms.deleteFromMasterServerHandler))
 	r.HandleFunc("/stats/health", ms.guard.WhiteList(statsHealthHandler))
 	r.HandleFunc("/stats/counter", ms.guard.WhiteList(statsCounterHandler))
 	r.HandleFunc("/stats/memory", ms.guard.WhiteList(statsMemoryHandler))

weed/server/master_server_handlers.go

@@ -58,15 +58,6 @@ func (ms *MasterServer) dirLookupHandler(w http.ResponseWriter, r *http.Request)
 	writeJsonQuiet(w, r, httpStatus, location)
 }
 
-// This can take batched volumeIds, &volumeId=x&volumeId=y&volumeId=z
-func (ms *MasterServer) volumeLookupHandler(w http.ResponseWriter, r *http.Request) {
-	r.ParseForm()
-	vids := r.Form["volumeId"]
-	collection := r.FormValue("collection") //optional, but can be faster if too many collections
-	volumeLocations := ms.lookupVolumeId(vids, collection)
-	writeJsonQuiet(w, r, http.StatusOK, volumeLocations)
-}
-
 func (ms *MasterServer) dirAssignHandler(w http.ResponseWriter, r *http.Request) {
 	stats.AssignRequest()
 	requestedCount, e := strconv.ParseUint(r.FormValue("count"), 10, 64)

weed/server/master_server_handlers_admin.go

@@ -1,6 +1,7 @@
 package weed_server
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"math/rand"
@@ -8,6 +9,8 @@ import (
 	"strconv"
 
 	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/operation"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/chrislusf/seaweedfs/weed/storage"
 	"github.com/chrislusf/seaweedfs/weed/topology"
 	"github.com/chrislusf/seaweedfs/weed/util"
@@ -20,7 +23,12 @@ func (ms *MasterServer) collectionDeleteHandler(w http.ResponseWriter, r *http.R
 		return
 	}
 	for _, server := range collection.ListVolumeServers() {
-		_, err := util.Get("http://" + server.Ip + ":" + strconv.Itoa(server.Port) + "/admin/delete_collection?collection=" + r.FormValue("collection"))
+		err := operation.WithVolumeServerClient(server.Url(), func(client volume_server_pb.VolumeServerClient) error {
+			_, deleteErr := client.DeleteCollection(context.Background(), &volume_server_pb.DeleteCollectionRequest{
+				Collection: collection.Name,
+			})
+			return deleteErr
+		})
 		if err != nil {
 			writeJsonError(w, r, http.StatusInternalServerError, err)
 			return
@@ -37,9 +45,15 @@ func (ms *MasterServer) dirStatusHandler(w http.ResponseWriter, r *http.Request)
 }
 
 func (ms *MasterServer) volumeVacuumHandler(w http.ResponseWriter, r *http.Request) {
-	gcThreshold := r.FormValue("garbageThreshold")
-	if gcThreshold == "" {
-		gcThreshold = ms.garbageThreshold
+	gcString := r.FormValue("garbageThreshold")
+	gcThreshold := ms.garbageThreshold
+	if gcString != "" {
+		var err error
+		gcThreshold, err = strconv.ParseFloat(gcString, 32)
+		if err != nil {
+			glog.V(0).Infof("garbageThreshold %s is not a valid float number: %v", gcString, err)
+			return
+		}
 	}
 	glog.Infoln("garbageThreshold =", gcThreshold)
 	ms.Topo.Vacuum(gcThreshold, ms.preallocate)
@@ -119,14 +133,6 @@ func (ms *MasterServer) submitFromMasterServerHandler(w http.ResponseWriter, r *
 	}
 }
 
-func (ms *MasterServer) deleteFromMasterServerHandler(w http.ResponseWriter, r *http.Request) {
-	if ms.Topo.IsLeader() {
-		deleteForClientHandler(w, r, ms.selfUrl(r))
-	} else {
-		deleteForClientHandler(w, r, ms.Topo.RaftServer.Leader())
-	}
-}
-
 func (ms *MasterServer) HasWritableVolume(option *topology.VolumeGrowOption) bool {
 	vl := ms.Topo.GetVolumeLayout(option.Collection, option.ReplicaPlacement, option.Ttl)
 	return vl.GetActiveVolumeCount(option) > 0

weed/server/volume_grpc_admin.go

@@ -0,0 +1,80 @@
+package weed_server
+
+import (
+	"context"
+
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/chrislusf/seaweedfs/weed/storage"
+)
+
+func (vs *VolumeServer) DeleteCollection(ctx context.Context, req *volume_server_pb.DeleteCollectionRequest) (*volume_server_pb.DeleteCollectionResponse, error) {
+
+	resp := &volume_server_pb.DeleteCollectionResponse{}
+
+	err := vs.store.DeleteCollection(req.Collection)
+
+	if err != nil {
+		glog.Errorf("delete collection %s: %v", req.Collection, err)
+	} else {
+		glog.V(2).Infof("delete collection %v", req)
+	}
+
+	return resp, err
+
+}
+
+func (vs *VolumeServer) AssignVolume(ctx context.Context, req *volume_server_pb.AssignVolumeRequest) (*volume_server_pb.AssignVolumeResponse, error) {
+
+	resp := &volume_server_pb.AssignVolumeResponse{}
+
+	err := vs.store.AddVolume(
+		storage.VolumeId(req.VolumdId),
+		req.Collection,
+		vs.needleMapKind,
+		req.Replication,
+		req.Ttl,
+		req.Preallocate,
+	)
+
+	if err != nil {
+		glog.Errorf("assign volume %v: %v", req, err)
+	} else {
+		glog.V(2).Infof("assign volume %v", req)
+	}
+
+	return resp, err
+
+}
+
+func (vs *VolumeServer) VolumeMount(ctx context.Context, req *volume_server_pb.VolumeMountRequest) (*volume_server_pb.VolumeMountResponse, error) {
+
+	resp := &volume_server_pb.VolumeMountResponse{}
+
+	err := vs.store.MountVolume(storage.VolumeId(req.VolumdId))
+
+	if err != nil {
+		glog.Errorf("volume mount %v: %v", req, err)
+	} else {
+		glog.V(2).Infof("volume mount %v", req)
+	}
+
+	return resp, err
+
+}
+
+func (vs *VolumeServer) VolumeUnmount(ctx context.Context, req *volume_server_pb.VolumeUnmountRequest) (*volume_server_pb.VolumeUnmountResponse, error) {
+
+	resp := &volume_server_pb.VolumeUnmountResponse{}
+
+	err := vs.store.UnmountVolume(storage.VolumeId(req.VolumdId))
+
+	if err != nil {
+		glog.Errorf("volume unmount %v: %v", req, err)
+	} else {
+		glog.V(2).Infof("volume unmount %v", req)
+	}
+
+	return resp, err
+
+}

weed/server/volume_grpc_sync.go

@@ -0,0 +1,87 @@
+package weed_server
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/chrislusf/seaweedfs/weed/storage"
+	"github.com/chrislusf/seaweedfs/weed/storage/types"
+)
+
+func (vs *VolumeServer) VolumeSyncStatus(ctx context.Context, req *volume_server_pb.VolumeSyncStatusRequest) (*volume_server_pb.VolumeSyncStatusResponse, error) {
+
+	v := vs.store.GetVolume(storage.VolumeId(req.VolumdId))
+	if v == nil {
+		return nil, fmt.Errorf("Not Found Volume Id %d", req.VolumdId)
+	}
+
+	resp := v.GetVolumeSyncStatus()
+
+	glog.V(2).Infof("volume sync status %d", req.VolumdId)
+
+	return resp, nil
+
+}
+
+func (vs *VolumeServer) VolumeSyncIndex(ctx context.Context, req *volume_server_pb.VolumeSyncIndexRequest) (*volume_server_pb.VolumeSyncIndexResponse, error) {
+
+	resp := &volume_server_pb.VolumeSyncIndexResponse{}
+
+	v := vs.store.GetVolume(storage.VolumeId(req.VolumdId))
+	if v == nil {
+		return nil, fmt.Errorf("Not Found Volume Id %d", req.VolumdId)
+	}
+
+	content, err := v.IndexFileContent()
+
+	if err != nil {
+		glog.Errorf("sync volume %d index: %v", req.VolumdId, err)
+	} else {
+		glog.V(2).Infof("sync volume %d index", req.VolumdId)
+	}
+
+	resp.IndexFileContent = content
+
+	return resp, nil
+
+}
+
+func (vs *VolumeServer) VolumeSyncData(ctx context.Context, req *volume_server_pb.VolumeSyncDataRequest) (*volume_server_pb.VolumeSyncDataResponse, error) {
+
+	resp := &volume_server_pb.VolumeSyncDataResponse{}
+
+	v := vs.store.GetVolume(storage.VolumeId(req.VolumdId))
+	if v == nil {
+		return nil, fmt.Errorf("Not Found Volume Id %d", req.VolumdId)
+	}
+
+	if uint32(v.SuperBlock.CompactRevision) != req.Revision {
+		return nil, fmt.Errorf("Requested Volume Revision is %d, but current revision is %d", req.Revision, v.SuperBlock.CompactRevision)
+	}
+
+	content, err := storage.ReadNeedleBlob(v.DataFile(), int64(req.Offset)*types.NeedlePaddingSize, req.Size, v.Version())
+	if err != nil {
+		return nil, fmt.Errorf("read offset:%d size:%d", req.Offset, req.Size)
+	}
+
+	id, err := types.ParseNeedleId(req.NeedleId)
+	if err != nil {
+		return nil, fmt.Errorf("parsing needle id %s: %v", req.NeedleId, err)
+	}
+	n := new(storage.Needle)
+	n.ParseNeedleHeader(content)
+	if id != n.Id {
+		return nil, fmt.Errorf("Expected file entry id %d, but found %d", id, n.Id)
+	}
+
+	if err != nil {
+		glog.Errorf("sync volume %d data: %v", req.VolumdId, err)
+	}
+
+	resp.FileContent = content
+
+	return resp, nil
+
+}

weed/server/volume_grpc_vacuum.go

@@ -0,0 +1,73 @@
+package weed_server
+
+import (
+	"context"
+
+	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
+	"github.com/chrislusf/seaweedfs/weed/storage"
+)
+
+func (vs *VolumeServer) VacuumVolumeCheck(ctx context.Context, req *volume_server_pb.VacuumVolumeCheckRequest) (*volume_server_pb.VacuumVolumeCheckResponse, error) {
+
+	resp := &volume_server_pb.VacuumVolumeCheckResponse{}
+
+	garbageRatio, err := vs.store.CheckCompactVolume(storage.VolumeId(req.VolumdId))
+
+	resp.GarbageRatio = garbageRatio
+
+	if err != nil {
+		glog.V(3).Infof("check volume %d: %v", req.VolumdId, err)
+	}
+
+	return resp, err
+
+}
+
+func (vs *VolumeServer) VacuumVolumeCompact(ctx context.Context, req *volume_server_pb.VacuumVolumeCompactRequest) (*volume_server_pb.VacuumVolumeCompactResponse, error) {
+
+	resp := &volume_server_pb.VacuumVolumeCompactResponse{}
+
+	err := vs.store.CompactVolume(storage.VolumeId(req.VolumdId), req.Preallocate)
+
+	if err != nil {
+		glog.Errorf("compact volume %d: %v", req.VolumdId, err)
+	} else {
+		glog.V(1).Infof("compact volume %d", req.VolumdId)
+	}
+
+	return resp, err
+
+}
+
+func (vs *VolumeServer) VacuumVolumeCommit(ctx context.Context, req *volume_server_pb.VacuumVolumeCommitRequest) (*volume_server_pb.VacuumVolumeCommitResponse, error) {
+
+	resp := &volume_server_pb.VacuumVolumeCommitResponse{}
+
+	err := vs.store.CommitCompactVolume(storage.VolumeId(req.VolumdId))
+
+	if err != nil {
+		glog.Errorf("commit volume %d: %v", req.VolumdId, err)
+	} else {
+		glog.V(1).Infof("commit volume %d", req.VolumdId)
+	}
+
+	return resp, err
+
+}
+
+func (vs *VolumeServer) VacuumVolumeCleanup(ctx context.Context, req *volume_server_pb.VacuumVolumeCleanupRequest) (*volume_server_pb.VacuumVolumeCleanupResponse, error) {
+
+	resp := &volume_server_pb.VacuumVolumeCleanupResponse{}
+
+	err := vs.store.CommitCleanupVolume(storage.VolumeId(req.VolumdId))
+
+	if err != nil {
+		glog.Errorf("cleanup volume %d: %v", req.VolumdId, err)
+	} else {
+		glog.V(1).Infof("cleanup volume %d", req.VolumdId)
+	}
+
+	return resp, err
+
+}

weed/server/volume_server.go

@@ -47,22 +47,9 @@ func NewVolumeServer(adminMux, publicMux *http.ServeMux, ip string,
 	handleStaticResources(adminMux)
 	adminMux.HandleFunc("/ui/index.html", vs.uiStatusHandler)
 	adminMux.HandleFunc("/status", vs.guard.WhiteList(vs.statusHandler))
-	adminMux.HandleFunc("/admin/assign_volume", vs.guard.WhiteList(vs.assignVolumeHandler))
-	adminMux.HandleFunc("/admin/vacuum/check", vs.guard.WhiteList(vs.vacuumVolumeCheckHandler))
-	adminMux.HandleFunc("/admin/vacuum/compact", vs.guard.WhiteList(vs.vacuumVolumeCompactHandler))
-	adminMux.HandleFunc("/admin/vacuum/commit", vs.guard.WhiteList(vs.vacuumVolumeCommitHandler))
-	adminMux.HandleFunc("/admin/vacuum/cleanup", vs.guard.WhiteList(vs.vacuumVolumeCleanupHandler))
-	adminMux.HandleFunc("/admin/delete_collection", vs.guard.WhiteList(vs.deleteCollectionHandler))
-	adminMux.HandleFunc("/admin/sync/status", vs.guard.WhiteList(vs.getVolumeSyncStatusHandler))
-	adminMux.HandleFunc("/admin/sync/index", vs.guard.WhiteList(vs.getVolumeIndexContentHandler))
-	adminMux.HandleFunc("/admin/sync/data", vs.guard.WhiteList(vs.getVolumeDataContentHandler))
-	adminMux.HandleFunc("/admin/volume/mount", vs.guard.WhiteList(vs.getVolumeMountHandler))
-	adminMux.HandleFunc("/admin/volume/unmount", vs.guard.WhiteList(vs.getVolumeUnmountHandler))
-	adminMux.HandleFunc("/admin/volume/delete", vs.guard.WhiteList(vs.getVolumeDeleteHandler))
 	adminMux.HandleFunc("/stats/counter", vs.guard.WhiteList(statsCounterHandler))
 	adminMux.HandleFunc("/stats/memory", vs.guard.WhiteList(statsMemoryHandler))
 	adminMux.HandleFunc("/stats/disk", vs.guard.WhiteList(vs.statsDiskHandler))
-	adminMux.HandleFunc("/delete", vs.guard.WhiteList(vs.batchDeleteHandler))
 	adminMux.HandleFunc("/", vs.privateStoreHandler)
 	if publicMux != adminMux {
 		// separated admin and public port

weed/server/volume_server_handlers_admin.go

@@ -1,14 +1,11 @@
 package weed_server
 
 import (
-	"fmt"
 	"net/http"
 	"path/filepath"
-	"strconv"
 
-	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/chrislusf/seaweedfs/weed/stats"
-	"github.com/chrislusf/seaweedfs/weed/storage"
 	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
@@ -19,46 +16,10 @@ func (vs *VolumeServer) statusHandler(w http.ResponseWriter, r *http.Request) {
 	writeJsonQuiet(w, r, http.StatusOK, m)
 }
 
-func (vs *VolumeServer) assignVolumeHandler(w http.ResponseWriter, r *http.Request) {
-	var err error
-	preallocate := int64(0)
-	if r.FormValue("preallocate") != "" {
-		preallocate, err = strconv.ParseInt(r.FormValue("preallocate"), 10, 64)
-		if err != nil {
-			glog.V(0).Infof("ignoring invalid int64 value for preallocate = %v", r.FormValue("preallocate"))
-		}
-	}
-	err = vs.store.AddVolume(
-		r.FormValue("volume"),
-		r.FormValue("collection"),
-		vs.needleMapKind,
-		r.FormValue("replication"),
-		r.FormValue("ttl"),
-		preallocate,
-	)
-	if err == nil {
-		writeJsonQuiet(w, r, http.StatusAccepted, map[string]string{"error": ""})
-	} else {
-		writeJsonError(w, r, http.StatusNotAcceptable, err)
-	}
-	glog.V(2).Infof("assign volume = %s, collection = %s , replication = %s, error = %v",
-		r.FormValue("volume"), r.FormValue("collection"), r.FormValue("replication"), err)
-}
-
-func (vs *VolumeServer) deleteCollectionHandler(w http.ResponseWriter, r *http.Request) {
-	err := vs.store.DeleteCollection(r.FormValue("collection"))
-	if err == nil {
-		writeJsonQuiet(w, r, http.StatusOK, map[string]string{"error": ""})
-	} else {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-	}
-	glog.V(2).Infof("deleting collection = %s, error = %v", r.FormValue("collection"), err)
-}
-
 func (vs *VolumeServer) statsDiskHandler(w http.ResponseWriter, r *http.Request) {
 	m := make(map[string]interface{})
 	m["Version"] = util.VERSION
-	var ds []*stats.DiskStatus
+	var ds []*volume_server_pb.DiskStatus
 	for _, loc := range vs.store.Locations {
 		if dir, e := filepath.Abs(loc.Directory); e == nil {
 			ds = append(ds, stats.NewDiskStatus(dir))
@@ -67,45 +28,3 @@ func (vs *VolumeServer) statsDiskHandler(w http.ResponseWriter, r *http.Request)
 	m["DiskStatuses"] = ds
 	writeJsonQuiet(w, r, http.StatusOK, m)
 }
-
-func (vs *VolumeServer) getVolume(volumeParameterName string, r *http.Request) (*storage.Volume, error) {
-	vid, err := vs.getVolumeId(volumeParameterName, r)
-	if err != nil {
-		return nil, err
-	}
-	v := vs.store.GetVolume(vid)
-	if v == nil {
-		return nil, fmt.Errorf("Not Found Volume Id %d", vid)
-	}
-	return v, nil
-}
-
-func (vs *VolumeServer) getVolumeMountHandler(w http.ResponseWriter, r *http.Request) {
-	vid, err := vs.getVolumeId("volume", r)
-	if err != nil {
-		writeJsonError(w, r, http.StatusNotFound, err)
-		return
-	}
-	vs.store.MountVolume(vid)
-	writeJsonQuiet(w, r, http.StatusOK, "Volume mounted")
-}
-
-func (vs *VolumeServer) getVolumeUnmountHandler(w http.ResponseWriter, r *http.Request) {
-	vid, err := vs.getVolumeId("volume", r)
-	if err != nil {
-		writeJsonError(w, r, http.StatusNotFound, err)
-		return
-	}
-	vs.store.UnmountVolume(vid)
-	writeJsonQuiet(w, r, http.StatusOK, "Volume unmounted")
-}
-
-func (vs *VolumeServer) getVolumeDeleteHandler(w http.ResponseWriter, r *http.Request) {
-	vid, err := vs.getVolumeId("volume", r)
-	if err != nil {
-		writeJsonError(w, r, http.StatusNotFound, err)
-		return
-	}
-	vs.store.DeleteVolume(vid)
-	writeJsonQuiet(w, r, http.StatusOK, "Volume deleted")
-}

weed/server/volume_server_handlers_read.go

@@ -92,7 +92,11 @@ func (vs *VolumeServer) GetOrHeadHandler(w http.ResponseWriter, r *http.Request)
 		w.WriteHeader(http.StatusNotModified)
 		return
 	}
-	setEtag(w, n.Etag())
+	if r.Header.Get("ETag-MD5") == "True" {
+		setEtag(w, n.MD5())
+	}else{
+		setEtag(w, n.Etag())
+	}
 
 	if n.HasPairs() {
 		pairMap := make(map[string]string)

weed/server/volume_server_handlers_sync.go

@@ -1,90 +0,0 @@
-package weed_server
-
-import (
-	"fmt"
-	"net/http"
-
-	"github.com/chrislusf/seaweedfs/weed/glog"
-	"github.com/chrislusf/seaweedfs/weed/storage"
-	"github.com/chrislusf/seaweedfs/weed/storage/types"
-	"github.com/chrislusf/seaweedfs/weed/util"
-)
-
-func (vs *VolumeServer) getVolumeSyncStatusHandler(w http.ResponseWriter, r *http.Request) {
-	v, err := vs.getVolume("volume", r)
-	if v == nil {
-		writeJsonError(w, r, http.StatusBadRequest, err)
-		return
-	}
-	syncStat := v.GetVolumeSyncStatus()
-	if syncStat.Error != "" {
-		writeJsonError(w, r, http.StatusInternalServerError, fmt.Errorf("Get Volume %d status error: %s", v.Id, syncStat.Error))
-		glog.V(2).Infoln("getVolumeSyncStatusHandler volume =", r.FormValue("volume"), ", error =", err)
-	} else {
-		writeJsonQuiet(w, r, http.StatusOK, syncStat)
-	}
-}
-
-func (vs *VolumeServer) getVolumeIndexContentHandler(w http.ResponseWriter, r *http.Request) {
-	v, err := vs.getVolume("volume", r)
-	if v == nil {
-		writeJsonError(w, r, http.StatusBadRequest, err)
-		return
-	}
-	content, err := v.IndexFileContent()
-	if err != nil {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-		return
-	}
-	w.Write(content)
-}
-
-func (vs *VolumeServer) getVolumeDataContentHandler(w http.ResponseWriter, r *http.Request) {
-	v, err := vs.getVolume("volume", r)
-	if v == nil {
-		writeJsonError(w, r, http.StatusBadRequest, fmt.Errorf("Not Found volume: %v", err))
-		return
-	}
-	if int(v.SuperBlock.CompactRevision) != util.ParseInt(r.FormValue("revision"), 0) {
-		writeJsonError(w, r, http.StatusExpectationFailed, fmt.Errorf("Requested Volume Revision is %s, but current revision is %d", r.FormValue("revision"), v.SuperBlock.CompactRevision))
-		return
-	}
-	offset := uint32(util.ParseUint64(r.FormValue("offset"), 0))
-	size := uint32(util.ParseUint64(r.FormValue("size"), 0))
-	content, err := storage.ReadNeedleBlob(v.DataFile(), int64(offset)*types.NeedlePaddingSize, size, v.Version())
-	if err != nil {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-		return
-	}
-
-	id, err := types.ParseNeedleId(r.FormValue("id"))
-	if err != nil {
-		writeJsonError(w, r, http.StatusBadRequest, err)
-		return
-	}
-	n := new(storage.Needle)
-	n.ParseNeedleHeader(content)
-	if id != n.Id {
-		writeJsonError(w, r, http.StatusNotFound, fmt.Errorf("Expected file entry id %d, but found %d", id, n.Id))
-		return
-	}
-
-	w.Write(content)
-}
-
-func (vs *VolumeServer) getVolumeId(volumeParameterName string, r *http.Request) (storage.VolumeId, error) {
-	volumeIdString := r.FormValue(volumeParameterName)
-
-	if volumeIdString == "" {
-		err := fmt.Errorf("Empty Volume Id: Need to pass in %s=the_volume_id.", volumeParameterName)
-		return 0, err
-	}
-
-	vid, err := storage.NewVolumeId(volumeIdString)
-	if err != nil {
-		err = fmt.Errorf("Volume Id %s is not a valid unsigned integer", volumeIdString)
-		return 0, err
-	}
-
-	return vid, err
-}

weed/server/volume_server_handlers_ui.go

@@ -5,6 +5,7 @@ import (
 	"path/filepath"
 	"time"
 
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	ui "github.com/chrislusf/seaweedfs/weed/server/volume_server_ui"
 	"github.com/chrislusf/seaweedfs/weed/stats"
 	"github.com/chrislusf/seaweedfs/weed/util"
@@ -13,7 +14,7 @@ import (
 func (vs *VolumeServer) uiStatusHandler(w http.ResponseWriter, r *http.Request) {
 	infos := make(map[string]interface{})
 	infos["Up Time"] = time.Now().Sub(startTime).String()
-	var ds []*stats.DiskStatus
+	var ds []*volume_server_pb.DiskStatus
 	for _, loc := range vs.store.Locations {
 		if dir, e := filepath.Abs(loc.Directory); e == nil {
 			ds = append(ds, stats.NewDiskStatus(dir))

weed/server/volume_server_handlers_vacuum.go

@@ -1,53 +0,0 @@
-package weed_server
-
-import (
-	"net/http"
-	"strconv"
-
-	"github.com/chrislusf/seaweedfs/weed/glog"
-)
-
-func (vs *VolumeServer) vacuumVolumeCheckHandler(w http.ResponseWriter, r *http.Request) {
-	err, ret := vs.store.CheckCompactVolume(r.FormValue("volume"), r.FormValue("garbageThreshold"))
-	if err == nil {
-		writeJsonQuiet(w, r, http.StatusOK, map[string]interface{}{"error": "", "result": ret})
-	} else {
-		writeJsonQuiet(w, r, http.StatusInternalServerError, map[string]interface{}{"error": err.Error(), "result": false})
-	}
-	glog.V(2).Infoln("checked compacting volume =", r.FormValue("volume"), "garbageThreshold =", r.FormValue("garbageThreshold"), "vacuum =", ret)
-}
-func (vs *VolumeServer) vacuumVolumeCompactHandler(w http.ResponseWriter, r *http.Request) {
-	var preallocate int64
-	var err error
-	if r.FormValue("preallocate") != "" {
-		preallocate, err = strconv.ParseInt(r.FormValue("preallocate"), 10, 64)
-		if err != nil {
-			glog.V(0).Infof("Failed to parse int64 preallocate = %s: %v", r.FormValue("preallocate"), err)
-		}
-	}
-	err = vs.store.CompactVolume(r.FormValue("volume"), preallocate)
-	if err == nil {
-		writeJsonQuiet(w, r, http.StatusOK, map[string]string{"error": ""})
-	} else {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-	}
-	glog.V(2).Infoln("compacted volume =", r.FormValue("volume"), ", error =", err)
-}
-func (vs *VolumeServer) vacuumVolumeCommitHandler(w http.ResponseWriter, r *http.Request) {
-	err := vs.store.CommitCompactVolume(r.FormValue("volume"))
-	if err == nil {
-		writeJsonQuiet(w, r, http.StatusOK, map[string]string{"error": ""})
-	} else {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-	}
-	glog.V(2).Infoln("commit compact volume =", r.FormValue("volume"), ", error =", err)
-}
-func (vs *VolumeServer) vacuumVolumeCleanupHandler(w http.ResponseWriter, r *http.Request) {
-	err := vs.store.CommitCleanupVolume(r.FormValue("volume"))
-	if err == nil {
-		writeJsonQuiet(w, r, http.StatusOK, map[string]string{"error": ""})
-	} else {
-		writeJsonError(w, r, http.StatusInternalServerError, err)
-	}
-	glog.V(2).Infoln("cleanup compact volume =", r.FormValue("volume"), ", error =", err)
-}

weed/server/volume_server_handlers_write.go

@@ -34,7 +34,7 @@ func (vs *VolumeServer) PostHandler(w http.ResponseWriter, r *http.Request) {
 	}
 
 	ret := operation.UploadResult{}
-	size, errorStatus := topology.ReplicatedWrite(vs.GetMaster(),
+	_, errorStatus := topology.ReplicatedWrite(vs.GetMaster(),
 		vs.store, volumeId, needle, r)
 	httpStatus := http.StatusCreated
 	if errorStatus != "" {
@@ -44,7 +44,7 @@ func (vs *VolumeServer) PostHandler(w http.ResponseWriter, r *http.Request) {
 	if needle.HasName() {
 		ret.Name = string(needle.Name)
 	}
-	ret.Size = size
+	ret.Size = needle.DataSize
 	setEtag(w, needle.Etag())
 	writeJsonQuiet(w, r, httpStatus, ret)
 }
@@ -55,7 +55,7 @@ func (vs *VolumeServer) DeleteHandler(w http.ResponseWriter, r *http.Request) {
 	volumeId, _ := storage.NewVolumeId(vid)
 	n.ParsePath(fid)
 
-	glog.V(2).Infof("volume %s deleting %s", vid, n)
+	// glog.V(2).Infof("volume %s deleting %s", vid, n)
 
 	cookie := n.Cookie
 
@@ -109,71 +109,6 @@ func (vs *VolumeServer) DeleteHandler(w http.ResponseWriter, r *http.Request) {
 
 }
 
-//Experts only: takes multiple fid parameters. This function does not propagate deletes to replicas.
-func (vs *VolumeServer) batchDeleteHandler(w http.ResponseWriter, r *http.Request) {
-	r.ParseForm()
-	var ret []operation.DeleteResult
-	now := uint64(time.Now().Unix())
-	for _, fid := range r.Form["fid"] {
-		vid, id_cookie, err := operation.ParseFileId(fid)
-		if err != nil {
-			ret = append(ret, operation.DeleteResult{
-				Fid:    fid,
-				Status: http.StatusBadRequest,
-				Error:  err.Error()})
-			continue
-		}
-		n := new(storage.Needle)
-		volumeId, _ := storage.NewVolumeId(vid)
-		n.ParsePath(id_cookie)
-		// glog.V(4).Infoln("batch deleting", n)
-		cookie := n.Cookie
-		if _, err := vs.store.ReadVolumeNeedle(volumeId, n); err != nil {
-			ret = append(ret, operation.DeleteResult{
-				Fid:    fid,
-				Status: http.StatusNotFound,
-				Error:  err.Error(),
-			})
-			continue
-		}
-
-		if n.IsChunkedManifest() {
-			ret = append(ret, operation.DeleteResult{
-				Fid:    fid,
-				Status: http.StatusNotAcceptable,
-				Error:  "ChunkManifest: not allowed in batch delete mode.",
-			})
-			continue
-		}
-
-		if n.Cookie != cookie {
-			ret = append(ret, operation.DeleteResult{
-				Fid:    fid,
-				Status: http.StatusBadRequest,
-				Error:  "File Random Cookie does not match.",
-			})
-			glog.V(0).Infoln("deleting", fid, "with unmaching cookie from ", r.RemoteAddr, "agent", r.UserAgent())
-			return
-		}
-		n.LastModified = now
-		if size, err := vs.store.Delete(volumeId, n); err != nil {
-			ret = append(ret, operation.DeleteResult{
-				Fid:    fid,
-				Status: http.StatusInternalServerError,
-				Error:  err.Error()},
-			)
-		} else {
-			ret = append(ret, operation.DeleteResult{
-				Fid:    fid,
-				Status: http.StatusAccepted,
-				Size:   int(size)},
-			)
-		}
-	}
-
-	writeJsonQuiet(w, r, http.StatusAccepted, ret)
-}
-
 func setEtag(w http.ResponseWriter, etag string) {
 	if etag != "" {
 		if strings.HasPrefix(etag, "\"") {

weed/stats/disk.go

@@ -1,14 +1,9 @@
 package stats
 
-type DiskStatus struct {
-	Dir  string
-	All  uint64
-	Used uint64
-	Free uint64
-}
+import "github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 
-func NewDiskStatus(path string) (disk *DiskStatus) {
-	disk = &DiskStatus{Dir: path}
-	disk.fillInStatus()
+func NewDiskStatus(path string) (disk *volume_server_pb.DiskStatus) {
+	disk = &volume_server_pb.DiskStatus{Dir: path}
+	fillInDiskStatus(disk)
 	return
 }

weed/stats/disk_notsupported.go

@@ -2,6 +2,8 @@
 
 package stats
 
-func (disk *DiskStatus) fillInStatus() {
+import "github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
+
+func fillInDiskStatus(status *volume_server_pb.DiskStatus) {
 	return
 }

weed/stats/disk_supported.go

@@ -4,9 +4,11 @@ package stats
 
 import (
 	"syscall"
+
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 )
 
-func (disk *DiskStatus) fillInStatus() {
+func fillInDiskStatus(disk *volume_server_pb.DiskStatus) {
 	fs := syscall.Statfs_t{}
 	err := syscall.Statfs(disk.Dir, &fs)
 	if err != nil {

weed/stats/memory.go

@@ -2,27 +2,19 @@ package stats
 
 import (
 	"runtime"
+
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 )
 
-type MemStatus struct {
-	Goroutines int
-	All        uint64
-	Used       uint64
-	Free       uint64
-	Self       uint64
-	Heap       uint64
-	Stack      uint64
-}
-
-func MemStat() MemStatus {
-	mem := MemStatus{}
-	mem.Goroutines = runtime.NumGoroutine()
+func MemStat() *volume_server_pb.MemStatus {
+	mem := &volume_server_pb.MemStatus{}
+	mem.Goroutines = int32(runtime.NumGoroutine())
 	memStat := new(runtime.MemStats)
 	runtime.ReadMemStats(memStat)
 	mem.Self = memStat.Alloc
 	mem.Heap = memStat.HeapAlloc
 	mem.Stack = memStat.StackInuse
 
-	mem.fillInStatus()
+	fillInMemStatus(mem)
 	return mem
 }

weed/stats/memory_notsupported.go

@@ -2,6 +2,8 @@
 
 package stats
 
-func (mem *MemStatus) fillInStatus() {
+import "github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
+
+func fillInMemStatus(status *volume_server_pb.MemStatus) {
 	return
 }

weed/stats/memory_supported.go

@@ -4,9 +4,11 @@ package stats
 
 import (
 	"syscall"
+
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 )
 
-func (mem *MemStatus) fillInStatus() {
+func fillInMemStatus(mem *volume_server_pb.MemStatus) {
 	//system memory usage
 	sysInfo := new(syscall.Sysinfo_t)
 	err := syscall.Sysinfo(sysInfo)

weed/storage/crc.go

@@ -2,8 +2,9 @@ package storage
 
 import (
 	"fmt"
-	"github.com/klauspost/crc32"
+	"crypto/md5"
 
+	"github.com/klauspost/crc32"
 	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
@@ -28,3 +29,13 @@ func (n *Needle) Etag() string {
 	util.Uint32toBytes(bits, uint32(n.Checksum))
 	return fmt.Sprintf("%x", bits)
 }
+
+func (n *Needle) MD5() string {
+
+	hash := md5.New()
+
+	hash.Write(n.Data)
+
+	return fmt.Sprintf("%x", hash.Sum(nil))
+
+}

weed/storage/store.go

@@ -2,9 +2,6 @@ package storage
 
 import (
 	"fmt"
-	"strconv"
-	"strings"
-
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/pb/master_pb"
 	. "github.com/chrislusf/seaweedfs/weed/storage/types"
@@ -49,7 +46,7 @@ func NewStore(port int, ip, publicUrl string, dirnames []string, maxVolumeCounts
 	s.DeletedVolumeIdChan = make(chan VolumeId, 3)
 	return
 }
-func (s *Store) AddVolume(volumeListString string, collection string, needleMapKind NeedleMapType, replicaPlacement string, ttlString string, preallocate int64) error {
+func (s *Store) AddVolume(volumeId VolumeId, collection string, needleMapKind NeedleMapType, replicaPlacement string, ttlString string, preallocate int64) error {
 	rt, e := NewReplicaPlacementFromString(replicaPlacement)
 	if e != nil {
 		return e
@@ -58,31 +55,7 @@ func (s *Store) AddVolume(volumeListString string, collection string, needleMapK
 	if e != nil {
 		return e
 	}
-	for _, range_string := range strings.Split(volumeListString, ",") {
-		if strings.Index(range_string, "-") < 0 {
-			id_string := range_string
-			id, err := NewVolumeId(id_string)
-			if err != nil {
-				return fmt.Errorf("Volume Id %s is not a valid unsigned integer!", id_string)
-			}
-			e = s.addVolume(VolumeId(id), collection, needleMapKind, rt, ttl, preallocate)
-		} else {
-			pair := strings.Split(range_string, "-")
-			start, start_err := strconv.ParseUint(pair[0], 10, 64)
-			if start_err != nil {
-				return fmt.Errorf("Volume Start Id %s is not a valid unsigned integer!", pair[0])
-			}
-			end, end_err := strconv.ParseUint(pair[1], 10, 64)
-			if end_err != nil {
-				return fmt.Errorf("Volume End Id %s is not a valid unsigned integer!", pair[1])
-			}
-			for id := start; id <= end; id++ {
-				if err := s.addVolume(VolumeId(id), collection, needleMapKind, rt, ttl, preallocate); err != nil {
-					e = err
-				}
-			}
-		}
-	}
+	e = s.addVolume(volumeId, collection, needleMapKind, rt, ttl, preallocate)
 	return e
 }
 func (s *Store) DeleteCollection(collection string) (e error) {
@@ -281,14 +254,3 @@ func (s *Store) UnmountVolume(i VolumeId) error {
 
 	return fmt.Errorf("Volume %d not found on disk", i)
 }
-
-func (s *Store) DeleteVolume(i VolumeId) error {
-	for _, location := range s.Locations {
-		if error := location.deleteVolumeById(i); error == nil {
-			s.DeletedVolumeIdChan <- VolumeId(i)
-			return nil
-		}
-	}
-
-	return fmt.Errorf("Volume %d not found on disk", i)
-}

weed/storage/store_vacuum.go

@@ -2,51 +2,29 @@ package storage
 
 import (
 	"fmt"
-	"strconv"
-
 	"github.com/chrislusf/seaweedfs/weed/glog"
 )
 
-func (s *Store) CheckCompactVolume(volumeIdString string, garbageThresholdString string) (error, bool) {
-	vid, err := NewVolumeId(volumeIdString)
-	if err != nil {
-		return fmt.Errorf("Volume Id %s is not a valid unsigned integer", volumeIdString), false
+func (s *Store) CheckCompactVolume(volumeId VolumeId) (float64, error) {
+	if v := s.findVolume(volumeId); v != nil {
+		glog.V(3).Infof("volumd %d garbage level: %f", volumeId, v.garbageLevel())
+		return v.garbageLevel(), nil
 	}
-	garbageThreshold, e := strconv.ParseFloat(garbageThresholdString, 32)
-	if e != nil {
-		return fmt.Errorf("garbageThreshold %s is not a valid float number", garbageThresholdString), false
-	}
-	if v := s.findVolume(vid); v != nil {
-		glog.V(3).Infoln(vid, "garbage level is", v.garbageLevel())
-		return nil, garbageThreshold < v.garbageLevel()
-	}
-	return fmt.Errorf("volume id %d is not found during check compact", vid), false
+	return 0, fmt.Errorf("volume id %d is not found during check compact", volumeId)
 }
-func (s *Store) CompactVolume(volumeIdString string, preallocate int64) error {
-	vid, err := NewVolumeId(volumeIdString)
-	if err != nil {
-		return fmt.Errorf("Volume Id %s is not a valid unsigned integer", volumeIdString)
-	}
+func (s *Store) CompactVolume(vid VolumeId, preallocate int64) error {
 	if v := s.findVolume(vid); v != nil {
 		return v.Compact(preallocate)
 	}
 	return fmt.Errorf("volume id %d is not found during compact", vid)
 }
-func (s *Store) CommitCompactVolume(volumeIdString string) error {
-	vid, err := NewVolumeId(volumeIdString)
-	if err != nil {
-		return fmt.Errorf("Volume Id %s is not a valid unsigned integer", volumeIdString)
-	}
+func (s *Store) CommitCompactVolume(vid VolumeId) error {
 	if v := s.findVolume(vid); v != nil {
 		return v.commitCompact()
 	}
 	return fmt.Errorf("volume id %d is not found during commit compact", vid)
 }
-func (s *Store) CommitCleanupVolume(volumeIdString string) error {
-	vid, err := NewVolumeId(volumeIdString)
-	if err != nil {
-		return fmt.Errorf("Volume Id %s is not a valid unsigned integer", volumeIdString)
-	}
+func (s *Store) CommitCleanupVolume(vid VolumeId) error {
 	if v := s.findVolume(vid); v != nil {
 		return v.cleanupCompact()
 	}

weed/storage/volume.go

@@ -57,6 +57,13 @@ func (v *Volume) Version() Version {
 }
 
 func (v *Volume) Size() int64 {
+	v.dataFileAccessLock.Lock()
+	defer v.dataFileAccessLock.Unlock()
+
+	if v.dataFile == nil {
+		return 0
+	}
+
 	stat, e := v.dataFile.Stat()
 	if e == nil {
 		return stat.Size()
@@ -69,8 +76,14 @@ func (v *Volume) Size() int64 {
 func (v *Volume) Close() {
 	v.dataFileAccessLock.Lock()
 	defer v.dataFileAccessLock.Unlock()
-	v.nm.Close()
-	_ = v.dataFile.Close()
+	if v.nm != nil {
+		v.nm.Close()
+		v.nm = nil
+	}
+	if v.dataFile != nil {
+		_ = v.dataFile.Close()
+		v.dataFile = nil
+	}
 }
 
 func (v *Volume) NeedToReplicate() bool {

weed/storage/volume_read_write.go

@@ -41,11 +41,10 @@ func (v *Volume) Destroy() (err error) {
 		return
 	}
 	v.Close()
-	err = os.Remove(v.dataFile.Name())
-	if err != nil {
-		return
+	os.Remove(v.FileName() + ".dat")
+	if v.nm!=nil{
+		err = v.nm.Destroy()
 	}
-	err = v.nm.Destroy()
 	os.Remove(v.FileName() + ".cpd")
 	os.Remove(v.FileName() + ".cpx")
 	os.Remove(v.FileName() + ".ldb")
@@ -193,6 +192,7 @@ func ScanVolumeFile(dirname string, collection string, id VolumeId,
 	if err = visitSuperBlock(v.SuperBlock); err != nil {
 		return fmt.Errorf("Failed to process volume %d super block: %v", id, err)
 	}
+	defer v.Close()
 
 	version := v.Version()
 

weed/storage/volume_sync.go

@@ -1,19 +1,16 @@
 package storage
 
 import (
+	"context"
 	"fmt"
-	"io"
-	"io/ioutil"
-	"net/url"
 	"os"
 	"sort"
-	"strconv"
 
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/operation"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/chrislusf/seaweedfs/weed/storage/needle"
 	. "github.com/chrislusf/seaweedfs/weed/storage/types"
-	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
 // The volume sync with a master volume via 2 steps:
@@ -121,7 +118,6 @@ func (v *Volume) trySynchronizing(volumeServer string, masterMap *needle.Compact
 
 	// make up the delta
 	fetchCount := 0
-	volumeDataContentHandlerUrl := "http://" + volumeServer + "/admin/sync/data"
 	for _, needleValue := range delta {
 		if needleValue.Size == 0 {
 			// remove file entry from local
@@ -129,7 +125,7 @@ func (v *Volume) trySynchronizing(volumeServer string, masterMap *needle.Compact
 			continue
 		}
 		// add master file entry to local data file
-		if err := v.fetchNeedle(volumeDataContentHandlerUrl, needleValue, compactRevision); err != nil {
+		if err := v.fetchNeedle(volumeServer, needleValue, compactRevision); err != nil {
 			glog.V(0).Infof("Fetch needle %v from %s: %v", needleValue, volumeServer, err)
 			return err
 		}
@@ -142,13 +138,13 @@ func (v *Volume) trySynchronizing(volumeServer string, masterMap *needle.Compact
 func fetchVolumeFileEntries(volumeServer string, vid VolumeId) (m *needle.CompactMap, lastOffset uint64, compactRevision uint16, err error) {
 	m = needle.NewCompactMap()
 
-	syncStatus, err := operation.GetVolumeSyncStatus(volumeServer, vid.String())
+	syncStatus, err := operation.GetVolumeSyncStatus(volumeServer, uint32(vid))
 	if err != nil {
 		return m, 0, 0, err
 	}
 
 	total := 0
-	err = operation.GetVolumeIdxEntries(volumeServer, vid.String(), func(key NeedleId, offset Offset, size uint32) {
+	err = operation.GetVolumeIdxEntries(volumeServer, uint32(vid), func(key NeedleId, offset Offset, size uint32) {
 		// println("remote key", key, "offset", offset*NeedlePaddingSize, "size", size)
 		if offset > 0 && size != TombstoneFileSize {
 			m.Set(NeedleId(key), offset, size)
@@ -159,17 +155,17 @@ func fetchVolumeFileEntries(volumeServer string, vid VolumeId) (m *needle.Compac
 	})
 
 	glog.V(2).Infof("server %s volume %d, entries %d, last offset %d, revision %d", volumeServer, vid, total, syncStatus.TailOffset, syncStatus.CompactRevision)
-	return m, syncStatus.TailOffset, syncStatus.CompactRevision, err
+	return m, syncStatus.TailOffset, uint16(syncStatus.CompactRevision), err
 
 }
 
-func (v *Volume) GetVolumeSyncStatus() operation.SyncVolumeResponse {
-	var syncStatus = operation.SyncVolumeResponse{}
+func (v *Volume) GetVolumeSyncStatus() *volume_server_pb.VolumeSyncStatusResponse {
+	var syncStatus = &volume_server_pb.VolumeSyncStatusResponse{}
 	if stat, err := v.dataFile.Stat(); err == nil {
 		syncStatus.TailOffset = uint64(stat.Size())
 	}
 	syncStatus.IdxFileSize = v.nm.IndexFileSize()
-	syncStatus.CompactRevision = v.SuperBlock.CompactRevision
+	syncStatus.CompactRevision = uint32(v.SuperBlock.CompactRevision)
 	syncStatus.Ttl = v.SuperBlock.Ttl.String()
 	syncStatus.Replication = v.SuperBlock.ReplicaPlacement.String()
 	return syncStatus
@@ -189,22 +185,21 @@ func (v *Volume) removeNeedle(key NeedleId) {
 // fetchNeedle fetches a remote volume needle by vid, id, offset
 // The compact revision is checked first in case the remote volume
 // is compacted and the offset is invalid any more.
-func (v *Volume) fetchNeedle(volumeDataContentHandlerUrl string,
-	needleValue needle.NeedleValue, compactRevision uint16) error {
-	// add master file entry to local data file
-	values := make(url.Values)
-	values.Add("revision", strconv.Itoa(int(compactRevision)))
-	values.Add("volume", v.Id.String())
-	values.Add("id", needleValue.Key.String())
-	values.Add("offset", strconv.FormatUint(uint64(needleValue.Offset), 10))
-	values.Add("size", strconv.FormatUint(uint64(needleValue.Size), 10))
-	glog.V(4).Infof("Fetch %+v", needleValue)
-	return util.GetUrlStream(volumeDataContentHandlerUrl, values, func(r io.Reader) error {
-		b, err := ioutil.ReadAll(r)
+func (v *Volume) fetchNeedle(volumeServer string, needleValue needle.NeedleValue, compactRevision uint16) error {
+
+	return operation.WithVolumeServerClient(volumeServer, func(client volume_server_pb.VolumeServerClient) error {
+		resp, err := client.VolumeSyncData(context.Background(), &volume_server_pb.VolumeSyncDataRequest{
+			VolumdId: uint32(v.Id),
+			Revision: uint32(compactRevision),
+			Offset:   uint32(needleValue.Offset),
+			Size:     uint32(needleValue.Size),
+			NeedleId: needleValue.Key.String(),
+		})
 		if err != nil {
-			return fmt.Errorf("Reading from %s error: %v", volumeDataContentHandlerUrl, err)
+			return err
 		}
-		offset, err := v.AppendBlob(b)
+
+		offset, err := v.AppendBlob(resp.FileContent)
 		if err != nil {
 			return fmt.Errorf("Appending volume %d error: %v", v.Id, err)
 		}
@@ -212,4 +207,5 @@ func (v *Volume) fetchNeedle(volumeDataContentHandlerUrl string,
 		v.nm.Put(needleValue.Key, Offset(offset/NeedlePaddingSize), needleValue.Size)
 		return nil
 	})
+
 }

weed/storage/volume_vacuum.go

@@ -18,7 +18,7 @@ func (v *Volume) garbageLevel() float64 {
 }
 
 func (v *Volume) Compact(preallocate int64) error {
-	glog.V(3).Infof("Compacting ...")
+	glog.V(3).Infof("Compacting volume %d ...", v.Id)
 	//no need to lock for copy on write
 	//v.accessLock.Lock()
 	//defer v.accessLock.Unlock()
@@ -32,23 +32,26 @@ func (v *Volume) Compact(preallocate int64) error {
 }
 
 func (v *Volume) Compact2() error {
-	glog.V(3).Infof("Compact2 ...")
+	glog.V(3).Infof("Compact2 volume %d ...", v.Id)
 	filePath := v.FileName()
 	glog.V(3).Infof("creating copies for volume %d ...", v.Id)
 	return v.copyDataBasedOnIndexFile(filePath+".cpd", filePath+".cpx")
 }
 
 func (v *Volume) commitCompact() error {
-	glog.V(0).Infof("Committing vacuuming...")
+	glog.V(0).Infof("Committing volume %d vacuuming...", v.Id)
 	v.dataFileAccessLock.Lock()
 	defer v.dataFileAccessLock.Unlock()
-	glog.V(3).Infof("Got Committing lock...")
+	glog.V(3).Infof("Got volume %d committing lock...", v.Id)
 	v.nm.Close()
-	_ = v.dataFile.Close()
+	if err := v.dataFile.Close(); err != nil {
+		glog.V(0).Infof("fail to close volume %d", v.Id)
+	}
+	v.dataFile = nil
 
 	var e error
 	if e = v.makeupDiff(v.FileName()+".cpd", v.FileName()+".cpx", v.FileName()+".dat", v.FileName()+".idx"); e != nil {
-		glog.V(0).Infof("makeupDiff in commitCompact failed %v", e)
+		glog.V(0).Infof("makeupDiff in commitCompact volume %d failed %v", v.Id, e)
 		e = os.Remove(v.FileName() + ".cpd")
 		if e != nil {
 			return e
@@ -60,10 +63,10 @@ func (v *Volume) commitCompact() error {
 	} else {
 		var e error
 		if e = os.Rename(v.FileName()+".cpd", v.FileName()+".dat"); e != nil {
-			return e
+			return fmt.Errorf("rename %s: %v", v.FileName()+".cpd", e)
 		}
 		if e = os.Rename(v.FileName()+".cpx", v.FileName()+".idx"); e != nil {
-			return e
+			return fmt.Errorf("rename %s: %v", v.FileName()+".cpx", e)
 		}
 	}
 
@@ -73,7 +76,7 @@ func (v *Volume) commitCompact() error {
 	os.RemoveAll(v.FileName() + ".ldb")
 	os.RemoveAll(v.FileName() + ".bdb")
 
-	glog.V(3).Infof("Loading Commit file...")
+	glog.V(3).Infof("Loading volume %d commit file...", v.Id)
 	if e = v.load(true, false, v.needleMapKind, 0); e != nil {
 		return e
 	}
@@ -81,7 +84,7 @@ func (v *Volume) commitCompact() error {
 }
 
 func (v *Volume) cleanupCompact() error {
-	glog.V(0).Infof("Cleaning up vacuuming...")
+	glog.V(0).Infof("Cleaning up volume %d vacuuming...", v.Id)
 
 	e1 := os.Remove(v.FileName() + ".cpd")
 	e2 := os.Remove(v.FileName() + ".cpx")
@@ -270,7 +273,7 @@ func (v *Volume) copyDataAndGenerateIndexFile(dstName, idxName string, prealloca
 					return fmt.Errorf("cannot append needle: %s", err)
 				}
 				new_offset += n.DiskSize(version)
-				glog.V(3).Infoln("saving key", n.Id, "volume offset", offset, "=>", new_offset, "data_size", n.Size)
+				glog.V(4).Infoln("saving key", n.Id, "volume offset", offset, "=>", new_offset, "data_size", n.Size)
 			}
 			return nil
 		})

weed/topology/allocate_volume.go

@@ -1,13 +1,11 @@
 package topology
 
 import (
-	"encoding/json"
-	"errors"
-	"fmt"
-	"net/url"
+	"context"
 
+	"github.com/chrislusf/seaweedfs/weed/operation"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/chrislusf/seaweedfs/weed/storage"
-	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
 type AllocateVolumeResult struct {
@@ -15,22 +13,16 @@ type AllocateVolumeResult struct {
 }
 
 func AllocateVolume(dn *DataNode, vid storage.VolumeId, option *VolumeGrowOption) error {
-	values := make(url.Values)
-	values.Add("volume", vid.String())
-	values.Add("collection", option.Collection)
-	values.Add("replication", option.ReplicaPlacement.String())
-	values.Add("ttl", option.Ttl.String())
-	values.Add("preallocate", fmt.Sprintf("%d", option.Prealloacte))
-	jsonBlob, err := util.Post("http://"+dn.Url()+"/admin/assign_volume", values)
-	if err != nil {
-		return err
-	}
-	var ret AllocateVolumeResult
-	if err := json.Unmarshal(jsonBlob, &ret); err != nil {
-		return fmt.Errorf("Invalid JSON result for %s: %s", "/admin/assign_volum", string(jsonBlob))
-	}
-	if ret.Error != "" {
-		return errors.New(ret.Error)
-	}
-	return nil
+
+	return operation.WithVolumeServerClient(dn.Url(), func(client volume_server_pb.VolumeServerClient) error {
+		_, deleteErr := client.AssignVolume(context.Background(), &volume_server_pb.AssignVolumeRequest{
+			VolumdId:    uint32(vid),
+			Collection:  option.Collection,
+			Replication: option.ReplicaPlacement.String(),
+			Ttl:         option.Ttl.String(),
+			Preallocate: option.Prealloacte,
+		})
+		return deleteErr
+	})
+
 }

weed/topology/topology_event_handling.go

@@ -8,7 +8,7 @@ import (
 	"github.com/chrislusf/seaweedfs/weed/storage"
 )
 
-func (t *Topology) StartRefreshWritableVolumes(garbageThreshold string, preallocate int64) {
+func (t *Topology) StartRefreshWritableVolumes(garbageThreshold float64, preallocate int64) {
 	go func() {
 		for {
 			if t.IsLeader() {
@@ -18,7 +18,7 @@ func (t *Topology) StartRefreshWritableVolumes(garbageThreshold string, prealloc
 			time.Sleep(time.Duration(float32(t.pulse*1e3)*(1+rand.Float32())) * time.Millisecond)
 		}
 	}()
-	go func(garbageThreshold string) {
+	go func(garbageThreshold float64) {
 		c := time.Tick(15 * time.Minute)
 		for _ = range c {
 			if t.IsLeader() {
@@ -40,6 +40,10 @@ func (t *Topology) SetVolumeCapacityFull(volumeInfo storage.VolumeInfo) bool {
 	if !vl.SetVolumeCapacityFull(volumeInfo.Id) {
 		return false
 	}
+
+	vl.accessLock.RLock()
+	defer vl.accessLock.RUnlock()
+
 	for _, dn := range vl.vid2location[volumeInfo.Id].list {
 		if !volumeInfo.ReadOnly {
 			dn.UpAdjustActiveVolumeCountDelta(-1)

weed/topology/topology_vacuum.go

@@ -1,28 +1,33 @@
 package topology
 
 import (
-	"encoding/json"
-	"errors"
-	"net/url"
+	"context"
 	"time"
 
-	"fmt"
 	"github.com/chrislusf/seaweedfs/weed/glog"
+	"github.com/chrislusf/seaweedfs/weed/operation"
+	"github.com/chrislusf/seaweedfs/weed/pb/volume_server_pb"
 	"github.com/chrislusf/seaweedfs/weed/storage"
-	"github.com/chrislusf/seaweedfs/weed/util"
 )
 
-func batchVacuumVolumeCheck(vl *VolumeLayout, vid storage.VolumeId, locationlist *VolumeLocationList, garbageThreshold string) bool {
+func batchVacuumVolumeCheck(vl *VolumeLayout, vid storage.VolumeId, locationlist *VolumeLocationList, garbageThreshold float64) bool {
 	ch := make(chan bool, locationlist.Length())
 	for index, dn := range locationlist.list {
 		go func(index int, url string, vid storage.VolumeId) {
-			//glog.V(0).Infoln(index, "Check vacuuming", vid, "on", dn.Url())
-			if e, ret := vacuumVolume_Check(url, vid, garbageThreshold); e != nil {
-				//glog.V(0).Infoln(index, "Error when checking vacuuming", vid, "on", url, e)
-				ch <- false
-			} else {
-				//glog.V(0).Infoln(index, "Checked vacuuming", vid, "on", url, "needVacuum", ret)
-				ch <- ret
+			err := operation.WithVolumeServerClient(url, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
+				resp, err := volumeServerClient.VacuumVolumeCheck(context.Background(), &volume_server_pb.VacuumVolumeCheckRequest{
+					VolumdId: uint32(vid),
+				})
+				if err != nil {
+					ch <- false
+					return err
+				}
+				isNeeded := resp.GarbageRatio > garbageThreshold
+				ch <- isNeeded
+				return nil
+			})
+			if err != nil {
+				glog.V(0).Infof("Checking vacuuming %d on %s: %v", vid, url, err)
 			}
 		}(index, dn.Url(), vid)
 	}
@@ -44,11 +49,17 @@ func batchVacuumVolumeCompact(vl *VolumeLayout, vid storage.VolumeId, locationli
 	for index, dn := range locationlist.list {
 		go func(index int, url string, vid storage.VolumeId) {
 			glog.V(0).Infoln(index, "Start vacuuming", vid, "on", url)
-			if e := vacuumVolume_Compact(url, vid, preallocate); e != nil {
-				glog.V(0).Infoln(index, "Error when vacuuming", vid, "on", url, e)
+			err := operation.WithVolumeServerClient(url, func(volumeServerClient volume_server_pb.VolumeServerClient) error {
+				_, err := volumeServerClient.VacuumVolumeCompact(context.Background(), &volume_server_pb.VacuumVolumeCompactRequest{
+					VolumdId: uint32(vid),
+				})
+				return err
+			})
+			if err != nil {
+				glog.Errorf("Error when vacuuming %d on %s: %v", vid, url, err)
 				ch <- false
 			} else {
-				glog.V(0).Infoln(index, "Complete vacuuming", vid, "on", url)
+				glog.V(0).Infof("Complete vacuuming %d on %s", vid, url)
 				ch <- true
 			}
 		}(index, dn.Url(), vid)
@@ -69,11 +80,17 @@ func batchVacuumVolumeCommit(vl *VolumeLayout, vid storage.VolumeId, locationlis
 	isCommitSuccess := true
 	for _, dn := range locationlist.list {
 		glog.V(0).Infoln("Start Commiting vacuum", vid, "on", dn.Url())
-		if e := vacuumVolume_Commit(dn.Url(), vid); e != nil {
-			glog.V(0).Infoln("Error when committing vacuum", vid, "on", dn.Url(), e)
+		err := operation.WithVolumeServerClient(dn.Url(), func(volumeServerClient volume_server_pb.VolumeServerClient) error {
+			_, err := volumeServerClient.VacuumVolumeCommit(context.Background(), &volume_server_pb.VacuumVolumeCommitRequest{
+				VolumdId: uint32(vid),
+			})
+			return err
+		})
+		if err != nil {
+			glog.Errorf("Error when committing vacuum %d on %s: %v", vid, dn.Url(), err)
 			isCommitSuccess = false
 		} else {
-			glog.V(0).Infoln("Complete Commiting vacuum", vid, "on", dn.Url())
+			glog.V(0).Infof("Complete Commiting vacuum %d on %s", vid, dn.Url())
 		}
 		if isCommitSuccess {
 			vl.SetVolumeAvailable(dn, vid)
@@ -84,113 +101,60 @@ func batchVacuumVolumeCommit(vl *VolumeLayout, vid storage.VolumeId, locationlis
 func batchVacuumVolumeCleanup(vl *VolumeLayout, vid storage.VolumeId, locationlist *VolumeLocationList) {
 	for _, dn := range locationlist.list {
 		glog.V(0).Infoln("Start cleaning up", vid, "on", dn.Url())
-		if e := vacuumVolume_Cleanup(dn.Url(), vid); e != nil {
-			glog.V(0).Infoln("Error when cleaning up", vid, "on", dn.Url(), e)
+		err := operation.WithVolumeServerClient(dn.Url(), func(volumeServerClient volume_server_pb.VolumeServerClient) error {
+			_, err := volumeServerClient.VacuumVolumeCleanup(context.Background(), &volume_server_pb.VacuumVolumeCleanupRequest{
+				VolumdId: uint32(vid),
+			})
+			return err
+		})
+		if err != nil {
+			glog.Errorf("Error when cleaning up vacuum %d on %s: %v", vid, dn.Url(), err)
 		} else {
-			glog.V(0).Infoln("Complete cleaning up", vid, "on", dn.Url())
+			glog.V(0).Infof("Complete cleaning up vacuum %d on %s", vid, dn.Url())
 		}
 	}
 }
 
-func (t *Topology) Vacuum(garbageThreshold string, preallocate int64) int {
-	glog.V(0).Infof("Start vacuum on demand with threshold:%s", garbageThreshold)
+func (t *Topology) Vacuum(garbageThreshold float64, preallocate int64) int {
+	glog.V(0).Infof("Start vacuum on demand with threshold: %f", garbageThreshold)
 	for _, col := range t.collectionMap.Items() {
 		c := col.(*Collection)
 		for _, vl := range c.storageType2VolumeLayout.Items() {
 			if vl != nil {
 				volumeLayout := vl.(*VolumeLayout)
-				for vid, locationlist := range volumeLayout.vid2location {
-
-					volumeLayout.accessLock.RLock()
-					isReadOnly, hasValue := volumeLayout.readonlyVolumes[vid]
-					volumeLayout.accessLock.RUnlock()
-
-					if hasValue && isReadOnly {
-						continue
-					}
-
-					glog.V(0).Infof("check vacuum on collection:%s volume:%d", c.Name, vid)
-					if batchVacuumVolumeCheck(volumeLayout, vid, locationlist, garbageThreshold) {
-						if batchVacuumVolumeCompact(volumeLayout, vid, locationlist, preallocate) {
-							batchVacuumVolumeCommit(volumeLayout, vid, locationlist)
-						}
-					}
-				}
+				vacuumOneVolumeLayout(volumeLayout, c, garbageThreshold, preallocate)
 			}
 		}
 	}
 	return 0
 }
 
-type VacuumVolumeResult struct {
-	Result bool
-	Error  string
-}
+func vacuumOneVolumeLayout(volumeLayout *VolumeLayout, c *Collection, garbageThreshold float64, preallocate int64) {
 
-func vacuumVolume_Check(urlLocation string, vid storage.VolumeId, garbageThreshold string) (error, bool) {
-	values := make(url.Values)
-	values.Add("volume", vid.String())
-	values.Add("garbageThreshold", garbageThreshold)
-	jsonBlob, err := util.Post("http://"+urlLocation+"/admin/vacuum/check", values)
-	if err != nil {
-		glog.V(0).Infoln("parameters:", values)
-		return err, false
+	volumeLayout.accessLock.RLock()
+	tmpMap := make(map[storage.VolumeId]*VolumeLocationList)
+	for vid, locationlist := range volumeLayout.vid2location {
+		tmpMap[vid] = locationlist
 	}
-	var ret VacuumVolumeResult
-	if err := json.Unmarshal(jsonBlob, &ret); err != nil {
-		return err, false
+	volumeLayout.accessLock.RUnlock()
+
+	for vid, locationlist := range tmpMap {
+
+		volumeLayout.accessLock.RLock()
+		isReadOnly, hasValue := volumeLayout.readonlyVolumes[vid]
+		volumeLayout.accessLock.RUnlock()
+
+		if hasValue && isReadOnly {
+			continue
+		}
+
+		glog.V(0).Infof("check vacuum on collection:%s volume:%d", c.Name, vid)
+		if batchVacuumVolumeCheck(volumeLayout, vid, locationlist, garbageThreshold) {
+			if batchVacuumVolumeCompact(volumeLayout, vid, locationlist, preallocate) {
+				batchVacuumVolumeCommit(volumeLayout, vid, locationlist)
+			} else {
+				batchVacuumVolumeCleanup(volumeLayout, vid, locationlist)
+			}
+		}
 	}
-	if ret.Error != "" {
-		return errors.New(ret.Error), false
-	}
-	return nil, ret.Result
-}
-func vacuumVolume_Compact(urlLocation string, vid storage.VolumeId, preallocate int64) error {
-	values := make(url.Values)
-	values.Add("volume", vid.String())
-	values.Add("preallocate", fmt.Sprintf("%d", preallocate))
-	jsonBlob, err := util.Post("http://"+urlLocation+"/admin/vacuum/compact", values)
-	if err != nil {
-		return err
-	}
-	var ret VacuumVolumeResult
-	if err := json.Unmarshal(jsonBlob, &ret); err != nil {
-		return err
-	}
-	if ret.Error != "" {
-		return errors.New(ret.Error)
-	}
-	return nil
-}
-func vacuumVolume_Commit(urlLocation string, vid storage.VolumeId) error {
-	values := make(url.Values)
-	values.Add("volume", vid.String())
-	jsonBlob, err := util.Post("http://"+urlLocation+"/admin/vacuum/commit", values)
-	if err != nil {
-		return err
-	}
-	var ret VacuumVolumeResult
-	if err := json.Unmarshal(jsonBlob, &ret); err != nil {
-		return err
-	}
-	if ret.Error != "" {
-		return errors.New(ret.Error)
-	}
-	return nil
-}
-func vacuumVolume_Cleanup(urlLocation string, vid storage.VolumeId) error {
-	values := make(url.Values)
-	values.Add("volume", vid.String())
-	jsonBlob, err := util.Post("http://"+urlLocation+"/admin/vacuum/cleanup", values)
-	if err != nil {
-		return err
-	}
-	var ret VacuumVolumeResult
-	if err := json.Unmarshal(jsonBlob, &ret); err != nil {
-		return err
-	}
-	if ret.Error != "" {
-		return errors.New(ret.Error)
-	}
-	return nil
 }

weed/util/constants.go

@@ -1,5 +1,5 @@
 package util
 
 const (
-	VERSION = "0.98"
+	VERSION = "1.05"
 )

weed/wdclient/vid_map.go

@@ -54,6 +54,18 @@ func (vc *vidMap) LookupFileId(fileId string) (fullUrl string, err error) {
 	return "http://" + serverUrl + "/" + fileId, nil
 }
 
+func (vc *vidMap) LookupVolumeServer(fileId string) (volumeServer string, err error) {
+	parts := strings.Split(fileId, ",")
+	if len(parts) != 2 {
+		return "", errors.New("Invalid fileId " + fileId)
+	}
+	serverUrl, lookupError := vc.LookupVolumeServerUrl(parts[0])
+	if lookupError != nil {
+		return "", lookupError
+	}
+	return serverUrl, nil
+}
+
 func (vc *vidMap) GetLocations(vid uint32) (locations []Location) {
 	vc.RLock()
 	defer vc.RUnlock()