minor

Merge branch 'master' of https://github.com/chrislusf/seaweedfs
remove dead code
2026-10-01 20:26:27 +00:00 · 2022-07-07 22:27:24 -07:00 · 2022-07-07 22:26:06 -07:00 · 2022-07-07 22:26:03 -07:00 · 2022-07-07 22:11:44 -07:00 · 2022-07-07 15:01:23 -07:00
60 changed files with 480 additions and 240 deletions
@@ -17,7 +17,7 @@ jobs:
    steps:

      - name: Delete old release assets
-        uses: mknejp/delete-release-assets@a8aaab13272b1eaac16cc46dddd3f725b97ee05a # v1
+        uses: mknejp/delete-release-assets@v1
        with:
          token: ${{ github.token }}
          tag: dev
@@ -6,8 +6,9 @@
 <h2 align="center">Generous Backers ($50+)</h2>

 - [Evercam Camera Management Software](https://evercam.io/)
+- [Spherical Elephant GmbH](https://www.sphericalelephant.com)

-<h2 align="center">Backers</h2>
+- <h2 align="center">Backers</h2>

 - [ColorfulClouds Tech Co. Ltd.](https://caiyunai.com/)
 - [Haravan - Ecommerce Platform](https://www.haravan.com)
@@ -28,6 +28,13 @@ cd $GOPATH/src/github.com/chrislusf/seaweedfs/docker
 make
 ```

+### S3 cmd
+
+list
+```
+s3cmd --no-ssl --host=127.0.0.1:8333 ls s3://
+```
+
 ## Build and push a multiarch build

 Make sure that `docker buildx` is supported (might be an experimental docker feature)
@@ -29,7 +29,7 @@ services:
      - 8111:8111
      - 8888:8888
      - 18888:18888
-    command: '-v=1 filer -master="master:9333" -iam'
+    command: '-v=1 filer -ip.bind=0.0.0.0 -master="master:9333" -iam -iam.ip=filer'
    depends_on:
      - master
      - volume
@@ -41,7 +41,7 @@ services:
    image: chrislusf/seaweedfs:local
    ports:
      - 8333:8333
-    command: '-v=1 s3 -filer="filer:8888"'
+    command: '-v=1 s3 -filer="filer:8888" -ip.bind=s3'
    depends_on:
      - master
      - volume
@@ -37,8 +37,8 @@ services:
    environment:
      S3TEST_CONF: "s3tests.conf"
      NOSETESTS_OPTIONS: "--verbose --logging-level=ERROR --with-xunit --failure-detail s3tests_boto3.functional.test_s3"
-      NOSETESTS_ATTR: "!tagging,!fails_on_aws,!encryption,!bucket-policy,!versioning,!fails_on_rgw,!bucket-policy,!fails_with_subdomain,!policy_status,!object-lock,!lifecycle,!cors,!user-policy"
-      NOSETESTS_EXCLUDE: "(get_bucket_encryption|delete_bucket_encryption|put_bucket_encryption|bucket_list_delimiter_basic|bucket_listv2_delimiter_basic|bucket_listv2_encoding_basic|bucket_list_encoding_basic|bucket_list_delimiter_prefix|bucket_listv2_delimiter_prefix_ends_with_delimiter|bucket_list_delimiter_prefix_ends_with_delimiter|bucket_list_delimiter_alt|bucket_listv2_delimiter_alt|bucket_list_delimiter_prefix_underscore|bucket_list_delimiter_percentage|bucket_listv2_delimiter_percentage|bucket_list_delimiter_whitespace|bucket_listv2_delimiter_whitespace|bucket_list_delimiter_dot|bucket_listv2_delimiter_dot|bucket_list_delimiter_unreadable|bucket_listv2_delimiter_unreadable|bucket_listv2_fetchowner_defaultempty|bucket_listv2_fetchowner_empty|bucket_list_prefix_delimiter_alt|bucket_listv2_prefix_delimiter_alt|bucket_list_prefix_delimiter_prefix_not_exist|bucket_listv2_prefix_delimiter_prefix_not_exist|bucket_list_prefix_delimiter_delimiter_not_exist|bucket_listv2_prefix_delimiter_delimiter_not_exist|bucket_list_prefix_delimiter_prefix_delimiter_not_exist|bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist|bucket_list_maxkeys_none|bucket_listv2_maxkeys_none|bucket_list_maxkeys_invalid|bucket_listv2_continuationtoken_empty|bucket_list_return_data|bucket_list_objects_anonymous|bucket_listv2_objects_anonymous|bucket_concurrent_set_canned_acl|object_write_to_nonexist_bucket|object_requestid_matches_header_on_error|object_set_get_metadata_none_to_good|object_set_get_metadata_none_to_empty|object_set_get_metadata_overwrite_to_empty|post_object_anonymous_request|post_object_authenticated_request|post_object_authenticated_no_content_type|post_object_authenticated_request_bad_access_key|post_object_set_success_code|post_object_set_invalid_success_code|post_object_upload_larger_than_chunk|post_object_set_key_from_filename|post_object_ignored_header|post_object_case_insensitive_condition_fields|post_object_escaped_field_values|post_object_success_redirect_action|post_object_invalid_signature|post_object_invalid_access_key|post_object_missing_policy_condition|post_object_user_specified_header|post_object_request_missing_policy_specified_field|post_object_expired_policy|post_object_invalid_request_field_value|get_object_ifunmodifiedsince_good|put_object_ifmatch_failed|object_raw_get_bucket_gone|object_delete_key_bucket_gone|object_raw_get_bucket_acl|object_raw_get_object_acl|object_raw_response_headers|object_raw_authenticated_bucket_gone|object_raw_get_x_amz_expires_out_max_range|object_raw_get_x_amz_expires_out_positive_range|object_anon_put_write_access|object_raw_put_authenticated_expired|bucket_create_exists|bucket_create_naming_bad_short_one|bucket_create_naming_bad_short_two|bucket_get_location|bucket_acl_default|bucket_acl_canned|bucket_acl_canned_publicreadwrite|bucket_acl_canned_authenticatedread|object_acl_default|object_acl_canned_during_create|object_acl_canned|object_acl_canned_publicreadwrite|object_acl_canned_authenticatedread|object_acl_canned_bucketownerread|object_acl_canned_bucketownerfullcontrol|object_acl_full_control_verify_attributes|bucket_acl_canned_private_to_private|bucket_acl_grant_nonexist_user|bucket_acl_no_grants|bucket_acl_grant_email_not_exist|bucket_acl_revoke_all|bucket_recreate_not_overriding|object_copy_verify_contenttype|object_copy_to_itself_with_metadata|object_copy_not_owned_bucket|object_copy_not_owned_object_bucket|object_copy_retaining_metadata|object_copy_replacing_metadata|multipart_upload_empty|multipart_copy_invalid_range|multipart_copy_special_names|multipart_upload_resend_part|multipart_upload_size_too_small|abort_multipart_upload_not_found|multipart_upload_missing_part|100_continue|ranged_request_invalid_range|ranged_request_empty_object|access_bucket|list_multipart_upload_owner|multipart_upload_small)"
+      NOSETESTS_ATTR: "!fails_on_aws,!encryption,!bucket-policy,!versioning,!fails_on_rgw,!bucket-policy,!fails_with_subdomain,!policy_status,!object-lock,!lifecycle,!cors,!user-policy"
+      NOSETESTS_EXCLUDE: "(post_object_tags_anonymous_request|get_obj_tagging|set_bucket_tagging|post_object_tags_authenticated_request|put_max_tags|put_modify_tags|test_put_obj_with_tags|get_bucket_encryption|delete_bucket_encryption|put_bucket_encryption|bucket_list_delimiter_basic|bucket_listv2_delimiter_basic|bucket_listv2_encoding_basic|bucket_list_encoding_basic|bucket_list_delimiter_prefix|bucket_listv2_delimiter_prefix_ends_with_delimiter|bucket_list_delimiter_prefix_ends_with_delimiter|bucket_list_delimiter_alt|bucket_listv2_delimiter_alt|bucket_list_delimiter_prefix_underscore|bucket_list_delimiter_percentage|bucket_listv2_delimiter_percentage|bucket_list_delimiter_whitespace|bucket_listv2_delimiter_whitespace|bucket_list_delimiter_dot|bucket_listv2_delimiter_dot|bucket_list_delimiter_unreadable|bucket_listv2_delimiter_unreadable|bucket_listv2_fetchowner_defaultempty|bucket_listv2_fetchowner_empty|bucket_list_prefix_delimiter_alt|bucket_listv2_prefix_delimiter_alt|bucket_list_prefix_delimiter_prefix_not_exist|bucket_listv2_prefix_delimiter_prefix_not_exist|bucket_list_prefix_delimiter_delimiter_not_exist|bucket_listv2_prefix_delimiter_delimiter_not_exist|bucket_list_prefix_delimiter_prefix_delimiter_not_exist|bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist|bucket_list_maxkeys_none|bucket_listv2_maxkeys_none|bucket_list_maxkeys_invalid|bucket_listv2_continuationtoken_empty|bucket_list_return_data|bucket_list_objects_anonymous|bucket_listv2_objects_anonymous|bucket_concurrent_set_canned_acl|object_write_to_nonexist_bucket|object_requestid_matches_header_on_error|object_set_get_metadata_none_to_good|object_set_get_metadata_none_to_empty|object_set_get_metadata_overwrite_to_empty|post_object_anonymous_request|post_object_authenticated_request|post_object_authenticated_no_content_type|post_object_authenticated_request_bad_access_key|post_object_set_success_code|post_object_set_invalid_success_code|post_object_upload_larger_than_chunk|post_object_set_key_from_filename|post_object_ignored_header|post_object_case_insensitive_condition_fields|post_object_escaped_field_values|post_object_success_redirect_action|post_object_invalid_signature|post_object_invalid_access_key|post_object_missing_policy_condition|post_object_user_specified_header|post_object_request_missing_policy_specified_field|post_object_expired_policy|post_object_invalid_request_field_value|get_object_ifunmodifiedsince_good|put_object_ifmatch_failed|object_raw_get_bucket_gone|object_delete_key_bucket_gone|object_raw_get_bucket_acl|object_raw_get_object_acl|object_raw_response_headers|object_raw_authenticated_bucket_gone|object_raw_get_x_amz_expires_out_max_range|object_raw_get_x_amz_expires_out_positive_range|object_anon_put_write_access|object_raw_put_authenticated_expired|bucket_create_exists|bucket_create_naming_bad_short_one|bucket_create_naming_bad_short_two|bucket_get_location|bucket_acl_default|bucket_acl_canned|bucket_acl_canned_publicreadwrite|bucket_acl_canned_authenticatedread|object_acl_default|object_acl_canned_during_create|object_acl_canned|object_acl_canned_publicreadwrite|object_acl_canned_authenticatedread|object_acl_canned_bucketownerread|object_acl_canned_bucketownerfullcontrol|object_acl_full_control_verify_attributes|bucket_acl_canned_private_to_private|bucket_acl_grant_nonexist_user|bucket_acl_no_grants|bucket_acl_grant_email_not_exist|bucket_acl_revoke_all|bucket_recreate_not_overriding|object_copy_verify_contenttype|object_copy_to_itself_with_metadata|object_copy_not_owned_bucket|object_copy_not_owned_object_bucket|object_copy_retaining_metadata|object_copy_replacing_metadata|multipart_upload_empty|multipart_copy_invalid_range|multipart_copy_special_names|multipart_upload_resend_part|multipart_upload_size_too_small|abort_multipart_upload_not_found|multipart_upload_missing_part|100_continue|ranged_request_invalid_range|ranged_request_empty_object|access_bucket|list_multipart_upload_owner|multipart_upload_small)"
    depends_on:
      - master
      - volume
@@ -20,7 +20,7 @@ services:
    ports:
      - 8888:8888
      - 18888:18888
-    command: 'filer -master="master:9333"'
+    command: 'filer -master="master:9333" -ip.bind=0.0.0.0'
    depends_on:
      - master
      - volume
@@ -28,7 +28,7 @@ services:
    image: chrislusf/seaweedfs:dev # use a remote dev image
    ports:
      - 8333:8333
-    command: 's3 -filer="filer:8888"'
+    command: 's3 -filer="filer:8888" -ip.bind=0.0.0.0'
    depends_on:
      - master
      - volume
@@ -3,14 +3,14 @@ module github.com/chrislusf/seaweedfs
 go 1.18

 require (
-	cloud.google.com/go v0.102.0 // indirect
-	cloud.google.com/go/pubsub v1.22.2
-	cloud.google.com/go/storage v1.22.1
+	cloud.google.com/go v0.102.1 // indirect
+	cloud.google.com/go/pubsub v1.23.1
+	cloud.google.com/go/storage v1.23.0
 	github.com/Azure/azure-pipeline-go v0.2.3
 	github.com/Azure/azure-storage-blob-go v0.15.0
 	github.com/OneOfOne/xxhash v1.2.8
 	github.com/Shopify/sarama v1.34.1
-	github.com/aws/aws-sdk-go v1.44.37
+	github.com/aws/aws-sdk-go v1.44.47
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/buraksezer/consistent v0.0.0-20191006190839-693edf70fd72
 	github.com/bwmarrin/snowflake v0.3.0
@@ -35,7 +35,7 @@ require (
 	github.com/fsnotify/fsnotify v1.5.4 // indirect
 	github.com/go-errors/errors v1.1.1 // indirect
 	github.com/go-redis/redis/v8 v8.11.5
-	github.com/go-redsync/redsync/v4 v4.5.0
+	github.com/go-redsync/redsync/v4 v4.5.1
 	github.com/go-sql-driver/mysql v1.6.0
 	github.com/go-stack/stack v1.8.1 // indirect
 	github.com/go-zookeeper/zk v1.0.2 // indirect
@@ -63,7 +63,7 @@ require (
 	github.com/karlseguin/ccache/v2 v2.0.8
 	github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
 	github.com/klauspost/compress v1.15.6 // indirect
-	github.com/klauspost/reedsolomon v1.9.16
+	github.com/klauspost/reedsolomon v1.10.0
 	github.com/kurin/blazer v0.5.3
 	github.com/lib/pq v1.10.6
 	github.com/linxGnu/grocksdb v1.7.3
@@ -96,7 +96,7 @@ require (
 	github.com/spf13/jwalterweatherman v1.1.0 // indirect
 	github.com/spf13/viper v1.12.0
 	github.com/streadway/amqp v1.0.0
-	github.com/stretchr/testify v1.7.3
+	github.com/stretchr/testify v1.8.0
 	github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203
 	github.com/syndtr/goleveldb v1.0.1-0.20190318030020-c3a204f8e965
 	github.com/tidwall/gjson v1.14.1
@@ -121,15 +121,15 @@ require (
 	golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 // indirect
 	golang.org/x/exp v0.0.0-20220414153411-bcd21879b8fd
 	golang.org/x/image v0.0.0-20200119044424-58c23975cae1
-	golang.org/x/net v0.0.0-20220607020251-c690dde0001d
-	golang.org/x/oauth2 v0.0.0-20220524215830-622c5d57e401 // indirect
-	golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a
+	golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e
+	golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2 // indirect
+	golang.org/x/sys v0.0.0-20220624220833-87e55d714810
 	golang.org/x/text v0.3.7 // indirect
 	golang.org/x/tools v0.1.8-0.20211029000441-d6a9af8af023
-	golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df // indirect
-	google.golang.org/api v0.83.0
+	golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f // indirect
+	google.golang.org/api v0.86.0
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/genproto v0.0.0-20220602131408-e326c6e8e9c8 // indirect
+	google.golang.org/genproto v0.0.0-20220624142145-8cd45d7dbd1f // indirect
 	google.golang.org/grpc v1.47.0
 	google.golang.org/protobuf v1.28.0
 	gopkg.in/inf.v0 v0.9.1 // indirect
@@ -140,7 +140,7 @@ require (
 	modernc.org/mathutil v1.4.1 // indirect
 	modernc.org/memory v1.1.1 // indirect
 	modernc.org/opt v0.1.1 // indirect
-	modernc.org/sqlite v1.17.2
+	modernc.org/sqlite v1.17.3
 	modernc.org/strutil v1.1.2
 	modernc.org/token v1.0.0 // indirect
 )
@@ -149,7 +149,7 @@ require (
 	github.com/Jille/raft-grpc-transport v1.2.0
 	github.com/arangodb/go-driver v1.3.2
 	github.com/fluent/fluent-logger-golang v1.9.0
-	github.com/hanwen/go-fuse/v2 v2.1.1-0.20220531082602-17a864ed5940
+	github.com/hanwen/go-fuse/v2 v2.1.1-0.20220627082937-d01fda7edf17
 	github.com/hashicorp/raft v1.3.9
 	github.com/hashicorp/raft-boltdb v0.0.0-20220329195025-15018e9b97e0
 	github.com/tikv/client-go/v2 v2.0.1
@@ -159,7 +159,7 @@ require (
 )

 require (
-	cloud.google.com/go/compute v1.6.1 // indirect
+	cloud.google.com/go/compute v1.7.0 // indirect
 	cloud.google.com/go/iam v0.3.0 // indirect
 	github.com/arangodb/go-velocypack v0.0.0-20200318135517-5af53c29c67e // indirect
 	github.com/armon/go-metrics v0.3.10 // indirect
@@ -184,6 +184,7 @@ require (
 	github.com/fatih/color v1.13.0 // indirect
 	github.com/fclairamb/go-log v0.3.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.1.0 // indirect
 	github.com/googleapis/go-type-adapters v1.0.0 // indirect
 	github.com/grpc-ecosystem/go-grpc-middleware v1.1.0 // indirect
 	github.com/hashicorp/go-hclog v1.2.0 // indirect
@@ -195,7 +196,7 @@ require (
 	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
 	github.com/jonboulle/clockwork v0.2.2 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
-	github.com/klauspost/cpuid/v2 v2.0.6 // indirect
+	github.com/klauspost/cpuid/v2 v2.0.14 // indirect
 	github.com/mattn/go-colorable v0.1.12 // indirect
 	github.com/mattn/go-runewidth v0.0.7 // indirect
 	github.com/mattn/go-sqlite3 v2.0.1+incompatible // indirect
@@ -32,8 +32,9 @@ cloud.google.com/go v0.97.0/go.mod h1:GF7l59pYBVlXQIBLx3a761cZ41F9bBH3JUlihCt2Ud
 cloud.google.com/go v0.99.0/go.mod h1:w0Xx2nLzqWJPuozYQX+hFfCSI8WioryfRDzkoI/Y2ZA=
 cloud.google.com/go v0.100.1/go.mod h1:fs4QogzfH5n2pBXBP9vRiU+eCny7lD2vmFZy79Iuw1U=
 cloud.google.com/go v0.100.2/go.mod h1:4Xra9TjzAeYHrl5+oeLlzbM2k3mjVhZh4UqTZ//w99A=
-cloud.google.com/go v0.102.0 h1:DAq3r8y4mDgyB/ZPJ9v/5VJNqjgJAxTn6ZYLlUywOu8=
 cloud.google.com/go v0.102.0/go.mod h1:oWcCzKlqJ5zgHQt9YsaeTY9KzIvjyy0ArmiBUgpQ+nc=
+cloud.google.com/go v0.102.1 h1:vpK6iQWv/2uUeFJth4/cBHsQAGjn1iIE6AAlxipRaA0=
+cloud.google.com/go v0.102.1/go.mod h1:XZ77E9qnTEnrgEOvr4xzfdX5TRo7fB4T2F4O6+34hIU=
 cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
 cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
 cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
@@ -45,8 +46,9 @@ cloud.google.com/go/compute v1.2.0/go.mod h1:xlogom/6gr8RJGBe7nT2eGsQYAFUbbv8dbC
 cloud.google.com/go/compute v1.3.0/go.mod h1:cCZiE1NHEtai4wiufUhW8I8S1JKkAnhnQJWM7YD99wM=
 cloud.google.com/go/compute v1.5.0/go.mod h1:9SMHyhJlzhlkJqrPAc839t2BZFTSk6Jdj6mkzQJeu0M=
 cloud.google.com/go/compute v1.6.0/go.mod h1:T29tfhtVbq1wvAPo0E3+7vhgmkOYeXjhFvz/FMzPu0s=
-cloud.google.com/go/compute v1.6.1 h1:2sMmt8prCn7DPaG4Pmh0N3Inmc8cT8ae5k1M6VJ9Wqc=
 cloud.google.com/go/compute v1.6.1/go.mod h1:g85FgpzFvNULZ+S8AYq87axRKuf2Kh7deLqV/jJ3thU=
+cloud.google.com/go/compute v1.7.0 h1:v/k9Eueb8aAJ0vZuxKMrgm6kPhCLZU9HxFU+AFDs9Uk=
+cloud.google.com/go/compute v1.7.0/go.mod h1:435lt8av5oL9P3fv1OEzSbSUe+ybHXGMPQHHZWZxy9U=
 cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
 cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
 cloud.google.com/go/firestore v1.6.1/go.mod h1:asNXNOzBdyVQmEU+ggO8UPodTkEVFW5Qx+rwHnAz+EY=
@@ -64,8 +66,8 @@ cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+
 cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
 cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
 cloud.google.com/go/pubsub v1.19.0/go.mod h1:/O9kmSe9bb9KRnIAWkzmqhPjHo6LtzGOBYd/kr06XSs=
-cloud.google.com/go/pubsub v1.22.2 h1:e6A4rhtMX4opff/jDWApl4HwLtsCdV9VULVfpFRp6eo=
-cloud.google.com/go/pubsub v1.22.2/go.mod h1:LBHGrtgM7+SGKCDKQu2pKIRtGwbZyJvRDkMk0594xdU=
+cloud.google.com/go/pubsub v1.23.1 h1:eVtkabVa+1M5ai67fGU+idws0hVb/KEPXiDmSS17+qc=
+cloud.google.com/go/pubsub v1.23.1/go.mod h1:ttM6nEGYK/2CnB36ndNySU3ZxPwpBk8cXM6+iOlxH9U=
 cloud.google.com/go/secretmanager v1.3.0/go.mod h1:+oLTkouyiYiabAQNugCeTS3PAArGiMJuBqvJnJsyH+U=
 cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
 cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
@@ -74,8 +76,9 @@ cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RX
 cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
 cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo=
 cloud.google.com/go/storage v1.21.0/go.mod h1:XmRlxkgPjlBONznT2dDUU/5XlpU2OjMnKuqnZI01LAA=
-cloud.google.com/go/storage v1.22.1 h1:F6IlQJZrZM++apn9V5/VfS3gbTUYg98PS3EMQAzqtfg=
 cloud.google.com/go/storage v1.22.1/go.mod h1:S8N1cAStu7BOeFfE8KAQzmyyLkK8p/vmRq6kuBTW58Y=
+cloud.google.com/go/storage v1.23.0 h1:wWRIaDURQA8xxHguFCshYepGlrWIrbBnAmc7wfg07qY=
+cloud.google.com/go/storage v1.23.0/go.mod h1:vOEEDNFnciUMhBeT6hsJIn3ieU5cFRmzeLgDvXzfIXc=
 cloud.google.com/go/trace v1.0.0/go.mod h1:4iErSByzxkyHWzzlAj63/Gmjz0NH1ASqhJguHpGcr6A=
 cloud.google.com/go/trace v1.2.0/go.mod h1:Wc8y/uYyOhPy12KEnXG9XGrvfMz5F5SrYecQlbW1rwM=
 contrib.go.opencensus.io/exporter/aws v0.0.0-20200617204711-c478e41e60e9/go.mod h1:uu1P0UCM/6RbsMrgPa98ll8ZcHM858i/AD06a9aLRCA=
@@ -154,8 +157,8 @@ github.com/armon/go-metrics v0.3.10/go.mod h1:4O98XIr/9W0sxpJ8UaYkvjk10Iff7SnFrb
 github.com/aws/aws-sdk-go v1.15.27/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0=
 github.com/aws/aws-sdk-go v1.37.0/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
 github.com/aws/aws-sdk-go v1.43.31/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
-github.com/aws/aws-sdk-go v1.44.37 h1:KvDxCX6dfJeEDC77U5GPGSP0ErecmNnhDHFxw+NIvlI=
-github.com/aws/aws-sdk-go v1.44.37/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
+github.com/aws/aws-sdk-go v1.44.47 h1:uyiNvoR4wfZ8Bp4ghgbyzGFIg5knjZMUAd5S9ba9qNU=
+github.com/aws/aws-sdk-go v1.44.47/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
 github.com/aws/aws-sdk-go-v2 v1.16.2 h1:fqlCk6Iy3bnCumtrLz9r3mJ/2gUT0pJ0wLFVIdWh+JA=
 github.com/aws/aws-sdk-go-v2 v1.16.2/go.mod h1:ytwTPBG6fXTZLxxeeCCWj2/EMYp/xDUgX+OET6TLNNU=
 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.1/go.mod h1:n8Bs1ElDD2wJ9kCRTczA83gYbBmjSwZp3umc6zF4EeM=
@@ -338,8 +341,8 @@ github.com/go-redis/redis/v7 v7.4.0/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRf
 github.com/go-redis/redis/v8 v8.11.4/go.mod h1:2Z2wHZXdQpCDXEGzqMockDpNyYvi2l4Pxt6RJr792+w=
 github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI=
 github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo=
-github.com/go-redsync/redsync/v4 v4.5.0 h1:kJjDzn/iEbU+K/6w+O8b1rzuYIK/nP9EQRc5nXKW9x4=
-github.com/go-redsync/redsync/v4 v4.5.0/go.mod h1:AfhgO1E6W3rlUTs6Zmz/B6qBZJFasV30lwo7nlizdDs=
+github.com/go-redsync/redsync/v4 v4.5.1 h1:T97UCaY8MfQg/6kB7MTuimF4tnLOCdJbsvIoN5KmjZE=
+github.com/go-redsync/redsync/v4 v4.5.1/go.mod h1:AfhgO1E6W3rlUTs6Zmz/B6qBZJFasV30lwo7nlizdDs=
 github.com/go-sql-driver/mysql v1.6.0 h1:BCTh4TKNUYmOmMUcQ3IipzF5prigylS7XXjEkfCHuOE=
 github.com/go-sql-driver/mysql v1.6.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
@@ -468,6 +471,9 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
 github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/wire v0.5.0 h1:I7ELFeVBr3yfPIcc8+MWvrjk+3VjbcSzoXm3JVa+jD8=
 github.com/google/wire v0.5.0/go.mod h1:ngWDr9Qvq3yZA10YrxfyGELY/AFWGVpy9c1LTRi1EoU=
+github.com/googleapis/enterprise-certificate-proxy v0.0.0-20220520183353-fd19c99a87aa/go.mod h1:17drOmN3MwGY7t0e+Ei9b45FFGA3fBs3x36SsCg1hq8=
+github.com/googleapis/enterprise-certificate-proxy v0.1.0 h1:zO8WHNx/MYiAKJ3d5spxZXZE6KHmIQGQcAzwUzV7qQw=
+github.com/googleapis/enterprise-certificate-proxy v0.1.0/go.mod h1:17drOmN3MwGY7t0e+Ei9b45FFGA3fBs3x36SsCg1hq8=
 github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
 github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
 github.com/googleapis/gax-go/v2 v2.1.0/go.mod h1:Q3nei7sK6ybPYH7twZdmQpAd1MKb7pfu6SK+H1/DsU0=
@@ -493,8 +499,8 @@ github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed h1:5upAirOpQc
 github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed/go.mod h1:tMWxXQ9wFIaZeTI9F+hmhFiGpFmhOHzyShyFUhRm0H4=
 github.com/hanwen/go-fuse v1.0.0/go.mod h1:unqXarDXqzAk0rt98O2tVndEPIpUgLD9+rwFisZH3Ok=
 github.com/hanwen/go-fuse/v2 v2.1.0/go.mod h1:oRyA5eK+pvJyv5otpO/DgccS8y/RvYMaO00GgRLGryc=
-github.com/hanwen/go-fuse/v2 v2.1.1-0.20220531082602-17a864ed5940 h1:fZ4Oe3ArCbIe5Fzxr5yrDtAipC619dKuGYnrVlv5Hck=
-github.com/hanwen/go-fuse/v2 v2.1.1-0.20220531082602-17a864ed5940/go.mod h1:B1nGE/6RBFyBRC1RRnf23UpwCdyJ31eukw34oAKukAc=
+github.com/hanwen/go-fuse/v2 v2.1.1-0.20220627082937-d01fda7edf17 h1:fOTuYWxywhaliwMobGTP/6NUCgGdal6pCpuch4MHCnU=
+github.com/hanwen/go-fuse/v2 v2.1.1-0.20220627082937-d01fda7edf17/go.mod h1:B1nGE/6RBFyBRC1RRnf23UpwCdyJ31eukw34oAKukAc=
 github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
 github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
 github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
@@ -620,10 +626,10 @@ github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47e
 github.com/klauspost/compress v1.15.1/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
 github.com/klauspost/compress v1.15.6 h1:6D9PcO8QWu0JyaQ2zUMmu16T1T+zjjEpP91guRsvDfY=
 github.com/klauspost/compress v1.15.6/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
-github.com/klauspost/cpuid/v2 v2.0.6 h1:dQ5ueTiftKxp0gyjKSx5+8BtPWkyQbd95m8Gys/RarI=
-github.com/klauspost/cpuid/v2 v2.0.6/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
-github.com/klauspost/reedsolomon v1.9.16 h1:mR0AwphBwqFv/I3B9AHtNKvzuowI1vrj8/3UX4XRmHA=
-github.com/klauspost/reedsolomon v1.9.16/go.mod h1:eqPAcE7xar5CIzcdfwydOEdcmchAKAP/qs14y4GCBOk=
+github.com/klauspost/cpuid/v2 v2.0.14 h1:QRqdp6bb9M9S5yyKeYteXKuoKE4p0tGlra81fKOpWH8=
+github.com/klauspost/cpuid/v2 v2.0.14/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
+github.com/klauspost/reedsolomon v1.10.0 h1:MonMtg979rxSHjwtsla5dZLhreS0Lu42AyQ20bhjIGg=
+github.com/klauspost/reedsolomon v1.10.0/go.mod h1:qHMIzMkuZUWqIh8mS/GruPdo3u0qwX2jk/LH440ON7Y=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
@@ -861,8 +867,8 @@ github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5
 github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/stretchr/testify v1.7.3 h1:dAm0YRdRQlWojc3CrCRgPBzG5f941d0zvAKu7qY4e+I=
-github.com/stretchr/testify v1.7.3/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
+github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
+github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203 h1:QVqDTf3h2WHt08YuiTGPZLls0Wq99X9bWd0Q5ZSBesM=
 github.com/stvp/tempredis v0.0.0-20181119212430-b82af8480203/go.mod h1:oqN97ltKNihBbwlX8dLpwxCl3+HnXKV/R0e+sRLd9C8=
 github.com/subosito/gotenv v1.3.0 h1:mjC+YW8QpAdXibNi+vNWgzmgBH4+5l5dCXv8cNysBLI=
@@ -1116,8 +1122,10 @@ golang.org/x/net v0.0.0-20220412020605-290c469a71a5/go.mod h1:CfG3xpIq0wQ8r1q4Su
 golang.org/x/net v0.0.0-20220420153159-1850ba15e1be/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
-golang.org/x/net v0.0.0-20220607020251-c690dde0001d h1:4SFsTMi4UahlKoloni7L4eYzhFRifURQLw+yv0QDCx8=
 golang.org/x/net v0.0.0-20220607020251-c690dde0001d/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/net v0.0.0-20220617184016-355a448f1bc9/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e h1:TsQ7F31D3bUCLeqPT0u+yjp1guoArKaNKmCr22PYgTQ=
+golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1139,8 +1147,9 @@ golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ
 golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/oauth2 v0.0.0-20220309155454-6242fa91716a/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
 golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
-golang.org/x/oauth2 v0.0.0-20220524215830-622c5d57e401 h1:zwrSfklXn0gxyLRX/aR+q6cgHbV/ItVyzbPlbA+dkAw=
-golang.org/x/oauth2 v0.0.0-20220524215830-622c5d57e401/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
+golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
+golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2 h1:+jnHzr9VPj32ykQVai5DNahi9+NSp7yYuCsl5eAQtL0=
+golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -1152,7 +1161,6 @@ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20220513210516-0976fa681c29/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f h1:Ax0t5p6N38Ga0dThY21weqDEyz2oklo4IvDkpigvkD8=
 golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -1249,8 +1257,11 @@ golang.org/x/sys v0.0.0-20220330033206-e17cdc41300f/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220502124256-b6088ccd6cba/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220610221304-9f5ed59c137d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220624220833-87e55d714810 h1:rHZQSjJdAI4Xf5Qzeh2bBc5YJIkPFVM6oDtMFYmgws0=
+golang.org/x/sys v0.0.0-20220624220833-87e55d714810/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -1269,8 +1280,8 @@ golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxb
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20211116232009-f0f3c7e86c11/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20220224211638-0e9765cccd65/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
-golang.org/x/time v0.0.0-20220411224347-583f2d630306 h1:+gHMid33q6pen7kv9xvT+JRinntgeXO2AeZVd0AWD3w=
-golang.org/x/time v0.0.0-20220411224347-583f2d630306/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/time v0.0.0-20220609170525-579cf78fd858 h1:Dpdu/EMxGMFgq0CeYMh4fazTD2vtlZRYE7wyynxJb9U=
+golang.org/x/time v0.0.0-20220609170525-579cf78fd858/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -1351,8 +1362,9 @@ golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8T
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df h1:5Pf6pFKu98ODmgnpvkJ3kFUOQGGLIzLIkbzUHp47618=
 golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
+golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f h1:uF6paiQQebLeSXkrTqHqz0MXhXXS1KgF41eUdBNvxK0=
+golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8=
 google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
 google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
@@ -1398,9 +1410,10 @@ google.golang.org/api v0.74.0/go.mod h1:ZpfMZOVRMywNyvJFeqL9HRWBgAuRfSjJFpe9QtRR
 google.golang.org/api v0.75.0/go.mod h1:pU9QmyHLnzlpar1Mjt4IbapUCy8J+6HD6GeELN69ljA=
 google.golang.org/api v0.78.0/go.mod h1:1Sg78yoMLOhlQTeF+ARBoytAcH1NNyyl390YMy6rKmw=
 google.golang.org/api v0.80.0/go.mod h1:xY3nI94gbvBrE0J6NHXhxOmW97HG7Khjkku6AFB3Hyg=
-google.golang.org/api v0.81.0/go.mod h1:FA6Mb/bZxj706H2j+j2d6mHEEaHBmbbWnkfvmorOCko=
-google.golang.org/api v0.83.0 h1:pMvST+6v+46Gabac4zlJlalxZjCeRcepwg2EdBU+nCc=
-google.golang.org/api v0.83.0/go.mod h1:CNywQoj/AfhTw26ZWAa6LwOv+6WFxHmeLPZq2uncLZk=
+google.golang.org/api v0.84.0/go.mod h1:NTsGnUFJMYROtiquksZHBWtHfeMC7iYthki7Eq3pa8o=
+google.golang.org/api v0.85.0/go.mod h1:AqZf8Ep9uZ2pyTvgL+x0D3Zt0eoT9b5E8fmzfu6FO2g=
+google.golang.org/api v0.86.0 h1:ZAnyOHQFIuWso1BodVfSaRyffD74T9ERGFa3k1fNk/U=
+google.golang.org/api v0.86.0/go.mod h1:+Sem1dnrKlrXMR/X0bPnMWyluQe4RsNoYfmNLhOIkzw=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -1503,10 +1516,12 @@ google.golang.org/genproto v0.0.0-20220421151946-72621c1f0bd3/go.mod h1:8w6bsBMX
 google.golang.org/genproto v0.0.0-20220429170224-98d788798c3e/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo=
 google.golang.org/genproto v0.0.0-20220505152158-f39f71e6c8f3/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4=
 google.golang.org/genproto v0.0.0-20220518221133-4f43b3371335/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4=
-google.golang.org/genproto v0.0.0-20220519153652-3a47de7e79bd/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4=
 google.golang.org/genproto v0.0.0-20220523171625-347a074981d8/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4=
-google.golang.org/genproto v0.0.0-20220602131408-e326c6e8e9c8 h1:qRu95HZ148xXw+XeZ3dvqe85PxH4X8+jIo0iRPKcEnM=
-google.golang.org/genproto v0.0.0-20220602131408-e326c6e8e9c8/go.mod h1:yKyY4AMRwFiC8yMMNaMi+RkCnjZJt9LoWuvhXjMs+To=
+google.golang.org/genproto v0.0.0-20220608133413-ed9918b62aac/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA=
+google.golang.org/genproto v0.0.0-20220616135557-88e70c0c3a90/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA=
+google.golang.org/genproto v0.0.0-20220617124728-180714bec0ad/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA=
+google.golang.org/genproto v0.0.0-20220624142145-8cd45d7dbd1f h1:hJ/Y5SqPXbarffmAsApliUlcvMU+wScNGfyop4bZm8o=
+google.golang.org/genproto v0.0.0-20220624142145-8cd45d7dbd1f/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
 google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
@@ -1626,8 +1641,8 @@ modernc.org/memory v1.1.1 h1:bDOL0DIDLQv7bWhP3gMvIrnoFw+Eo6F7a2QK9HPDiFU=
 modernc.org/memory v1.1.1/go.mod h1:/0wo5ibyrQiaoUoH7f9D8dnglAmILJ5/cxZlRECf+Nw=
 modernc.org/opt v0.1.1 h1:/0RX92k9vwVeDXj+Xn23DKp2VJubL7k8qNffND6qn3A=
 modernc.org/opt v0.1.1/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
-modernc.org/sqlite v1.17.2 h1:TjmF36Wi5QcPYqRoAacV1cAyJ7xB/CD0ExpVUEMebnw=
-modernc.org/sqlite v1.17.2/go.mod h1:GOQmuiXd6pTTes1Fi2s9apiCcD/wbKQtBZ0Nw6/etjM=
+modernc.org/sqlite v1.17.3 h1:iE+coC5g17LtByDYDWKpR6m2Z9022YrSh3bumwOnIrI=
+modernc.org/sqlite v1.17.3/go.mod h1:10hPVYar9C0kfXuTWGz8s0XtB8uAGymUy51ZzStYe3k=
 modernc.org/strutil v1.1.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
 modernc.org/strutil v1.1.1/go.mod h1:DE+MQQ/hjKBZS2zNInV5hhcipt5rLPWkmpbGeW5mmdw=
 modernc.org/strutil v1.1.2 h1:iFBDH6j1Z0bN/Q9udJnnFoFpENA4252qe/7/5woE5MI=
@@ -1,5 +1,5 @@
 apiVersion: v1
 description: SeaweedFS
 name: seaweedfs
-appVersion: "3.13"
-version: "3.13"
+appVersion: "3.14"
+version: "3.14"
@@ -5,7 +5,7 @@

    <groupId>com.github.chrislusf</groupId>
    <artifactId>seaweedfs-client</artifactId>
-    <version>2.85</version>
+    <version>3.13</version>

    <parent>
        <groupId>org.sonatype.oss</groupId>
@@ -5,7 +5,7 @@

    <groupId>com.github.chrislusf</groupId>
    <artifactId>seaweedfs-client</artifactId>
-    <version>2.85</version>
+    <version>3.13</version>

    <parent>
        <groupId>org.sonatype.oss</groupId>
@@ -5,7 +5,7 @@

    <groupId>com.github.chrislusf</groupId>
    <artifactId>seaweedfs-client</artifactId>
-    <version>2.85</version>
+    <version>3.13</version>

    <parent>
        <groupId>org.sonatype.oss</groupId>
@@ -75,7 +75,6 @@ public class SeaweedOutputStream extends OutputStream {
                    .setIsDirectory(false)
                    .setAttributes(FilerProto.FuseAttributes.newBuilder()
                            .setFileMode(0755)
-                            .setReplication(replication)
                            .setCrtime(now)
                            .setMtime(now)
                            .clearGroupName()
@@ -11,13 +11,13 @@
        <dependency>
            <groupId>com.github.chrislusf</groupId>
            <artifactId>seaweedfs-client</artifactId>
-            <version>2.85</version>
+            <version>3.13</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>com.github.chrislusf</groupId>
            <artifactId>seaweedfs-hadoop2-client</artifactId>
-            <version>2.85</version>
+            <version>3.13</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
@@ -124,7 +124,7 @@
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
-      <version>2.9.2</version>
+      <version>2.10.1</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
@@ -156,7 +156,7 @@
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
-      <version>2.9.2</version>
+      <version>2.10.1</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
@@ -231,6 +231,14 @@
          <artifactId>commons-configuration</artifactId>
          <groupId>commons-configuration</groupId>
        </exclusion>
+        <exclusion>
+          <artifactId>commons-digester</artifactId>
+          <groupId>commons-digester</groupId>
+        </exclusion>
+        <exclusion>
+          <artifactId>commons-beanutils</artifactId>
+          <groupId>commons-beanutils</groupId>
+        </exclusion>
        <exclusion>
          <artifactId>commons-lang3</artifactId>
          <groupId>org.apache.commons</groupId>
@@ -301,7 +309,7 @@
    </snapshotRepository>
  </distributionManagement>
  <properties>
-    <seaweedfs.client.version>2.85</seaweedfs.client.version>
-    <hadoop.version>2.9.2</hadoop.version>
+    <seaweedfs.client.version>3.13</seaweedfs.client.version>
+    <hadoop.version>2.10.1</hadoop.version>
  </properties>
 </project>
@@ -5,7 +5,7 @@
    <modelVersion>4.0.0</modelVersion>

    <properties>
-        <seaweedfs.client.version>2.85</seaweedfs.client.version>
+        <seaweedfs.client.version>3.13</seaweedfs.client.version>
        <hadoop.version>2.10.1</hadoop.version>
    </properties>

@@ -200,7 +200,6 @@ public class SeaweedFileSystemStore {
                entry.getAttributesBuilder().setMtime(now);
                LOG.debug("createFile merged entry path:{} entry:{} from:{}", path, entry, existingEntry);
                writePosition = SeaweedRead.fileSize(existingEntry);
-                replication = existingEntry.getAttributes().getReplication();
            }
        }
        if (entry == null) {
@@ -209,7 +208,6 @@ public class SeaweedFileSystemStore {
                .setIsDirectory(false)
                .setAttributes(FilerProto.FuseAttributes.newBuilder()
                    .setFileMode(permissionToMode(permission, false))
-                    .setReplication(replication)
                    .setCrtime(now)
                    .setMtime(now)
                    .setUserName(userGroupInformation.getUserName())
@@ -124,7 +124,7 @@
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
-      <version>3.1.1</version>
+      <version>3.2.3</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
@@ -156,7 +156,7 @@
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
-      <version>3.1.1</version>
+      <version>3.2.3</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
@@ -183,6 +183,10 @@
          <artifactId>javax.servlet-api</artifactId>
          <groupId>javax.servlet</groupId>
        </exclusion>
+        <exclusion>
+          <artifactId>javax.activation-api</artifactId>
+          <groupId>javax.activation</groupId>
+        </exclusion>
        <exclusion>
          <artifactId>jetty-server</artifactId>
          <groupId>org.eclipse.jetty</groupId>
@@ -223,10 +227,6 @@
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
-        <exclusion>
-          <artifactId>commons-lang</artifactId>
-          <groupId>commons-lang</groupId>
-        </exclusion>
        <exclusion>
          <artifactId>commons-beanutils</artifactId>
          <groupId>commons-beanutils</groupId>
@@ -239,6 +239,10 @@
          <artifactId>commons-lang3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
+        <exclusion>
+          <artifactId>commons-text</artifactId>
+          <groupId>org.apache.commons</groupId>
+        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
@@ -295,6 +299,10 @@
          <artifactId>woodstox-core</artifactId>
          <groupId>com.fasterxml.woodstox</groupId>
        </exclusion>
+        <exclusion>
+          <artifactId>dnsjava</artifactId>
+          <groupId>dnsjava</groupId>
+        </exclusion>
        <exclusion>
          <artifactId>hadoop-annotations</artifactId>
          <groupId>org.apache.hadoop</groupId>
@@ -309,7 +317,7 @@
    </snapshotRepository>
  </distributionManagement>
  <properties>
-    <seaweedfs.client.version>2.85</seaweedfs.client.version>
-    <hadoop.version>3.1.1</hadoop.version>
+    <seaweedfs.client.version>3.13</seaweedfs.client.version>
+    <hadoop.version>3.2.3</hadoop.version>
  </properties>
 </project>
@@ -5,7 +5,7 @@
    <modelVersion>4.0.0</modelVersion>

    <properties>
-        <seaweedfs.client.version>2.85</seaweedfs.client.version>
+        <seaweedfs.client.version>3.13</seaweedfs.client.version>
        <hadoop.version>3.2.3</hadoop.version>
    </properties>

@@ -200,7 +200,6 @@ public class SeaweedFileSystemStore {
                entry.getAttributesBuilder().setMtime(now);
                LOG.debug("createFile merged entry path:{} entry:{} from:{}", path, entry, existingEntry);
                writePosition = SeaweedRead.fileSize(existingEntry);
-                replication = existingEntry.getAttributes().getReplication();
            }
        }
        if (entry == null) {
@@ -209,7 +208,6 @@ public class SeaweedFileSystemStore {
                .setIsDirectory(false)
                .setAttributes(FilerProto.FuseAttributes.newBuilder()
                    .setFileMode(permissionToMode(permission, false))
-                    .setReplication(replication)
                    .setCrtime(now)
                    .setMtime(now)
                    .setUserName(userGroupInformation.getUserName())
@@ -46,8 +46,6 @@ func NewCluster() *Cluster {
 }

 func (cluster *Cluster) getFilers(filerGroup FilerGroup, createIfNotFound bool) *Filers {
-	cluster.filersLock.Lock()
-	defer cluster.filersLock.Unlock()
 	filers, found := cluster.filerGroup2filers[filerGroup]
 	if !found && createIfNotFound {
 		filers = &Filers{
@@ -63,6 +61,8 @@ func (cluster *Cluster) AddClusterNode(ns, nodeType string, address pb.ServerAdd
 	filerGroup := FilerGroup(ns)
 	switch nodeType {
 	case FilerType:
+		cluster.filersLock.Lock()
+		defer cluster.filersLock.Unlock()
 		filers := cluster.getFilers(filerGroup, true)
 		if existingNode, found := filers.filers[address]; found {
 			existingNode.counter++
@@ -115,6 +115,8 @@ func (cluster *Cluster) RemoveClusterNode(ns string, nodeType string, address pb
 	filerGroup := FilerGroup(ns)
 	switch nodeType {
 	case FilerType:
+		cluster.filersLock.Lock()
+		defer cluster.filersLock.Unlock()
 		filers := cluster.getFilers(filerGroup, false)
 		if filers == nil {
 			return nil
@@ -165,12 +167,12 @@ func (cluster *Cluster) RemoveClusterNode(ns string, nodeType string, address pb
 func (cluster *Cluster) ListClusterNode(filerGroup FilerGroup, nodeType string) (nodes []*ClusterNode) {
 	switch nodeType {
 	case FilerType:
+		cluster.filersLock.RLock()
+		defer cluster.filersLock.RUnlock()
 		filers := cluster.getFilers(filerGroup, false)
 		if filers == nil {
 			return
 		}
-		cluster.filersLock.RLock()
-		defer cluster.filersLock.RUnlock()
 		for _, node := range filers.filers {
 			nodes = append(nodes, node)
 		}
@@ -3,6 +3,8 @@ package cluster
 import (
 	"github.com/chrislusf/seaweedfs/weed/pb"
 	"github.com/stretchr/testify/assert"
+	"strconv"
+	"sync"
 	"testing"
 )

@@ -45,3 +47,35 @@ func TestClusterAddRemoveNodes(t *testing.T) {
 	c.RemoveClusterNode("", "filer", pb.ServerAddress("111:1"))

 }
+
+func TestConcurrentAddRemoveNodes(t *testing.T) {
+	c := NewCluster()
+	var wg sync.WaitGroup
+	for i := 0; i < 50; i++ {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			address := strconv.Itoa(i)
+			c.AddClusterNode("", "filer", pb.ServerAddress(address), "23.45")
+		}(i)
+	}
+	wg.Wait()
+
+	for i := 0; i < 50; i++ {
+		wg.Add(1)
+		go func(i int) {
+			defer wg.Done()
+			address := strconv.Itoa(i)
+			node := c.RemoveClusterNode("", "filer", pb.ServerAddress(address))
+
+			if len(node) == 0 {
+				t.Errorf("TestConcurrentAddRemoveNodes: node[%s] not found", address)
+				return
+			} else if node[0].ClusterNodeUpdate.Address != address {
+				t.Errorf("TestConcurrentAddRemoveNodes: expect:%s, actual:%s", address, node[0].ClusterNodeUpdate.Address)
+				return
+			}
+		}(i)
+	}
+	wg.Wait()
+}
@@ -6,10 +6,13 @@ import (
 	"net/http"
 	"os"
 	"runtime"
+	"sort"
+	"strings"
 	"time"

 	"google.golang.org/grpc/reflection"

+	"github.com/chrislusf/seaweedfs/weed/filer"
 	"github.com/chrislusf/seaweedfs/weed/glog"
 	"github.com/chrislusf/seaweedfs/weed/pb"
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
@@ -114,10 +117,8 @@ func init() {
 	filerIamOptions.port = cmdFiler.Flag.Int("iam.port", 8111, "iam server http listen port")
 }

-var cmdFiler = &Command{
-	UsageLine: "filer -port=8888 -master=<ip:port>[,<ip:port>]*",
-	Short:     "start a file server that points to a master server, or a list of master servers",
-	Long: `start a file server which accepts REST operation for any files.
+func filerLongDesc() string {
+	desc := `start a file server which accepts REST operation for any files.

 	//create or overwrite the file, the directories /path/to will be automatically created
 	POST /path/to/file
@@ -133,7 +134,22 @@ var cmdFiler = &Command{

 	The example filer.toml configuration file can be generated by "weed scaffold -config=filer"

-`,
+Supported Filer Stores:
+`
+
+	storeNames := make([]string, len(filer.Stores))
+	for i, store := range filer.Stores {
+		storeNames[i] = "\t" + store.GetName()
+	}
+	sort.Strings(storeNames)
+	storeList := strings.Join(storeNames, "\n")
+	return desc + storeList
+}
+
+var cmdFiler = &Command{
+	UsageLine: "filer -port=8888 -master=<ip:port>[,<ip:port>]*",
+	Short:     "start a file server that points to a master server, or a list of master servers",
+	Long:      filerLongDesc(),
 }

 func runFiler(cmd *Command, args []string) bool {
@@ -9,6 +9,7 @@ import (
 	"github.com/chrislusf/seaweedfs/weed/pb/remote_pb"
 	"github.com/chrislusf/seaweedfs/weed/remote_storage"
 	"github.com/chrislusf/seaweedfs/weed/replication/source"
+	"github.com/chrislusf/seaweedfs/weed/s3api/s3_constants"
 	"github.com/chrislusf/seaweedfs/weed/util"
 	"github.com/golang/protobuf/proto"
 	"math"
@@ -181,7 +182,7 @@ func (option *RemoteGatewayOptions) makeBucketedEventProcessor(filerSource *sour
 			if message.NewParentPath == option.bucketsDir {
 				return handleCreateBucket(message.NewEntry)
 			}
-			if strings.HasPrefix(message.NewParentPath, option.bucketsDir) && strings.Contains(message.NewParentPath, "/.uploads/") {
+			if strings.HasPrefix(message.NewParentPath, option.bucketsDir) && strings.Contains(message.NewParentPath, "/"+s3_constants.MultipartUploadsFolder+"/") {
 				return nil
 			}
 			if !filer.HasData(message.NewEntry) {
@@ -215,10 +215,10 @@ func doSubscribeFilerMetaChanges(clientId int32, grpcDialOption grpc.DialOption,
 		return persistEventFn(resp)
 	}

-	var lastLogTsNs = time.Now().Nanosecond()
+	var lastLogTsNs = time.Now().UnixNano()
 	var clientName = fmt.Sprintf("syncFrom_%s_To_%s", string(sourceFiler), string(targetFiler))
 	processEventFnWithOffset := pb.AddOffsetFunc(processEventFn, 3*time.Second, func(counter int64, lastTsNs int64) error {
-		now := time.Now().Nanosecond()
+		now := time.Now().UnixNano()
 		glog.V(0).Infof("sync %s to %s progressed to %v %0.2f/sec", sourceFiler, targetFiler, time.Unix(0, lastTsNs), float64(counter)/(float64(now-lastLogTsNs)/1e9))
 		lastLogTsNs = now
 		// collect synchronous offset
@@ -3,8 +3,6 @@ default = "c1"

 [cluster.c1]
 master = "localhost:9333"    # comma-separated master servers
-filer = "localhost:8888"     # filer host and port

 [cluster.c2]
 master = ""
-filer = ""
@@ -1,5 +1,5 @@
-//go:build elastic && ydb && gocdk && hdfs
-// +build elastic,ydb,gocdk,hdfs
+//go:build elastic && ydb && gocdk && tikv
+// +build elastic,ydb,gocdk,tikv

 package command

@@ -3,11 +3,9 @@ package filer
 import (
 	"bytes"
 	"fmt"
-	"math"
-	"sync"
-
 	"github.com/chrislusf/seaweedfs/weed/wdclient"
 	"golang.org/x/exp/slices"
+	"math"

 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/util"
@@ -54,11 +52,11 @@ func ETagChunks(chunks []*filer_pb.FileChunk) (etag string) {
 	if len(chunks) == 1 {
 		return fmt.Sprintf("%x", util.Base64Md5ToBytes(chunks[0].ETag))
 	}
-	md5_digests := [][]byte{}
+	var md5Digests [][]byte
 	for _, c := range chunks {
-		md5_digests = append(md5_digests, util.Base64Md5ToBytes(c.ETag))
+		md5Digests = append(md5Digests, util.Base64Md5ToBytes(c.ETag))
 	}
-	return fmt.Sprintf("%x-%d", util.Md5(bytes.Join(md5_digests, nil)), len(chunks))
+	return fmt.Sprintf("%x-%d", util.Md5(bytes.Join(md5Digests, nil)), len(chunks))
 }

 func CompactFileChunks(lookupFileIdFn wdclient.LookupFileIdFunctionType, chunks []*filer_pb.FileChunk) (compacted, garbage []*filer_pb.FileChunk) {
@@ -189,12 +187,6 @@ func logPrintf(name string, visibles []VisibleInterval) {
 	*/
 }

-var bufPool = sync.Pool{
-	New: func() interface{} {
-		return new(VisibleInterval)
-	},
-}
-
 func MergeIntoVisibles(visibles []VisibleInterval, chunk *filer_pb.FileChunk) (newVisibles []VisibleInterval) {

 	newV := newVisibleInterval(chunk.Offset, chunk.Offset+int64(chunk.Size), chunk.GetFileIdString(), chunk.Mtime, 0, chunk.Size, chunk.CipherKey, chunk.IsCompressed)
@@ -100,7 +100,8 @@ func (ma *MetaAggregator) loopSubscribeToOnefiler(f *Filer, self pb.ServerAddres
 		}
 		if err != nil {
 			glog.V(0).Infof("subscribing remote %s meta change: %v", peer, err)
-		} else if lastTsNs < nextLastTsNs {
+		}
+		if lastTsNs < nextLastTsNs {
 			lastTsNs = nextLastTsNs
 		}
 		time.Sleep(1733 * time.Millisecond)
@@ -210,10 +211,9 @@ func (ma *MetaAggregator) doSubscribeToOneFiler(f *Filer, self pb.ServerAddress,
 			if err := processEventFn(resp); err != nil {
 				return fmt.Errorf("process %v: %v", resp, err)
 			}
-			lastTsNs = resp.TsNs

 			f.onMetadataChangeEvent(resp)
-
+			lastTsNs = resp.TsNs
 		}
 	})
 	return lastTsNs, err
@@ -164,6 +164,10 @@ func (c *ChunkReadAt) doReadAt(p []byte, offset int64) (n int, err error) {
 func (c *ChunkReadAt) readChunkSliceAt(buffer []byte, chunkView *ChunkView, nextChunkViews []*ChunkView, offset uint64) (n int, err error) {

 	if c.readerPattern.IsRandomMode() {
+		n, err := c.readerCache.chunkCache.ReadChunkAt(buffer, chunkView.FileId, offset)
+		if n > 0 {
+			return n, err
+		}
 		return fetchChunkRange(buffer, c.readerCache.lookupFileIdFn, chunkView.FileId, chunkView.CipherKey, chunkView.IsGzipped, int64(offset))
 	}

@@ -6,7 +6,6 @@ import (
 	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"github.com/chrislusf/seaweedfs/weed/util"
 	"golang.org/x/exp/slices"
-	"io"
 	"sync"
 )

@@ -24,7 +23,7 @@ type FileHandle struct {
 	dirtyMetadata  bool
 	dirtyPages     *PageWriter
 	entryViewCache []filer.VisibleInterval
-	reader         io.ReaderAt
+	reader         *filer.ChunkReadAt
 	contentType    string
 	handle         uint64
 	sync.Mutex
@@ -99,8 +98,16 @@ func (fh *FileHandle) AddChunks(chunks []*filer_pb.FileChunk) {
 	fh.entryViewCache = nil
 }

+func (fh *FileHandle) SetReader(reader *filer.ChunkReadAt) {
+	if fh.reader != nil {
+		fh.reader.Close()
+	}
+	fh.reader = reader
+}
+
 func (fh *FileHandle) Release() {
 	fh.dirtyPages.Destroy()
+	fh.SetReader(nil)
 }

 func lessThan(a, b *filer_pb.FileChunk) bool {
@@ -62,7 +62,7 @@ func (fh *FileHandle) readFromChunks(buff []byte, offset int64) (int64, error) {
 		if chunkResolveErr != nil {
 			return 0, fmt.Errorf("fail to resolve chunk manifest: %v", chunkResolveErr)
 		}
-		fh.reader = nil
+		fh.SetReader(nil)
 	}

 	reader := fh.reader
@@ -74,7 +74,7 @@ func (fh *FileHandle) readFromChunks(buff []byte, offset int64) (int64, error) {
 		}
 		reader = filer.NewChunkReaderAtFromClient(fh.wfs.LookupFn(), chunkViews, fh.wfs.chunkCache, fileSize)
 	}
-	fh.reader = reader
+	fh.SetReader(reader)

 	totalRead, err := reader.ReadAt(buff, offset)

@@ -46,9 +46,9 @@ func NewGrpcServer(opts ...grpc.ServerOption) *grpc.Server {
 	var options []grpc.ServerOption
 	options = append(options,
 		grpc.KeepaliveParams(keepalive.ServerParameters{
-			Time:             10 * time.Second, // wait time before ping if no activity
-			Timeout:          20 * time.Second, // ping timeout
-			MaxConnectionAge: 10 * time.Hour,
+			Time:    10 * time.Second, // wait time before ping if no activity
+			Timeout: 20 * time.Second, // ping timeout
+			// MaxConnectionAge: 10 * time.Hour,
 		}),
 		grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
 			MinTime:             60 * time.Second, // min time a client should wait before sending a ping
@@ -7,6 +7,7 @@ import (
 	"github.com/chrislusf/seaweedfs/weed/replication/repl_util"
 	"github.com/chrislusf/seaweedfs/weed/replication/sink"
 	"github.com/chrislusf/seaweedfs/weed/replication/source"
+	"github.com/chrislusf/seaweedfs/weed/s3api/s3_constants"
 	"github.com/chrislusf/seaweedfs/weed/util"
 	"os"
 	"path/filepath"
@@ -32,7 +33,7 @@ func (localsink *LocalSink) GetName() string {
 }

 func (localsink *LocalSink) isMultiPartEntry(key string) bool {
-	return strings.HasSuffix(key, ".part") && strings.Contains(key, "/.uploads/")
+	return strings.HasSuffix(key, ".part") && strings.Contains(key, "/"+s3_constants.MultipartUploadsFolder+"/")
 }

 func (localsink *LocalSink) initialize(dir string, isIncremental bool) error {
@@ -46,17 +46,17 @@ func QueueDeclareAndBind(conn *amqp.Connection, exchangeUrl string, queueUrl str
 	}
 	defer ch.Close()
 	if err := ch.ExchangeDeclare(
-		exchangeNameDLX, "fanout", false, false, false, false, nil); err != nil {
+		exchangeNameDLX, "fanout", true, false, false, false, nil); err != nil {
 		glog.Error(err)
 		return err
 	}
 	if err := ch.ExchangeDeclare(
-		exchangeName, "fanout", false, false, false, false, nil); err != nil {
+		exchangeName, "fanout", true, false, false, false, nil); err != nil {
 		glog.Error(err)
 		return err
 	}
 	if _, err := ch.QueueDeclare(
-		queueName, false, false, false, false,
+		queueName, true, false, false, false,
 		amqp.Table{"x-dead-letter-exchange": exchangeNameDLX}); err != nil {
 		glog.Error(err)
 		return err
@@ -66,7 +66,7 @@ func QueueDeclareAndBind(conn *amqp.Connection, exchangeUrl string, queueUrl str
 		return err
 	}
 	if _, err := ch.QueueDeclare(
-		queueNameDLX, false, false, false, false,
+		queueNameDLX, true, false, false, false,
 		amqp.Table{"x-dead-letter-exchange": exchangeName, "x-message-ttl": 600000}); err != nil {
 		glog.Error(err)
 		return err
@@ -8,4 +8,5 @@ const (
 	ACTION_LIST    = "List"

 	SeaweedStorageDestinationHeader = "x-seaweedfs-destination"
+	MultipartUploadsFolder          = ".uploads"
 )
@@ -150,12 +150,14 @@ func (s3a *S3ApiServer) DeleteBucketHandler(w http.ResponseWriter, r *http.Reque

 	err := s3a.WithFilerClient(false, func(client filer_pb.SeaweedFilerClient) error {
 		if !s3a.option.AllowDeleteBucketNotEmpty {
-			entries, _, err := s3a.list(s3a.option.BucketsPath+"/"+bucket, "", "", false, 1)
+			entries, _, err := s3a.list(s3a.option.BucketsPath+"/"+bucket, "", "", false, 2)
 			if err != nil {
 				return fmt.Errorf("failed to list bucket %s: %v", bucket, err)
 			}
-			if len(entries) > 0 {
-				return errors.New(s3err.GetAPIError(s3err.ErrBucketNotEmpty).Code)
+			for _, entry := range entries {
+				if entry.Name != s3_constants.MultipartUploadsFolder {
+					return errors.New(s3err.GetAPIError(s3err.ErrBucketNotEmpty).Code)
+				}
 			}
 		}

@@ -37,7 +37,7 @@ func NewCircuitBreaker(option *S3ApiServerOption) *CircuitBreaker {
 	})

 	if err != nil {
-		glog.Warningf("fail to load config: %v", err)
+		glog.Infof("s3 circuit breaker not configured: %v", err)
 	}

 	return cb
@@ -45,7 +45,12 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
 			s3err.WriteErrorResponse(w, r, s3err.ErrInvalidCopySource)
 			return
 		}
-		entry.Extended = processMetadataBytes(r.Header, entry.Extended, replaceMeta, replaceTagging)
+		entry.Extended, err = processMetadataBytes(r.Header, entry.Extended, replaceMeta, replaceTagging)
+		if err != nil {
+			glog.Errorf("CopyObjectHandler ValidateTags error %s: %v", r.URL, err)
+			s3err.WriteErrorResponse(w, r, s3err.ErrInvalidTag)
+			return
+		}
 		err = s3a.touch(dir, name, entry)
 		if err != nil {
 			s3err.WriteErrorResponse(w, r, s3err.ErrInvalidCopySource)
@@ -75,8 +80,8 @@ func (s3a *S3ApiServer) CopyObjectHandler(w http.ResponseWriter, r *http.Request
 		return
 	}

-	dstUrl := fmt.Sprintf("http://%s%s/%s%s?collection=%s",
-		s3a.option.Filer.ToHttpAddress(), s3a.option.BucketsPath, dstBucket, urlPathEscape(dstObject), dstBucket)
+	dstUrl := fmt.Sprintf("http://%s%s/%s%s",
+		s3a.option.Filer.ToHttpAddress(), s3a.option.BucketsPath, dstBucket, urlPathEscape(dstObject))
 	srcUrl := fmt.Sprintf("http://%s%s/%s%s",
 		s3a.option.Filer.ToHttpAddress(), s3a.option.BucketsPath, srcBucket, urlPathEscape(srcObject))

@@ -164,8 +169,8 @@ func (s3a *S3ApiServer) CopyObjectPartHandler(w http.ResponseWriter, r *http.Req

 	rangeHeader := r.Header.Get("x-amz-copy-source-range")

-	dstUrl := fmt.Sprintf("http://%s%s/%s/%04d.part?collection=%s",
-		s3a.option.Filer.ToHttpAddress(), s3a.genUploadsFolder(dstBucket), uploadID, partID, dstBucket)
+	dstUrl := fmt.Sprintf("http://%s%s/%s/%04d.part",
+		s3a.option.Filer.ToHttpAddress(), s3a.genUploadsFolder(dstBucket), uploadID, partID)
 	srcUrl := fmt.Sprintf("http://%s%s/%s%s",
 		s3a.option.Filer.ToHttpAddress(), s3a.option.BucketsPath, srcBucket, urlPathEscape(srcObject))

@@ -252,7 +257,7 @@ func processMetadata(reqHeader, existing http.Header, replaceMeta, replaceTaggin
 	return
 }

-func processMetadataBytes(reqHeader http.Header, existing map[string][]byte, replaceMeta, replaceTagging bool) (metadata map[string][]byte) {
+func processMetadataBytes(reqHeader http.Header, existing map[string][]byte, replaceMeta, replaceTagging bool) (metadata map[string][]byte, err error) {
 	metadata = make(map[string][]byte)

 	if sc := existing[s3_constants.AmzStorageClass]; len(sc) > 0 {
@@ -277,16 +282,18 @@ func processMetadataBytes(reqHeader http.Header, existing map[string][]byte, rep
 			}
 		}
 	}
-
 	if replaceTagging {
 		if tags := reqHeader.Get(s3_constants.AmzObjectTagging); tags != "" {
-			for _, v := range strings.Split(tags, "&") {
-				tag := strings.Split(v, "=")
-				if len(tag) == 2 {
-					metadata[s3_constants.AmzObjectTagging+"-"+tag[0]] = []byte(tag[1])
-				} else if len(tag) == 1 {
-					metadata[s3_constants.AmzObjectTagging+"-"+tag[0]] = nil
-				}
+			parsedTags, err := parseTagsHeader(tags)
+			if err != nil {
+				return nil, err
+			}
+			err = ValidateTags(parsedTags)
+			if err != nil {
+				return nil, err
+			}
+			for k, v := range parsedTags {
+				metadata[s3_constants.AmzObjectTagging+"-"+k] = []byte(v)
 			}
 		}
 	} else {
@@ -332,6 +332,19 @@ var processMetadataBytesTestCases = []struct {
 			"X-Amz-Tagging-type": "request",
 		},
 	},
+
+	{
+		108,
+		H{
+			"User-Agent":                           "firefox",
+			"X-Amz-Meta-My-Meta":                   "request",
+			"X-Amz-Tagging":                        "A=B&a=b&type=request*",
+			s3_constants.AmzUserMetaDirective:      DirectiveReplace,
+			s3_constants.AmzObjectTaggingDirective: DirectiveReplace,
+		},
+		H{},
+		H{},
+	},
 }

 func TestProcessMetadata(t *testing.T) {
@@ -339,7 +352,6 @@ func TestProcessMetadata(t *testing.T) {
 		reqHeader := transferHToHeader(tc.request)
 		existing := transferHToHeader(tc.existing)
 		replaceMeta, replaceTagging := replaceDirective(reqHeader)
-
 		err := processMetadata(reqHeader, existing, replaceMeta, replaceTagging, func(_ string, _ string) (tags map[string]string, err error) {
 			return tc.getTags, nil
 		}, "", "")
@@ -367,7 +379,7 @@ func TestProcessMetadataBytes(t *testing.T) {
 		reqHeader := transferHToHeader(tc.request)
 		existing := transferHToBytesArr(tc.existing)
 		replaceMeta, replaceTagging := replaceDirective(reqHeader)
-		extends := processMetadataBytes(reqHeader, existing, replaceMeta, replaceTagging)
+		extends, _ := processMetadataBytes(reqHeader, existing, replaceMeta, replaceTagging)

 		result := transferBytesArrToH(extends)
 		fmtTagging(result, tc.want)
@@ -247,8 +247,8 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ

 	glog.V(2).Infof("PutObjectPartHandler %s %s %04d", bucket, uploadID, partID)

-	uploadUrl := fmt.Sprintf("http://%s%s/%s/%04d.part?collection=%s",
-		s3a.option.Filer.ToHttpAddress(), s3a.genUploadsFolder(bucket), uploadID, partID, bucket)
+	uploadUrl := fmt.Sprintf("http://%s%s/%s/%04d.part",
+		s3a.option.Filer.ToHttpAddress(), s3a.genUploadsFolder(bucket), uploadID, partID)

 	if partID == 1 && r.Header.Get("Content-Type") == "" {
 		dataReader = mimeDetect(r, dataReader)
@@ -268,7 +268,7 @@ func (s3a *S3ApiServer) PutObjectPartHandler(w http.ResponseWriter, r *http.Requ
 }

 func (s3a *S3ApiServer) genUploadsFolder(bucket string) string {
-	return fmt.Sprintf("%s/%s/.uploads", s3a.option.BucketsPath, bucket)
+	return fmt.Sprintf("%s/%s/%s", s3a.option.BucketsPath, bucket, s3_constants.MultipartUploadsFolder)
 }

 // Generate uploadID hash string from object
@@ -62,23 +62,12 @@ func (s3a *S3ApiServer) PutObjectTaggingHandler(w http.ResponseWriter, r *http.R
 		return
 	}
 	tags := tagging.ToTags()
-	if len(tags) > 10 {
-		glog.Errorf("PutObjectTaggingHandler tags %s: %d tags more than 10", r.URL, len(tags))
+	err = ValidateTags(tags)
+	if err != nil {
+		glog.Errorf("PutObjectTaggingHandler ValidateTags error %s: %v", r.URL, err)
 		s3err.WriteErrorResponse(w, r, s3err.ErrInvalidTag)
 		return
 	}
-	for k, v := range tags {
-		if len(k) > 128 {
-			glog.Errorf("PutObjectTaggingHandler tags %s: tag key %s longer than 128", r.URL, k)
-			s3err.WriteErrorResponse(w, r, s3err.ErrInvalidTag)
-			return
-		}
-		if len(v) > 256 {
-			glog.Errorf("PutObjectTaggingHandler tags %s: tag value %s longer than 256", r.URL, v)
-			s3err.WriteErrorResponse(w, r, s3err.ErrInvalidTag)
-			return
-		}
-	}

 	if err = s3a.setTags(dir, name, tagging.ToTags()); err != nil {
 		if err == filer_pb.ErrNotFound {
@@ -254,20 +254,23 @@ func (s3a *S3ApiServer) doListFilerEntries(client filer_pb.SeaweedFilerClient, d
 		sepIndex := strings.Index(marker, "/")
 		if sepIndex != -1 {
 			subPrefix, subMarker := marker[0:sepIndex], marker[sepIndex+1:]
-			subDir := fmt.Sprintf("%s/%s", dir[0:bucketPrefixLen-1], subPrefix)
-			if strings.HasPrefix(subDir, dir) {
-				subCounter, subIsTruncated, subNextMarker, subErr := s3a.doListFilerEntries(client, subDir, "", maxKeys, subMarker, delimiter, false, false, bucketPrefixLen, eachEntryFn)
-				if subErr != nil {
-					err = subErr
-					return
-				}
-				counter += subCounter
-				isTruncated = isTruncated || subIsTruncated
-				maxKeys -= subCounter
-				nextMarker = subNextMarker
-				// finished processing this sub directory
-				marker = subPrefix
+			var subDir string
+			if len(dir) > bucketPrefixLen && dir[bucketPrefixLen:] == subPrefix {
+				subDir = dir
+			} else {
+				subDir = fmt.Sprintf("%s/%s", dir, subPrefix)
 			}
+			subCounter, subIsTruncated, subNextMarker, subErr := s3a.doListFilerEntries(client, subDir, "", maxKeys, subMarker, delimiter, false, false, bucketPrefixLen, eachEntryFn)
+			if subErr != nil {
+				err = subErr
+				return
+			}
+			counter += subCounter
+			isTruncated = isTruncated || subIsTruncated
+			maxKeys -= subCounter
+			nextMarker = subNextMarker
+			// finished processing this sub directory
+			marker = subPrefix
 		}
 	}
 	if maxKeys <= 0 {
@@ -278,7 +281,7 @@ func (s3a *S3ApiServer) doListFilerEntries(client filer_pb.SeaweedFilerClient, d
 	request := &filer_pb.ListEntriesRequest{
 		Directory:          dir,
 		Prefix:             prefix,
-		Limit:              uint32(maxKeys + 1),
+		Limit:              uint32(maxKeys + 2), // bucket root directory needs to skip additional s3_constants.MultipartUploadsFolder folder
 		StartFromFileName:  marker,
 		InclusiveStartFrom: inclusiveStartFrom,
 	}
@@ -309,7 +312,7 @@ func (s3a *S3ApiServer) doListFilerEntries(client filer_pb.SeaweedFilerClient, d
 		nextMarker = dir + "/" + entry.Name
 		if entry.IsDirectory {
 			// println("ListEntries", dir, "dir:", entry.Name)
-			if entry.Name == ".uploads" { // FIXME no need to apply to all directories. this extra also affects maxKeys
+			if entry.Name == s3_constants.MultipartUploadsFolder { // FIXME no need to apply to all directories. this extra also affects maxKeys
 				continue
 			}
 			if delimiter == "" {
@@ -2,6 +2,9 @@ package s3api

 import (
 	"encoding/xml"
+	"fmt"
+	"regexp"
+	"strings"
 )

 type Tag struct {
@@ -37,3 +40,40 @@ func FromTags(tags map[string]string) (t *Tagging) {
 	}
 	return
 }
+
+func parseTagsHeader(tags string) (map[string]string, error) {
+	parsedTags := make(map[string]string)
+	for _, v := range strings.Split(tags, "&") {
+		tag := strings.Split(v, "=")
+		if len(tag) == 2 {
+			parsedTags[tag[0]] = tag[1]
+		} else if len(tag) == 1 {
+			parsedTags[tag[0]] = ""
+		}
+	}
+	return parsedTags, nil
+}
+
+func ValidateTags(tags map[string]string) error {
+	if len(tags) > 10 {
+		return fmt.Errorf("validate tags: %d tags more than 10", len(tags))
+	}
+	for k, v := range tags {
+		if len(k) > 128 {
+			return fmt.Errorf("validate tags: tag key longer than 128")
+		}
+		validateKey, err := regexp.MatchString(`^([\p{L}\p{Z}\p{N}_.:/=+\-@]*)$`, k)
+		if !validateKey || err != nil {
+			return fmt.Errorf("validate tags key %s error, incorrect key", k)
+		}
+		if len(v) > 256 {
+			return fmt.Errorf("validate tags: tag value longer than 256")
+		}
+		validateValue, err := regexp.MatchString(`^([\p{L}\p{Z}\p{N}_.:/=+\-@]*)$`, v)
+		if !validateValue || err != nil {
+			return fmt.Errorf("validate tags value %s error, incorrect value", v)
+		}
+	}
+
+	return nil
+}
@@ -50,3 +50,65 @@ func TestXMLMarshall(t *testing.T) {
 	assert.Equal(t, expected, actual)

 }
+
+type TestTags map[string]string
+
+var ValidateTagsTestCases = []struct {
+	testCaseID    int
+	tags          TestTags
+	wantErrString string
+}{
+	{
+		1,
+		TestTags{"key-1": "value-1"},
+		"",
+	},
+	{
+		2,
+		TestTags{"key-1": "valueOver256R59YI9bahPwAVqvLeKCvM2S1RjzgP8fNDKluCbol0XTTFY6VcMwTBmdnqjsddilXztSGfEoZS1wDAIMBA0rW0CLNSoE2zNg4TT0vDbLHEtZBoZjdZ5E0JNIAqwb9ptIk2VizYmhWjb1G4rJ0CqDGWxcy3usXaQg6Dk6kU8N4hlqwYWeGw7uqdghcQ3ScfF02nHW9QFMN7msLR5fe90mbFBBp3Tjq34i0LEr4By2vxoRa2RqdBhEJhi23Tm"},
+		"validate tags: tag value longer than 256",
+	},
+	{
+		3,
+		TestTags{"keyLenOver128a5aUUGcPexMELsz3RyROzIzfO6BKABeApH2nbbagpOxZh2MgBWYDZtFxQaCuQeP1xR7dUJLwfFfDHguVIyxvTStGDk51BemKETIwZ0zkhR7lhfHBp2y0nFnV": "value-1"},
+		"validate tags: tag key longer than 128",
+	},
+	{
+		4,
+		TestTags{"key-1*": "value-1"},
+		"validate tags key key-1* error, incorrect key",
+	},
+	{
+		5,
+		TestTags{"key-1": "value-1?"},
+		"validate tags value value-1? error, incorrect value",
+	},
+	{
+		6,
+		TestTags{
+			"key-1":  "value",
+			"key-2":  "value",
+			"key-3":  "value",
+			"key-4":  "value",
+			"key-5":  "value",
+			"key-6":  "value",
+			"key-7":  "value",
+			"key-8":  "value",
+			"key-9":  "value",
+			"key-10": "value",
+			"key-11": "value",
+		},
+		"validate tags: 11 tags more than 10",
+	},
+}
+
+func TestValidateTags(t *testing.T) {
+	for _, testCase := range ValidateTagsTestCases {
+		err := ValidateTags(testCase.tags)
+		if testCase.wantErrString == "" {
+			assert.NoErrorf(t, err, "no error")
+		} else {
+			assert.EqualError(t, err, testCase.wantErrString)
+		}
+	}
+}
@@ -35,7 +35,7 @@ func LoadServerTLS(config *util.ViperProxy, component string) (grpc.ServerOption

 	serverIdentityProvider, err := pemfile.NewProvider(serverOptions)
 	if err != nil {
-		glog.Warningf("pemfile.NewProvider(%v) failed: %v", serverOptions, err)
+		glog.Warningf("pemfile.NewProvider(%v) %v failed: %v", serverOptions, component, err)
 		return nil, nil
 	}

@@ -263,8 +263,12 @@ func (ms *MasterServer) KeepConnected(stream master_pb.Seaweed_KeepConnectedServ
 		}
 		ms.deleteClient(clientName)
 	}()
-
-	for _, message := range ms.Topo.ToVolumeLocations() {
+	for i, message := range ms.Topo.ToVolumeLocations() {
+		if i == 0 {
+			if leader, err := ms.Topo.Leader(); err == nil {
+				message.Leader = string(leader)
+			}
+		}
 		if sendErr := stream.Send(&master_pb.KeepConnectedResponse{VolumeLocation: message}); sendErr != nil {
 			return sendErr
 		}
@@ -48,6 +48,13 @@ type WebDavServer struct {
 	Handler        *webdav.Handler
 }

+func max(x, y int64) int64 {
+	if x <= y {
+		return y
+	}
+	return x
+}
+
 func NewWebDavServer(option *WebDavOption) (ws *WebDavServer, err error) {

 	fs, _ := NewWebDavFileSystem(option)
@@ -496,6 +503,7 @@ func (f *WebDavFile) Write(buf []byte) (int, error) {
 	written, err := f.bufWriter.Write(buf)

 	if err == nil {
+		f.entry.Attributes.FileSize = uint64(max(f.off+int64(written), int64(f.entry.Attributes.FileSize)))
 		glog.V(3).Infof("WebDavFileSystem.Write %v: written [%d,%d)", f.name, f.off, f.off+int64(len(buf)))
 		f.off += int64(written)
 	}
@@ -45,7 +45,7 @@ func (c *commandFsDu) Do(args []string, commandEnv *CommandEnv, writer io.Writer
 	blockCount, byteCount, err = duTraverseDirectory(writer, commandEnv, dir, name)

 	if name == "" && err == nil {
-		fmt.Fprintf(writer, "block:%4d\tbyte:%10d\t%s\n", blockCount, byteCount, dir)
+		fmt.Fprintf(writer, "block:%4d\tlogical size:%10d\t%s\n", blockCount, byteCount, dir)
 	}

 	return
@@ -76,7 +76,7 @@ func duTraverseDirectory(writer io.Writer, filerClient filer_pb.FilerClient, dir
 		}

 		if name != "" && !entry.IsDirectory {
-			fmt.Fprintf(writer, "block:%4d\tbyte:%10d\t%s/%s\n", fileBlockCount, fileByteCount, dir, entry.Name)
+			fmt.Fprintf(writer, "block:%4d\tlogical size:%10d\t%s/%s\n", fileBlockCount, fileByteCount, dir, entry.Name)
 		}
 		return nil
 	})
@@ -4,8 +4,9 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 	"io"
+
+	"github.com/chrislusf/seaweedfs/weed/pb/filer_pb"
 )

 func init() {
@@ -23,7 +24,7 @@ func (c *commandS3BucketQuota) Help() string {
 	return `set/remove/enable/disable quota for a bucket

 	Example:
-		s3.bucket.quota -name=<bucket_name> -operation=set -sizeMB=1024
+		s3.bucket.quota -name=<bucket_name> -op=set -sizeMB=1024
 `
 }

@@ -31,7 +32,7 @@ func (c *commandS3BucketQuota) Do(args []string, commandEnv *CommandEnv, writer

 	bucketCommand := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
 	bucketName := bucketCommand.String("name", "", "bucket name")
-	operationName := bucketCommand.String("op", "set", "operation name [set|remove|enable|disable]")
+	operationName := bucketCommand.String("op", "set", "operation name [set|get|remove|enable|disable]")
 	sizeMB := bucketCommand.Int64("sizeMB", 0, "bucket quota size in MiB")
 	if err = bucketCommand.Parse(args); err != nil {
 		return nil
@@ -63,6 +64,9 @@ func (c *commandS3BucketQuota) Do(args []string, commandEnv *CommandEnv, writer
 		switch *operationName {
 		case "set":
 			bucketEntry.Quota = *sizeMB * 1024 * 1024
+		case "get":
+			fmt.Fprintf(writer, "bucket quota: %dMiB \n", bucketEntry.Quota/1024/1024)
+			return nil
 		case "remove":
 			bucketEntry.Quota = 0
 		case "enable":
@@ -3,6 +3,7 @@ package shell
 import (
 	"flag"
 	"fmt"
+	"github.com/chrislusf/seaweedfs/weed/s3api/s3_constants"
 	"github.com/chrislusf/seaweedfs/weed/security"
 	"github.com/chrislusf/seaweedfs/weed/util"
 	"io"
@@ -68,7 +69,7 @@ func (c *commandS3CleanUploads) Do(args []string, commandEnv *CommandEnv, writer
 }

 func (c *commandS3CleanUploads) cleanupUploads(commandEnv *CommandEnv, writer io.Writer, filerBucketsPath string, bucket string, timeAgo time.Duration, signingKey string) error {
-	uploadsDir := filerBucketsPath + "/" + bucket + "/.uploads"
+	uploadsDir := filerBucketsPath + "/" + bucket + "/" + s3_constants.MultipartUploadsFolder
 	var staleUploads []string
 	now := time.Now()
 	err := filer_pb.List(commandEnv, uploadsDir, "", func(entry *filer_pb.Entry, isLast bool) error {
@@ -259,6 +259,10 @@ func balanceSelectedVolume(commandEnv *CommandEnv, diskType types.DiskType, volu
 		slices.SortFunc(nodesWithCapacity, func(a, b *Node) bool {
 			return a.localVolumeRatio(capacityFunc) < b.localVolumeRatio(capacityFunc)
 		})
+		if len(nodesWithCapacity) == 0 {
+			fmt.Printf("no volume server found with capacity for %s", diskType.ReadableString())
+			return nil
+		}
 		fullNode := nodesWithCapacity[len(nodesWithCapacity)-1]
 		var candidateVolumes []*master_pb.VolumeInformationMessage
 		for _, v := range fullNode.selectedVolumes {
@@ -18,6 +18,7 @@ func init() {
 }

 type commandVolumeServerEvacuate struct {
+	targetServer string
 }

 func (c *commandVolumeServerEvacuate) Name() string {
@@ -46,6 +47,7 @@ func (c *commandVolumeServerEvacuate) Do(args []string, commandEnv *CommandEnv,

 	vsEvacuateCommand := flag.NewFlagSet(c.Name(), flag.ContinueOnError)
 	volumeServer := vsEvacuateCommand.String("node", "", "<host>:<port> of the volume server")
+	c.targetServer = *vsEvacuateCommand.String("target", "", "<host>:<port> of target volume")
 	skipNonMoveable := vsEvacuateCommand.Bool("skipNonMoveable", false, "skip volumes that can not be moved")
 	applyChange := vsEvacuateCommand.Bool("force", false, "actually apply the changes")
 	retryCount := vsEvacuateCommand.Int("retry", 0, "how many times to retry")
@@ -61,9 +63,8 @@ func (c *commandVolumeServerEvacuate) Do(args []string, commandEnv *CommandEnv,
 	if *volumeServer == "" {
 		return fmt.Errorf("need to specify volume server by -node=<host>:<port>")
 	}
-
 	for i := 0; i < *retryCount+1; i++ {
-		if err = volumeServerEvacuate(commandEnv, *volumeServer, *skipNonMoveable, *applyChange, writer); err == nil {
+		if err = c.volumeServerEvacuate(commandEnv, *volumeServer, *skipNonMoveable, *applyChange, writer); err == nil {
 			return nil
 		}
 	}
@@ -72,7 +73,7 @@ func (c *commandVolumeServerEvacuate) Do(args []string, commandEnv *CommandEnv,

 }

-func volumeServerEvacuate(commandEnv *CommandEnv, volumeServer string, skipNonMoveable, applyChange bool, writer io.Writer) (err error) {
+func (c *commandVolumeServerEvacuate) volumeServerEvacuate(commandEnv *CommandEnv, volumeServer string, skipNonMoveable, applyChange bool, writer io.Writer) (err error) {
 	// 1. confirm the volume server is part of the cluster
 	// 2. collect all other volume servers, sort by empty slots
 	// 3. move to any other volume server as long as it satisfy the replication requirements
@@ -84,18 +85,18 @@ func volumeServerEvacuate(commandEnv *CommandEnv, volumeServer string, skipNonMo
 		return err
 	}

-	if err := evacuateNormalVolumes(commandEnv, topologyInfo, volumeServer, skipNonMoveable, applyChange, writer); err != nil {
+	if err := c.evacuateNormalVolumes(commandEnv, topologyInfo, volumeServer, skipNonMoveable, applyChange, writer); err != nil {
 		return err
 	}

-	if err := evacuateEcVolumes(commandEnv, topologyInfo, volumeServer, skipNonMoveable, applyChange, writer); err != nil {
+	if err := c.evacuateEcVolumes(commandEnv, topologyInfo, volumeServer, skipNonMoveable, applyChange, writer); err != nil {
 		return err
 	}

 	return nil
 }

-func evacuateNormalVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyInfo, volumeServer string, skipNonMoveable, applyChange bool, writer io.Writer) error {
+func (c *commandVolumeServerEvacuate) evacuateNormalVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyInfo, volumeServer string, skipNonMoveable, applyChange bool, writer io.Writer) error {
 	// find this volume server
 	volumeServers := collectVolumeServersByDc(topologyInfo, "")
 	thisNode, otherNodes := nodesOtherThan(volumeServers, volumeServer)
@@ -124,7 +125,7 @@ func evacuateNormalVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.Topol
 	return nil
 }

-func evacuateEcVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyInfo, volumeServer string, skipNonMoveable, applyChange bool, writer io.Writer) error {
+func (c *commandVolumeServerEvacuate) evacuateEcVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyInfo, volumeServer string, skipNonMoveable, applyChange bool, writer io.Writer) error {
 	// find this ec volume server
 	ecNodes, _ := collectEcVolumeServersByDc(topologyInfo, "")
 	thisNode, otherNodes := ecNodesOtherThan(ecNodes, volumeServer)
@@ -135,7 +136,7 @@ func evacuateEcVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyI
 	// move away ec volumes
 	for _, diskInfo := range thisNode.info.DiskInfos {
 		for _, ecShardInfo := range diskInfo.EcShardInfos {
-			hasMoved, err := moveAwayOneEcVolume(commandEnv, ecShardInfo, thisNode, otherNodes, applyChange)
+			hasMoved, err := c.moveAwayOneEcVolume(commandEnv, ecShardInfo, thisNode, otherNodes, applyChange)
 			if err != nil {
 				return fmt.Errorf("move away volume %d from %s: %v", ecShardInfo.Id, volumeServer, err)
 			}
@@ -151,7 +152,7 @@ func evacuateEcVolumes(commandEnv *CommandEnv, topologyInfo *master_pb.TopologyI
 	return nil
 }

-func moveAwayOneEcVolume(commandEnv *CommandEnv, ecShardInfo *master_pb.VolumeEcShardInformationMessage, thisNode *EcNode, otherNodes []*EcNode, applyChange bool) (hasMoved bool, err error) {
+func (c *commandVolumeServerEvacuate) moveAwayOneEcVolume(commandEnv *CommandEnv, ecShardInfo *master_pb.VolumeEcShardInformationMessage, thisNode *EcNode, otherNodes []*EcNode, applyChange bool) (hasMoved bool, err error) {

 	for _, shardId := range erasure_coding.ShardBits(ecShardInfo.EcIndexBits).ShardIds() {
 		slices.SortFunc(otherNodes, func(a, b *EcNode) bool {
@@ -159,6 +160,9 @@ func moveAwayOneEcVolume(commandEnv *CommandEnv, ecShardInfo *master_pb.VolumeEc
 		})
 		for i := 0; i < len(otherNodes); i++ {
 			emptyNode := otherNodes[i]
+			if c.targetServer != "" && c.targetServer != emptyNode.info.Id {
+				continue
+			}
 			collectionPrefix := ""
 			if ecShardInfo.Collection != "" {
 				collectionPrefix = ecShardInfo.Collection + "_"
@@ -188,7 +192,7 @@ func moveAwayOneNormalVolume(commandEnv *CommandEnv, volumeReplicas map[uint32][
 		})
 	}
 	slices.SortFunc(otherNodes, func(a, b *Node) bool {
-		return a.localVolumeRatio(fn) > b.localVolumeRatio(fn)
+		return a.localVolumeRatio(fn) < b.localVolumeRatio(fn)
 	})
 	for i := 0; i < len(otherNodes); i++ {
 		emptyNode := otherNodes[i]
@@ -10,7 +10,8 @@ func TestVolumeServerEvacuate(t *testing.T) {

 	volumeServer := "192.168.1.4:8080"

-	if err := evacuateNormalVolumes(nil, topologyInfo, volumeServer, true, false, os.Stdout); err != nil {
+	c := commandVolumeServerEvacuate{}
+	if err := c.evacuateNormalVolumes(nil, topologyInfo, volumeServer, true, false, os.Stdout); err != nil {
 		t.Errorf("evacuate: %v", err)
 	}

@@ -81,5 +81,6 @@ func (df *DiskFile) Name() string {
 }

 func (df *DiskFile) Sync() error {
-	return df.File.Sync()
+	return nil
+	// return df.File.Sync()
 }
@@ -44,13 +44,11 @@ func GenerateDirUuid(dir string) (dirUuidString string, err error) {
 		dirUuidString = dirUuid.String()
 		writeErr := util.WriteFile(fileName, []byte(dirUuidString), 0644)
 		if writeErr != nil {
-			glog.Warningf("failed to write uuid to %s : %v", fileName, writeErr)
 			return "", fmt.Errorf("failed to write uuid to %s : %v", fileName, writeErr)
 		}
 	} else {
 		uuidData, readErr := os.ReadFile(fileName)
 		if readErr != nil {
-			glog.Warningf("failed to read uuid from %s : %v", fileName, readErr)
 			return "", fmt.Errorf("failed to read uuid from %s : %v", fileName, readErr)
 		}
 		dirUuidString = string(uuidData)
@@ -65,7 +63,10 @@ func NewDiskLocation(dir string, maxVolumeCount int, minFreeSpace util.MinFreeSp
 	} else {
 		idxDir = util.ResolvePath(idxDir)
 	}
-	dirUuid, _ := GenerateDirUuid(dir)
+	dirUuid, err := GenerateDirUuid(dir)
+	if err != nil {
+		glog.Fatalf("cannot generate uuid of dir %s: %v", dir, err)
+	}
 	location := &DiskLocation{
 		Directory:              dir,
 		DirectoryUuid:          dirUuid,
@@ -317,9 +317,6 @@ func (v *Volume) makeupDiff(newDatFileName, newIdxFileName, oldDatFileName, oldI
 		if err != nil {
 			return fmt.Errorf("cannot write indexfile %s: %v", newIdxFileName, err)
 		}
-		if err := idx.Sync(); err != nil {
-			return fmt.Errorf("cannot sync indexfile %s: %v", newIdxFileName, err)
-		}
 	}

 	return nil
@@ -82,12 +82,13 @@ func (dnll *VolumeLocationList) Refresh(freshThreshHold int64) {
 	}
 }

+// Stats returns logic size and count
 func (dnll *VolumeLocationList) Stats(vid needle.VolumeId, freshThreshHold int64) (size uint64, fileCount int) {
 	for _, dnl := range dnll.list {
 		if dnl.LastSeen < freshThreshHold {
 			vinfo, err := dnl.GetVolumesById(vid)
 			if err == nil {
-				return (vinfo.Size - vinfo.DeletedByteCount) * uint64(len(dnll.list)), vinfo.FileCount - vinfo.DeleteCount
+				return (vinfo.Size - vinfo.DeletedByteCount), vinfo.FileCount - vinfo.DeleteCount
 			}
 		}
 	}
@@ -5,7 +5,7 @@ import (
 )

 var (
-	VERSION_NUMBER = fmt.Sprintf("%.02f", 3.13)
+	VERSION_NUMBER = fmt.Sprintf("%.02f", 3.14)
 	VERSION        = sizeLimit + " " + VERSION_NUMBER
 	COMMIT         = ""
 )
@@ -168,15 +168,19 @@ func (mc *MasterClient) tryConnectToMaster(master pb.ServerAddress) (nextHintedL
 		}

 		// check if it is the leader to determine whether to reset the vidMap
-		if resp.VolumeLocation != nil && resp.VolumeLocation.Leader != "" {
-			glog.V(0).Infof("redirected to leader %v", resp.VolumeLocation.Leader)
-			nextHintedLeader = pb.ServerAddress(resp.VolumeLocation.Leader)
-			stats.MasterClientConnectCounter.WithLabelValues(stats.RedirectedToleader).Inc()
-			return nil
+		if resp.VolumeLocation != nil {
+			if resp.VolumeLocation.Leader != "" && string(master) != resp.VolumeLocation.Leader {
+				glog.V(0).Infof("master %v redirected to leader %v", master, resp.VolumeLocation.Leader)
+				nextHintedLeader = pb.ServerAddress(resp.VolumeLocation.Leader)
+				stats.MasterClientConnectCounter.WithLabelValues(stats.RedirectedToleader).Inc()
+				return nil
+			}
+			mc.vidMap = newVidMap("")
+			mc.updateVidMap(resp)
+		} else {
+			mc.vidMap = newVidMap("")
 		}
-
 		mc.currentMaster = master
-		mc.vidMap = newVidMap("")

 		for {
 			resp, err := stream.Recv()
@@ -188,36 +192,14 @@ func (mc *MasterClient) tryConnectToMaster(master pb.ServerAddress) (nextHintedL

 			if resp.VolumeLocation != nil {
 				// maybe the leader is changed
-				if resp.VolumeLocation.Leader != "" {
-					glog.V(0).Infof("redirected to leader %v", resp.VolumeLocation.Leader)
+				if resp.VolumeLocation.Leader != "" && string(mc.currentMaster) != resp.VolumeLocation.Leader {
+					glog.V(0).Infof("currentMaster %v redirected to leader %v", mc.currentMaster, resp.VolumeLocation.Leader)
 					nextHintedLeader = pb.ServerAddress(resp.VolumeLocation.Leader)
 					stats.MasterClientConnectCounter.WithLabelValues(stats.RedirectedToleader).Inc()
 					return nil
 				}

-				// process new volume location
-				loc := Location{
-					Url:        resp.VolumeLocation.Url,
-					PublicUrl:  resp.VolumeLocation.PublicUrl,
-					DataCenter: resp.VolumeLocation.DataCenter,
-					GrpcPort:   int(resp.VolumeLocation.GrpcPort),
-				}
-				for _, newVid := range resp.VolumeLocation.NewVids {
-					glog.V(1).Infof("%s.%s: %s masterClient adds volume %d", mc.FilerGroup, mc.clientType, loc.Url, newVid)
-					mc.addLocation(newVid, loc)
-				}
-				for _, deletedVid := range resp.VolumeLocation.DeletedVids {
-					glog.V(1).Infof("%s.%s: %s masterClient removes volume %d", mc.FilerGroup, mc.clientType, loc.Url, deletedVid)
-					mc.deleteLocation(deletedVid, loc)
-				}
-				for _, newEcVid := range resp.VolumeLocation.NewEcVids {
-					glog.V(1).Infof("%s.%s: %s masterClient adds ec volume %d", mc.FilerGroup, mc.clientType, loc.Url, newEcVid)
-					mc.addEcLocation(newEcVid, loc)
-				}
-				for _, deletedEcVid := range resp.VolumeLocation.DeletedEcVids {
-					glog.V(1).Infof("%s.%s: %s masterClient removes ec volume %d", mc.FilerGroup, mc.clientType, loc.Url, deletedEcVid)
-					mc.deleteEcLocation(deletedEcVid, loc)
-				}
+				mc.updateVidMap(resp)
 			}

 			if resp.ClusterNodeUpdate != nil {
@@ -245,6 +227,32 @@ func (mc *MasterClient) tryConnectToMaster(master pb.ServerAddress) (nextHintedL
 	return
 }

+func (mc *MasterClient) updateVidMap(resp *master_pb.KeepConnectedResponse) {
+	// process new volume location
+	loc := Location{
+		Url:        resp.VolumeLocation.Url,
+		PublicUrl:  resp.VolumeLocation.PublicUrl,
+		DataCenter: resp.VolumeLocation.DataCenter,
+		GrpcPort:   int(resp.VolumeLocation.GrpcPort),
+	}
+	for _, newVid := range resp.VolumeLocation.NewVids {
+		glog.V(1).Infof("%s.%s: %s masterClient adds volume %d", mc.FilerGroup, mc.clientType, loc.Url, newVid)
+		mc.addLocation(newVid, loc)
+	}
+	for _, deletedVid := range resp.VolumeLocation.DeletedVids {
+		glog.V(1).Infof("%s.%s: %s masterClient removes volume %d", mc.FilerGroup, mc.clientType, loc.Url, deletedVid)
+		mc.deleteLocation(deletedVid, loc)
+	}
+	for _, newEcVid := range resp.VolumeLocation.NewEcVids {
+		glog.V(1).Infof("%s.%s: %s masterClient adds ec volume %d", mc.FilerGroup, mc.clientType, loc.Url, newEcVid)
+		mc.addEcLocation(newEcVid, loc)
+	}
+	for _, deletedEcVid := range resp.VolumeLocation.DeletedEcVids {
+		glog.V(1).Infof("%s.%s: %s masterClient removes ec volume %d", mc.FilerGroup, mc.clientType, loc.Url, deletedEcVid)
+		mc.deleteEcLocation(deletedEcVid, loc)
+	}
+}
+
 func (mc *MasterClient) WithClient(streamingMode bool, fn func(client master_pb.SeaweedClient) error) error {
 	return util.Retry("master grpc", func() error {
 		for mc.currentMaster == "" {
Author	SHA1	Message	Date
chrislu	a85ed3fe8f	minor	2022-07-07 22:27:24 -07:00
chrislu	1f8e73ede7	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-07-07 22:26:06 -07:00
chrislu	2ea18cdcc8	remove dead code	2022-07-07 22:26:03 -07:00
Chris LuandGitHub	fa8e33ccb8	Update binaries_dev.yml	2022-07-07 22:11:44 -07:00
chrislu	b7de5c6c43	shell: remove unused filer parameter from shell.toml	2022-07-07 15:01:23 -07:00
chrislu	f2f0482dd3	mount: random read also try to use the local cache first	2022-07-07 11:50:28 -07:00
chrislu	05c53820b9	mount: file handler release memory	2022-07-07 11:49:40 -07:00
chrislu	69ef6459c3	S3 ListObjectsV2 fix pagination under bucket root fix https://github.com/chrislusf/seaweedfs/issues/3166	2022-07-06 02:28:34 -07:00
Chris LuandGitHub	bfea7008f3	Merge pull request #3277 from kmlebedev/rabbitry_enable_durable	2022-07-06 00:19:49 -07:00
Konstantin Lebedev	785223e587	rabbitpubsub enable durable	2022-07-06 10:05:29 +05:00
chrislu	57975b93ff	adjust used size reporting	2022-07-04 13:47:42 -07:00
chrislu	a79f5bd0d5	adjust used size reporting	2022-07-04 13:46:32 -07:00
chrislu	3c79c77056	3.14	2022-07-04 13:15:49 -07:00
Chris LuandGitHub	05510630fd	Merge pull request #3270 from chrislusf/dependabot/go_modules/cloud.google.com/go/pubsub-1.23.1	2022-07-04 12:09:06 -07:00
dependabot[bot]andGitHub	2c031f2a7c	Bump cloud.google.com/go/pubsub from 1.22.2 to 1.23.1 Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.22.2 to 1.23.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.22.2...pubsub/v1.23.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-07-04 19:08:32 +00:00
Chris LuandGitHub	f69a8127a4	Merge pull request #3271 from chrislusf/dependabot/go_modules/github.com/stretchr/testify-1.8.0	2022-07-04 12:06:46 -07:00
Chris LuandGitHub	9ea1d093b0	Merge pull request #3272 from chrislusf/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.47	2022-07-04 12:06:36 -07:00
Chris LuandGitHub	c2c9c6d60a	Merge pull request #3273 from chrislusf/dependabot/go_modules/modernc.org/sqlite-1.17.3	2022-07-04 12:06:27 -07:00
Chris LuandGitHub	00d5be61d3	Merge pull request #3274 from chrislusf/dependabot/go_modules/google.golang.org/api-0.86.0	2022-07-04 12:06:13 -07:00
dependabot[bot]andGitHub	bf924e37c1	Bump google.golang.org/api from 0.85.0 to 0.86.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.85.0 to 0.86.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.85.0...v0.86.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-07-04 15:45:24 +00:00
dependabot[bot]andGitHub	8f7273a13c	Bump modernc.org/sqlite from 1.17.2 to 1.17.3 Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.17.2 to 1.17.3. - [Release notes](https://gitlab.com/cznic/sqlite/tags) - [Commits](https://gitlab.com/cznic/sqlite/compare/v1.17.2...v1.17.3) --- updated-dependencies: - dependency-name: modernc.org/sqlite dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2022-07-04 15:45:13 +00:00
dependabot[bot]andGitHub	d243fa49cf	Bump github.com/aws/aws-sdk-go from 1.44.37 to 1.44.47 Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.37 to 1.44.47. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.37...v1.44.47) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2022-07-04 15:44:49 +00:00
dependabot[bot]andGitHub	92f62165ef	Bump github.com/stretchr/testify from 1.7.5 to 1.8.0 Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.5 to 1.8.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](https://github.com/stretchr/testify/compare/v1.7.5...v1.8.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-07-04 15:44:39 +00:00
Chris LuandGitHub	a122baa427	Merge pull request #3268 from guol-fnst/fix_usage add getting bucket quota	2022-07-04 00:44:43 -07:00
guol-fnst	5bc895673e	add getting bucket quota	2022-07-04 15:36:44 +08:00
Chris LuandGitHub	267d45c313	Merge pull request #3266 from guol-fnst/fix_usage fix usage	2022-07-03 22:11:37 -07:00
guol-fnst	dd4664e610	fix usage	2022-07-04 11:31:39 +08:00
Chris LuandGitHub	c349f81195	Update binaries_dev.yml	2022-07-03 16:16:44 -07:00
chrislu	d0eb65ae7b	update go-fuse library version Make OSXFUSE handle timeouts the same way as Linux	2022-07-03 16:09:03 -07:00
chrislu	141f662734	edge case checking when volume server does not have capacity to balance fix https://github.com/chrislusf/seaweedfs/issues/3257	2022-07-01 12:16:18 -07:00
chrislu	bee3a7c798	fix tests	2022-07-01 12:09:42 -07:00
Chris LuandGitHub	ddb0f6bc74	Merge pull request #3258 from kmlebedev/issues_3255 volumeServer.evacuate fix select empty node	2022-07-01 02:36:30 -07:00
Chris LuandGitHub	96c9d16c11	Merge pull request #3263 from kmlebedev/fix_pagination fix pagination	2022-07-01 02:23:45 -07:00
Konstantin Lebedev	3d5f1cbc5e	fix pagination	2022-07-01 14:17:49 +05:00
chrislu	06a8b174b5	also remove Sync() for idx file	2022-06-30 13:50:53 -07:00
chrislu	52580743b9	stuck with file.Sync() starting to get weed/storage () - (master) > go test ./... I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648 I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959 I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000 I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming... panic: test timed out after 10m0s goroutine 61 [running]: testing.(M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2029 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:176 +0x32 goroutine 1 [chan receive, 10 minutes]: testing.(T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1487 +0x37a testing.runTests.func1(0xc0000cd520?) /usr/local/go/src/testing/testing.go:1839 +0x6e testing.tRunner(0xc0000cd520, 0xc00039fcd8) /usr/local/go/src/testing/testing.go:1439 +0x102 testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?}) /usr/local/go/src/testing/testing.go:1837 +0x457 testing.(M).Run(0xc0000dadc0) /usr/local/go/src/testing/testing.go:1719 +0x5d9 main.main() _testmain.go:57 +0x1aa goroutine 21 [chan receive]: github.com/chrislusf/seaweedfs/weed/glog.(loggingT).flushDaemon(0x0?) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a created by github.com/chrislusf/seaweedfs/weed/glog.init.0 /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf goroutine 55 [syscall]: syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?) /usr/local/go/src/runtime/sys_darwin.go:22 +0x4e syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?) /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30 internal/poll.(FD).Fsync.func1(...) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:18 internal/poll.ignoringEINTR(...) /usr/local/go/src/internal/poll/fd_posix.go:74 internal/poll.(FD).Fsync(0xc00038f1e0?) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc os.(File).Sync(0xc00012a030) /usr/local/go/src/os/file_posix.go:168 +0x4e github.com/chrislusf/seaweedfs/weed/storage/backend.(DiskFile).Sync(...) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84 github.com/chrislusf/seaweedfs/weed/storage.(Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52}) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa github.com/chrislusf/seaweedfs/weed/storage.(Volume).CommitCompact(0xc0000bb440) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8 github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305 testing.tRunner(0xc00025a000, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1439 +0x102 created by testing.(T).Run /usr/local/go/src/testing/testing.go:1486 +0x35f goroutine 56 [chan receive, 10 minutes]: github.com/chrislusf/seaweedfs/weed/storage.(Volume).startWorker.func1() /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56 FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s ? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached) ? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached) ? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached) ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached) ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached) ? github.com/chrislusf/seaweedfs/weed/storage/types [no test files] ? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files] FAIL	2022-06-30 12:32:55 -07:00
Konstantin Lebedev	5ed8165161	fix logic add option targetServer https://github.com/chrislusf/seaweedfs/issues/3255	2022-06-30 17:58:24 +05:00
Chris LuandGitHub	413cb3d389	Merge pull request #3256 from shichanglin5/fix_keep_connected fixed `volume xx not found` caused by missing VolumeLocation events	2022-06-30 00:10:09 -07:00
石昌林	200a862701	fixed `volume xx not found` caused by missing VolumeLocation events When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost	2022-06-30 13:41:56 +08:00
chrislu	ffaa2163e0	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-29 12:50:33 -07:00
chrislu	f3c4b07b8a	Update backers.md	2022-06-29 12:50:31 -07:00
Chris LuandGitHub	4ae9884a6e	Merge pull request #3254 from shichanglin5/fix_subscribe_meta [Critical] Avoid synchronizing from scratch due to err	2022-06-29 12:36:28 -07:00
Chris LuandGitHub	b98556cff5	Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder use const multipart uploads folder	2022-06-29 12:09:03 -07:00
石昌林	2b5b37231b	Make sure that onChangeEvent is called successfully before updating lastTsNs	2022-06-29 23:57:07 +08:00
石昌林	e17e6ed542	Avoid synchronizing from scratch due to err	2022-06-29 23:31:04 +08:00
Konstantin Lebedev	bcbdc4cb37	use const multipart uploads folder avoid error bucket NotEmpty if multipart uploads folder exist	2022-06-29 16:21:16 +05:00
chrislu	a8abab2412	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-29 02:44:14 -07:00
chrislu	449ecc8890	remove max connection age related to https://github.com/chrislusf/seaweedfs/pull/3226	2022-06-29 02:44:12 -07:00
Chris LuandGitHub	c6e6e303db	Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy validate tags on copy object and add regex for validating tags	2022-06-29 01:37:14 -07:00
Chris LuandGitHub	e22c2c8012	Merge pull request #3188 from kmlebedev/fix_make_dev fix ip.bind cmd for docker compose	2022-06-29 01:36:59 -07:00
Lapshin Vitaliy	7c3d9d0535	remove println	2022-06-29 10:43:02 +03:00
Lapshin Vitaliy	61b2e3f9a2	fix return	2022-06-28 18:31:46 +03:00
Lapshin Vitaliy	606252472c	fix return	2022-06-28 18:24:03 +03:00
Lapshin Vitaliy	e969370913	fix parseTagging function and add test for validate tags	2022-06-28 14:48:55 +03:00
Chris LuandGitHub	a79bf9d92a	Merge pull request #3252 from guol-fnst/update_tbk remove hdfs, add tikv for full version update	2022-06-27 22:37:41 -07:00
guol-fnst	74c9f39d48	remove hdfs, add tikv for full version update	2022-06-28 12:35:07 +08:00
Chris LuandGitHub	8d810f8305	Merge pull request #3251 from shichanglin5/fix_concurrent_write_read	2022-06-27 21:23:48 -07:00
石昌林	e1b94eb6b9	fix `error: concurrent map writes` when add or remove cluster node	2022-06-28 10:34:59 +08:00
chrislu	4b1f48a399	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-27 12:40:50 -07:00
chrislu	4a65159250	fix reading time	2022-06-27 12:40:47 -07:00
Chris LuandGitHub	dc6b1a52e8	Merge pull request #3250 from xdadrm/xdadrm-patch-xweo-byte-files-from-webdav Fixes WebDAV 0-bytes files	2022-06-27 11:33:35 -07:00
xdadrmandGitHub	3a42269da8	Fixes WebDAV 0-bytes files Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.	2022-06-27 20:19:21 +02:00
Chris LuandGitHub	4be406a095	Merge pull request #3249 from kmlebedev/issues_2589 avoid set currentMaster svc.local k8s discovery service domains	2022-06-27 11:05:07 -07:00
Chris LuandGitHub	265b05c8fc	Merge pull request #3245 from chrislusf/dependabot/go_modules/github.com/klauspost/reedsolomon-1.10.0 Bump github.com/klauspost/reedsolomon from 1.9.16 to 1.10.0	2022-06-27 10:15:23 -07:00
Chris LuandGitHub	f5273cdd5c	Merge pull request #3244 from chrislusf/dependabot/go_modules/github.com/go-redsync/redsync/v4-4.5.1 Bump github.com/go-redsync/redsync/v4 from 4.5.0 to 4.5.1	2022-06-27 10:03:32 -07:00
Chris LuandGitHub	a9389377b5	Merge pull request #3247 from chrislusf/dependabot/go_modules/google.golang.org/api-0.85.0 Bump google.golang.org/api from 0.83.0 to 0.85.0	2022-06-27 10:02:19 -07:00
Konstantin Lebedev	9bff097b01	currentMaster redirected	2022-06-27 21:55:16 +05:00
Konstantin Lebedev	6c20a3b622	avoid set currentMaster k8s svc.local discoveruy service domains https://github.com/chrislusf/seaweedfs/issues/2589	2022-06-27 21:47:05 +05:00
Chris LuandGitHub	201023dd15	Merge pull request #3242 from garenchan/ck-dev1 fix 3238: handle errors for GenerateDirUuid method	2022-06-27 09:46:46 -07:00
Chris LuandGitHub	d172d07212	Merge pull request #3246 from chrislusf/dependabot/go_modules/github.com/stretchr/testify-1.7.5 Bump github.com/stretchr/testify from 1.7.3 to 1.7.5	2022-06-27 09:44:41 -07:00
Chris LuandGitHub	2e3d6104e4	Merge pull request #3243 from chrislusf/dependabot/go_modules/cloud.google.com/go/storage-1.23.0 Bump cloud.google.com/go/storage from 1.22.1 to 1.23.0	2022-06-27 09:44:25 -07:00
Lapshin Vitaliy	1a5981d583	fix test	2022-06-27 18:46:21 +03:00
dependabot[bot]andGitHub	983686bb83	Bump google.golang.org/api from 0.83.0 to 0.85.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.83.0 to 0.85.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.83.0...v0.85.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-06-27 15:24:20 +00:00
dependabot[bot]andGitHub	b5d70669ad	Bump github.com/stretchr/testify from 1.7.3 to 1.7.5 Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.7.3 to 1.7.5. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](https://github.com/stretchr/testify/compare/v1.7.3...v1.7.5) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2022-06-27 15:24:14 +00:00
dependabot[bot]andGitHub	787103b032	Bump github.com/klauspost/reedsolomon from 1.9.16 to 1.10.0 Bumps [github.com/klauspost/reedsolomon](https://github.com/klauspost/reedsolomon) from 1.9.16 to 1.10.0. - [Release notes](https://github.com/klauspost/reedsolomon/releases) - [Commits](https://github.com/klauspost/reedsolomon/compare/v1.9.16...v1.10.0) --- updated-dependencies: - dependency-name: github.com/klauspost/reedsolomon dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-06-27 15:24:07 +00:00
dependabot[bot]andGitHub	cfd7c0fa78	Bump github.com/go-redsync/redsync/v4 from 4.5.0 to 4.5.1 Bumps [github.com/go-redsync/redsync/v4](https://github.com/go-redsync/redsync) from 4.5.0 to 4.5.1. - [Release notes](https://github.com/go-redsync/redsync/releases) - [Commits](https://github.com/go-redsync/redsync/compare/v4.5.0...v4.5.1) --- updated-dependencies: - dependency-name: github.com/go-redsync/redsync/v4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>	2022-06-27 15:24:02 +00:00
dependabot[bot]andGitHub	e90a6af65e	Bump cloud.google.com/go/storage from 1.22.1 to 1.23.0 Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.22.1 to 1.23.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.22.1...pubsub/v1.23.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>	2022-06-27 15:23:46 +00:00
garenchan	8aa19577f4	fix 3238: handle errors for GenerateDirUuid method	2022-06-27 22:04:50 +08:00
Lapshin Vitaliy	d7c3493d15	Merge branch 'master' into bufix/validate-tags-on-copy	2022-06-27 13:53:57 +03:00
chrislu	b9f7b6fb9a	adjust log message	2022-06-26 23:12:16 -07:00
Chris LuandGitHub	fdd880b111	Merge pull request #3237 from shichanglin5/remove_query_collection Remove the collection query param of s3api and let the collection be …	2022-06-26 23:02:11 -07:00
石昌林	15bfc60a88	Remove the collection query param of s3api and let the collection be determined by the filer	2022-06-27 13:47:53 +08:00
Chris LuandGitHub	6a2afc3a7e	Merge pull request #3235 from blacktear23/list-filer-stores Improve filer command help text, add supported filer store list	2022-06-26 22:20:38 -07:00
yulai.li	af23e63e3f	Improve filer command help, add supported filer store list	2022-06-27 12:09:16 +08:00
chrislu	d003bb0166	java 3.13	2022-06-26 20:41:51 -07:00
Lapshin Vitaliy	5f5fd0bc48	validate tags on copy object and add regex for validating tags	2022-06-20 11:32:58 +03:00
Konstantin Lebedev	a9f6c4b43d	fix ip.bind cmd for docker compose	2022-06-15 16:49:53 +05:00