Commit Graph
7511 Commits
Author SHA1 Message Date
chrislu 0778298f5e simplify 2025-07-15 08:20:33 -07:00
chrislu 21aaddf665 fix tests 2025-07-15 08:10:42 -07:00
chrislu 28ca4cf635 remove unused functions 2025-07-15 07:57:25 -07:00
chrislu 71aface936 fix tests 2025-07-15 00:12:48 -07:00
chrislu a9f1dd21d3 fix tests 2025-07-15 00:06:46 -07:00
chrislu 925a183203 simplify 2025-07-14 23:56:36 -07:00
76eafeae77 Update weed/s3api/s3api_server.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:53:45 -07:00
aab2df4a5e Update weed/s3api/s3api_object_handlers.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:44:02 -07:00
5e8bbf94dd Update weed/s3api/s3api_bucket_config.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:43:33 -07:00
chrislu 68d9245129 refactor 2025-07-14 23:40:25 -07:00
chrislu 27ec13ca23 fix 2025-07-14 23:26:39 -07:00
43bd62d661 Update weed/s3api/s3api_object_handlers.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:25:08 -07:00
456c3ce717 Update weed/s3api/s3api_object_handlers.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:22:10 -07:00
4d4463afca Update weed/s3api/s3api_bucket_config.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:21:42 -07:00
chrislu b7294b3905 address comment
Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements.
2025-07-14 23:18:56 -07:00
212dab3a34 Update weed/s3api/s3api_bucket_config.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-14 23:14:50 -07:00
chrislu 0a68aaea66 refactor 2025-07-14 23:10:09 -07:00
chrislu c61ae43f96 path building 2025-07-14 23:06:30 -07:00
chrislu d2184282d7 address comments 2025-07-14 22:58:04 -07:00
chrislu b6fbedde4a address comments 2025-07-14 22:48:37 -07:00
chrislu 902a5f689b address comments 2025-07-14 22:25:33 -07:00
chrislu c64c96dac8 fix for tests 2025-07-14 22:23:47 -07:00
chrislu 4630eea10a address comments 2025-07-14 22:14:53 -07:00
chrislu f3ba114909 optimize matchesWildcard 2025-07-14 22:05:55 -07:00
chrislu 7c22a23964 address some comments 2025-07-14 22:03:40 -07:00
chrislu 5ba551c1d8 adding cors support 2025-07-14 21:46:09 -07:00
chrislu 44dfa793d5 Collecting volume locations for volumes before EC encoding
fix https://github.com/seaweedfs/seaweedfs/issues/6963
2025-07-14 12:17:33 -07:00
chrislu a51d993aa9 ensure bucket exists
related to https://github.com/seaweedfs/seaweedfs/issues/6971
2025-07-14 09:55:35 -07:00
chrislu 406aaf7c14 increase upload limit via browser 2025-07-14 08:42:15 -07:00
chrislu 24eff93d9a 3.94 2025-07-13 20:31:31 -07:00
chrislu e7dfc3552c admin ui adds object lock permissions 2025-07-13 20:29:25 -07:00
Chris LuandGitHub 7cb1ca1308 Add policy engine (#6970) 2025-07-13 16:21:36 -07:00
1549ee2e15 implement PubObjectRetention and WORM (#6969)
* implement PubObjectRetention and WORM

* Update s3_worm_integration_test.go

* avoid previous buckets

* Update s3-versioning-tests.yml

* address comments

* address comments

* rename to ExtObjectLockModeKey

* only checkObjectLockPermissions if versioningEnabled

* address comments

* comments

* Revert "comments"

This reverts commit 6736434176.

* Update s3api_object_handlers_skip.go

* Update s3api_object_retention_test.go

* add version id to ObjectIdentifier

* address comments

* add comments

* Add proper error logging for timestamp parsing failures

* address comments

* add version id to the error

* Update weed/s3api/s3api_object_retention_test.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_retention.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* constants

* fix comments

* address comments

* address comment

* refactor out handleObjectLockAvailabilityCheck

* errors.Is ErrBucketNotFound

* better error checking

* address comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-12 21:58:55 -07:00
Chris LuandGitHub 687a6a6c1d Admin UI: Add policies (#6968)
* add policies to UI, accessing filer directly

* view, edit policies

* add back buttons for "users" page

* remove unused

* fix ui dark mode when modal is closed

* bucket view details button

* fix browser buttons

* filer action button works

* clean up masters page

* fix volume servers action buttons

* fix collections page action button

* fix properties page

* more obvious

* fix directory creation file mode

* Update file_browser_handlers.go

* directory permission
2025-07-12 01:13:11 -07:00
chrislu 49d43003e1 show volume size limit on dashboard 2025-07-11 19:37:09 -07:00
Chris LuandGitHub d892538d32 More efficient copy object (#6665)
* it compiles

* refactored

* reduce to 4 concurrent chunk upload

* CopyObjectPartHandler

* copy a range of the chunk data, fix offset size in copied chunks

* Update s3api_object_handlers_copy.go

What the PR Accomplishes:
CopyObjectHandler - Now copies entire objects by copying chunks individually instead of downloading/uploading the entire file
CopyObjectPartHandler - Handles copying parts of objects for multipart uploads by copying only the relevant chunk portions
Efficient Chunk Copying - Uses direct chunk-to-chunk copying with proper volume assignment and concurrent processing (limited to 4 concurrent operations)
Range Support - Properly handles range-based copying for partial object copies

* fix compilation

* fix part destination

* handling small objects

* use mkFile

* copy to existing file or part

* add testing tools

* adjust tests

* fix chunk lookup

* refactoring

* fix TestObjectCopyRetainingMetadata

* ensure bucket name not conflicting

* fix conditional copying tests

* remove debug messages

* add custom s3 copy tests
2025-07-11 18:51:32 -07:00
chrislu 4fcbdc1f61 tweaking dashboard UI 2025-07-11 13:11:39 -07:00
chrislu 3d4a9bdac0 upgrade templ version from v0.3.833 to v0.3.906
// templ: version: v0.3.833
// templ: version: v0.3.906

fix https://github.com/seaweedfs/seaweedfs/issues/6966#issuecomment-3063449163
2025-07-11 13:03:04 -07:00
Chris LuandGitHub 51543bbb87 Admin UI: Add message queue to admin UI (#6958)
* add a menu item "Message Queue"

* add a menu item "Message Queue"
  * move the "brokers" link under it.
  * add "topics", "subscribers". Add pages for them.

* refactor

* show topic details

* admin display publisher and subscriber info

* remove publisher and subscribers from the topic row pull down

* collecting more stats from publishers and subscribers

* fix layout

* fix publisher name

* add local listeners for mq broker and agent

* render consumer group offsets

* remove subscribers from left menu

* topic with retention

* support editing topic retention

* show retention when listing topics

* create bucket

* Update s3_buckets_templ.go

* embed the static assets into the binary

fix https://github.com/seaweedfs/seaweedfs/issues/6964
2025-07-11 10:19:27 -07:00
Ibrahim KonsowaandGitHub 93bbaa1fb4 [Notifications] Support webhook notifications (#6962)
Add webhook notification support
2025-07-10 09:22:05 -07:00
chaletandGitHub 804979d68b [Enhancement] support fix for remote files with command fix (#6961) 2025-07-10 06:13:16 -07:00
Joon Young BaikandGitHub c04b7b411c refactor: Performance and readability improvement on isDefaultPort (#6960) 2025-07-10 05:50:20 -07:00
chrislu 14859f0e8c add mq agent options to server.go 2025-07-09 09:02:26 -07:00
cf5a24983a S3: add object versioning (#6945)
* add object versioning

* add missing file

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* ListObjectVersionsResult is better to show multiple version entries

* fix test

* Update weed/s3api/s3api_object_handlers_put.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update weed/s3api/s3api_object_versioning.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* multiple improvements

* move PutBucketVersioningHandler into weed/s3api/s3api_bucket_handlers.go file
* duplicated code for reading bucket config, versioningEnabled, etc. try to use functions
* opportunity to cache bucket config

* error handling if bucket is not found

* in case bucket is not found

* fix build

* add object versioning tests

* remove non-existent tests

* add tests

* add versioning tests

* skip a new test

* ensure .versions directory exists before saving info into it

* fix creating version entry

* logging on creating version directory

* Update s3api_object_versioning_test.go

* retry and wait for directory creation

* revert add more logging

* Update s3api_object_versioning.go

* more debug messages

* clean up logs, and touch directory correctly

* log the .versions creation and then parent directory listing

* use mkFile instead of touch

touch is for update

* clean up data

* add versioning test in go

* change location

* if modified, latest version is moved to .versions directory, and create a new latest version

 Core versioning functionality: WORKING
TestVersioningBasicWorkflow - PASS
TestVersioningDeleteMarkers - PASS
TestVersioningMultipleVersionsSameObject - PASS
TestVersioningDeleteAndRecreate - PASS
TestVersioningListWithPagination - PASS
 Some advanced features still failing:
ETag calculation issues (using mtime instead of proper MD5)
Specific version retrieval (EOF error)
Version deletion (internal errors)
Concurrent operations (race conditions)

* calculate multi chunk md5

Test Results - All Passing:
 TestBucketListReturnDataVersioning - PASS
 TestVersioningCreateObjectsInOrder - PASS
 TestVersioningBasicWorkflow - PASS
 TestVersioningMultipleVersionsSameObject - PASS
 TestVersioningDeleteMarkers - PASS

* dedupe

* fix TestVersioningErrorCases

* fix eof error of reading old versions

* get specific version also check current version

* enable integration tests for versioning

* trigger action to work for now

* Fix GitHub Actions S3 versioning tests workflow

- Fix syntax error (incorrect indentation)
- Update directory paths from weed/s3api/versioning_tests/ to test/s3/versioning/
- Add push trigger for add-object-versioning branch to enable CI during development
- Update artifact paths to match correct directory structure

* Improve CI robustness for S3 versioning tests

Makefile improvements:
- Increase server startup timeout from 30s to 90s for CI environments
- Add progressive timeout reporting (logs at 30s, full logs at 90s)
- Better error handling with server logs on failure
- Add server PID tracking for debugging
- Improved test failure reporting

GitHub Actions workflow improvements:
- Increase job timeouts to account for CI environment delays
- Add system information logging (memory, disk space)
- Add detailed failure reporting with server logs
- Add process and network diagnostics on failure
- Better error messaging and log collection

These changes should resolve the 'Server failed to start within 30 seconds' issue
that was causing the CI tests to fail.

* adjust testing volume size

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update s3-versioning-tests.yml

* Update s3api_object_versioning.go

* Update Makefile

* do not clean up

* log received version id

* more logs

* printout response

* print out list version response

* use tmp files when put versioned object

* change to versions folder layout

* Delete weed-test.log

* test with mixed versioned and unversioned objects

* remove versionDirCache

* remove unused functions

* remove unused function

* remove fallback checking

* minor

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-09 01:51:45 -07:00
zuzuviewerandGitHub 8fa1a69f8c * Fix undefined http serve behaiver (#6943) 2025-07-07 22:48:12 -07:00
chrislu 39b7e44fb5 embed static assets
fix https://github.com/seaweedfs/seaweedfs/issues/6946
2025-07-07 12:42:13 -07:00
chrislu 592b6a1e98 less aggressive volume server shutdown on same uuid
related to https://github.com/seaweedfs/seaweedfs/issues/5439
2025-07-07 01:22:17 -07:00
chrislu 8493871406 remove sqlite due to GOOS support for freebsd and openbsd 2025-07-06 21:50:08 -07:00
chrislu a7c57caa3f 3.93 2025-07-06 21:06:50 -07:00
fd4154cfed fix: s3 return BadDigest (#6714)
* fix: s3 return BadDigest

* adjust error message checking

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
2025-07-06 14:18:57 -07:00