Commit Graph
157 Commits
Author SHA1 Message Date
Chris Lu 3e5d34dd67 skip md5 validation if Content-MD5 is not provided 2026-01-18 15:04:56 -08:00
Chris LuandGitHub a473278bfa Fix: Fail fast on unsupported volume versions (#8047)
* Fix: Fail fast when initializing volume with Version 0

* Fix: Fail fast when loading unsupported volume version (e.g. 0 or 4)

* Refactor: Use IsSupportedVersion helper function for version validation
2026-01-16 19:19:18 -08:00
9012069bd7 chore: execute goimports to format the code (#7983)
* chore: execute goimports to format the code

Signed-off-by: promalert <promalert@outlook.com>

* goimports -w .

---------

Signed-off-by: promalert <promalert@outlook.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-01-07 13:06:08 -08:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
ec1c27a4b3 storage/needle: add bounds check for WriteNeedleBlob buffer (#7973)
* storage/needle: add bounds check for WriteNeedleBlob buffer

* storage/needle: use int offsets when checking/writing Version3 timestamp

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-01-05 19:21:57 -08:00
Chris LuandGitHub 0ede657a34 fix: return error on size mismatch in ReadNeedleMeta for consistency (#7687)
* fix: return error on size mismatch in ReadNeedleMeta for consistency

When ReadNeedleMeta encounters a size mismatch at offset >= MaxPossibleVolumeSize,
it previously just continued without returning an error, potentially using wrong data.

This fix makes ReadNeedleMeta consistent with ReadBytes (needle_read.go), which
properly returns an error in both cases:
- ErrorSizeMismatch when offset < MaxPossibleVolumeSize (to trigger retry at offset+32GB)
- A descriptive error when offset >= MaxPossibleVolumeSize (after retry failed)

Fixes #7673

* refactor: use more accurate error message for size mismatch
2025-12-09 13:17:18 -08:00
Chris LuGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
cea12ba3c4 fix: prevent makeslice panic in ReadNeedleMeta with corrupted needle (#7675)
* fix: prevent makeslice panic in ReadNeedleMeta with corrupted needle

When a needle's DataSize in the .dat file is corrupted to a very large
value, the calculation of metaSize can become negative, causing a panic
with 'makeslice: len out of range' when creating the metadata slice.

This fix adds validation to check if metaSize is negative before
creating the slice, returning a descriptive error instead of panicking.

Fixes #7475

* Update weed/storage/needle/needle_read_page.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-08 22:26:03 -08:00
Chris LuandGitHub 28ac536280 fix: normalize Windows backslash paths in weed admin file uploads (#7636)
fix: normalize Windows backslash paths in file uploads

When uploading files from a Windows client to a Linux server,
file paths containing backslashes were not being properly interpreted as
directory separators. This caused files intended for subdirectories to be
created in the root directory with backslashes in their filenames.

Changes:
- Add util.CleanWindowsPath and util.CleanWindowsPathBase helper functions
  in weed/util/fullpath.go for reusable path normalization
- Use path.Join/path.Clean/path.Base instead of filepath equivalents
  for URL path semantics (filepath is OS-specific)
- Apply normalization in weed admin handlers and filer upload parsing

Fixes #7628
2025-12-05 17:40:32 -08:00
Chris LuandGitHub 69553e5ba6 convert error fromating to %w everywhere (#6995) 2025-07-16 23:39:27 -07:00
chrislu d2be5822a1 refactoring 2025-06-16 22:25:22 -07:00
chrislu e71d681fee refactor 2025-06-11 20:46:13 -07:00
chrislu 7c4d98446b refactor 2025-06-11 20:13:06 -07:00
chrislu f27e195354 refactoring 2025-06-11 20:13:06 -07:00
chrislu 33ecc8442e refactor 2025-06-08 22:11:09 -07:00
chrislu 60f11f6510 add a readme file for volume needle data layout 2025-06-07 15:52:51 -07:00
chrislu 61c4f01e05 refactor 2025-06-06 08:55:32 -07:00
chrislu a489d99333 fix tests 2025-06-06 08:25:04 -07:00
chrislu 7439af0eca refactoring 2025-06-06 01:35:48 -07:00
chrislu cc135c63f7 a bit refactoring 2025-06-06 01:26:54 -07:00
chrislu c4695fc3b3 refactor needle write for different versions 2025-06-06 00:35:13 -07:00
ff878a542d correctly report volume with input/output error to master (#6790)
* correctly capture io error and report to master

* code fix

* check io error by error.Is

---------

Co-authored-by: dongxu_feng <dongxu_feng@intsig.net>
2025-05-15 00:56:43 -07:00
bwlfhuandGitHub 0e08b83521 Fix volume ttl (#6683) 2025-04-02 07:59:21 -07:00
chrislu c9f3448692 ReadAt may return io.EOF t end of file
related to https://github.com/seaweedfs/seaweedfs/issues/6219
2024-11-21 00:37:38 -08:00
chrislu be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
Kevin BulteelandGitHub 4a0dc883cf Fix/parse upload filename (#6241)
* fix: parse filename in PUT + refactor

* fix: master iu public url with http

* fix: better parsing and handle disposition header

* fix: take mime type from file extension if not set
2024-11-15 11:25:18 -08:00
Eugeniy E. MikhailovandGitHub 3003c9e17e added backward compatible CRC check and extra explanatory comments (#6233) 2024-11-12 13:56:14 -08:00
0b00706454 EC volume supports expiration and displays expiration message when executing volume.list (#5895)
* ec volume expire

* volume.list show DestroyTime

* comments

* code optimization

---------

Co-authored-by: xuwenfeng <xuwenfeng1@zto.com>
2024-08-16 00:20:00 -07:00
Konstantin LebedevandGitHub 5ffacbb6ea refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
Kevin BulteelandGitHub a3dab1fcd0 feat: upload files directly without multipart/form-data (#5539) 2024-04-26 16:03:40 -07:00
Konstantin LebedevandGitHub 5189a09de0 [volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
316ae45795 can vacuum volume when size mismatch (#5200)
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-01-15 16:16:46 -08:00
0e8a54f6f6 fix write volume over size MaxPossibleVolumeSize (#5190)
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-01-11 20:23:46 -08:00
Konstantin LebedevandGitHub a7fc723ae0 chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
bd6f2716d3 fix_read_tomb_needle_meta (#4263)
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:26:55 -08:00
51d462f204 ADHOC: volume fsck using append at ns (#3906)
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
6253058d9d ensure monotonic n.AppendAtNs in each place (#3880)
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin LebedevandGitHub 452202f35b ReadNeedleMeta recovery panic occurred (#3883)
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu e3ea544d40 skip reading needle if size is invalid
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
Ryan RussellandGitHub 277976bd76 refactor(storage): readability improvements (#3703)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:17 -07:00
chrislu 2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
b324a6536c ADHOC: add read needle meta grpc (#3581)
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-06 23:51:27 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 36d2756978 volume: avoid writing too much data for range requests in large files
related to https://github.com/chrislusf/seaweedfs/issues/3178
2022-06-14 11:16:47 -07:00
chrislu 11f1e7996b fix test 2022-06-05 18:50:41 -07:00
chrislu 056c480eb0 volume: checksum remove the hashing step in Value() 2022-06-05 15:24:02 -07:00
chrislu ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu 685643d506 needle read into a writer 2022-06-04 19:10:52 -07:00
chrislu 663bc5dc23 add needle reading in chunks 2022-06-04 18:15:39 -07:00
chrislu 18b29f709c refactoring, split file 2022-06-03 23:31:14 -07:00
chrislu 433fde4b18 move error to a separate file
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Konstantin Lebedev 3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00