mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-07-29 03:23:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,512 Commits 156 Branches 353 Tags
refactor-volume-write
Commit Graph
921 Commits
Author SHA1 Message Date
chrislu 33ecc8442e refactor 2025-06-08 22:11:09 -07:00
chrislu 60f11f6510 add a readme file for volume needle data layout 2025-06-07 15:52:51 -07:00
chrislu 61c4f01e05 refactor 2025-06-06 08:55:32 -07:00
chrislu a489d99333 fix tests 2025-06-06 08:25:04 -07:00
Lisandro PinandGitHub 00c621abb8 Fix dumb typo in 08556257 (#6844) 2025-06-06 05:59:11 -07:00
chrislu 7439af0eca refactoring 2025-06-06 01:35:48 -07:00
chrislu cc135c63f7 a bit refactoring 2025-06-06 01:26:54 -07:00
chrislu c4695fc3b3 refactor needle write for different versions 2025-06-06 00:35:13 -07:00
Lisandro PinandGitHub bed0a64693 New needle_map.CompactMap() implementation for reduced memory usage (#6842) 
* Rework `needle_map.CompactMap()` to maximize memory efficiency.

* Use a memory-efficient structure for `CompactMap` needle value entries.

This slightly complicates the code, but makes a **massive** difference
in memory efficiency - preliminary results show a ~30% reduction in
heap usage, with no measurable performance impact otherwise.

* Clean up type for `CompactMap` chunk IDs.

* Add a small comment description for `CompactMap()`.

* Add the old version of `CompactMap()` for comparison purposes.
 2025-06-05 14:03:29 -07:00
chrislu 35f0daa198 the isFsync parameter is essentially IsAsyncWrite and it needs to be turned off if s.isStopping 
https://github.com/seaweedfs/seaweedfs/commit/d8c574a5ef1a811f9a0d447097d9edfcc0c1d84c#r159132764
 2025-06-05 00:19:10 -07:00
chrislu bd4891a117 change version directory 2025-06-03 22:46:10 -07:00
Chris Lu 7151a54b28 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2025-06-02 23:57:54 -07:00
Chris Lu b25561d0d7 3.89 2025-06-02 23:56:58 -07:00
Chris LuandGitHub d40746f34e fix insert beyond look back window (#6838) 2025-06-02 23:43:01 -07:00
Lisandro PinandGitHub 7204731749 Minor fix for the CompactMap() performance test. (#6836) 
Per-entry memory usage is based on `TotalAllocs`, which is incorrect - that
value is a cummulative of heap usage, which doesn't decrease when objects
are freeed.

`Allocs` is instead an accurate represeentation of actual memory usage
at the time metrics are reported.
 2025-06-02 17:09:01 -07:00
Chris LuandGitHub 90802cb201 revert part of d8c574a5ef (#6829) 2025-06-01 12:27:49 -07:00
Lisandro PinandGitHub 9ffc8bcb54 Further improve memory usage of needle_map.CompactMap(). (#6825) 2025-05-28 11:42:00 -07:00
Lisandro PinandGitHub 2e1506c31e Rewrite needle_map.CompactMap() for more efficient memory usage (#6813) 2025-05-23 07:05:08 -07:00
dongxufengGitHubdongxu_feng
 		ff878a542d correctly report volume with input/output error to master (#6790) 
* correctly capture io error and report to master

* code fix

* check io error by error.Is

---------

Co-authored-by: dongxu_feng <dongxu_feng@intsig.net>
 2025-05-15 00:56:43 -07:00
chrislu d8c574a5ef fix fsync logic 2025-05-14 01:33:36 -07:00
Lisandro PinandGitHub dddb0f0ae5 Fix update of SeaweedFS_volumeServer_volumes gauge metrics when EC shards are unmounted (#6776) 2025-05-09 10:15:34 -07:00
Quentin D.andGitHub 2ae5b480a6 Use the correct constant when computing the offset in SearchNeedleFromSortedIndex (#6771) 
NeedleHeaderSize happen to have the same size as NeedleMapEntrySize, except when running the 5 bytes offset variant of Seaweedfs, because it does not contain OffsetSize. This causes ECX corruption on deletes, due to the drifting offset computation (offset is always computed on a basis of 16 bytes per record instead of 17 bytes)

Signed-off-by: Quentin Devos <4972091+Okhoshi@users.noreply.github.com>
 2025-05-09 08:47:53 -07:00
bwlfhuandGitHub 0e08b83521 Fix volume ttl (#6683) 2025-04-02 07:59:21 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>chrislu
 		216c52e377 chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679) 
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0

Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: gocloud.dev
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix error

* fix printing errors

* Update go.mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-03-31 21:42:54 -07:00
NinaLuaandGitHub 271505717e chore: make function comment match function name (#6607) 2025-03-07 00:45:47 -08:00
Aleksey KosovandGitHub ef4eda0761 added re-generating and writing the Volume UUID if it is empty (#6568) 2025-02-24 07:58:43 -08:00
Lisandro PinandGitHub 7913681297 ec.encode: Display a warning on EC balancing if no replica placement settings are found. (#6487) 2025-01-29 08:50:19 -08:00
chrislu be15fee8e7 show error if backend is mis-configured 
related to https://github.com/seaweedfs/seaweedfs/discussions/6472
 2025-01-23 09:18:54 -08:00
ludwigxiaandGitHub 5452405a81 cloud tier: support for Alibaba Cloud OSS (#6466) 2025-01-22 00:48:55 -08:00
chrislu de09a82a32 go fmt 2024-12-19 19:25:27 -08:00
chrislu bfd59dd579 adjust import 2024-12-19 19:25:06 -08:00
chrislu ec155022e7 "golang.org/x/exp/slices" => "slices" and go fmt 2024-12-19 19:25:06 -08:00
coffeecloudgitandGitHub c531e2eb0e Update disk_location_ec.go (#6359) 
Add lock for l.ecVolumes
 2024-12-15 22:05:47 -08:00
coffeecloudgitandGitHub 7eb8257126 Update store.go (#6358) 
Add lock for location.ecVolumes And single delete
 2024-12-15 22:05:22 -08:00
Konstantin LebedevandGitHub ff1392f7f4 [shell] use constant for hdd of type (#6337) 
use constant for hdd of type
 2024-12-10 08:43:59 -08:00
chrislu f3dde99796 adjust error message 2024-12-05 09:33:50 -08:00
Trim21andGitHub fd6c787c02 fix EcVolumes sorting in volume UI (#6275) 
* Update store_ec.go

* fix: EcVolume sorting not work

* use stdlib

* revert
 2024-11-23 22:43:08 -08:00
chrislu 75f5afa571 fix compilation 2024-11-21 00:41:22 -08:00
chrislu 1c143cfcea display actual offset value 2024-11-21 00:38:03 -08:00
chrislu c9f3448692 ReadAt may return io.EOF t end of file 
related to https://github.com/seaweedfs/seaweedfs/issues/6219
 2024-11-21 00:37:38 -08:00
chrislu be09d08eb6 adjust error message 2024-11-20 23:49:01 -08:00
chrislu 96f1a2e01a adjust error message 2024-11-19 14:16:06 -08:00
Lisandro PinandGitHub f2db746690 Introduce logic to resolve volume replica placement within EC rebalancing. (#6254) 
* Rename `command_ec_encode_test.go` to `command_ec_common_test.go`.

All tests defined in this file are now for `command_ec_common.go`.

* Minor code cleanups.

- Fix broken `ec.balance` test.
- Rework integer ceiling division to not use floats, which can introduce precision errors.

* Introduce logic to resolve volume replica placement within EC rebalancing.

This will be used to make rebalancing logic topology-aware.

* Give shell.EcNode.dc a dedicated DataCenterId type.
 2024-11-18 18:05:06 -08:00
Kevin BulteelandGitHub 4a0dc883cf Fix/parse upload filename (#6241) 
* fix: parse filename in PUT + refactor

* fix: master iu public url with http

* fix: better parsing and handle disposition header

* fix: take mime type from file extension if not set
 2024-11-15 11:25:18 -08:00
Eugeniy E. MikhailovandGitHub 3003c9e17e added backward compatible CRC check and extra explanatory comments (#6233) 2024-11-12 13:56:14 -08:00
chrislu 9873b033d1 backward compatible vif loading 2024-10-28 19:44:30 -07:00
chrislu ae5bd0667a rename proto field from DestroyTime to expire_at_sec 
For TTL volume converted into EC volume, this change may leave the volumes staying.
 2024-10-24 21:35:11 -07:00
chrislu d38456dc06 do not delete collection metrics when unmount a volume 
fix https://github.com/seaweedfs/seaweedfs/issues/6139
 2024-10-22 00:44:52 -07:00
dsdGitHub邓书东Chris Lu
 		1e13b6879c fix(volume): to avoid duplicate write a same needle (#6138) 
fix WriteNeedleBlob to avoid duplicate write a same needle

Co-authored-by: 邓书东 <shudong_deng@hhnb2024010108.intsig.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2024-10-18 01:20:50 -07:00
chrislu f11567816e Revert "fix: volume data integrity check (#6118)" 
This reverts commit e12b9ab746.
 2024-10-17 18:52:30 -07:00