* fix(volume.list): show one entry per physical disk on multi-disk nodes
DataNodeInfo.DiskInfos is keyed by disk type, so several same-type
physical disks on one node collapse to a single map entry at the master.
volume.list iterated that map directly and reported one "Disk hdd ...
id:0" line per node, hiding the per-disk volume and shard layout. EC
operators on multi-disk volume servers had no way to verify which
physical disk a shard landed on.
Lift the per-physical-disk split into a DiskInfo.SplitByPhysicalDisk()
method on the proto type so consumers outside admin/topology can use
it. Apply it in writeDataNodeInfo so the verbose Disk block shows one
entry per physical disk, ordered by DiskId. Capacity counters are
split evenly across reconstructed disks since the wire format doesn't
carry per-disk capacity yet.
This is a display-only change. ActiveTopology already did the split on
its own and is now updated to call the shared helper.
* fix(volume.list): preserve totals, count active/remote exactly, dedupe header
Address review feedback on the per-physical-disk split:
- share() truncated remainders so reconstructed per-disk counters could
sum to less than the original aggregate (10 / 3 = 3+3+3). Distribute
the remainder to the lowest disk ids so MaxVolumeCount and
FreeVolumeCount sum exactly back to the node totals.
- ActiveVolumeCount and RemoteVolumeCount are derivable per disk from
the VolumeInfos already grouped by DiskId, so count them exactly
(ReadOnly=false and RemoteStorageName!="" respectively) instead of
approximating with an even split.
- writeDataNodeInfo's per-disk callback fired the DataNode header on
every iteration after the split, so a node with 6 physical disks
emitted 6 DataNode headers. Guard the callback with headerPrinted so
the header still appears at most once per node.
- Sort split disks deterministically using explicit DiskId comparison
to avoid int overflow risk on 32-bit systems.
- Tighten the volume.list test substring to "id:N\n" so unrelated
tokens like "ec volume id:101" don't accidentally match the id:1
needle, and assert the rack callback fires once.
* opt: reduce ShardsInfo memory usage with bitmap and sorted slice
- Replace map[ShardId]*ShardInfo with sorted []ShardInfo slice
- Add ShardBits (uint32) bitmap for O(1) existence checks
- Use binary search for O(log n) lookups by shard ID
- Maintain sorted order for efficient iteration
- Add comprehensive unit tests and benchmarks
Memory savings:
- Map overhead: ~48 bytes per entry eliminated
- Pointers: 8 bytes per entry eliminated
- Total: ~56 bytes per shard saved
Performance improvements:
- Has(): O(1) using bitmap
- Size(): O(log n) using binary search (was O(1), acceptable tradeoff)
- Count(): O(1) using popcount on bitmap
- Iteration: Faster due to cache locality
* refactor: add methods to ShardBits type
- Add Has(), Set(), Clear(), and Count() methods to ShardBits
- Simplify ShardsInfo methods by using ShardBits methods
- Improves code readability and encapsulation
* opt: use ShardBits directly in ShardsCountFromVolumeEcShardInformationMessage
Avoid creating a full ShardsInfo object just to count shards.
Directly cast vi.EcIndexBits to ShardBits and use Count() method.
* opt: use strings.Builder in ShardsInfo.String() for efficiency
* refactor: change AsSlice to return []ShardInfo (values instead of pointers)
This completes the memory optimization by avoiding unnecessary pointer slices and potential allocations.
* refactor: rename ShardsCountFromVolumeEcShardInformationMessage to GetShardCount
* fix: prevent deadlock in Add and Subtract methods
Copy shards data from 'other' before releasing its lock to avoid
potential deadlock when a.Add(b) and b.Add(a) are called concurrently.
The previous implementation held other's lock while calling si.Set/Delete,
which acquires si's lock. This could deadlock if two goroutines tried to
add/subtract each other concurrently.
* opt: avoid unnecessary locking in constructor functions
ShardsInfoFromVolume and ShardsInfoFromVolumeEcShardInformationMessage
now build shards slice and bitmap directly without calling Set(), which
acquires a lock on every call. Since the object is local and not yet
shared, locking is unnecessary and adds overhead.
This improves performance during object construction.
* fix: rename 'copy' variable to avoid shadowing built-in function
The variable name 'copy' in TestShardsInfo_Copy shadowed the built-in
copy() function, which is confusing and bad practice. Renamed to 'siCopy'.
* opt: use math/bits.OnesCount32 and reorganize types
1. Replace manual popcount loop with math/bits.OnesCount32 for better
performance and idiomatic Go code
2. Move ShardSize type definition to ec_shards_info.go for better code
organization since it's primarily used there
* refactor: Set() now accepts ShardInfo for future extensibility
Changed Set(id ShardId, size ShardSize) to Set(shard ShardInfo) to
support future additions to ShardInfo without changing the API.
This makes the code more extensible as new fields can be added to
ShardInfo (e.g., checksum, location, etc.) without breaking the Set API.
* refactor: move ShardInfo and ShardSize to separate file
Created ec_shard_info.go to hold the basic shard types (ShardInfo and
ShardSize) for better code organization and separation of concerns.
* refactor: add ShardInfo constructor and helper functions
Added NewShardInfo() constructor and IsValid() method to better
encapsulate ShardInfo creation and validation. Updated code to use
the constructor for cleaner, more maintainable code.
* fix: update remaining Set() calls to use NewShardInfo constructor
Fixed compilation errors in storage and shell packages where Set() calls
were not updated to use the new NewShardInfo() constructor.
* fix: remove unreachable code in filer backup commands
Removed unreachable return statements after infinite loops in
filer_backup.go and filer_meta_backup.go to fix compilation errors.
* fix: rename 'new' variable to avoid shadowing built-in
Renamed 'new' to 'result' in MinusParityShards, Plus, and Minus methods
to avoid shadowing Go's built-in new() function.
* fix: update remaining test files to use NewShardInfo constructor
Fixed Set() calls in command_volume_list_test.go and
ec_rebalance_slots_test.go to use NewShardInfo() constructor.
Improve EC shards rebalancing logic across racks.
- Favor target shards with less preexisting shards, to ensure a fair distribution.
- Randomize selection when multiple possible target shards are available.
- Add logic to account for replication settings when selecting target shards (currently disabled).
