5d5fcdf07b fix(filer): bound aggregated metadata reads by peer watermarks (#10803)
* fix(filer): watermark-bound aggregated metadata subscription against multi-source merge races

The aggregated metadata subscription (SubscribeMetadata) merges per-filer
sources that become readable at independent paces, but tracks its progress
with a single scalar cursor. Once the cursor passes a timestamp T, anything
a source materializes below T afterwards is silently skipped: a peer
recovering from a stall re-inserts its backlog late (late ring merge), and
a source's flush can land a log file, or a later chunk of the same file,
after a subscriber's disk pass listed the files (late persisted-log
landing). This is the residual documented in #10501.

Bound the subscriber's two read paths by what every source has provably
made visible, each with its own watermark:

- Delivery low-watermark -> in-memory reads. The meta aggregator tracks,
  per subscribed peer (self included), the newest timestamp received on
  that peer's stream - real events, or idle heartbeats (peer streams now
  opt into ClientSupportsIdleHeartbeat). The aggregated ring is complete
  up to the minimum across peers; in-memory reads hold at it.
- Flush low-watermark -> persisted-log reads. Each filer reports its local
  log-buffer flush watermark on its stream: a new flushed_ts_ns response
  field, carried on idle heartbeats and on periodic flush reports (gated
  on ClientSupportsIdleHeartbeat). Disk passes freeze the minimum across
  peers before listing the log files and hold at it; the day-boundary
  cursor jump and the metadata-chunks ref listing are bounded the same
  way, the latter at minute-file granularity.
- Held reads keep the cursor at the last entry actually delivered and
  retry; the retry re-lists the log files, which is what picks up a
  late-landing file. Both watermarks are relaxed by the settled horizon
  (2 x LogFlushInterval) as a liveness escape, so a peer stalled beyond it
  delays subscribers by at most the horizon instead of forever - any loss
  that escape allows was unconditional before.

With reads held at the flush watermark, a disk advance below it is proven
complete on every peer's disk, so the unproven-crossing counter now only
counts crossings the horizon escape allowed past a stalled peer.

Live delivery on the aggregated stream may lag by up to the idle-heartbeat
interval when some peers are quiet; SubscribeLocalMetadata consumers are
unaffected.

* fix(filer): resume evicted aggregated readers from an original-space disk anchor

The aggregated ring rewrites out-of-order peer arrivals to its head, so a
subscriber tailing it advances its cursor in bumped (arrival) timestamps,
while persisted logs keep original timestamps. When a slow reader's unread
window is evicted (e.g. a peer backlog flooding in after a stall) and the
reader falls back to disk, resuming from the bumped cursor skips every
original-space entry below it that memory never delivered - reproduced as
a ~66% silent loss on a 3-filer cluster with one peer's stream frozen for
~70s while the subscriber lagged.

Track a disk anchor: the newest original-space position the stream is
proven complete through. Disk passes advance it directly; contiguous
memory reads advance it to the peers' delivery low-watermark observed
before the read (per-peer streams are ordered, so everything with an
original timestamp at or below that watermark had already arrived and was
delivered). A reader kicked off the ring resumes the disk pass from the
anchor instead of the bumped cursor - redelivering what memory already
sent is within the subscription's at-least-once contract, skipping what
it never sent is not.

* fix(filer): close review findings on the peer-watermark subscription bounds

Four correctness holes found in review, one generated-file cleanup:

- The flush-through claim could assert durability for events still on
  their way into the buffer: an event is timestamped before notification
  work that can block, and only then appended. Track stamped-but-unappended
  events on the Filer (the stamp shares a lock with the reader, and appends
  are bumped monotonically past the buffer head), and cap the reported
  flush watermark just below the oldest in-flight stamp.

- Removing a peer deleted its watermark entries while its stream kept
  running: its next signal recreated the deleted entry, which then pinned
  the low-watermark forever once the stream died. Watermarks now advance
  only for tracked peers, and peer removal cancels the subscription
  context so the stream stops feeding the aggregated buffer promptly.

- The pipelined sender folded flush reports (TsNs 0 reads as far behind)
  into batch Events tails, where the aggregator's nil-notification guard
  dropped them - a busy backlog replay could starve the flush watermark
  until the settled-horizon escape opened a loss window. Control messages
  are now unbatchable on the sender, and the receiver also reads watermark
  state off nested batch entries as belt and braces.

- A give-up skip's cursor was not anchored, so the next eviction rewind
  undid the counted decision and re-entered the same park forever when the
  evicted window carried bumped timestamps. The anchor now follows give-up
  skips; an anchored cursor makes the rewind a no-op and keeps the gap
  machinery's re-arm onto the retained window reachable.

- Regenerated-file churn from a different protoc-gen-go-vtproto version is
  dropped: the vtproto file is upstream's, plus only the flushed_ts_ns
  marshal/size/unmarshal cases in the same generator style.

New tests pin the in-flight floor, the no-resurrection rule for removed
peers, and that control messages are never nested in batches.

* fix(filer): keep a removed peer's watermarks through a grace period

Deleting a peer's watermark entries the moment the master removes it
reopened the loss the watermarks exist to prevent: a filer frozen or
partitioned long enough to miss master heartbeats is removed from the
cluster, its unflushed events still exist, and with its entries gone the
low-watermarks snap forward to the healthy peers - subscribers advance
past the absent peer's window and its late-landing log files are silently
skipped. Reproduced on a 3-filer cluster: freezing two filers for ~70s got
them removed ~28s in, and a catching-up subscriber lost their entire
overlapping window.

Removal now only marks the peer; its watermarks keep participating in the
low-watermarks for a grace period (2 x LogFlushInterval, matching the
subscribe loops' settled horizon, which already bounds a stale watermark's
influence meanwhile). A re-added peer clears the mark and continues its
values monotonically - the flap case costs nothing. A peer that stays gone
is dropped when the grace expires, so a decommission cannot pin the
low-watermarks, and a dropped peer's straggling signals cannot resurrect
its entry.

* fix(filer): cap delivery heartbeats by the in-flight floor; harden stamps

Second review pass on the watermark bounds:

- Idle heartbeats on the local stream claimed delivery-completeness
  through "now" while an event could still sit stamped-but-unappended
  behind blocking notification work. A peer aggregator turns that claim
  into its delivery low-watermark, so it could advance (and anchor
  credits with it) past an event that had not been streamed yet. The
  heartbeat timestamp is now capped just below the oldest in-flight
  stamp, like the flush claim already was.

- In-flight stamps are forced monotonic against the registry's own
  history, so a wall-clock step backwards cannot slip a new stamp under
  an already-sampled floor. The cross-goroutine ordering still shares
  the meta log's global forward-clock assumption; the comments now say
  so instead of overclaiming.

- Duplicate removal notifications no longer refresh a removed peer's
  grace deadline: the first removal time wins, so a decommissioned peer
  cannot sit in the watermark sets forever on repeated updates.

- A failed buffer append clears the event's in-flight stamp on purpose:
  the event is dropped from the change stream entirely (a pre-existing
  defect of the append path, loudly logged), and a watermark waiting for
  it would pin this filer's claims forever. The comments now state the
  decision instead of implying the failure cannot happen.

* docs(filer): tighten the watermark comments

Comment-only: compress the narrative comments added on this branch down
to their load-bearing invariants, and fix one stale sentence (peer
removal no longer deletes the watermark entries immediately). No code
changes.

* fix(filer): subscribe to the local filer before remote peers

Self's events reach the aggregated buffer only through the aggregator's
own subscription to it, but bootstrap only seeded the peers the master
already listed - and self's master registration races that listing, so
the watermark set could hold remote peers without self. Once the remotes
signalled, the low-watermarks would claim completeness for a stream that
was still missing a merge source, letting aggregated subscribers advance
past the local filer's events before its subscription started.

Seed self first, unconditionally: before that the watermark set is empty
(a documented safe state - reads hold at the settled horizon), and after
it the set can never be remotes-only. The later master update for self,
or a duplicate in the listed peers, is a no-op via the already-followed
check in OnPeerUpdate.

* fix(filer): fence watermark claims against wall-clock regression

Record issued heartbeat/flush claims in the in-flight registry and stamp
later events above them, so a backward clock step cannot land an event
under a watermark a peer has already advanced to.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(filer): re-check the buffer head after fencing heartbeat claims

An event appended between the caught-up check and the delivery claim
was covered by the claim but not yet sent on the stream. The claims
fence later stamps, so re-checking the head after them proves every
covered event was already sent before the heartbeat.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(filer): cross the aggregated ring's pre-subscription range only on proof

The eviction gate and the gap proofs read "nothing evicted yet" as "memory
holds everything after the cursor". That is false for the merge-fed
aggregated ring, which is born empty while every peer's history sits on
disk: before the ring's first real eviction, a subscriber whose cursor was
still below the bounded chunk pass's listing stop was served the ring's
earliest entry inclusively, silently skipping the withheld pre-restart
files - and the idle-wait callback credited the delivery low-watermark to
the disk anchor in the same disconnected state.

Mark everything at or below the subscriptions' start as evicted when the
aggregator is built, credit the anchor only once the run is connected to
the ring, and give the aggregated gap pass a real proof to cross the
marked boundary with: each disk pass's proven coverage (the peer flush
low-watermark capped by the pass's listing bound). An empty pass whose
proof reaches the eviction watermark crosses to it silently - no park, no
loss counter - so the mark costs a bounded catch-up delay instead of the
15-minute give-up.

* fix(filer): keep shipped chunk tails at or below the hold point

A log file spans past its named minute (window start plus up to a flush
interval), and chunk-mode clients apply a shipped file whole - so a file
tail past the hold point can become a persisted client checkpoint beyond
what every peer has proven, and a crash inside that window resumes past
another peer's late-but-in-contract flush. Stop the ref listing a minute
plus a flush interval below the hold; the withheld band is served by the
memory pass (ring retention far exceeds it) or by later passes as the
hold advances, so freshness is unchanged. A frozen peer flushing one
window that spans its whole freeze can still overshoot; that residual is
bounded by the freeze and needs a crash inside it.

* docs(filer): trim the review-fix comments

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-08-19 18:38:46 -07:00
2026-08-17 16:11:27 -07:00
2026-08-17 15:39:20 -07:00
2026-08-17 15:39:20 -07:00
2023-01-05 11:01:22 -08:00

SeaweedFS

Slack Twitter Build Status GoDoc Wiki Docker Pulls SeaweedFS on Maven Central Artifact Hub

SeaweedFS Logo

Sponsor SeaweedFS via Patreon

SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible entirely thanks to the support of these awesome backers. If you'd like to grow SeaweedFS even stronger, please consider joining our sponsors on Patreon.

Your support will be really appreciated by me and other supporters!

Gold Sponsors

nodion piknik keepsec zyner


Table of Contents

Quick Start

Quick Start with weed mini

Download the latest binary from https://github.com/seaweedfs/seaweedfs/releases and unzip the single weed (or weed.exe) file, or run go install github.com/seaweedfs/seaweedfs/weed@latest. Then start a ready-to-use S3 object store with credentials and a pre-created bucket in one command:

AWS_ACCESS_KEY_ID=admin \
AWS_SECRET_ACCESS_KEY=secret \
S3_BUCKET=my-bucket \
./weed mini -dir=/data

That's it — the S3 endpoint is at http://localhost:8333, my-bucket already exists, and admin/secret are valid credentials. S3_BUCKET accepts a comma-separated list (e.g. raw,processed); use S3_TABLE_BUCKET for S3 Tables (Iceberg) buckets. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).

The same command starts everything else too:

macOS: if the binary is quarantined, run xattr -d com.apple.quarantine ./weed first.

Perfect for development, testing, learning SeaweedFS, and single-node deployments. To scale out, add more volume servers by running weed volume -dir="/some/data/dir2" -master="<master_host>:9333" -port=8081 locally, on another machine, or on thousands of machines.

Quick Start for S3 API on Docker

docker run -p 8333:8333 \
  -e AWS_ACCESS_KEY_ID=admin \
  -e AWS_SECRET_ACCESS_KEY=secret \
  -e S3_BUCKET=my-bucket \
  chrislusf/seaweedfs

Same behavior as the weed mini command above — the S3 endpoint is at http://localhost:8333 with my-bucket pre-created. Drop the env vars to run anonymously for development.

Introduction

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:

  1. to store billions of files!
  2. to serve the files fast!

SeaweedFS started as a blob store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages volumes on volume servers, and these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation).

There is only 40 bytes of disk storage overhead for each file's metadata. It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.

SeaweedFS started by implementing Facebook's Haystack design paper. Also, SeaweedFS implements erasure coding with ideas from f4: Facebooks Warm BLOB Storage System, and has a lot of similarities with Facebooks Tectonic Filesystem and Google's Colossus File System

On top of the blob store, optional Filer can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql, Postgres, Redis, Cassandra, HBase, Mongodb, Elastic Search, LevelDB, RocksDB, Sqlite, MemSql, TiDB, Etcd, CockroachDB, YDB, etc.

SeaweedFS can transparently integrate with the cloud. With hot data on local cluster, and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity. What's more, the cloud storage access API cost is minimized. Faster and cheaper than direct cloud storage!

SeaweedFS also ships a built-in Iceberg REST Catalog, turning the same cluster into a self-contained lakehouse. Spark, Trino, Dremio, DuckDB, and RisingWave can query Iceberg tables directly — no Hive Metastore, Glue, or external catalog service required. Storage and table metadata live in one system, simplifying on-prem and small-team analytics stacks.

Back to TOC

Features

Additional Blob Store Features

  • Support different replication levels, with rack and data center aware.
  • Automatic master servers failover - no single point of failure (SPOF).
  • Automatic compression depending on file MIME type.
  • Automatic compaction to reclaim disk space after deletion or update.
  • Automatic entry TTL expiration.
  • Flexible Capacity Expansion: Any server with some disk space can add to the total storage space.
  • Adding/Removing servers does not cause any data re-balancing unless triggered by admin commands.
  • Optional picture resizing.
  • Support ETag, Accept-Range, Last-Modified, etc.
  • Support in-memory/leveldb/readonly mode tuning for memory/performance balance.
  • Support rebalancing the writable and readonly volumes.
  • Customizable Multiple Storage Tiers: Customizable storage disk types to balance performance and cost.
  • Transparent cloud integration: unlimited capacity via tiered cloud storage for warm data.
  • Erasure Coding for warm storage Rack-Aware 10.4 erasure coding reduces storage cost and increases availability. Enterprise version can customize EC ratio.

Back to TOC

Filer Features

Data Lakehouse Features

Kubernetes

Back to TOC

Example: Using Seaweed Blob Store

By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.

SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.

Start Master Server

> ./weed master

Start Volume Servers

> weed volume -dir="/tmp/data1" -max=5  -master="localhost:9333" -port=8080 &
> weed volume -dir="/tmp/data2" -max=10 -master="localhost:9333" -port=8081 &

Write A Blob

A blob, also referred as a needle, a chunk, or mistakenly as a file, is just a byte array. It can have attributes, such as name, mime type, create or update time, etc. But basically it is just a byte array of a relatively small size, such as 2 MB ~ 64 MB. The size is not fixed.

To upload a blob: first, send a HTTP POST, PUT, or GET request to /dir/assign to get an fid and a volume server URL:

> curl http://localhost:9333/dir/assign
{"count":1,"fid":"3,01637037d6","url":"127.0.0.1:8080","publicUrl":"localhost:8080"}

Second, to store the blob content, send a HTTP multi-part POST request to url + '/' + fid from the response:

> curl -F file=@/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6
{"name":"myphoto.jpg","size":43234,"eTag":"1cc0118e"}

To update, send another POST request with updated blob content.

For deletion, send an HTTP DELETE request to the same url + '/' + fid URL:

> curl -X DELETE http://127.0.0.1:8080/3,01637037d6

Save Blob Id

Now, you can save the fid, 3,01637037d6 in this case, to a database field.

The number 3 at the start represents a volume id. After the comma, it's one file key, 01, and a file cookie, 637037d6.

The volume id is an unsigned 32-bit integer. The file key is an unsigned 64-bit integer. The file cookie is an unsigned 32-bit integer, used to prevent URL guessing.

The file key and file cookie are both coded in hex. You can store the <volume id, file key, file cookie> tuple in your own format, or simply store the fid as a string.

If stored as a string, in theory, you would need 8+1+16+8=33 bytes. A char(33) would be enough, if not more than enough, since most uses will not need 2^32 volumes.

If space is really a concern, you can store the file id in the binary format. You would need one 4-byte integer for volume id, 8-byte long number for file key, and a 4-byte integer for the file cookie. So 16 bytes are more than enough.

Read a Blob

Here is an example of how to render the URL.

First look up the volume server's URLs by the file's volumeId:

> curl http://localhost:9333/dir/lookup?volumeId=3
{"volumeId":"3","locations":[{"publicUrl":"localhost:8080","url":"localhost:8080"}]}

Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.

Now you can take the public URL, render the URL or directly read from the volume server via URL:

 http://localhost:8080/3,01637037d6.jpg

Notice we add a file extension ".jpg" here. It's optional and just one way for the client to specify the file content type.

If you want a nicer URL, you can use one of these alternative URL formats:

 http://localhost:8080/3/01637037d6/my_preferred_name.jpg
 http://localhost:8080/3/01637037d6.jpg
 http://localhost:8080/3,01637037d6.jpg
 http://localhost:8080/3/01637037d6
 http://localhost:8080/3,01637037d6

If you want to get a scaled version of an image, you can add some params:

http://localhost:8080/3/01637037d6.jpg?height=200&width=200
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fit
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fill

Rack-Aware and Data Center-Aware Replication

SeaweedFS applies the replication strategy at a volume level. So, when you are getting a blob id, you can specify the replication strategy. For example:

curl http://localhost:9333/dir/assign?replication=001

The replication parameter options are:

000: no replication
001: replicate once on the same rack
010: replicate once on a different rack, but same data center
100: replicate once on a different data center
200: replicate twice on two different data center
110: replicate once on a different rack, and once on a different data center

More details about replication can be found on the wiki.

You can also set the default replication strategy when starting the master server.

Allocate Blob Key on Specific Data Center

Volume servers can be started with a specific data center name:

 weed volume -dir=/tmp/1 -port=8080 -dataCenter=dc1
 weed volume -dir=/tmp/2 -port=8081 -dataCenter=dc2

When requesting a blob key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specifies that the assigned volume should be limited to 'dc1':

 http://localhost:9333/dir/assign?dataCenter=dc1

Other Features

Back to TOC

Blob Store Architecture

Usually distributed file systems split each file into chunks. A central server keeps a mapping of filenames to chunks, and also which chunks each chunk server has.

The main drawback is that the central server can't handle many small files efficiently, and since all read requests need to go through the central master, so it might not scale well for many concurrent users.

Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is 32GB in size, and can hold a lot of blobs. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is a fairly small amount of data and is generally stable.

The actual blob metadata, which are the blob volume, offset, and size, is stored in each volume on volume servers. Since each volume server only manages metadata of blobs on its own disk, with only 16 bytes for each blob, all access can read the metadata just from memory and only needs one disk operation to actually read file data.

For comparison, consider that an xfs inode structure in Linux is 536 bytes.

Master Server and Volume Server

The architecture is fairly simple. The actual data is stored in volumes on storage nodes. One volume server can have multiple volumes, and can both support read and write access with basic authentication.

All volumes are managed by a master server. The master server contains the volume id to volume server mapping. This is fairly static information, and can be easily cached.

On each write request, the master server also generates a file key, which is a growing 64-bit unsigned integer. Since write requests are not generally as frequent as read requests, one master server should be able to handle the concurrency well.

Write and Read files

When a client sends a write request, the master server returns (volume id, file key, file cookie, volume node URL) for the blob. The client then contacts the volume node and POSTs the blob content.

When a client needs to read a blob based on (volume id, file key, file cookie), it asks the master server by the volume id for the (volume node URL, volume node public URL), or retrieves this from a cache. Then the client can GET the content, or just render the URL on web pages and let browsers fetch the content.

Saving memory

All blob metadata stored on a volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.

Tiered Storage to the cloud

The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.

Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.

With the O(1) access time, the network latency cost is kept at minimum.

If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.

Back to TOC

SeaweedFS Filer

Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases.

The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.

Compared to Other File Systems

Most other distributed file systems seem more complicated than necessary.

SeaweedFS is meant to be fast and simple, in both setup and operation. If you do not understand how it works when you reach here, we've failed! Please raise an issue with any questions or update this file with clarifications.

SeaweedFS is constantly moving forward. Same with other systems. These comparisons can be outdated quickly. Please help to keep them updated.

Back to TOC

Compared to HDFS

HDFS uses the chunk approach for each file, and is ideal for storing large files.

SeaweedFS is ideal for serving relatively smaller files quickly and concurrently.

SeaweedFS can also store extra large files by splitting them into manageable data chunks, and store the file ids of the data chunks into a meta chunk. This is managed by "weed upload/download" tool, and the weed master or volume servers are agnostic about it.

Back to TOC

Compared to GlusterFS, Ceph

The architectures are mostly the same. SeaweedFS aims to store and read files fast, with a simple and flat architecture. The main differences are

  • SeaweedFS optimizes for small files, ensuring O(1) disk seek operation, and can also handle large files.
  • SeaweedFS statically assigns a volume id for a file. Locating file content becomes just a lookup of the volume id, which can be easily cached.
  • SeaweedFS Filer metadata store can be any well-known and proven data store, e.g., Redis, Cassandra, HBase, Mongodb, Elastic Search, MySql, Postgres, Sqlite, MemSql, TiDB, CockroachDB, Etcd, YDB etc, and is easy to customize.
  • SeaweedFS Volume server also communicates directly with clients via HTTP, supporting range queries, direct uploads, etc.
System File Metadata File Content Read POSIX REST API Optimized for large number of small files
SeaweedFS lookup volume id, cacheable O(1) disk seek Yes Yes
SeaweedFS Filer Linearly Scalable, Customizable O(1) disk seek FUSE Yes Yes
GlusterFS hashing FUSE, NFS
Ceph hashing + rules FUSE Yes
MooseFS in memory FUSE No
MinIO separate meta file per drive for each file Yes No
RustFS separate meta file per drive for each file Yes No

Back to TOC

Compared to GlusterFS

GlusterFS stores files, both directories and content, in configurable volumes called "bricks".

GlusterFS hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".

Back to TOC

Compared to MooseFS

MooseFS chooses to neglect small file issue. From moosefs 3.0 manual, "even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header", because it "was initially designed for keeping large amounts (like several thousands) of very big files"

MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.

Back to TOC

Compared to Ceph

Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. Here is a more detailed comparison

SeaweedFS has a centralized master group to look up free volumes, while Ceph uses hashing and metadata servers to locate its objects. Having a centralized master makes it easy to code and manage.

Ceph, like SeaweedFS, is based on the object store RADOS. Ceph is rather complicated with mixed reviews.

Ceph uses CRUSH hashing to automatically manage data placement, which is efficient to locate the data. But the data has to be placed according to the CRUSH algorithm. Any wrong configuration would cause data loss. Topology changes, such as adding new servers to increase capacity, will cause data migration with high IO cost to fit the CRUSH algorithm. SeaweedFS places data by assigning them to any writable volumes. If writes to one volume failed, just pick another volume to write. Adding more volumes is also as simple as it can be.

SeaweedFS is optimized for small files. Small files are stored as one continuous block of content, with at most 8 unused bytes between files. Small file access is O(1) disk read.

SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mongodb, Redis, Elastic Search, Cassandra, HBase, MemSql, TiDB, CockroachCB, Etcd, YDB, to manage file directories. These stores are proven, scalable, and easier to manage.

SeaweedFS comparable to Ceph advantage
Master MDS simpler
Volume OSD optimized for small files
Filer Ceph FS linearly scalable, Customizable, O(1) or O(logN)

Back to TOC

Compared to MinIO, RustFS

Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs. RustFS is a MinIO reimplementation in Rust, Apache 2.0 licensed and still developed, keeping MinIO's storage model down to a byte-compatible on-disk format. So the points below apply to both.

MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.

The metadata are in simple files. Each file write incurs extra writes to the corresponding meta file, on every drive of the erasure set. Changing only tags or retention rewrites that meta file on all of them, so the write amplification does not shrink with object size.

There is no optimization for lots of small files. The files are simply stored as is to local disks. Plus the extra meta file and shards for erasure coding, it only amplifies the LOSF problem.

Multiple disk IO are needed to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.

Erasure coding is full-time. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.

No POSIX-like API support.

There are specific requirements on storage layout, which makes it hard to scale out and to maintain. An erasure set must be 2 to 16 drives and must divide the drive list symmetrically, and capacity grows or shrinks a whole pool at a time. In SeaweedFS, just start one volume server pointing to the master. That's all.

Back to TOC

Dev Plan

  • More tools and documentation, on how to manage and scale the system.
  • Read and write stream data.
  • Support structured data.

This is a super exciting project! And we need helpers and support!

Back to TOC

Installation Guide

Installation guide for users who are not familiar with golang

Step 1: install go on your machine and setup the environment by following the instructions at:

https://golang.org/doc/install

make sure to define your $GOPATH

Step 2: checkout this repo:

git clone https://github.com/seaweedfs/seaweedfs.git

Step 3: download, compile, and install the project by executing the following command

cd seaweedfs/weed && make install

Once this is done, you will find the executable "weed" in your $GOPATH/bin directory

For more installation options, including how to run with Docker, see the Getting Started guide.

Back to TOC

Hard Drive Performance

When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.

Solid State Disk

To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.

Back to TOC

Benchmark

My Own Unscientific Single Machine Results on Mac Book with Solid State Disk, CPU: 1 Intel Core i7 2.6GHz.

Write 1 million 1KB file:

Concurrency Level:      16
Time taken for tests:   66.753 seconds
Completed requests:      1048576
Failed requests:        0
Total transferred:      1106789009 bytes
Requests per second:    15708.23 [#/sec]
Transfer rate:          16191.69 [Kbytes/sec]

Connection Times (ms)
              min      avg        max      std
Total:        0.3      1.0       84.3      0.9

Percentage of the requests served within a certain time (ms)
   50%      0.8 ms
   66%      1.0 ms
   75%      1.1 ms
   80%      1.2 ms
   90%      1.4 ms
   95%      1.7 ms
   98%      2.1 ms
   99%      2.6 ms
  100%     84.3 ms

Randomly read 1 million files:

Concurrency Level:      16
Time taken for tests:   22.301 seconds
Completed requests:      1048576
Failed requests:        0
Total transferred:      1106812873 bytes
Requests per second:    47019.38 [#/sec]
Transfer rate:          48467.57 [Kbytes/sec]

Connection Times (ms)
              min      avg        max      std
Total:        0.0      0.3       54.1      0.2

Percentage of the requests served within a certain time (ms)
   50%      0.3 ms
   90%      0.4 ms
   98%      0.6 ms
   99%      0.7 ms
  100%     54.1 ms

Run WARP and launch a mixed benchmark.

make benchmark
warp: Benchmark data written to "warp-mixed-2025-12-05[194844]-kBpU.csv.zst"

Mixed operations.
Operation: DELETE, 10%, Concurrency: 20, Ran 42s.
 * Throughput: 55.13 obj/s

Operation: GET, 45%, Concurrency: 20, Ran 42s.
 * Throughput: 2477.45 MiB/s, 247.75 obj/s

Operation: PUT, 15%, Concurrency: 20, Ran 42s.
 * Throughput: 825.85 MiB/s, 82.59 obj/s

Operation: STAT, 30%, Concurrency: 20, Ran 42s.
 * Throughput: 165.27 obj/s

Cluster Total: 3302.88 MiB/s, 550.51 obj/s over 43s.

Back to TOC

Enterprise

For enterprise users, please visit seaweedfs.com for the SeaweedFS Enterprise Edition, which has advanced features, including data recovery, self-healing storage, customizable erasure coding, EC vacuum and repair, etc.

Back to TOC

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The text of this page is available for modification and reuse under the terms of the Creative Commons Attribution-Sharealike 3.0 Unported License and the GNU Free Documentation License (unversioned, with no invariant sections, front-cover texts, or back-cover texts).

Back to TOC

Stargazers over time

Stargazers over time

S
Description
No description provided
Readme Apache-2.0
428 MiB
Languages
Go 83.8%
Rust 6.9%
templ 3.1%
Java 2.1%
Shell 1.5%
Other 2.4%