* admin: add visual iam policy editor Add a structured, tabbed editor (Editor / JSON) for creating and editing IAM policies in the admin dashboard, alongside the existing raw-JSON textarea: - policies.templ: per-statement cards for Sid, Effect, Action, and Resource, with unmanaged fields (Principal, NotPrincipal, NotResource, Condition, or anything else) preserved verbatim in a per-statement "advanced fields" JSON box so nothing is lost on round-trip. Switching tabs commits and reparses in both directions. Restored the "Use Sample Policy" button, now filling both the structured editor and the JSON tab. The "Validate" button now calls the existing but previously unused POST /api/object-store/policies/validate endpoint instead of doing JS-only checks. - Progressive Resource ARN autocomplete: suggests bucket names first, then once "bucket/" is typed, suggests bucket/* plus the bucket's direct subfolders, drilling down one path segment at a time as the user types further "/" characters. - New GET /api/files/list-folders endpoint (file_browser_handlers.go) backing the folder autocomplete: wraps the existing file browser data function and returns just the subdirectory names as JSON, scoped to paths under /buckets. - Action-name suggestions (datalist) for the Action field, sourced from the existing s3_constants.S3_ACTION_* constants plus new s3_constants.S3TABLES_ACTION_* constants (extracted from the s3tables operation dispatch switch) so the suggestion list can't drift from the strings the engines actually understand. - policy_handlers.go: ValidatePolicy now accepts a statement with only NotResource set (previously required Resource), matching policy_engine.validateStatement and the fact the new editor makes such statements reachable from the UI. - Tests: ValidatePolicy behavior, route registration for the policy API and the new list-folders endpoint, list-folders path scoping, and the action-suggestion list's shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: fix XSS, cache poisoning, and cap overshoot in policy editor Address code review findings on the IAM policy editor added in the previous commit: - policies.templ (displayPolicyDetails): escape every interpolated policy value (Sid, Effect, Action, Resource, policy name, and the raw JSON document) before assigning to innerHTML. Policy documents can come from other admins or an import, so an unescaped field could execute script when the "View" modal renders it. - policies.templ (policyEditorStateToDoc): reject JSON arrays in a statement's "advanced fields" box, not just invalid JSON. `typeof [] === 'object'` was true, so a JSON array was assigned to the statement; subsequent property assignments (Sid, Effect, ...) landed on the array object but JSON.stringify of an array only serializes numeric indices, silently dropping them. - policies.templ (loadPolicyFolderNames): on a failed folder lookup, remove the cache entry instead of permanently caching the empty fallback, so a transient network/server error doesn't block retries for the rest of the page's lifetime. - file_browser_handlers.go (ListFolders): stop appending directory names as soon as the running count reaches maxListFoldersEntries, instead of only checking the cap after a full page is processed, so the returned list never exceeds the configured cap. Regenerated policies_templ.go with the already-stamped templ v0.3.1001 to keep the diff scoped to this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: stop policy editor from clobbering the active tab and dropping malformed advanced fields Address two review findings on the IAM policy editor (Issue 3, stored-XSS in displayPolicyDetails, was already fixed by the previous commit and is unchanged here): - createPolicy, updatePolicy, and validatePolicyDocument always committed the structured editor's (possibly stale) state into the JSON textarea before submitting, even when the user had just edited the JSON tab directly. That silently discarded the user's JSON edits and validated/saved the old structured-editor state instead, which could leave broader permissions in force than intended. Added commitPolicyActiveTab(which), which commits whichever tab is currently visible into the other side instead of unconditionally overwriting the JSON tab from the editor: if the JSON tab is active it parses that JSON back into the structured editor (without touching the textarea itself), otherwise it serializes the structured editor into the textarea as before. All three call sites, plus the JSON-tab "show.bs.tab" handler, now use this and abort with an alert if the currently active tab's content can't be committed. - policyEditorStateToDoc silently continued with an empty object when a statement's "advanced fields" box held invalid JSON, so switching tabs, validating, or saving would drop Principal/NotResource/Condition from that statement without telling the user. It now throws (with the statement number and parse error) on invalid or non-object JSON there, and callers surface that via showAlert and abort instead of proceeding. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: keep unmanaged top-level policy fields across editor tab switches policyDocToEditorState only carried Version and Statement into editor state, so any other top-level key (e.g. Id) present in the JSON tab was silently rewritten away as soon as the user switched to the Editor tab and back. Capture those keys in state.otherFields and merge them back in policyEditorStateToDoc before Version and Statement are written, so the two tabs stay faithful to each other and the editor never rewrites text the user typed. Note this is editor fidelity only: the admin API's policy_engine.PolicyDocument carries just Version and Statement, and DocumentJSON is never populated, so such fields are still discarded by the server once a policy is saved. Making them survive a save would require a backend change, which is out of scope here. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: warn before a policy save discards unsupported top-level fields The editor round-trips unmanaged top-level keys (e.g. Id) between the Editor and JSON tabs, but the admin API's policy_engine.PolicyDocument carries only Version and Statement, so the server drops them on save and the user saw no indication. Added confirmPolicyFieldDiscard(), called from createPolicy and updatePolicy after the active tab is committed (so the field list is accurate whichever tab is showing). It names the fields that will be lost and lets the user confirm or cancel. Not wired into validatePolicyDocument, which doesn't persist anything. Chose the warning over the alternative of persisting these fields through the backend: policy_engine.PolicyDocument is shared by the S3 bucket-policy engine and IAM evaluation, so extending it would change the stored document shape for every policy in the codebase - far beyond the scope of this editor. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: reject malformed Effect and Resource/NotResource conflicts in policy editor Two review findings on the IAM policy editor: - policyDocToEditorState defaulted any non-"Deny" Effect (missing, misspelled, wrong case) to "Allow". A statement meant to be "Deny" with a typo like "deny" would silently become a permissive "Allow" instead of being rejected. It now throws on anything but an exact "Allow" or "Deny", naming the offending statement and value. commitPolicyTextareaToEditor catches this the same way it already catches invalid JSON: alert the user and keep the JSON tab active instead of switching to the Editor tab with wrong data. - policyEditorStateToDoc could save a statement with both Resource (from the structured field) and NotResource (surviving in the "advanced fields" extras from before the user switched to using Resource) set at once - a contradictory combination neither the admin's ValidatePolicy handler nor policy_engine's evaluator rejected. When the structured Resource field is non-empty it now deletes any leftover NotResource from extras, consistent with the file's existing rule that structured fields take precedence over extras. Mirrored the existing Principal/NotPrincipal exclusivity check in weed/admin/handlers/policy_handlers.go's ValidatePolicy to reject the same combination server-side, since create/update perform no validation at all. Deliberately left policy_engine.validateStatement (used by the S3 bucket-policy PUT handler for every bucket policy in the product) unchanged - extending that shared validator is a larger, separate change outside this admin-editor fix's scope. Added a handler test for the new Resource+NotResource rejection. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * admin: add NotResource support to the visual policy editor Since Resource and NotResource are mutually exclusive (enforced by a previous fix), NotResource could previously only be set through the raw JSON in a statement's "advanced fields" box. Promote it to a first-class mode of the structured editor: - The static "Resources" label is now a Resource/NotResource dropdown; the same list of values underneath is reused for either key depending on the selected mode, with a short form-text explaining the semantics. - NotResource is added to POLICY_STATEMENT_KNOWN_KEYS, since it's now a managed field like Resource rather than something that falls through to extras. - policyDocToEditorState derives resourceMode from which key is present on load, and throws (same handling as the existing malformed-Effect case: alert, keep the JSON tab active) if a hand-edited document has both Resource and NotResource on one statement, since that can't be represented by the dropdown. - policyEditorStateToDoc writes only the key matching the selected mode, replacing the previous one-directional "delete NotResource whenever Resource is set" fix with mode-driven logic that also deletes Resource when NotResource is selected. - displayPolicyDetails (the read-only View modal) now shows the actual NotResource values with a distinct label instead of a static "(NotResource used instead)" placeholder. No backend changes: the server-side "cannot specify both" check added previously in policy_handlers.go's ValidatePolicy already covers this. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: reject non-object policy documents; catch Principal/NotPrincipal conflicts server-side Two review findings: - policyDocToEditorState treated a top-level JSON value that wasn't an object (null, or a bare string/number/boolean) as an empty statement list instead of failing explicitly. If the user typed e.g. "hello" or 42 in the JSON tab and switched to the Editor tab, their input was silently discarded and replaced with an empty policy - the same class of "guess instead of reject" bug fixed for malformed Effect and Resource/NotResource conflicts previously. Added an explicit check that throws for null/scalar input, while leaving array and object document shapes accepted exactly as before. - weed/admin/handlers/policy_handlers.go's ValidatePolicy checked the Resource/NotResource conflict by non-empty length (len(...Strings()) > 0), which misses a statement where Resource is explicitly present but an empty list (e.g. "Resource": []) alongside a non-empty NotResource. Switched that check to field presence (!= nil), matching how policy_engine's own validateStatement already treats Principal/NotPrincipal exclusivity. Also added the equivalent Principal/NotPrincipal presence check to this handler, which had none before - the advanced-fields box in the visual editor lets a user set both today, and nothing server-side caught it. The existing non-empty "Resource or NotResource is required" check is left as a length check, since an empty array shouldn't count as "provided". Added test cases for both conflict checks in policy_handlers_test.go. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: add Principal/NotPrincipal support to the visual policy editor (v1, AWS-only) Adds a first, deliberately narrow structured editor for a statement's Principal/NotPrincipal, left out when NotResource support was added: - A Principal/NotPrincipal mode dropdown mirrors the existing Resource/NotResource one (same mutual-exclusivity handling: the two fields can't be set at once, and switching modes reuses the same value list). - A simple repeatable text-value list feeds a single {"AWS": [...]} object on save - always the AWS type, never the "bare" (untyped) SeaweedFS-extension shape. Per policy_engine's allowedPrincipalKeys, Service/Federated/CanonicalUser also parse successfully, but nothing in the S3 bucket-policy evaluation path ever sets a real caller's principal to a service name, an OIDC provider ARN, or a canonical user ID, so only AWS is functionally meaningful today - out of scope for this v1. - On load, only the exact {"AWS": ...} single-key shape is unwrapped into the structured field and removed from "extras". Anything else (bare string/array, a different single type key, or several type keys at once) is left untouched in "extras" exactly as before, with a visible warning under the dropdown so the user knows a Principal/NotPrincipal exists but isn't shown there. Saving with the structured field left empty never touches whatever's already in extras, so a preserved complex form isn't silently dropped just because the user didn't touch this field. - The read-only View modal now displays Principal/NotPrincipal for any shape (via a small generic summarizer), not just the AWS-simple one. - Generalized the action/resource field-to-state-key mapping (used by commitPolicyEditorForm and the add/remove-item click handler) into a shared lookup table instead of stacking another ternary, now that a third field (principal) exists. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: support the bare "*" wildcard Principal in the visual editor "Principal": "*" (and NotPrincipal: "*") is the standard AWS shorthand for "everyone" and is common in real bucket policies, but the v1 Principal/NotPrincipal editor only recognized the {"AWS": ...} object form, leaving a bare "*" statement's principal hidden in Advanced fields. parseSimpleAwsPrincipal now also accepts the bare string "*" as a simple, structurally-editable value. On save, a principal value list containing exactly ["*"] is written back as the bare "*" string (matching the common convention) rather than wrapped as {"AWS": "*"}; anything else still wraps under AWS as before. Updated the field's form-text hint accordingly. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: add Principal field autocomplete backed by users + IAM roles Adds a datalist-backed autocomplete for the policy editor's Principal/ NotPrincipal text fields, sourced from a new API listing existing identities: - weed/admin/dash/principal_suggestions.go: AdminServer.GetPrincipalSuggestions combines S3 user ARNs (via the existing GetObjectStoreUsers + iam.UserArn) with IAM role ARNs (via integration.NewFilerRoleStore / ListRoles, reusing the exact same construction already used in iam_manager.go - no new dependency risk introduced). Role ARNs are reconstructed from the role name using SeaweedFS's default arn:aws:iam::role/<name> convention rather than fetching each role's stored definition, since this only backs a suggestion list. Role listing failures are logged and swallowed rather than failing the whole request - an incomplete suggestion list is fine, blocking policy editing over it is not. Service accounts are deliberately not listed separately: a service account's ARN is identical to its parent user's, already covered by the user list. - weed/admin/handlers/policy_handlers.go: GetPrincipalSuggestions handler exposing this as {"principals": [...]}. - Route registered at the API root (GET /api/principals) rather than under policyApi's "/object-store/policies" prefix, since that subrouter's existing "/{name}" GET route would shadow any single-segment GET route registered after it (the same class of gotcha previously seen with "/validate"). - weed/admin/view/app/policies.templ: a shared, lazily-fetched-once policyPrincipalSuggestions datalist (flat list - unlike the progressive per-folder Resource ARN autocomplete, users/roles aren't hierarchical), wired into policyListRowHtml for field:"principal" and populated on input/focus, with "*" always offered first. Added tests for the new ARN-construction helper and route registration. Regenerated policies_templ.go with the already-stamped templ v0.3.1001. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: fix fieldset/legend styling in the structured policy editor Bootstrap's form reset stretches <legend> to the fieldset's full width (float: left; width: 100%), which loses the native "notch in the border" look and makes each section's label bar as wide as the card. Add two scoped classes: .policy-stmt-fieldset (border, rounded corners, spacing between sections) and .policy-stmt-legend (undoes the float/width so the legend hugs its content, with a little padding). Applied to the three per-statement sections (Actions, Resource/NotResource, Principal/NotPrincipal), replacing the ad hoc "border rounded" utility classes that were doubling up with the fieldset's own border. Also gave the "Advanced fields" <details> a small top margin to match the new spacing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: suggest bucket/* alongside the bucket itself in Resource autocomplete At the bucket-name stage of the Resource field's progressive autocomplete, only "arn:aws:s3:::bucket" was offered. Add "arn:aws:s3:::bucket/*" right alongside it, since granting access to everything in a bucket is the more common case and previously required typing a "/" first to reach the folder-level "*" suggestion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Eb2a51LciCsyY35sqDoGNe * admin: keep an unparseable policy in the JSON tab instead of wiping it editPolicy() built the structured state inside the fetch .then, so a policy the editor cannot model threw into the sibling .catch, which alerted and called hide(). Showing the alert at that moment left the modal on screen with an empty editor and the document only in the JSON tab, and Save Changes then serialized the empty state over the policy. Reachable two ways, since neither create path rejects these: the admin API never validates on create, so "Effect":"allow" is stored as typed, and policy_engine.validateStatement lets Resource and NotResource sit in the same statement. Hand the document to the JSON tab instead, which is what that tab is for, and mark the state so nothing serializes the placeholder over it. * admin: validate a policy document before saving it Validation was wired only to the Validate button, so nothing stopped a document the server's own validator rejects from being stored. With the structured editor supplying the boilerplate and required dropped from the textarea, opening the modal, typing a name and clicking Create Policy was enough to save a statement-less policy. Share validatePolicyJSON with the two save paths and abort on failure. * admin: bound the folder autocomplete listing maxListFoldersEntries caps the folders collected, but nothing capped the entries paged through to find them, so a bucket holding only flat object keys - no subfolders to count - was walked to the end, 200 entries per round trip, behind one keystroke. Measured against an in-process filer: 6 entries 0.5ms, 3k entries 7.7ms, 30k entries 53ms, all of it linear in the directory rather than in the answer. Cap the scan as well, and let GetFileBrowser take a prefix so the segment the user is still typing is filtered by the filer instead of by paging. The same 30k directory now answers in 0.6ms once a prefix is typed. * admin: clean the path before scoping list-folders to /buckets util.CleanWindowsPath only rewrites backslashes, so "/buckets/../etc" walked straight past the prefix check the endpoint relies on for its scope. Nothing leaked - filer paths are literal keys, so the traversal resolved to nothing - but the check reads as a boundary and wasn't one, and the test asserting it didn't cover the one input that would try. validateAndCleanFilePath in the same file already does this. * admin: stringify policy values before escaping them escapeHtml calls text.replace directly, and the Sid, the per-item action and resource inputs, and the View modal's Resource/NotResource all pass values straight out of JSON.parse. A policy carrying "Sid": 5 or "Action": [1] threw "text.replace is not a function" and took the render with it. escapedJoin already coerced; use it everywhere and coerce the editor state at the point it's built. * admin: only show the NotResource hint in NotResource mode The hint rendered unconditionally, so it sat under a selector reading "Resource" telling the user the statement applies to everything except what they'd listed. Redraw the card when the selector changes so it follows the mode. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Chris Lu <chris.lu@gmail.com>
SeaweedFS
Sponsor SeaweedFS via Patreon
SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible entirely thanks to the support of these awesome backers. If you'd like to grow SeaweedFS even stronger, please consider joining our sponsors on Patreon.
Your support will be really appreciated by me and other supporters!
Gold Sponsors
- Download Binaries for different platforms
- SeaweedFS on Slack
- SeaweedFS on Twitter
- SeaweedFS on Telegram
- SeaweedFS on Reddit
- SeaweedFS Mailing List
- Wiki Documentation
- SeaweedFS White Paper
- SeaweedFS Introduction Slides 2025.5
- SeaweedFS Introduction Slides 2021.5
- SeaweedFS Introduction Slides 2019.3
Table of Contents
- Quick Start
- Introduction
- Features
- Example: Using Seaweed Blob Store
- Architecture
- Compared to Other File Systems
- Dev Plan
- Installation Guide
- Disk Related Topics
- Benchmark
- Enterprise
- License
Quick Start
Quick Start with weed mini
Download the latest binary from https://github.com/seaweedfs/seaweedfs/releases and unzip the single weed (or weed.exe) file, or run go install github.com/seaweedfs/seaweedfs/weed@latest. Then start a ready-to-use S3 object store with credentials and a pre-created bucket in one command:
AWS_ACCESS_KEY_ID=admin \
AWS_SECRET_ACCESS_KEY=secret \
S3_BUCKET=my-bucket \
./weed mini -dir=/data
That's it — the S3 endpoint is at http://localhost:8333, my-bucket already exists, and admin/secret are valid credentials. S3_BUCKET accepts a comma-separated list (e.g. raw,processed); use S3_TABLE_BUCKET for S3 Tables buckets, each name or name:FORMAT where the format is ICEBERG (the default) or LANCE. Drop any of the env vars to skip that piece (no AWS keys → S3 runs in unauthenticated "Allow All" mode for development).
The same command starts everything else too:
- S3 Endpoint: http://localhost:8333
- Master UI: http://localhost:9333
- Volume Server: http://localhost:9340
- Filer UI: http://localhost:8888
- WebDAV: http://localhost:7333
- Admin UI: http://localhost:23646
macOS: if the binary is quarantined, run
xattr -d com.apple.quarantine ./weedfirst.
Perfect for development, testing, learning SeaweedFS, and single-node deployments. To scale out, add more volume servers by running weed volume -dir="/some/data/dir2" -master="<master_host>:9333" -port=8081 locally, on another machine, or on thousands of machines.
Quick Start for S3 API on Docker
docker run -p 8333:8333 \
-e AWS_ACCESS_KEY_ID=admin \
-e AWS_SECRET_ACCESS_KEY=secret \
-e S3_BUCKET=my-bucket \
chrislusf/seaweedfs
Same behavior as the weed mini command above — the S3 endpoint is at http://localhost:8333 with my-bucket pre-created. Drop the env vars to run anonymously for development.
Introduction
SeaweedFS is a simple and highly scalable distributed file system. There are two objectives:
- to store billions of files!
- to serve the files fast!
SeaweedFS started as a blob store to handle small files efficiently. Instead of managing all file metadata in a central master, the central master only manages volumes on volume servers, and these volume servers manage files and their metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers, allowing faster file access (O(1), usually just one disk read operation).
There is only 40 bytes of disk storage overhead for each file's metadata. It is so simple with O(1) disk reads that you are welcome to challenge the performance with your actual use cases.
SeaweedFS started by implementing Facebook's Haystack design paper. Also, SeaweedFS implements erasure coding with ideas from f4: Facebook’s Warm BLOB Storage System, and has a lot of similarities with Facebook’s Tectonic Filesystem and Google's Colossus File System
On top of the blob store, optional Filer can support directories and POSIX attributes. Filer is a separate linearly-scalable stateless server with customizable metadata stores, e.g., MySql, Postgres, Redis, Cassandra, HBase, Mongodb, Elastic Search, LevelDB, RocksDB, Sqlite, MemSql, TiDB, Etcd, CockroachDB, YDB, etc.
SeaweedFS can transparently integrate with the cloud. With hot data on local cluster, and warm data on the cloud with O(1) access time, SeaweedFS can achieve both fast local access time and elastic cloud storage capacity. What's more, the cloud storage access API cost is minimized. Faster and cheaper than direct cloud storage!
SeaweedFS also ships a built-in Iceberg REST Catalog, turning the same cluster into a self-contained lakehouse. Spark, Trino, Dremio, DuckDB, and RisingWave can query Iceberg tables directly — no Hive Metastore, Glue, or external catalog service required. Storage and table metadata live in one system, simplifying on-prem and small-team analytics stacks.
Features
Additional Blob Store Features
- Support different replication levels, with rack and data center aware.
- Automatic master servers failover - no single point of failure (SPOF).
- Automatic compression depending on file MIME type.
- Automatic compaction to reclaim disk space after deletion or update.
- Automatic entry TTL expiration.
- Flexible Capacity Expansion: Any server with some disk space can add to the total storage space.
- Adding/Removing servers does not cause any data re-balancing unless triggered by admin commands.
- Optional picture resizing.
- Support ETag, Accept-Range, Last-Modified, etc.
- Support in-memory/leveldb/readonly mode tuning for memory/performance balance.
- Support rebalancing the writable and readonly volumes.
- Customizable Multiple Storage Tiers: Customizable storage disk types to balance performance and cost.
- Transparent cloud integration: unlimited capacity via tiered cloud storage for warm data.
- Erasure Coding for warm storage Rack-Aware 10.4 erasure coding reduces storage cost and increases availability. Enterprise version can customize EC ratio.
Filer Features
- Filer server provides "normal" directories and files via HTTP.
- File TTL automatically expires file metadata and actual file data.
- Mount filer reads and writes files directly as a local directory via FUSE.
- Filer Store Replication enables HA for filer meta data stores.
- Active-Active Replication enables asynchronous one-way or two-way cross cluster continuous replication.
- Amazon S3 compatible API accesses files with S3 tooling.
- Hadoop Compatible File System accesses files from Hadoop/Spark/Flink/etc or even runs HBase.
- Async Replication To Cloud has extremely fast local access and backups to Amazon S3, Google Cloud Storage, Azure, BackBlaze.
- WebDAV accesses as a mapped drive on Mac and Windows, or from mobile devices.
- AES256-GCM Encrypted Storage safely stores the encrypted data.
- Super Large Files stores large or super large files in tens of TB.
- Cloud Drive mounts cloud storage to local cluster, cached for fast read and write with asynchronous write back.
- Gateway to Remote Object Store mirrors bucket operations to remote object storage, in addition to Cloud Drive
Data Lakehouse Features
- S3 Table Buckets expose a dedicated namespace for Iceberg tables with strict layout validation.
- Built-in Iceberg REST Catalog runs alongside the S3 endpoint — no external metastore needed.
- Native integrations with Apache Spark, Trino, Dremio, DuckDB, and RisingWave.
- Automated table maintenance: compaction, snapshot expiration, orphan removal, manifest rewriting.
- Granular IAM at the bucket, namespace, and table level via standard S3 bucket policies.
Kubernetes
- Kubernetes CSI Driver A Container Storage Interface (CSI) Driver.
- SeaweedFS Operator
Example: Using Seaweed Blob Store
By default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and two volume nodes on port 8080 and 8081. Ideally, they should be started from different machines. We'll use localhost as an example.
SeaweedFS uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.
Start Master Server
> ./weed master
Start Volume Servers
> weed volume -dir="/tmp/data1" -max=5 -master="localhost:9333" -port=8080 &
> weed volume -dir="/tmp/data2" -max=10 -master="localhost:9333" -port=8081 &
Write A Blob
A blob, also referred as a needle, a chunk, or mistakenly as a file, is just a byte array. It can have attributes, such as name, mime type, create or update time, etc. But basically it is just a byte array of a relatively small size, such as 2 MB ~ 64 MB. The size is not fixed.
To upload a blob: first, send a HTTP POST, PUT, or GET request to /dir/assign to get an fid and a volume server URL:
> curl http://localhost:9333/dir/assign
{"count":1,"fid":"3,01637037d6","url":"127.0.0.1:8080","publicUrl":"localhost:8080"}
Second, to store the blob content, send a HTTP multi-part POST request to url + '/' + fid from the response:
> curl -F file=@/home/chris/myphoto.jpg http://127.0.0.1:8080/3,01637037d6
{"name":"myphoto.jpg","size":43234,"eTag":"1cc0118e"}
To update, send another POST request with updated blob content.
For deletion, send an HTTP DELETE request to the same url + '/' + fid URL:
> curl -X DELETE http://127.0.0.1:8080/3,01637037d6
Save Blob Id
Now, you can save the fid, 3,01637037d6 in this case, to a database field.
The number 3 at the start represents a volume id. After the comma, it's one file key, 01, and a file cookie, 637037d6.
The volume id is an unsigned 32-bit integer. The file key is an unsigned 64-bit integer. The file cookie is an unsigned 32-bit integer, used to prevent URL guessing.
The file key and file cookie are both coded in hex. You can store the <volume id, file key, file cookie> tuple in your own format, or simply store the fid as a string.
If stored as a string, in theory, you would need 8+1+16+8=33 bytes. A char(33) would be enough, if not more than enough, since most uses will not need 2^32 volumes.
If space is really a concern, you can store the file id in the binary format. You would need one 4-byte integer for volume id, 8-byte long number for file key, and a 4-byte integer for the file cookie. So 16 bytes are more than enough.
Read a Blob
Here is an example of how to render the URL.
First look up the volume server's URLs by the file's volumeId:
> curl http://localhost:9333/dir/lookup?volumeId=3
{"volumeId":"3","locations":[{"publicUrl":"localhost:8080","url":"localhost:8080"}]}
Since (usually) there are not too many volume servers, and volumes don't move often, you can cache the results most of the time. Depending on the replication type, one volume can have multiple replica locations. Just randomly pick one location to read.
Now you can take the public URL, render the URL or directly read from the volume server via URL:
http://localhost:8080/3,01637037d6.jpg
Notice we add a file extension ".jpg" here. It's optional and just one way for the client to specify the file content type.
If you want a nicer URL, you can use one of these alternative URL formats:
http://localhost:8080/3/01637037d6/my_preferred_name.jpg
http://localhost:8080/3/01637037d6.jpg
http://localhost:8080/3,01637037d6.jpg
http://localhost:8080/3/01637037d6
http://localhost:8080/3,01637037d6
If you want to get a scaled version of an image, you can add some params:
http://localhost:8080/3/01637037d6.jpg?height=200&width=200
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fit
http://localhost:8080/3/01637037d6.jpg?height=200&width=200&mode=fill
Rack-Aware and Data Center-Aware Replication
SeaweedFS applies the replication strategy at a volume level. So, when you are getting a blob id, you can specify the replication strategy. For example:
curl http://localhost:9333/dir/assign?replication=001
The replication parameter options are:
000: no replication
001: replicate once on the same rack
010: replicate once on a different rack, but same data center
100: replicate once on a different data center
200: replicate twice on two different data center
110: replicate once on a different rack, and once on a different data center
More details about replication can be found on the wiki.
You can also set the default replication strategy when starting the master server.
Allocate Blob Key on Specific Data Center
Volume servers can be started with a specific data center name:
weed volume -dir=/tmp/1 -port=8080 -dataCenter=dc1
weed volume -dir=/tmp/2 -port=8081 -dataCenter=dc2
When requesting a blob key, an optional "dataCenter" parameter can limit the assigned volume to the specific data center. For example, this specifies that the assigned volume should be limited to 'dc1':
http://localhost:9333/dir/assign?dataCenter=dc1
Other Features
- No Single Point of Failure
- Insert with your own keys
- Chunking large files
- Collection as a Simple Name Space
Blob Store Architecture
Usually distributed file systems split each file into chunks. A central server keeps a mapping of filenames to chunks, and also which chunks each chunk server has.
The main drawback is that the central server can't handle many small files efficiently, and since all read requests need to go through the central master, so it might not scale well for many concurrent users.
Instead of managing chunks, SeaweedFS manages data volumes in the master server. Each data volume is 32GB in size, and can hold a lot of blobs. And each storage node can have many data volumes. So the master node only needs to store the metadata about the volumes, which is a fairly small amount of data and is generally stable.
The actual blob metadata, which are the blob volume, offset, and size, is stored in each volume on volume servers. Since each volume server only manages metadata of blobs on its own disk, with only 16 bytes for each blob, all access can read the metadata just from memory and only needs one disk operation to actually read file data.
For comparison, consider that an xfs inode structure in Linux is 536 bytes.
Master Server and Volume Server
The architecture is fairly simple. The actual data is stored in volumes on storage nodes. One volume server can have multiple volumes, and can both support read and write access with basic authentication.
All volumes are managed by a master server. The master server contains the volume id to volume server mapping. This is fairly static information, and can be easily cached.
On each write request, the master server also generates a file key, which is a growing 64-bit unsigned integer. Since write requests are not generally as frequent as read requests, one master server should be able to handle the concurrency well.
Write and Read files
When a client sends a write request, the master server returns (volume id, file key, file cookie, volume node URL) for the blob. The client then contacts the volume node and POSTs the blob content.
When a client needs to read a blob based on (volume id, file key, file cookie), it asks the master server by the volume id for the (volume node URL, volume node public URL), or retrieves this from a cache. Then the client can GET the content, or just render the URL on web pages and let browsers fetch the content.
Saving memory
All blob metadata stored on a volume server is readable from memory without disk access. Each file takes just a 16-byte map entry of <64bit key, 32bit offset, 32bit size>. Of course, each map entry has its own space cost for the map. But usually the disk space runs out before the memory does.
Tiered Storage to the cloud
The local volume servers are much faster, while cloud storages have elastic capacity and are actually more cost-efficient if not accessed often (usually free to upload, but relatively costly to access). With the append-only structure and O(1) access time, SeaweedFS can take advantage of both local and cloud storage by offloading the warm data to the cloud.
Usually hot data are fresh and warm data are old. SeaweedFS puts the newly created volumes on local servers, and optionally upload the older volumes on the cloud. If the older data are accessed less often, this literally gives you unlimited capacity with limited local servers, and still fast for new data.
With the O(1) access time, the network latency cost is kept at minimum.
If the hot/warm data is split as 20/80, with 20 servers, you can achieve storage capacity of 100 servers. That's a cost saving of 80%! Or you can repurpose the 80 servers to store new data also, and get 5X storage throughput.
SeaweedFS Filer
Built on top of the blob store, SeaweedFS Filer adds directory structure to create a file system. The directory structure is an interface that is implemented in many key-value stores or databases.
The content of a file is mapped to one or many blobs, distributed to multiple volumes on multiple volume servers.
Compared to Other File Systems
Most other distributed file systems seem more complicated than necessary.
SeaweedFS is meant to be fast and simple, in both setup and operation. If you do not understand how it works when you reach here, we've failed! Please raise an issue with any questions or update this file with clarifications.
SeaweedFS is constantly moving forward. Same with other systems. These comparisons can be outdated quickly. Please help to keep them updated.
Compared to HDFS
HDFS uses the chunk approach for each file, and is ideal for storing large files.
SeaweedFS is ideal for serving relatively smaller files quickly and concurrently.
SeaweedFS can also store extra large files by splitting them into manageable data chunks, and store the file ids of the data chunks into a meta chunk. This is managed by "weed upload/download" tool, and the weed master or volume servers are agnostic about it.
Compared to GlusterFS, Ceph
The architectures are mostly the same. SeaweedFS aims to store and read files fast, with a simple and flat architecture. The main differences are
- SeaweedFS optimizes for small files, ensuring O(1) disk seek operation, and can also handle large files.
- SeaweedFS statically assigns a volume id for a file. Locating file content becomes just a lookup of the volume id, which can be easily cached.
- SeaweedFS Filer metadata store can be any well-known and proven data store, e.g., Redis, Cassandra, HBase, Mongodb, Elastic Search, MySql, Postgres, Sqlite, MemSql, TiDB, CockroachDB, Etcd, YDB etc, and is easy to customize.
- SeaweedFS Volume server also communicates directly with clients via HTTP, supporting range queries, direct uploads, etc.
| System | File Metadata | File Content Read | POSIX | REST API | Optimized for large number of small files |
|---|---|---|---|---|---|
| SeaweedFS | lookup volume id, cacheable | O(1) disk seek | Yes | Yes | |
| SeaweedFS Filer | Linearly Scalable, Customizable | O(1) disk seek | FUSE | Yes | Yes |
| GlusterFS | hashing | FUSE, NFS | |||
| Ceph | hashing + rules | FUSE | Yes | ||
| MooseFS | in memory | FUSE | No | ||
| MinIO | separate meta file per drive for each file | Yes | No | ||
| RustFS | separate meta file per drive for each file | Yes | No |
Compared to GlusterFS
GlusterFS stores files, both directories and content, in configurable volumes called "bricks".
GlusterFS hashes the path and filename into ids, and assigned to virtual volumes, and then mapped to "bricks".
Compared to MooseFS
MooseFS chooses to neglect small file issue. From moosefs 3.0 manual, "even a small file will occupy 64KiB plus additionally 4KiB of checksums and 1KiB for the header", because it "was initially designed for keeping large amounts (like several thousands) of very big files"
MooseFS Master Server keeps all meta data in memory. Same issue as HDFS namenode.
Compared to Ceph
Ceph can be setup similar to SeaweedFS as a key->blob store. It is much more complicated, with the need to support layers on top of it. Here is a more detailed comparison
SeaweedFS has a centralized master group to look up free volumes, while Ceph uses hashing and metadata servers to locate its objects. Having a centralized master makes it easy to code and manage.
Ceph, like SeaweedFS, is based on the object store RADOS. Ceph is rather complicated with mixed reviews.
Ceph uses CRUSH hashing to automatically manage data placement, which is efficient to locate the data. But the data has to be placed according to the CRUSH algorithm. Any wrong configuration would cause data loss. Topology changes, such as adding new servers to increase capacity, will cause data migration with high IO cost to fit the CRUSH algorithm. SeaweedFS places data by assigning them to any writable volumes. If writes to one volume failed, just pick another volume to write. Adding more volumes is also as simple as it can be.
SeaweedFS is optimized for small files. Small files are stored as one continuous block of content, with at most 8 unused bytes between files. Small file access is O(1) disk read.
SeaweedFS Filer uses off-the-shelf stores, such as MySql, Postgres, Sqlite, Mongodb, Redis, Elastic Search, Cassandra, HBase, MemSql, TiDB, CockroachCB, Etcd, YDB, to manage file directories. These stores are proven, scalable, and easier to manage.
| SeaweedFS | comparable to Ceph | advantage |
|---|---|---|
| Master | MDS | simpler |
| Volume | OSD | optimized for small files |
| Filer | Ceph FS | linearly scalable, Customizable, O(1) or O(logN) |
Compared to MinIO, RustFS
Please note, as Apr 25, 2026 MinIO ceased development. It's strongly discouraged to use that unmaintained software with multiple security bugs. RustFS is a MinIO reimplementation in Rust, Apache 2.0 licensed and still developed, keeping MinIO's storage model down to a byte-compatible on-disk format. So the points below apply to both.
MinIO followed AWS S3 closely and was ideal for testing for S3 API. It had good UI, policies, versionings, etc. SeaweedFS is trying to catch up here.
The metadata are in simple files. Each file write incurs extra writes to the corresponding meta file, on every drive of the erasure set. Changing only tags or retention rewrites that meta file on all of them, so the write amplification does not shrink with object size.
There is no optimization for lots of small files. The files are simply stored as is to local disks. Plus the extra meta file and shards for erasure coding, it only amplifies the LOSF problem.
Multiple disk IO are needed to read one file. SeaweedFS has O(1) disk reads, even for erasure coded files.
Erasure coding is full-time. SeaweedFS uses replication on hot data for faster speed and optionally applies erasure coding on warm data.
No POSIX-like API support.
There are specific requirements on storage layout, which makes it hard to scale out and to maintain. An erasure set must be 2 to 16 drives and must divide the drive list symmetrically, and capacity grows or shrinks a whole pool at a time. In SeaweedFS, just start one volume server pointing to the master. That's all.
Dev Plan
- More tools and documentation, on how to manage and scale the system.
- Read and write stream data.
- Support structured data.
This is a super exciting project! And we need helpers and support!
Installation Guide
Installation guide for users who are not familiar with golang
Step 1: install go on your machine and setup the environment by following the instructions at:
https://golang.org/doc/install
make sure to define your $GOPATH
Step 2: checkout this repo:
git clone https://github.com/seaweedfs/seaweedfs.git
Step 3: download, compile, and install the project by executing the following command
cd seaweedfs/weed && make install
Once this is done, you will find the executable "weed" in your $GOPATH/bin directory
For more installation options, including how to run with Docker, see the Getting Started guide.
Disk Related Topics
Hard Drive Performance
When testing read performance on SeaweedFS, it basically becomes a performance test of your hard drive's random read speed. Hard drives usually get 100MB/s~200MB/s.
Solid State Disk
To modify or delete small files, SSD must delete a whole block at a time, and move content in existing blocks to a new block. SSD is fast when brand new, but will get fragmented over time and you have to garbage collect, compacting blocks. SeaweedFS is friendly to SSD since it is append-only. Deletion and compaction are done on volume level in the background, not slowing reading and not causing fragmentation.
Benchmark
My Own Unscientific Single Machine Results on Mac Book with Solid State Disk, CPU: 1 Intel Core i7 2.6GHz.
Write 1 million 1KB file:
Concurrency Level: 16
Time taken for tests: 66.753 seconds
Completed requests: 1048576
Failed requests: 0
Total transferred: 1106789009 bytes
Requests per second: 15708.23 [#/sec]
Transfer rate: 16191.69 [Kbytes/sec]
Connection Times (ms)
min avg max std
Total: 0.3 1.0 84.3 0.9
Percentage of the requests served within a certain time (ms)
50% 0.8 ms
66% 1.0 ms
75% 1.1 ms
80% 1.2 ms
90% 1.4 ms
95% 1.7 ms
98% 2.1 ms
99% 2.6 ms
100% 84.3 ms
Randomly read 1 million files:
Concurrency Level: 16
Time taken for tests: 22.301 seconds
Completed requests: 1048576
Failed requests: 0
Total transferred: 1106812873 bytes
Requests per second: 47019.38 [#/sec]
Transfer rate: 48467.57 [Kbytes/sec]
Connection Times (ms)
min avg max std
Total: 0.0 0.3 54.1 0.2
Percentage of the requests served within a certain time (ms)
50% 0.3 ms
90% 0.4 ms
98% 0.6 ms
99% 0.7 ms
100% 54.1 ms
Run WARP and launch a mixed benchmark.
make benchmark
warp: Benchmark data written to "warp-mixed-2025-12-05[194844]-kBpU.csv.zst"
Mixed operations.
Operation: DELETE, 10%, Concurrency: 20, Ran 42s.
* Throughput: 55.13 obj/s
Operation: GET, 45%, Concurrency: 20, Ran 42s.
* Throughput: 2477.45 MiB/s, 247.75 obj/s
Operation: PUT, 15%, Concurrency: 20, Ran 42s.
* Throughput: 825.85 MiB/s, 82.59 obj/s
Operation: STAT, 30%, Concurrency: 20, Ran 42s.
* Throughput: 165.27 obj/s
Cluster Total: 3302.88 MiB/s, 550.51 obj/s over 43s.
Enterprise
For enterprise users, please visit seaweedfs.com for the SeaweedFS Enterprise Edition, which has advanced features, including data recovery, self-healing storage, customizable erasure coding, EC vacuum and repair, etc.
License
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
The text of this page is available for modification and reuse under the terms of the Creative Commons Attribution-Sharealike 3.0 Unported License and the GNU Free Documentation License (unversioned, with no invariant sections, front-cover texts, or back-cover texts).




