mirrors/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-08-25 00:27:11 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
7ebf2ebac3ab35ec1ebf01fa3558ff6891eb0fe7
seaweedfs/test/tus
T
History
Chris LuandGitHub c1a993bc3b filer: keep the TUS sub-chunks that already landed when a write fails (#10876) 
* filer: keep the TUS sub-chunks that already landed when a write fails

A PATCH is split into 4MB sub-chunks, and each one is recorded in the
session as soon as it is stored. The session listing is what HEAD reports
as Upload-Offset and what the final entry is assembled from, so a record
is a promise that the data behind it exists.

When a later sub-chunk failed - a read-only volume, or a client that hung
up mid-body - the error path deleted the needles of every sub-chunk the
same PATCH had written but left their records in place. The resuming
client was then told to continue past bytes the filer had just queued for
deletion, and the upload completed into a gapless manifest pointing at
needles that were gone: HEAD returned the right size, GET died mid-body
once a vacuum reclaimed them.

Recorded sub-chunks now stay, which is what resumption expects: the
client picks up at the offset the session reports, and an upload that is
abandoned frees its chunks with the session.

* filer: drop a TUS chunk's record before freeing its data

filer.CreateEntry can return an error with the entry already inserted -
the parent-directory pass runs after the insert and keeps the entry when
it fails. A failed saveTusChunk therefore does not mean the record is
absent, and deleting the needle outright left the same corruption the
resume path used to cause: a session record pointing at data that is gone.

Remove the record first and only free the needle once it is gone. A
record lost with its data still stored merely leaks, which the vacuum and
fsck paths already account for.

* test: cover a TUS PATCH that is cut off mid-body

Resets the connection after one 4MB sub-chunk has landed, resumes from the
offset the session reports, and vacuums before reading the file back, so
anything the filer deleted behind a kept record shows up as a short read.
2026-08-22 00:30:14 -07:00
..
Makefile
Add TUS protocol support for resumable uploads (#7592)
2025-12-14 21:56:07 -08:00
README.md
Add TUS protocol support for resumable uploads (#7592)
2025-12-14 21:56:07 -08:00
tus_integration_test.go
filer: keep the TUS sub-chunks that already landed when a write fails (#10876)
2026-08-22 00:30:14 -07:00

README.md

TUS Protocol Integration Tests

This directory contains integration tests for the TUS (resumable upload) protocol support in SeaweedFS Filer.

Overview

TUS is an open protocol for resumable file uploads over HTTP. It allows clients to upload files in chunks and resume uploads after network failures or interruptions.

Why TUS?

  • Resumable uploads: Resume interrupted uploads without re-sending data
  • Chunked uploads: Upload large files in smaller pieces
  • Simple protocol: Standard HTTP methods with custom headers
  • Wide client support: Libraries available for JavaScript, Python, Go, and more

TUS Protocol Endpoints

Method Path Description
OPTIONS /.tus/ Server capability discovery
POST /.tus/{path} Create new upload session
HEAD /.tus/.uploads/{id} Get current upload offset
PATCH /.tus/.uploads/{id} Upload data at offset
DELETE /.tus/.uploads/{id} Cancel upload

TUS Headers

Request Headers:

  • Tus-Resumable: 1.0.0 - Protocol version (required)
  • Upload-Length - Total file size in bytes (required on POST)
  • Upload-Offset - Current byte offset (required on PATCH)
  • Upload-Metadata - Base64-encoded key-value pairs (optional)
  • Content-Type: application/offset+octet-stream (required on PATCH)

Response Headers:

  • Tus-Resumable - Protocol version
  • Tus-Version - Supported versions
  • Tus-Extension - Supported extensions
  • Tus-Max-Size - Maximum upload size
  • Upload-Offset - Current byte offset
  • Location - Upload URL (on POST)

Enabling TUS

TUS protocol support is enabled by default at /.tus path. You can customize the path using the -tusBasePath flag:

# Start filer with default TUS path (/.tus)
weed filer -master=localhost:9333

# Use a custom path (leading slash added automatically if missing)
weed filer -master=localhost:9333 -tusBasePath=/.uploads/tus

# Disable TUS by setting empty path
weed filer -master=localhost:9333 -tusBasePath=

Test Structure

Integration Tests

The tests cover:

  1. Basic Functionality

    • TestTusOptionsHandler - Capability discovery
    • TestTusBasicUpload - Simple complete upload
    • TestTusCreationWithUpload - Creation-with-upload extension

  2. Chunked Uploads

    • TestTusChunkedUpload - Upload in multiple chunks

  3. Resumable Uploads

    • TestTusHeadRequest - Offset tracking
    • TestTusResumeAfterInterruption - Resume after failure

  4. Error Handling

    • TestTusInvalidOffset - Offset mismatch (409 Conflict)
    • TestTusUploadNotFound - Missing upload (404 Not Found)
    • TestTusDeleteUpload - Upload cancellation

Running Tests

Prerequisites

  1. Build SeaweedFS:
make build-weed
# or
cd ../../weed && go build -o weed

Using Makefile

# Show available targets
make help

# Run all tests with automatic server management
make test-with-server

# Run all tests (requires running server)
make test

# Run specific test categories
make test-basic      # Basic upload tests
make test-chunked    # Chunked upload tests
make test-resume     # Resume/HEAD tests
make test-errors     # Error handling tests

# Manual testing
make manual-start    # Start SeaweedFS for manual testing
make manual-stop     # Stop and cleanup

Using Go Test Directly

# Run all TUS tests
go test -v ./test/tus/...

# Run specific test
go test -v ./test/tus -run TestTusBasicUpload

# Skip integration tests (short mode)
go test -v -short ./test/tus/...

Debug

# View server logs
make debug-logs

# Check process and port status
make debug-status

Test Environment

Each test run:

  1. Starts a SeaweedFS cluster (master, volume, filer)
  2. Creates uploads using TUS protocol
  3. Verifies files are stored correctly
  4. Cleans up test data

Default Ports

Service Port
Master 19333
Volume 18080
Filer 18888

Configuration

Override defaults via environment or Makefile variables:

FILER_PORT=8889 MASTER_PORT=9334 make test

Example Usage

Create Upload

curl -X POST http://localhost:18888/.tus/mydir/file.txt \
  -H "Tus-Resumable: 1.0.0" \
  -H "Upload-Length: 1000" \
  -H "Upload-Metadata: filename dGVzdC50eHQ="

Upload Data

curl -X PATCH http://localhost:18888/.tus/.uploads/{upload-id} \
  -H "Tus-Resumable: 1.0.0" \
  -H "Upload-Offset: 0" \
  -H "Content-Type: application/offset+octet-stream" \
  --data-binary @file.txt

Check Offset

curl -I http://localhost:18888/.tus/.uploads/{upload-id} \
  -H "Tus-Resumable: 1.0.0"

Cancel Upload

curl -X DELETE http://localhost:18888/.tus/.uploads/{upload-id} \
  -H "Tus-Resumable: 1.0.0"

TUS Extensions Supported

  • creation: Create new uploads with POST
  • creation-with-upload: Send data in creation request
  • termination: Cancel uploads with DELETE

Architecture

Client                         Filer                      Volume Servers
  |                              |                              |
  |-- POST /.tus/path/file.mp4 ->|                              |
  |                              |-- Create session dir ------->|
  |<-- 201 Location: /.../{id} --|                              |
  |                              |                              |
  |-- PATCH /.tus/.uploads/{id} >|                              |
  |   Upload-Offset: 0           |-- Assign volume ------------>|
  |   [chunk data]               |-- Upload chunk ------------->|
  |<-- 204 Upload-Offset: N -----|                              |
  |                              |                              |
  |   (network failure)          |                              |
  |                              |                              |
  |-- HEAD /.tus/.uploads/{id} ->|                              |
  |<-- Upload-Offset: N ---------|                              |
  |                              |                              |
  |-- PATCH (resume) ----------->|-- Upload remaining -------->|
  |<-- 204 (complete) -----------|-- Assemble final file ----->|

Comparison with S3 Multipart

Feature TUS S3 Multipart
Protocol Custom HTTP headers S3 API
Session Init POST with Upload-Length CreateMultipartUpload
Upload Data PATCH with offset UploadPart with partNumber
Resume HEAD to get offset ListParts
Complete Automatic at final offset CompleteMultipartUpload
Ordering Sequential (offset-based) Parallel (part numbers)

Related Resources

Reference in New Issue View Git Blame Copy Permalink