Compare commits

..
Author SHA1 Message Date
Chris Lu 0a1b54b146 s3api: order ListObjectVersions newest-first 2026-02-24 14:30:45 -08:00
b4c7d42a06 fix(admin): release mutex before disk I/O in maintenance queue; remove per-request LoadAllTaskStates (#8433)
* fix(admin): release mutex before disk I/O in maintenance queue

saveTaskState performs synchronous BoltDB writes. Calling it while
holding mq.mutex.Lock() in AddTask, GetNextTask, and CompleteTask
blocks all readers (GetTasks via RLock) for the full disk write
duration on every task state change.

During a maintenance scan AddTasksFromResults calls AddTask for every
volume — potentially hundreds of times — meaning the write lock is
held almost continuously. The HTTP handler for /maintenance calls
GetTasks which blocks on RLock, exceeding the 30s timeout and
returning 408 to the browser.

Fix: update in-memory state (mq.tasks, mq.pendingTasks) under the
lock as before, then unlock before calling saveTaskState. In-memory
state is the authoritative source; persistence is crash-recovery only
and does not require lock protection during the write.

* fix(admin): add mutex to ConfigPersistence to synchronize tasks/ filesystem ops

saveTaskState is now called outside mq.mutex, meaning SaveTaskState,
LoadAllTaskStates, DeleteTaskState, and CleanupCompletedTasks can be
invoked concurrently from multiple goroutines. ConfigPersistence had no
internal synchronization, creating races on the tasks/ directory:

- concurrent os.WriteFile + os.ReadFile on the same .pb file could
  yield a partial read and unmarshal error
- LoadAllTaskStates (ReadDir + per-file ReadFile) could see a
  directory entry for a file being written or deleted concurrently
- CleanupCompletedTasks (LoadAllTaskStates + DeleteTaskState) could
  race with SaveTaskState on the same file

Fix: add tasksMu sync.Mutex to ConfigPersistence, acquired at the top
of SaveTaskState, LoadTaskState, LoadAllTaskStates, DeleteTaskState,
and CleanupCompletedTasks. Extract private Locked helpers so that
CleanupCompletedTasks (which holds tasksMu) can call them internally
without deadlocking.

---------

Co-authored-by: Anton Ustyugov <anton@devops>
2026-02-24 13:41:41 -08:00
Chris Lu cba69f4593 Update layout_templ.go 2026-02-24 13:22:12 -08:00
Chris Lu 91f59e73e5 close ports 2026-02-24 13:20:21 -08:00
Chris LuandGitHub 98d89ffad7 s3api: preserve Host header port in signature verification (#8434)
Avoid stripping default ports (80/443) from the Host header in extractHostHeader.
This fixes SignatureDoesNotMatch errors when SeaweedFS is accessed via a proxy
(like Kong Ingress) that explicitly includes the port in the Host header or
X-Forwarded-Host, which S3 clients sign.

Also cleaned up unused variables and logic after refactoring.
2026-02-24 13:09:40 -08:00
Peter DoddandGitHub f4af1cc0ba feat(helm): annotations for service account (#8429) 2026-02-24 07:35:13 -08:00
Xiao WeiandGitHub 9fa95dd2c6 fix: unload leveldb not take effect (#8431) 2026-02-24 07:32:13 -08:00
Plamen NikolovandGitHub ff84ef880d fix(s3api): make ListObjectsV1 namespaced and prevent marker-echo pagination loops (#8409)
* fix(s3api): make ListObjectsV1 namespaced and stop marker-echo pagination loops

* test(s3api): harden marker-echo coverage and align V1 encoding tag

* test(s3api): cover encoded marker matching and trim redundant setup

* refactor(s3api): tighten V1 list helper visibility and test mock docs
2026-02-23 23:45:08 -08:00
Chris LuandGitHub 2d65d7f499 Embed role policies in AssumeRole STS tokens (#8421)
* Embed role policies in AssumeRole STS tokens

* Log STS policy lookup failures

* Use IAMManager provider

* Guard policy embedding role lookup
2026-02-23 22:59:53 -08:00
Chris LuandGitHub 3f58e3bf8f Use master shard sizes for EC volumes (#8423)
* Use master shard sizes for EC volumes

* Remove EC volume shard size fallback

* Remove unused EC dash imports
2026-02-23 21:59:09 -08:00
Justin CichraandGitHub 016391530b fix: plural clientType on ListExistingPeerUpdates (#8422) 2026-02-23 20:19:12 -08:00
Chris LuandGitHub 8d59ef41d5 Admin UI: replace gin with mux (#8420)
* Replace admin gin router with mux

* Update layout_templ.go

* Harden admin handlers

* Add login CSRF handling

* Fix filer copy naming conflict

* address comments

* address comments
2026-02-23 19:11:17 -08:00
Chris LuandGitHub e596542295 Move SQL engine and PostgreSQL server to their own binaries (#8417)
* Drop SQL engine and PostgreSQL server

* Split SQL tooling into weed-db and weed-sql

* move

* fix building
2026-02-23 16:27:08 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
61db4d0966 build(deps): bump github.com/rclone/rclone from 1.72.1 to 1.73.1 (#8416)
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.72.1 to 1.73.1.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.72.1...v1.73.1)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-version: 1.73.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:42:12 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
74de694447 build(deps): bump modernc.org/sqlite from 1.44.3 to 1.46.1 (#8415)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.44.3 to 1.46.1.
- [Changelog](https://gitlab.com/cznic/sqlite/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.44.3...v1.46.1)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-version: 1.46.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:42:05 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
c5e8e4f049 build(deps): bump actions/dependency-review-action from 4.8.2 to 4.8.3 (#8414)
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 4.8.2 to 4.8.3.
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](https://github.com/actions/dependency-review-action/compare/3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261...05fe4576374b728f0c523d6a13d64c25081e0803)

---
updated-dependencies:
- dependency-name: actions/dependency-review-action
  dependency-version: 4.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:57 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
c96b0913ed build(deps): bump golang.org/x/image from 0.35.0 to 0.36.0 (#8413)
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.35.0 to 0.36.0.
- [Commits](https://github.com/golang/image/compare/v0.35.0...v0.36.0)

---
updated-dependencies:
- dependency-name: golang.org/x/image
  dependency-version: 0.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:49 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b033823611 build(deps): bump helm/kind-action from 1.13.0 to 1.14.0 (#8412)
Bumps [helm/kind-action](https://github.com/helm/kind-action) from 1.13.0 to 1.14.0.
- [Release notes](https://github.com/helm/kind-action/releases)
- [Commits](https://github.com/helm/kind-action/compare/v1.13.0...v1.14.0)

---
updated-dependencies:
- dependency-name: helm/kind-action
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:40 -08:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
3044b51e7c build(deps): bump github.com/pierrec/lz4/v4 from 4.1.22 to 4.1.25 (#8411)
Bumps [github.com/pierrec/lz4/v4](https://github.com/pierrec/lz4) from 4.1.22 to 4.1.25.
- [Release notes](https://github.com/pierrec/lz4/releases)
- [Commits](https://github.com/pierrec/lz4/compare/v4.1.22...v4.1.25)

---
updated-dependencies:
- dependency-name: github.com/pierrec/lz4/v4
  dependency-version: 4.1.25
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-23 13:41:25 -08:00
66 changed files with 3156 additions and 2111 deletions
+1 -1
View File
@@ -11,4 +11,4 @@ jobs:
- name: 'Checkout Repository'
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8
- name: 'Dependency Review'
uses: actions/dependency-review-action@3c4e3dcb1aa7874d2c16be7d79418e9b7efd6261
uses: actions/dependency-review-action@05fe4576374b728f0c523d6a13d64c25081e0803
+1 -1
View File
@@ -119,7 +119,7 @@ jobs:
echo "✅ All template rendering tests passed!"
- name: Create kind cluster
uses: helm/kind-action@v1.13.0
uses: helm/kind-action@v1.14.0
- name: Run chart-testing (install)
run: ct install --target-branch ${{ github.event.repository.default_branch }} --all --chart-dirs k8s/charts
+4 -1
View File
@@ -1,4 +1,4 @@
.PHONY: test admin-generate admin-build admin-clean admin-dev admin-run admin-test admin-fmt admin-help
.PHONY: test admin-generate admin-build admin-clean admin-dev admin-run admin-test admin-fmt admin-help weed-commands
BINARY = weed
ADMIN_DIR = weed/admin
@@ -11,6 +11,9 @@ all: install
install: admin-generate
cd weed; go install
weed-commands:
cd weed && $(MAKE) weed-db weed-sql
warp_install:
go install github.com/minio/warp@v0.7.6
+116 -101
View File
@@ -1,10 +1,11 @@
package command
package main
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"io"
"os"
"os/signal"
"strings"
@@ -13,43 +14,12 @@ import (
"github.com/seaweedfs/seaweedfs/weed/server/postgres"
"github.com/seaweedfs/seaweedfs/weed/util"
flag "github.com/seaweedfs/seaweedfs/weed/util/fla9"
)
var (
dbOptions DBOptions
)
const usageLine = "weed-db -port=5432 -master=<master_server>"
type DBOptions struct {
host *string
port *int
masterAddr *string
authMethod *string
users *string
database *string
maxConns *int
idleTimeout *string
tlsCert *string
tlsKey *string
}
func init() {
cmdDB.Run = runDB // break init cycle
dbOptions.host = cmdDB.Flag.String("host", "localhost", "Database server host")
dbOptions.port = cmdDB.Flag.Int("port", 5432, "Database server port")
dbOptions.masterAddr = cmdDB.Flag.String("master", "localhost:9333", "SeaweedFS master server address")
dbOptions.authMethod = cmdDB.Flag.String("auth", "trust", "Authentication method: trust, password, md5")
dbOptions.users = cmdDB.Flag.String("users", "", "User credentials for auth (JSON format '{\"user1\":\"pass1\",\"user2\":\"pass2\"}' or file '@/path/to/users.json')")
dbOptions.database = cmdDB.Flag.String("database", "default", "Default database name")
dbOptions.maxConns = cmdDB.Flag.Int("max-connections", 100, "Maximum concurrent connections per server")
dbOptions.idleTimeout = cmdDB.Flag.String("idle-timeout", "1h", "Connection idle timeout")
dbOptions.tlsCert = cmdDB.Flag.String("tls-cert", "", "TLS certificate file path")
dbOptions.tlsKey = cmdDB.Flag.String("tls-key", "", "TLS private key file path")
}
var cmdDB = &Command{
UsageLine: "db -port=5432 -master=<master_server>",
Short: "start a PostgreSQL-compatible database server for SQL queries",
Long: `Start a PostgreSQL wire protocol compatible database server that provides SQL query access to SeaweedFS.
const longHelp = `Start a PostgreSQL wire protocol compatible database server that provides SQL query access to SeaweedFS.
This database server enables any PostgreSQL client, tool, or application to connect to SeaweedFS
and execute SQL queries against MQ topics. It implements the PostgreSQL wire protocol for maximum
@@ -58,25 +28,25 @@ compatibility with the existing PostgreSQL ecosystem.
Examples:
# Start database server on default port 5432
weed db
weed-db
# Start with MD5 authentication using JSON format (recommended)
weed db -auth=md5 -users='{"admin":"secret","readonly":"view123"}'
weed-db -auth=md5 -users='{"admin":"secret","readonly":"view123"}'
# Start with complex passwords using JSON format
weed db -auth=md5 -users='{"admin":"pass;with;semicolons","user":"password:with:colons"}'
weed-db -auth=md5 -users='{"admin":"pass;with;semicolons","user":"password:with:colons"}'
# Start with credentials from JSON file (most secure)
weed db -auth=md5 -users="@/etc/seaweedfs/users.json"
weed-db -auth=md5 -users="@/etc/seaweedfs/users.json"
# Start with custom port and master
weed db -port=5433 -master=master1:9333
weed-db -port=5433 -master=master1:9333
# Allow connections from any host
weed db -host=0.0.0.0 -port=5432
weed-db -host=0.0.0.0 -port=5432
# Start with TLS encryption
weed db -tls-cert=server.crt -tls-key=server.key
weed-db -tls-cert=server.crt -tls-key=server.key
Client Connection Examples:
@@ -95,7 +65,7 @@ Programming Language Examples:
# Python (psycopg2)
import psycopg2
conn = psycopg2.connect(
host="localhost", port=5432,
host="localhost", port=5432,
user="seaweedfs", database="default"
)
@@ -116,7 +86,7 @@ Supported SQL Operations:
- SELECT queries on MQ topics
- DESCRIBE/DESC table_name commands
- EXPLAIN query execution plans
- SHOW DATABASES/TABLES commands
- SHOW DATABASES/TABLES commands
- Aggregation functions (COUNT, SUM, AVG, MIN, MAX)
- WHERE clauses with filtering
- System columns (_timestamp_ns, _key, _source)
@@ -149,50 +119,95 @@ Performance Features:
- PostgreSQL wire protocol
- Query result streaming
`,
`
type Options struct {
Host string
Port int
MasterAddr string
AuthMethod string
Users string
Database string
MaxConns int
IdleTimeout string
TLSCert string
TLSKey string
}
func runDB(cmd *Command, args []string) bool {
// Run executes the weed-db CLI.
func Run(args []string) int {
fs := flag.NewFlagSet("weed-db", flag.ContinueOnError)
usageWriter := io.Writer(os.Stderr)
fs.SetOutput(usageWriter)
var opts Options
fs.StringVar(&opts.Host, "host", "localhost", "Database server host")
fs.IntVar(&opts.Port, "port", 5432, "Database server port")
fs.StringVar(&opts.MasterAddr, "master", "localhost:9333", "SeaweedFS master server address")
fs.StringVar(&opts.AuthMethod, "auth", "trust", "Authentication method: trust, password, md5")
fs.StringVar(&opts.Users, "users", "", "User credentials for auth (JSON format '{\"user1\":\"pass1\",\"user2\":\"pass2\"}' or file '@/path/to/users.json')")
fs.StringVar(&opts.Database, "database", "default", "Default database name")
fs.IntVar(&opts.MaxConns, "max-connections", 100, "Maximum concurrent connections per server")
fs.StringVar(&opts.IdleTimeout, "idle-timeout", "1h", "Connection idle timeout")
fs.StringVar(&opts.TLSCert, "tls-cert", "", "TLS certificate file path")
fs.StringVar(&opts.TLSKey, "tls-key", "", "TLS private key file path")
fs.Usage = func() {
fmt.Fprintf(usageWriter, "Usage: %s\n\n%s\n", usageLine, longHelp)
fmt.Fprintln(usageWriter, "Default Parameters:")
fs.PrintDefaults()
}
if err := fs.Parse(args); err != nil {
return 2
}
if !runWithOptions(&opts) {
return 1
}
return 0
}
func runWithOptions(opts *Options) bool {
util.LoadConfiguration("security", false)
// Validate options
if *dbOptions.masterAddr == "" {
// Validate options.
if opts.MasterAddr == "" {
fmt.Fprintf(os.Stderr, "Error: master address is required\n")
return false
}
// Parse authentication method
authMethod, err := parseAuthMethod(*dbOptions.authMethod)
// Parse authentication method.
authMethod, err := parseAuthMethod(opts.AuthMethod)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
return false
}
// Parse user credentials
users, err := parseUsers(*dbOptions.users, authMethod)
// Parse user credentials.
users, err := parseUsers(opts.Users, authMethod)
if err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
return false
}
// Parse idle timeout
idleTimeout, err := time.ParseDuration(*dbOptions.idleTimeout)
// Parse idle timeout.
idleTimeout, err := time.ParseDuration(opts.IdleTimeout)
if err != nil {
fmt.Fprintf(os.Stderr, "Error parsing idle timeout: %v\n", err)
return false
}
// Validate port number
if err := validatePortNumber(*dbOptions.port); err != nil {
// Validate port number.
if err := validatePortNumber(opts.Port); err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
return false
}
// Setup TLS if requested
// Setup TLS if requested.
var tlsConfig *tls.Config
if *dbOptions.tlsCert != "" && *dbOptions.tlsKey != "" {
cert, err := tls.LoadX509KeyPair(*dbOptions.tlsCert, *dbOptions.tlsKey)
if opts.TLSCert != "" && opts.TLSKey != "" {
cert, err := tls.LoadX509KeyPair(opts.TLSCert, opts.TLSKey)
if err != nil {
fmt.Fprintf(os.Stderr, "Error loading TLS certificates: %v\n", err)
return false
@@ -202,34 +217,34 @@ func runDB(cmd *Command, args []string) bool {
}
}
// Create server configuration
// Create server configuration.
config := &postgres.PostgreSQLServerConfig{
Host: *dbOptions.host,
Port: *dbOptions.port,
Host: opts.Host,
Port: opts.Port,
AuthMethod: authMethod,
Users: users,
Database: *dbOptions.database,
MaxConns: *dbOptions.maxConns,
Database: opts.Database,
MaxConns: opts.MaxConns,
IdleTimeout: idleTimeout,
TLSConfig: tlsConfig,
}
// Create database server
dbServer, err := postgres.NewPostgreSQLServer(config, *dbOptions.masterAddr)
// Create database server.
dbServer, err := postgres.NewPostgreSQLServer(config, opts.MasterAddr)
if err != nil {
fmt.Fprintf(os.Stderr, "Error creating database server: %v\n", err)
return false
}
// Print startup information
// Print startup information.
fmt.Printf("Starting SeaweedFS Database Server...\n")
fmt.Printf("Host: %s\n", *dbOptions.host)
fmt.Printf("Port: %d\n", *dbOptions.port)
fmt.Printf("Master: %s\n", *dbOptions.masterAddr)
fmt.Printf("Database: %s\n", *dbOptions.database)
fmt.Printf("Auth Method: %s\n", *dbOptions.authMethod)
fmt.Printf("Max Connections: %d\n", *dbOptions.maxConns)
fmt.Printf("Idle Timeout: %s\n", *dbOptions.idleTimeout)
fmt.Printf("Host: %s\n", opts.Host)
fmt.Printf("Port: %d\n", opts.Port)
fmt.Printf("Master: %s\n", opts.MasterAddr)
fmt.Printf("Database: %s\n", opts.Database)
fmt.Printf("Auth Method: %s\n", opts.AuthMethod)
fmt.Printf("Max Connections: %d\n", opts.MaxConns)
fmt.Printf("Idle Timeout: %s\n", opts.IdleTimeout)
if tlsConfig != nil {
fmt.Printf("TLS: Enabled\n")
} else {
@@ -240,15 +255,15 @@ func runDB(cmd *Command, args []string) bool {
}
fmt.Printf("\nDatabase Connection Examples:\n")
fmt.Printf(" psql -h %s -p %d -U seaweedfs -d %s\n", *dbOptions.host, *dbOptions.port, *dbOptions.database)
fmt.Printf(" psql -h %s -p %d -U seaweedfs -d %s\n", opts.Host, opts.Port, opts.Database)
if len(users) > 0 {
// Show first user as example
// Show first user as example.
for username := range users {
fmt.Printf(" psql -h %s -p %d -U %s -d %s\n", *dbOptions.host, *dbOptions.port, username, *dbOptions.database)
fmt.Printf(" psql -h %s -p %d -U %s -d %s\n", opts.Host, opts.Port, username, opts.Database)
break
}
}
fmt.Printf(" postgresql://%s:%d/%s\n", *dbOptions.host, *dbOptions.port, *dbOptions.database)
fmt.Printf(" postgresql://%s:%d/%s\n", opts.Host, opts.Port, opts.Database)
fmt.Printf("\nSupported Operations:\n")
fmt.Printf(" - SELECT queries on MQ topics\n")
@@ -261,26 +276,26 @@ func runDB(cmd *Command, args []string) bool {
fmt.Printf("\nReady for database connections!\n\n")
// Start the server
// Start the server.
err = dbServer.Start()
if err != nil {
fmt.Fprintf(os.Stderr, "Error starting database server: %v\n", err)
return false
}
// Set up signal handling for graceful shutdown
// Set up signal handling for graceful shutdown.
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
// Wait for shutdown signal
// Wait for shutdown signal.
<-sigChan
fmt.Printf("\nReceived shutdown signal, stopping database server...\n")
// Create context with timeout for graceful shutdown
// Create context with timeout for graceful shutdown.
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// Stop the server with timeout
// Stop the server with timeout.
done := make(chan error, 1)
go func() {
done <- dbServer.Stop()
@@ -301,7 +316,7 @@ func runDB(cmd *Command, args []string) bool {
return true
}
// parseAuthMethod parses the authentication method string
// parseAuthMethod parses the authentication method string.
func parseAuthMethod(method string) (postgres.AuthMethod, error) {
switch strings.ToLower(method) {
case "trust":
@@ -315,7 +330,7 @@ func parseAuthMethod(method string) (postgres.AuthMethod, error) {
}
}
// parseUsers parses the user credentials string with support for secure formats only
// parseUsers parses the user credentials string with support for secure formats only.
// Supported formats:
// 1. JSON format: {"username":"password","username2":"password2"}
// 2. File format: /path/to/users.json or @/path/to/users.json
@@ -323,41 +338,41 @@ func parseUsers(usersStr string, authMethod postgres.AuthMethod) (map[string]str
users := make(map[string]string)
if usersStr == "" {
// No users specified
// No users specified.
if authMethod != postgres.AuthTrust {
return nil, fmt.Errorf("users must be specified when auth method is not 'trust'")
}
return users, nil
}
// Trim whitespace
// Trim whitespace.
usersStr = strings.TrimSpace(usersStr)
// Determine format and parse accordingly
// Determine format and parse accordingly.
if strings.HasPrefix(usersStr, "{") && strings.HasSuffix(usersStr, "}") {
// JSON format
// JSON format.
return parseUsersJSON(usersStr, authMethod)
}
// Check if it's a file path (with or without @ prefix) before declaring invalid format
// Check if it's a file path (with or without @ prefix) before declaring invalid format.
filePath := strings.TrimPrefix(usersStr, "@")
if _, err := os.Stat(filePath); err == nil {
// File format
return parseUsersFile(usersStr, authMethod) // Pass original string to preserve @ handling
// File format.
return parseUsersFile(usersStr, authMethod) // Pass original string to preserve @ handling.
}
// Invalid format
// Invalid format.
return nil, fmt.Errorf("invalid user credentials format. Use JSON format '{\"user\":\"pass\"}' or file format '@/path/to/users.json' or 'path/to/users.json'. Legacy semicolon-separated format is no longer supported")
}
// parseUsersJSON parses user credentials from JSON format
// parseUsersJSON parses user credentials from JSON format.
func parseUsersJSON(jsonStr string, authMethod postgres.AuthMethod) (map[string]string, error) {
var users map[string]string
if err := json.Unmarshal([]byte(jsonStr), &users); err != nil {
return nil, fmt.Errorf("invalid JSON format for users: %v", err)
}
// Validate users
// Validate users.
for username, password := range users {
if username == "" {
return nil, fmt.Errorf("empty username in JSON user specification")
@@ -370,12 +385,12 @@ func parseUsersJSON(jsonStr string, authMethod postgres.AuthMethod) (map[string]
return users, nil
}
// parseUsersFile parses user credentials from a JSON file
// parseUsersFile parses user credentials from a JSON file.
func parseUsersFile(filePath string, authMethod postgres.AuthMethod) (map[string]string, error) {
// Remove @ prefix if present
// Remove @ prefix if present.
filePath = strings.TrimPrefix(filePath, "@")
// Read file content
// Read file content.
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("failed to read users file '%s': %v", filePath, err)
@@ -383,16 +398,16 @@ func parseUsersFile(filePath string, authMethod postgres.AuthMethod) (map[string
contentStr := strings.TrimSpace(string(content))
// File must contain JSON format
// File must contain JSON format.
if !strings.HasPrefix(contentStr, "{") || !strings.HasSuffix(contentStr, "}") {
return nil, fmt.Errorf("users file '%s' must contain JSON format: {\"user\":\"pass\"}. Legacy formats are no longer supported", filePath)
}
// Parse as JSON
// Parse as JSON.
return parseUsersJSON(contentStr, authMethod)
}
// validatePortNumber validates that the port number is reasonable
// validatePortNumber validates that the port number is reasonable.
func validatePortNumber(port int) error {
if port < 1 || port > 65535 {
return fmt.Errorf("port number must be between 1 and 65535, got %d", port)
+7
View File
@@ -0,0 +1,7 @@
package main
import "os"
func main() {
os.Exit(Run(os.Args[1:]))
}
+7
View File
@@ -0,0 +1,7 @@
package main
import "os"
func main() {
os.Exit(Run(os.Args[1:]))
}
+134 -107
View File
@@ -1,4 +1,4 @@
package command
package main
import (
"context"
@@ -13,28 +13,24 @@ import (
"github.com/peterh/liner"
"github.com/seaweedfs/seaweedfs/weed/query/engine"
flag "github.com/seaweedfs/seaweedfs/weed/util/fla9"
"github.com/seaweedfs/seaweedfs/weed/util/grace"
"github.com/seaweedfs/seaweedfs/weed/util/sqlutil"
)
func init() {
cmdSql.Run = runSql
}
const usageLine = "weed-sql [-master=localhost:9333] [-interactive] [-file=query.sql] [-output=table|json|csv] [-database=dbname] [-query=\"SQL\"]"
var cmdSql = &Command{
UsageLine: "sql [-master=localhost:9333] [-interactive] [-file=query.sql] [-output=table|json|csv] [-database=dbname] [-query=\"SQL\"]",
Short: "advanced SQL query interface for SeaweedFS MQ topics with multiple execution modes",
Long: `Enhanced SQL interface for SeaweedFS Message Queue topics with multiple execution modes.
const longHelp = `Enhanced SQL interface for SeaweedFS Message Queue topics with multiple execution modes.
Execution Modes:
- Interactive shell (default): weed sql -interactive
- Single query: weed sql -query "SELECT * FROM user_events"
- Batch from file: weed sql -file queries.sql
- Context switching: weed sql -database analytics -interactive
- Interactive shell (default): weed-sql -interactive
- Single query: weed-sql -query "SELECT * FROM user_events"
- Batch from file: weed-sql -file queries.sql
- Context switching: weed-sql -database analytics -interactive
Output Formats:
- table: ASCII table format (default for interactive)
- json: JSON format (default for non-interactive)
- json: JSON format (default for non-interactive)
- csv: Comma-separated values
Features:
@@ -45,24 +41,23 @@ Features:
- Database context switching
Examples:
weed sql -interactive
weed sql -query "SHOW DATABASES" -output json
weed sql -file batch_queries.sql -output csv
weed sql -database analytics -query "SELECT COUNT(*) FROM metrics"
weed sql -master broker1:9333 -interactive
`,
weed-sql -interactive
weed-sql -query "SHOW DATABASES" -output json
weed-sql -file batch_queries.sql -output csv
weed-sql -database analytics -query "SELECT COUNT(*) FROM metrics"
weed-sql -master broker1:9333 -interactive
`
type Options struct {
Master string
Interactive bool
File string
Output string
Database string
Query string
}
var (
sqlMaster = cmdSql.Flag.String("master", "localhost:9333", "SeaweedFS master server HTTP address")
sqlInteractive = cmdSql.Flag.Bool("interactive", false, "start interactive shell mode")
sqlFile = cmdSql.Flag.String("file", "", "execute SQL queries from file")
sqlOutput = cmdSql.Flag.String("output", "", "output format: table, json, csv (auto-detected if not specified)")
sqlDatabase = cmdSql.Flag.String("database", "", "default database context")
sqlQuery = cmdSql.Flag.String("query", "", "execute single SQL query")
)
// OutputFormat represents different output formatting options
// OutputFormat represents different output formatting options.
type OutputFormat string
const (
@@ -71,50 +66,82 @@ const (
OutputCSV OutputFormat = "csv"
)
// SQLContext holds the execution context for SQL operations
// SQLContext holds the execution context for SQL operations.
type SQLContext struct {
engine *engine.SQLEngine
currentDatabase string
outputFormat OutputFormat
interactive bool
master string
}
func runSql(command *Command, args []string) bool {
// Initialize SQL engine with master address for service discovery
sqlEngine := engine.NewSQLEngine(*sqlMaster)
// Run executes the weed-sql CLI.
func Run(args []string) int {
fs := flag.NewFlagSet("weed-sql", flag.ContinueOnError)
usageWriter := io.Writer(os.Stderr)
fs.SetOutput(usageWriter)
// Determine execution mode and output format
interactive := *sqlInteractive || (*sqlQuery == "" && *sqlFile == "")
outputFormat := determineOutputFormat(*sqlOutput, interactive)
var opts Options
fs.StringVar(&opts.Master, "master", "localhost:9333", "SeaweedFS master server HTTP address")
fs.BoolVar(&opts.Interactive, "interactive", false, "start interactive shell mode")
fs.StringVar(&opts.File, "file", "", "execute SQL queries from file")
fs.StringVar(&opts.Output, "output", "", "output format: table, json, csv (auto-detected if not specified)")
fs.StringVar(&opts.Database, "database", "", "default database context")
fs.StringVar(&opts.Query, "query", "", "execute single SQL query")
// Create SQL context
fs.Usage = func() {
fmt.Fprintf(usageWriter, "Usage: %s\n\n%s\n", usageLine, longHelp)
fmt.Fprintln(usageWriter, "Default Parameters:")
fs.PrintDefaults()
}
if err := fs.Parse(args); err != nil {
return 2
}
if !runWithOptions(&opts) {
return 1
}
return 0
}
func runWithOptions(opts *Options) bool {
// Initialize SQL engine with master address for service discovery.
sqlEngine := engine.NewSQLEngine(opts.Master)
// Determine execution mode and output format.
interactive := opts.Interactive || (opts.Query == "" && opts.File == "")
outputFormat := determineOutputFormat(opts.Output, interactive)
// Create SQL context.
ctx := &SQLContext{
engine: sqlEngine,
currentDatabase: *sqlDatabase,
currentDatabase: opts.Database,
outputFormat: outputFormat,
interactive: interactive,
master: opts.Master,
}
// Set current database in SQL engine if specified via command line
if *sqlDatabase != "" {
ctx.engine.GetCatalog().SetCurrentDatabase(*sqlDatabase)
// Set current database in SQL engine if specified via command line.
if opts.Database != "" {
ctx.engine.GetCatalog().SetCurrentDatabase(opts.Database)
}
// Execute based on mode
// Execute based on mode.
switch {
case *sqlQuery != "":
// Single query mode
return executeSingleQuery(ctx, *sqlQuery)
case *sqlFile != "":
// Batch file mode
return executeFileQueries(ctx, *sqlFile)
case opts.Query != "":
// Single query mode.
return executeSingleQuery(ctx, opts.Query)
case opts.File != "":
// Batch file mode.
return executeFileQueries(ctx, opts.File)
default:
// Interactive mode
// Interactive mode.
return runInteractiveShell(ctx)
}
}
// determineOutputFormat selects the appropriate output format
// determineOutputFormat selects the appropriate output format.
func determineOutputFormat(specified string, interactive bool) OutputFormat {
switch strings.ToLower(specified) {
case "table":
@@ -124,7 +151,7 @@ func determineOutputFormat(specified string, interactive bool) OutputFormat {
case "csv":
return OutputCSV
default:
// Auto-detect based on mode
// Auto-detect based on mode.
if interactive {
return OutputTable
}
@@ -132,18 +159,18 @@ func determineOutputFormat(specified string, interactive bool) OutputFormat {
}
}
// executeSingleQuery executes a single query and outputs the result
// executeSingleQuery executes a single query and outputs the result.
func executeSingleQuery(ctx *SQLContext, query string) bool {
if ctx.outputFormat != OutputTable {
// Suppress banner for non-interactive output
// Suppress banner for non-interactive output.
return executeAndDisplay(ctx, query, false)
}
fmt.Printf("Executing query against %s...\n", *sqlMaster)
fmt.Printf("Executing query against %s...\n", ctx.master)
return executeAndDisplay(ctx, query, true)
}
// executeFileQueries processes SQL queries from a file
// executeFileQueries processes SQL queries from a file.
func executeFileQueries(ctx *SQLContext, filename string) bool {
content, err := os.ReadFile(filename)
if err != nil {
@@ -152,10 +179,10 @@ func executeFileQueries(ctx *SQLContext, filename string) bool {
}
if ctx.outputFormat == OutputTable && ctx.interactive {
fmt.Printf("Executing queries from %s against %s...\n", filename, *sqlMaster)
fmt.Printf("Executing queries from %s against %s...\n", filename, ctx.master)
}
// Split file content into individual queries (robust approach)
// Split file content into individual queries (robust approach).
queries := sqlutil.SplitStatements(string(content))
for i, query := range queries {
@@ -176,11 +203,11 @@ func executeFileQueries(ctx *SQLContext, filename string) bool {
return true
}
// runInteractiveShell starts the enhanced interactive shell with readline support
// runInteractiveShell starts the enhanced interactive shell with readline support.
func runInteractiveShell(ctx *SQLContext) bool {
fmt.Println("SeaweedFS Enhanced SQL Interface")
fmt.Println("Type 'help;' for help, 'exit;' to quit")
fmt.Printf("Connected to master: %s\n", *sqlMaster)
fmt.Printf("Connected to master: %s\n", ctx.master)
if ctx.currentDatabase != "" {
fmt.Printf("Current database: %s\n", ctx.currentDatabase)
}
@@ -188,24 +215,24 @@ func runInteractiveShell(ctx *SQLContext) bool {
fmt.Println("Use up/down arrows for command history")
fmt.Println()
// Initialize liner for readline functionality
// Initialize liner for readline functionality.
line := liner.NewLiner()
defer line.Close()
// Handle Ctrl+C gracefully
// Handle Ctrl+C gracefully.
line.SetCtrlCAborts(true)
grace.OnInterrupt(func() {
line.Close()
})
// Load command history
// Load command history.
historyPath := path.Join(os.TempDir(), "weed-sql-history")
if f, err := os.Open(historyPath); err == nil {
line.ReadHistory(f)
f.Close()
}
// Save history on exit
// Save history on exit.
defer func() {
if f, err := os.Create(historyPath); err == nil {
line.WriteHistory(f)
@@ -216,7 +243,7 @@ func runInteractiveShell(ctx *SQLContext) bool {
var queryBuffer strings.Builder
for {
// Show prompt with current database context
// Show prompt with current database context.
var prompt string
if queryBuffer.Len() == 0 {
if ctx.currentDatabase != "" {
@@ -225,10 +252,10 @@ func runInteractiveShell(ctx *SQLContext) bool {
prompt = "seaweedfs> "
}
} else {
prompt = " -> " // Continuation prompt
prompt = " -> " // Continuation prompt.
}
// Read line with readline support
// Read line with readline support.
input, err := line.Prompt(prompt)
if err != nil {
if err == liner.ErrPromptAborted {
@@ -244,30 +271,30 @@ func runInteractiveShell(ctx *SQLContext) bool {
lineStr := strings.TrimSpace(input)
// Handle empty lines
// Handle empty lines.
if lineStr == "" {
continue
}
// Accumulate lines in query buffer
// Accumulate lines in query buffer.
if queryBuffer.Len() > 0 {
queryBuffer.WriteString(" ")
}
queryBuffer.WriteString(lineStr)
// Check if we have a complete statement (ends with semicolon or special command)
// Check if we have a complete statement (ends with semicolon or special command).
fullQuery := strings.TrimSpace(queryBuffer.String())
isComplete := strings.HasSuffix(lineStr, ";") ||
isSpecialCommand(fullQuery)
if !isComplete {
continue // Continue reading more lines
continue // Continue reading more lines.
}
// Add completed command to history
// Add completed command to history.
line.AppendHistory(fullQuery)
// Handle special commands (with or without semicolon)
// Handle special commands (with or without semicolon).
cleanQuery := strings.TrimSuffix(fullQuery, ";")
cleanQuery = strings.TrimSpace(cleanQuery)
@@ -282,19 +309,19 @@ func runInteractiveShell(ctx *SQLContext) bool {
continue
}
// Handle database switching - use proper SQL parser instead of manual parsing
// Handle database switching - use proper SQL parser instead of manual parsing.
if strings.HasPrefix(strings.ToUpper(cleanQuery), "USE ") {
// Execute USE statement through the SQL engine for proper parsing
// Execute USE statement through the SQL engine for proper parsing.
result, err := ctx.engine.ExecuteSQL(context.Background(), cleanQuery)
if err != nil {
fmt.Printf("Error: %v\n\n", err)
} else if result.Error != nil {
fmt.Printf("Error: %v\n\n", result.Error)
} else {
// Extract the database name from the result message for CLI context
// Extract the database name from the result message for CLI context.
if len(result.Rows) > 0 && len(result.Rows[0]) > 0 {
message := result.Rows[0][0].ToString()
// Extract database name from "Database changed to: dbname"
// Extract database name from "Database changed to: dbname".
if strings.HasPrefix(message, "Database changed to: ") {
ctx.currentDatabase = strings.TrimPrefix(message, "Database changed to: ")
}
@@ -305,7 +332,7 @@ func runInteractiveShell(ctx *SQLContext) bool {
continue
}
// Handle output format switching
// Handle output format switching.
if strings.HasPrefix(strings.ToUpper(cleanQuery), "\\FORMAT ") {
format := strings.TrimSpace(strings.TrimPrefix(strings.ToUpper(cleanQuery), "\\FORMAT "))
switch format {
@@ -325,22 +352,22 @@ func runInteractiveShell(ctx *SQLContext) bool {
continue
}
// Execute SQL query (without semicolon)
// Execute SQL query (without semicolon).
executeAndDisplay(ctx, cleanQuery, true)
// Reset buffer for next query
// Reset buffer for next query.
queryBuffer.Reset()
}
return true
}
// isSpecialCommand checks if a command is a special command that doesn't require semicolon
// isSpecialCommand checks if a command is a special command that doesn't require semicolon.
func isSpecialCommand(query string) bool {
cleanQuery := strings.TrimSuffix(strings.TrimSpace(query), ";")
cleanQuery = strings.ToLower(cleanQuery)
// Special commands that work with or without semicolon
// Special commands that work with or without semicolon.
specialCommands := []string{
"exit", "quit", "\\q", "help",
}
@@ -351,7 +378,7 @@ func isSpecialCommand(query string) bool {
}
}
// Commands that are exactly specific commands (not just prefixes)
// Commands that are exactly specific commands (not just prefixes).
parts := strings.Fields(strings.ToUpper(cleanQuery))
if len(parts) == 0 {
return false
@@ -360,11 +387,11 @@ func isSpecialCommand(query string) bool {
strings.HasPrefix(strings.ToUpper(cleanQuery), "\\FORMAT ")
}
// executeAndDisplay executes a query and displays the result in the specified format
// executeAndDisplay executes a query and displays the result in the specified format.
func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
startTime := time.Now()
// Execute the query
// Execute the query.
execCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
@@ -397,7 +424,7 @@ func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
return false
}
// Display results in the specified format
// Display results in the specified format.
switch ctx.outputFormat {
case OutputTable:
displayTableResult(result)
@@ -407,8 +434,8 @@ func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
displayCSVResult(result)
}
// Show execution time for interactive/table mode
// Only show timing if there are columns or if result is truly empty
// Show execution time for interactive/table mode.
// Only show timing if there are columns or if result is truly empty.
if showTiming && ctx.outputFormat == OutputTable && (len(result.Columns) > 0 || len(result.Rows) == 0) {
elapsed := time.Since(startTime)
fmt.Printf("\n(%d rows in set, %.3f sec)\n\n", len(result.Rows), elapsed.Seconds())
@@ -417,20 +444,20 @@ func executeAndDisplay(ctx *SQLContext, query string, showTiming bool) bool {
return true
}
// displayTableResult formats and displays query results in ASCII table format
// displayTableResult formats and displays query results in ASCII table format.
func displayTableResult(result *engine.QueryResult) {
if len(result.Columns) == 0 {
fmt.Println("Empty result set")
return
}
// Calculate column widths for formatting
// Calculate column widths for formatting.
colWidths := make([]int, len(result.Columns))
for i, col := range result.Columns {
colWidths[i] = len(col)
}
// Check data for wider columns
// Check data for wider columns.
for _, row := range result.Rows {
for i, val := range row {
if i < len(colWidths) {
@@ -442,28 +469,28 @@ func displayTableResult(result *engine.QueryResult) {
}
}
// Print header separator
// Print header separator.
fmt.Print("+")
for _, width := range colWidths {
fmt.Print(strings.Repeat("-", width+2) + "+")
}
fmt.Println()
// Print column headers
// Print column headers.
fmt.Print("|")
for i, col := range result.Columns {
fmt.Printf(" %-*s |", colWidths[i], col)
}
fmt.Println()
// Print separator
// Print separator.
fmt.Print("+")
for _, width := range colWidths {
fmt.Print(strings.Repeat("-", width+2) + "+")
}
fmt.Println()
// Print data rows
// Print data rows.
for _, row := range result.Rows {
fmt.Print("|")
for i, val := range row {
@@ -474,7 +501,7 @@ func displayTableResult(result *engine.QueryResult) {
fmt.Println()
}
// Print bottom separator
// Print bottom separator.
fmt.Print("+")
for _, width := range colWidths {
fmt.Print(strings.Repeat("-", width+2) + "+")
@@ -482,16 +509,16 @@ func displayTableResult(result *engine.QueryResult) {
fmt.Println()
}
// displayJSONResult outputs query results in JSON format
// displayJSONResult outputs query results in JSON format.
func displayJSONResult(result *engine.QueryResult) {
// Convert result to JSON-friendly format
// Convert result to JSON-friendly format.
jsonResult := map[string]interface{}{
"columns": result.Columns,
"rows": make([]map[string]interface{}, len(result.Rows)),
"count": len(result.Rows),
}
// Convert rows to JSON objects
// Convert rows to JSON objects.
for i, row := range result.Rows {
rowObj := make(map[string]interface{})
for j, val := range row {
@@ -502,7 +529,7 @@ func displayJSONResult(result *engine.QueryResult) {
jsonResult["rows"].([]map[string]interface{})[i] = rowObj
}
// Marshal and print JSON
// Marshal and print JSON.
jsonBytes, err := json.MarshalIndent(jsonResult, "", " ")
if err != nil {
fmt.Printf("Error formatting JSON: %v\n", err)
@@ -512,11 +539,11 @@ func displayJSONResult(result *engine.QueryResult) {
fmt.Println(string(jsonBytes))
}
// displayCSVResult outputs query results in CSV format
// displayCSVResult outputs query results in CSV format.
func displayCSVResult(result *engine.QueryResult) {
// Handle execution plan results specially to avoid CSV quoting issues
// Handle execution plan results specially to avoid CSV quoting issues.
if len(result.Columns) == 1 && result.Columns[0] == "Query Execution Plan" {
// For execution plans, output directly without CSV encoding to avoid quotes
// For execution plans, output directly without CSV encoding to avoid quotes.
for _, row := range result.Rows {
if len(row) > 0 {
fmt.Println(row[0].ToString())
@@ -525,17 +552,17 @@ func displayCSVResult(result *engine.QueryResult) {
return
}
// Standard CSV output for regular query results
// Standard CSV output for regular query results.
writer := csv.NewWriter(os.Stdout)
defer writer.Flush()
// Write headers
// Write headers.
if err := writer.Write(result.Columns); err != nil {
fmt.Printf("Error writing CSV headers: %v\n", err)
return
}
// Write data rows
// Write data rows.
for _, row := range result.Rows {
csvRow := make([]string, len(row))
for i, val := range row {
@@ -553,7 +580,7 @@ func showEnhancedHelp() {
METADATA OPERATIONS:
SHOW DATABASES; - List all MQ namespaces
SHOW TABLES; - List all topics in current namespace
SHOW TABLES; - List all topics in current namespace
SHOW TABLES FROM database; - List topics in specific namespace
DESCRIBE table_name; - Show table schema
@@ -581,7 +608,7 @@ SPECIAL COMMANDS:
EXTENDED WHERE OPERATORS:
=, <, >, <=, >= - Comparison operators
!=, <> - Not equal operators
!=, <> - Not equal operators
LIKE 'pattern%' - Pattern matching (% = any chars, _ = single char)
IN (value1, value2, ...) - Multi-value matching
AND, OR - Logical operators
+14 -19
View File
@@ -94,7 +94,7 @@ require (
gocloud.dev/pubsub/rabbitpubsub v0.44.0
golang.org/x/crypto v0.48.0
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546
golang.org/x/image v0.35.0
golang.org/x/image v0.36.0
golang.org/x/net v0.49.0
golang.org/x/oauth2 v0.34.0
golang.org/x/sys v0.41.0
@@ -109,7 +109,7 @@ require (
modernc.org/b v1.0.0 // indirect
modernc.org/mathutil v1.7.1
modernc.org/memory v1.11.0 // indirect
modernc.org/sqlite v1.44.3
modernc.org/sqlite v1.46.1
modernc.org/strutil v1.2.1
)
@@ -131,8 +131,6 @@ require (
github.com/cognusion/imaging v1.0.2
github.com/fluent/fluent-logger-golang v1.10.1
github.com/getsentry/sentry-go v0.42.0
github.com/gin-contrib/sessions v1.0.4
github.com/gin-gonic/gin v1.11.0
github.com/go-ldap/ldap/v3 v3.4.12
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/flatbuffers/go v0.0.0-20230108230133-3b8644d32c50
@@ -148,7 +146,7 @@ require (
github.com/parquet-go/parquet-go v0.26.4
github.com/pkg/sftp v1.13.10
github.com/rabbitmq/amqp091-go v1.10.0
github.com/rclone/rclone v1.72.1
github.com/rclone/rclone v1.73.1
github.com/rdleal/intervalst v1.5.0
github.com/redis/go-redis/v9 v9.18.0
github.com/schollz/progressbar/v3 v3.19.0
@@ -180,6 +178,7 @@ require (
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/to v0.4.1 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 // indirect
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 // indirect
github.com/anchore/go-lzo v0.1.0 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
@@ -212,6 +211,8 @@ require (
github.com/docker/docker v28.5.0+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dromara/dongle v1.0.1 // indirect
github.com/gin-gonic/gin v1.11.0 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/goccy/go-yaml v1.18.0 // indirect
@@ -226,6 +227,7 @@ require (
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.7 // indirect
github.com/hashicorp/hcl v1.0.1-vault-7 // indirect
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
@@ -257,8 +259,9 @@ require (
github.com/pierrre/geohash v1.0.0 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/pterm/pterm v0.12.81 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/quic-go/quic-go v0.57.0 // indirect
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 // indirect
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
@@ -269,6 +272,7 @@ require (
github.com/substrait-io/substrait-protobuf/go v0.71.0 // indirect
github.com/twpayne/go-geom v1.4.1 // indirect
github.com/twpayne/go-kml v1.5.2 // indirect
github.com/tyler-smith/go-bip39 v1.1.0 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
@@ -279,6 +283,7 @@ require (
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.37.0 // indirect
go.opentelemetry.io/otel/exporters/zipkin v1.36.0 // indirect
go.opentelemetry.io/proto/otlp v1.7.0 // indirect
go.uber.org/mock v0.5.2 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.32.0 // indirect
@@ -341,15 +346,12 @@ require (
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
github.com/bytedance/sonic v1.14.0 // indirect
github.com/bytedance/sonic/loader v0.3.0 // indirect
github.com/calebcase/tmpfile v1.0.3 // indirect
github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.13.0 // indirect
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
github.com/cloudwego/base64x v0.1.6 // indirect
github.com/cncf/xds/go v0.0.0-20251022180443-0feb69152e9f // indirect
github.com/colinmarc/hdfs/v2 v2.4.0 // indirect
github.com/creasty/defaults v1.8.0 // indirect
@@ -369,8 +371,7 @@ require (
github.com/flynn/noise v1.1.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.11 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/gin-contrib/sse v1.1.0 // indirect
github.com/go-chi/chi/v5 v5.2.3 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
@@ -389,10 +390,9 @@ require (
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
github.com/gorilla/context v1.1.2 // indirect
github.com/gorilla/schema v1.4.1 // indirect
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/gorilla/sessions v1.4.0 // indirect
github.com/gorilla/sessions v1.4.0
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
@@ -402,8 +402,6 @@ require (
github.com/hashicorp/go-msgpack/v2 v2.1.2 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
github.com/hashicorp/golang-lru v0.6.0 // indirect
github.com/henrybear327/Proton-API-Bridge v1.0.0 // indirect
github.com/henrybear327/go-proton-api v1.0.0 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
@@ -445,7 +443,7 @@ require (
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/pierrec/lz4/v4 v4.1.22
github.com/pierrec/lz4/v4 v4.1.25
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c // indirect
github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c // indirect
github.com/pingcap/kvproto v0.0.0-20230403051650-e166ae588106 // indirect
@@ -476,9 +474,7 @@ require (
github.com/tinylib/msgp v1.5.0 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/twmb/murmur3 v1.1.8 // indirect
github.com/ugorji/go/codec v1.3.0 // indirect
github.com/unknwon/goconfig v1.0.0 // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
@@ -504,7 +500,6 @@ require (
go.opentelemetry.io/otel/trace v1.38.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.1 // indirect
golang.org/x/arch v0.20.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/time v0.14.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20251124214823-79d6a2a48846 // indirect
+24 -18
View File
@@ -596,6 +596,8 @@ github.com/DataDog/datadog-go v3.2.0+incompatible/go.mod h1:LButxg5PwREeZtORoXG3
github.com/DataDog/zstd v1.5.2/go.mod h1:g4AWEaM3yOg3HYfnJ3YIawPnVdXJh9QME85blwSAmyw=
github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e h1:rd4bOvKmDIx0WeTv9Qz+hghsgyjikFiPrseXHlKepO0=
github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e/go.mod h1:blbwPQh4DTlCZEfk1BLU4oMIhLda2U+A840Uag9DsZw=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 h1:W8S9TrAyZ4//3PXsU6+Bi+fe/6uIL986GyS7PVzIDL4=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
github.com/Files-com/files-sdk-go/v3 v3.2.264 h1:lMHTplAYI9FtmCo/QOcpRxmPA5REVAct1r2riQmDQKw=
github.com/Files-com/files-sdk-go/v3 v3.2.264/go.mod h1:wGqkOzRu/ClJibvDgcfuJNAqI2nLhe8g91tPlDKRCdE=
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0 h1:sBEjpZlNHzK1voKq9695PJSX2o5NEXl7/OL3coiIY0c=
@@ -962,6 +964,8 @@ github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDD
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
github.com/dromara/dongle v1.0.1 h1:si/7UP/EXxnFVZok1cNos70GiMGxInAYMilHQFP5dJs=
github.com/dromara/dongle v1.0.1/go.mod h1:ebFhTaDgxaDIKppycENTWlBsxz8mWCPWOLnsEgDpMv4=
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 h1:FT+t0UEDykcor4y3dMVKXIiWJETBpRgERYTGlmMd7HU=
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5/go.mod h1:rSS3kM9XMzSQ6pw91Qgd6yB5jdt70N4OdtrAf74As5M=
github.com/dsnet/try v0.0.3 h1:ptR59SsrcFUYbT/FhAbKTV6iLkeD6O18qfIWRml2fqI=
@@ -989,6 +993,7 @@ github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff h1:4N8wnS3f1hNHS
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff/go.mod h1:HMJKR5wlh/ziNp+sHEDV2ltblO4JD2+IdDOWtGcQBTM=
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/emmansun/gmsm v0.15.5/go.mod h1:2m4jygryohSWkaSduFErgCwQKab5BNjURoFrn2DNwyU=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
@@ -1056,16 +1061,14 @@ github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNe
github.com/getsentry/sentry-go v0.42.0 h1:eeFMACuZTbUQf90RE8dE4tXeSe4CZyfvR1MBL7RLEt8=
github.com/getsentry/sentry-go v0.42.0/go.mod h1:eRXCoh3uvmjQLY6qu63BjUZnaBu5L5WhMV1RwYO8W5s=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sessions v1.0.4 h1:ha6CNdpYiTOK/hTp05miJLbpTSNfOnFg5Jm2kbcqy8U=
github.com/gin-contrib/sessions v1.0.4/go.mod h1:ccmkrb2z6iU2osiAHZG3x3J4suJK+OU27oqzlWOqQgs=
github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w=
github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM=
github.com/gin-gonic/gin v1.11.0 h1:OW/6PLjyusp2PPXtyxKHU0RbX6I/l28FTdDlae5ueWk=
github.com/gin-gonic/gin v1.11.0/go.mod h1:+iq/FyxlGzII0KHiBGjuNn4UNENUlKbGlNmc+W50Dls=
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 h1:BP4M0CvQ4S3TGls2FvczZtj5Re/2ZzkV9VwqPHH/3Bo=
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-chi/chi/v5 v5.2.3 h1:WQIt9uxdsAbgIYgid+BpYc+liqQZGMHRaUwp0JUcvdE=
github.com/go-chi/chi/v5 v5.2.3/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-errors/errors v1.5.1 h1:ZwEMSLRCapFLflTpT7NKaAc7ukJ8ZPEjzlxt8rPN8bk=
@@ -1309,8 +1312,6 @@ github.com/gookit/color v1.5.4 h1:FZmqs7XOyGgCAxmWyPslpiok1k05wmY3SJTytgvYFs0=
github.com/gookit/color v1.5.4/go.mod h1:pZJOeOS8DM43rXbp4AZo1n9zCU2qjpcRko0b6/QJi9w=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/gorilla/context v1.1.2 h1:WRkNAv2uoa03QNIc1A6u4O7DAGMUVoopZhkiXWA2V1o=
github.com/gorilla/context v1.1.2/go.mod h1:KDPwT9i/MeWHiLl90fuTgrt4/wPcv75vFAZLaOOcbxM=
github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/schema v1.4.1 h1:jUg5hUjCSDZpNGLuXQOgIWGdlgrIdYvgQ0wZtdK1M3E=
@@ -1390,10 +1391,6 @@ github.com/hashicorp/raft-boltdb/v2 v2.3.1 h1:ackhdCNPKblmOhjEU9+4lHSJYFkJd6Jqyv
github.com/hashicorp/raft-boltdb/v2 v2.3.1/go.mod h1:n4S+g43dXF1tqDT+yzcXHhXM6y7MrlUd3TTwGRcUvQE=
github.com/hashicorp/vault/api v1.22.0 h1:+HYFquE35/B74fHoIeXlZIP2YADVboaPjaSicHEZiH0=
github.com/hashicorp/vault/api v1.22.0/go.mod h1:IUZA2cDvr4Ok3+NtK2Oq/r+lJeXkeCrHRmqdyWfpmGM=
github.com/henrybear327/Proton-API-Bridge v1.0.0 h1:gjKAaWfKu++77WsZTHg6FUyPC5W0LTKWQciUm8PMZb0=
github.com/henrybear327/Proton-API-Bridge v1.0.0/go.mod h1:gunH16hf6U74W2b9CGDaWRadiLICsoJ6KRkSt53zLts=
github.com/henrybear327/go-proton-api v1.0.0 h1:zYi/IbjLwFAW7ltCeqXneUGJey0TN//Xo851a/BgLXw=
github.com/henrybear327/go-proton-api v1.0.0/go.mod h1:w63MZuzufKcIZ93pwRgiOtxMXYafI8H74D77AxytOBc=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
@@ -1410,6 +1407,8 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/inhies/go-bytesize v0.0.0-20220417184213-4913239db9cf h1:FtEj8sfIcaaBfAKrE1Cwb61YDtYq9JxChK1c7AKce7s=
github.com/inhies/go-bytesize v0.0.0-20220417184213-4913239db9cf/go.mod h1:yrqSXGoD/4EKfF26AOGzscPOgTTJcyAwM2rpixWT+t4=
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c h1:r+KtxPyrhsYeNbsfeqTfEM8xRdwgV6LuNhLZxpXecb4=
github.com/internxt/rclone-adapter v0.0.0-20260213125353-6f59c89fcb7c/go.mod h1:vdPya4AIcDjvng4ViaAzqjegJf0VHYpYHQguFx5xBp0=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
@@ -1714,8 +1713,8 @@ github.com/phpdave11/gofpdf v1.4.2/go.mod h1:zpO6xFn9yxo3YLyMvW8HcKWVdbNqgIfOOp2
github.com/phpdave11/gofpdi v1.0.12/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
github.com/phpdave11/gofpdi v1.0.13/go.mod h1:vBmVV0Do6hSBHC8uKUQ71JGW+ZGQq74llk/7bXwjDoI=
github.com/pierrec/lz4/v4 v4.1.15/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0=
github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pierrre/compare v1.0.2 h1:k4IUsHgh+dbcAOIWCfxVa/7G6STjADH2qmhomv+1quc=
github.com/pierrre/compare v1.0.2/go.mod h1:8UvyRHH+9HS8Pczdd2z5x/wvv67krDwVxoOndaIIDVU=
github.com/pierrre/geohash v1.0.0 h1:f/zfjdV4rVofTCz1FhP07T+EMQAvcMM2ioGZVt+zqjI=
@@ -1808,8 +1807,12 @@ github.com/r3labs/sse v0.0.0-20210224172625-26fe804710bc h1:zAsgcP8MhzAbhMnB1QQ2
github.com/r3labs/sse v0.0.0-20210224172625-26fe804710bc/go.mod h1:S8xSOnV3CgpNrWd0GQ/OoQfMtlg2uPRSuTzcSGrzwK8=
github.com/rabbitmq/amqp091-go v1.10.0 h1:STpn5XsHlHGcecLmMFCtg7mqq0RnD+zFr4uzukfVhBw=
github.com/rabbitmq/amqp091-go v1.10.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
github.com/rclone/rclone v1.72.1 h1:Cc/NshKd3/TP3CC0cx9Jg9nTLG8YQ8yLYMTm6Z/LdHk=
github.com/rclone/rclone v1.72.1/go.mod h1:QjmSgz98CjQZZJhROIeYHYjpN5kN7rTA+jtChj/+3Do=
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 h1:4MI2alxM/Ye2gIRBlYf28JGWTipZ4Zz7yAziPKrttjs=
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11/go.mod h1:3HLX7dwZgvB7nt+Yl/xdzVPcargQ1yBmJEUg3n+jMKM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 h1:Lc+d3ISfQaMJKWZOE7z4ZSY4RVmdzbn1B0IM8xN18qM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18/go.mod h1:LB2kCEaZMzNn3ocdz+qYfxXmuLxxN0ka62KJd2x53Bc=
github.com/rclone/rclone v1.73.1 h1:FCGZUd7sXO8+nIpbuXGPkd+f23o/ZACefrHtRGDAiDM=
github.com/rclone/rclone v1.73.1/go.mod h1:T9HeSXv/SI9qJFkYLqsjY/5ljIFWJTVmTrG7H/4JrPY=
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 h1:N/ElC8H3+5XpJzTSTfLsJV/mx9Q9g7kxmchpfZyxgzM=
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/rdleal/intervalst v1.5.0 h1:SEB9bCFz5IqD1yhfH1Wv8IBnY/JQxDplwkxHjT6hamU=
@@ -2011,6 +2014,8 @@ github.com/twpayne/go-kml v1.5.2 h1:rFMw2/EwgkVssGS2MT6YfWSPZz6BgcJkLxQ53jnE8rQ=
github.com/twpayne/go-kml v1.5.2/go.mod h1:kz8jAiIz6FIdU2Zjce9qGlVtgFYES9vt7BTPBHf5jl4=
github.com/twpayne/go-polyline v1.0.0/go.mod h1:ICh24bcLYBX8CknfvNPKqoTbe+eg+MX1NPyJmSBo7pU=
github.com/twpayne/go-waypoint v0.0.0-20200706203930-b263a7f6e4e8/go.mod h1:qj5pHncxKhu9gxtZEYWypA/z097sxhFlbTyOyt9gcnU=
github.com/tyler-smith/go-bip39 v1.1.0 h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8=
github.com/tyler-smith/go-bip39 v1.1.0/go.mod h1:gUYDtqQw1JS3ZJ8UWVcGTGqqr6YIN3CWg+kkNaLt55U=
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43 h1:QEePdg0ty2r0t1+qwfZmQ4OOl/MB2UXIeJSpIZv56lg=
github.com/tylertreat/BoomFilters v0.0.0-20210315201527-1a82519a3e43/go.mod h1:OYRfF6eb5wY9VRFkXJH8FFBi3plw2v+giaIu7P054pM=
github.com/ugorji/go/codec v1.3.0 h1:Qd2W2sQawAfG8XSvzwhBeoGq71zXOC/Q1E9y/wUcsUA=
@@ -2221,6 +2226,7 @@ golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.4.0/go.mod h1:3quD/ATkf6oY+rnes5c3ExXTbLc8mueNue5/DoinL80=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
@@ -2260,8 +2266,8 @@ golang.org/x/image v0.0.0-20210607152325-775e3b0c77b9/go.mod h1:023OzeP/+EPmXeap
golang.org/x/image v0.0.0-20210628002857-a66eb6448b8d/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/image v0.0.0-20211028202545-6944b10bf410/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/image v0.0.0-20220302094943-723b81ca9867/go.mod h1:023OzeP/+EPmXeapQh35lcL3II3LrY8Ic+EFFKVhULM=
golang.org/x/image v0.35.0 h1:LKjiHdgMtO8z7Fh18nGY6KDcoEtVfsgLDPeLyguqb7I=
golang.org/x/image v0.35.0/go.mod h1:MwPLTVgvxSASsxdLzKrl8BRFuyqMyGhLwmC+TO1Sybk=
golang.org/x/image v0.36.0 h1:Iknbfm1afbgtwPTmHnS2gTM/6PPZfH+z2EFuOkSbqwc=
golang.org/x/image v0.36.0/go.mod h1:YsWD2TyyGKiIX1kZlu9QfKIsQ4nAAK9bdgdrIsE7xy4=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -3047,8 +3053,8 @@ modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.18.1/go.mod h1:6ho+Gow7oX5V+OiOQ6Tr4xeqbx13UZ6t+Fw9IRUG4d4=
modernc.org/sqlite v1.44.3 h1:+39JvV/HWMcYslAwRxHb8067w+2zowvFOUrOWIy9PjY=
modernc.org/sqlite v1.44.3/go.mod h1:CzbrU2lSB1DKUusvwGz7rqEKIq+NUd8GWuBBZDs9/nA=
modernc.org/sqlite v1.46.1 h1:eFJ2ShBLIEnUWlLy12raN0Z1plqmFX9Qe3rjQTKt6sU=
modernc.org/sqlite v1.46.1/go.mod h1:CzbrU2lSB1DKUusvwGz7rqEKIq+NUd8GWuBBZDs9/nA=
modernc.org/strutil v1.1.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs=
modernc.org/strutil v1.1.1/go.mod h1:DE+MQQ/hjKBZS2zNInV5hhcipt5rLPWkmpbGeW5mmdw=
modernc.org/strutil v1.1.3/go.mod h1:MEHNA7PdEnEwLvspRMtWTNnp2nnyvMfkimT1NKNAGbw=
@@ -3,6 +3,10 @@ kind: ServiceAccount
metadata:
name: {{ include "seaweedfs.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- with .Values.global.serviceAccountAnnotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
app.kubernetes.io/name: {{ template "seaweedfs.name" . }}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
+1
View File
@@ -20,6 +20,7 @@ global:
filerRead: false
# we will use this serviceAccountName for all ClusterRoles/ClusterRoleBindings
serviceAccountName: "seaweedfs"
serviceAccountAnnotations: {}
automountServiceAccountToken: true
certificates:
duration: 87600h
+4 -2
View File
@@ -16,8 +16,9 @@ RUN go mod download
# Copy source code
COPY . .
# Build the weed binary with Kafka gateway support
# Build the weed binaries with Kafka gateway support
RUN CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o weed ./weed
RUN CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o weed-sql ./cmd/weed-sql
# Final stage
FROM alpine:latest
@@ -32,8 +33,9 @@ RUN addgroup -g 1000 seaweedfs && \
# Set working directory
WORKDIR /usr/bin
# Copy binary from builder
# Copy binaries from builder
COPY --from=builder /app/weed .
COPY --from=builder /app/weed-sql .
# Create data directory
RUN mkdir -p /data && chown seaweedfs:seaweedfs /data
+1 -2
View File
@@ -14,8 +14,7 @@ sleep 2
echo "Querying messages..."
cd /Users/chrislu/go/src/github.com/seaweedfs/seaweedfs/test/kafka/kafka-client-loadtest
docker compose exec kafka-gateway /usr/local/bin/weed sql \
docker compose exec kafka-gateway /usr/bin/weed-sql \
-master=seaweedfs-master:9333 \
-database=kafka \
-query="SELECT id, timestamp, producer_id, counter, user_id, event_type FROM \"test-json-topic\" LIMIT 5;"
+5 -3
View File
@@ -13,8 +13,9 @@ RUN go mod download
# Copy source code
COPY . .
# Build the weed binary without CGO
# Build the weed binaries without CGO
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-s -w" -o weed ./weed/
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags "-s -w" -o weed-db ./cmd/weed-db
# Final stage - minimal runtime image
FROM alpine:latest
@@ -24,11 +25,12 @@ RUN apk --no-cache add ca-certificates netcat-openbsd curl
WORKDIR /root/
# Copy the weed binary from builder stage
# Copy the binaries from builder stage
COPY --from=builder /app/weed .
COPY --from=builder /app/weed-db .
# Make it executable
RUN chmod +x ./weed
RUN chmod +x ./weed ./weed-db
# Expose ports
EXPOSE 9333 8888 8333 8085 9533 5432
+1 -1
View File
@@ -280,7 +280,7 @@ This test setup proves:
- Comprehensive error handling
### ✅ Performance and Scalability
- Direct SQL engine integration (same as `weed sql`)
- Direct SQL engine integration (same as `weed-sql`)
- No translation overhead for real queries
- Efficient data access from stored formats
- Scalable architecture with service discovery
+1 -1
View File
@@ -54,7 +54,7 @@ services:
seaweedfs:
condition: service_healthy
command: >
./weed db
./weed-db
-host=0.0.0.0
-port=5432
-master=seaweedfs:9333
Executable → Regular
View File
Executable → Regular
View File
@@ -40,6 +40,7 @@ type TestEnvironment struct {
dockerAvailable bool
accessKey string
secretKey string
closers []io.Closer
}
func TestTrinoIcebergCatalog(t *testing.T) {
@@ -118,30 +119,66 @@ func NewTestEnvironment(t *testing.T) *TestEnvironment {
bindIP := testutil.FindBindIP()
masterPort, masterGrpcPort := mustFreePortPair(t, "Master")
volumePort, volumeGrpcPort := mustFreePortPair(t, "Volume")
filerPort, filerGrpcPort := mustFreePortPair(t, "Filer")
s3Port, s3GrpcPort := mustFreePortPair(t, "S3")
icebergPort := mustFreePort(t, "Iceberg")
return &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: bindIP,
s3Port: s3Port,
s3GrpcPort: s3GrpcPort,
icebergPort: icebergPort,
masterPort: masterPort,
masterGrpcPort: masterGrpcPort,
filerPort: filerPort,
filerGrpcPort: filerGrpcPort,
volumePort: volumePort,
volumeGrpcPort: volumeGrpcPort,
dockerAvailable: hasDocker(),
accessKey: "AKIAIOSFODNN7EXAMPLE",
secretKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
env := &TestEnvironment{
seaweedDir: seaweedDir,
weedBinary: weedBinary,
dataDir: dataDir,
bindIP: bindIP,
closers: []io.Closer{},
}
env.masterPort, env.masterGrpcPort = env.mustFreePortPair("Master")
env.volumePort, env.volumeGrpcPort = env.mustFreePortPair("Volume")
env.filerPort, env.filerGrpcPort = env.mustFreePortPair("Filer")
env.s3Port, env.s3GrpcPort = env.mustFreePortPair("S3")
env.icebergPort = env.mustFreePort("Iceberg")
env.dockerAvailable = hasDocker()
env.accessKey = "AKIAIOSFODNN7EXAMPLE"
env.secretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
return env
}
func (env *TestEnvironment) mustFreePort(name string) int {
port, closer, err := getFreePort()
if err != nil {
panic(fmt.Sprintf("Failed to get free port for %s: %v", name, err))
}
env.closers = append(env.closers, closer)
return port
}
func (env *TestEnvironment) mustFreePortPair(name string) (int, int) {
httpPort, httpCloser, grpcPort, grpcCloser, err := findAvailablePortPair()
if err != nil {
panic(fmt.Sprintf("Failed to get free port pair for %s: %v", name, err))
}
env.closers = append(env.closers, httpCloser, grpcCloser)
return httpPort, grpcPort
}
func mustFreePort(t *testing.T, name string) int {
t.Helper()
port, closer, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for %s: %v", name, err)
}
closer.Close()
return port
}
func mustFreePortPair(t *testing.T, name string) (int, int) {
t.Helper()
httpPort, httpCloser, grpcPort, grpcCloser, err := findAvailablePortPair()
if err != nil {
t.Fatalf("Failed to get free port pair for %s: %v", name, err)
}
httpCloser.Close()
grpcCloser.Close()
return httpPort, grpcPort
}
func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
@@ -161,6 +198,12 @@ func (env *TestEnvironment) StartSeaweedFS(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
env.weedCancel = cancel
// Close all port listeners right before starting the weed process
for _, closer := range env.closers {
closer.Close()
}
env.closers = nil
cmd := exec.CommandContext(ctx, env.weedBinary, "mini",
"-master.port", fmt.Sprintf("%d", env.masterPort),
"-master.port.grpc", fmt.Sprintf("%d", env.masterGrpcPort),
@@ -471,47 +514,27 @@ func hasDocker() bool {
return cmd.Run() == nil
}
func mustFreePort(t *testing.T, name string) int {
t.Helper()
port, err := getFreePort()
func findAvailablePortPair() (int, io.Closer, int, io.Closer, error) {
httpPort, httpCloser, err := getFreePort()
if err != nil {
t.Fatalf("Failed to get free port for %s: %v", name, err)
return 0, nil, 0, nil, err
}
return port
grpcPort, grpcCloser, err := getFreePort()
if err != nil {
httpCloser.Close()
return 0, nil, 0, nil, err
}
return httpPort, httpCloser, grpcPort, grpcCloser, nil
}
func mustFreePortPair(t *testing.T, name string) (int, int) {
t.Helper()
httpPort, grpcPort, err := findAvailablePortPair()
if err != nil {
t.Fatalf("Failed to get free port pair for %s: %v", name, err)
}
return httpPort, grpcPort
}
func findAvailablePortPair() (int, int, error) {
httpPort, err := getFreePort()
if err != nil {
return 0, 0, err
}
grpcPort, err := getFreePort()
if err != nil {
return 0, 0, err
}
return httpPort, grpcPort, nil
}
func getFreePort() (int, error) {
func getFreePort() (int, io.Closer, error) {
listener, err := net.Listen("tcp", "0.0.0.0:0")
if err != nil {
return 0, err
return 0, nil, err
}
defer listener.Close()
addr := listener.Addr().(*net.TCPAddr)
return addr.Port, nil
return addr.Port, listener, nil
}
func randomString(length int) string {
+8
View File
@@ -9,6 +9,14 @@ all: install
install:
go install -ldflags="-s -w"
.PHONY: weed-db weed-sql
weed-db:
go build -ldflags="-s -w" -o weed-db ./cmd/weed-db
weed-sql:
go build -ldflags="-s -w" -o weed-sql ./cmd/weed-sql
build_docker:
CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w"
+7 -8
View File
@@ -6,7 +6,6 @@ import (
"sort"
"time"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/cluster"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/iam"
@@ -185,28 +184,28 @@ func (s *AdminServer) GetAdminData(username string) (AdminData, error) {
}
// ShowAdmin displays the main admin page (now uses GetAdminData)
func (s *AdminServer) ShowAdmin(c *gin.Context) {
username := c.GetString("username")
func (s *AdminServer) ShowAdmin(w http.ResponseWriter, r *http.Request) {
username := UsernameFromContext(r.Context())
adminData, err := s.GetAdminData(username)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get admin data: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get admin data: "+err.Error())
return
}
// Return JSON for API calls
c.JSON(http.StatusOK, adminData)
writeJSON(w, http.StatusOK, adminData)
}
// ShowOverview displays cluster overview
func (s *AdminServer) ShowOverview(c *gin.Context) {
func (s *AdminServer) ShowOverview(w http.ResponseWriter, r *http.Request) {
topology, err := s.GetClusterTopology()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
c.JSON(http.StatusOK, topology)
writeJSON(w, http.StatusOK, topology)
}
// getMasterNodesStatus checks status of all master nodes
+5 -7
View File
@@ -7,8 +7,6 @@ import (
"sort"
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/admin/maintenance"
adminplugin "github.com/seaweedfs/seaweedfs/weed/admin/plugin"
"github.com/seaweedfs/seaweedfs/weed/cluster"
@@ -817,18 +815,18 @@ func (s *AdminServer) GetClusterBrokers() (*ClusterBrokersData, error) {
// VacuumVolume method moved to volume_management.go
// TriggerTopicRetentionPurgeAPI triggers topic retention purge via HTTP API
func (as *AdminServer) TriggerTopicRetentionPurgeAPI(c *gin.Context) {
func (as *AdminServer) TriggerTopicRetentionPurgeAPI(w http.ResponseWriter, r *http.Request) {
err := as.TriggerTopicRetentionPurge()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"message": "Topic retention purge triggered successfully"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Topic retention purge triggered successfully"})
}
// GetConfigInfo returns information about the admin configuration
func (as *AdminServer) GetConfigInfo(c *gin.Context) {
func (as *AdminServer) GetConfigInfo(w http.ResponseWriter, r *http.Request) {
configInfo := as.configPersistence.GetConfigInfo()
// Add additional admin server info
@@ -846,7 +844,7 @@ func (as *AdminServer) GetConfigInfo(c *gin.Context) {
configInfo["maintenance_running"] = false
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"config_info": configInfo,
"title": "Configuration Information",
})
+53 -43
View File
@@ -4,81 +4,91 @@ import (
"crypto/subtle"
"net/http"
"github.com/gin-contrib/sessions"
"github.com/gin-gonic/gin"
"github.com/gorilla/sessions"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
// ShowLogin displays the login page
func (s *AdminServer) ShowLogin(c *gin.Context) {
// If authentication is not required, redirect to admin
session := sessions.Default(c)
if session.Get("authenticated") == true {
c.Redirect(http.StatusSeeOther, "/admin")
return
}
// For now, return a simple login form as JSON
c.HTML(http.StatusOK, "login.html", gin.H{
"title": "SeaweedFS Admin Login",
"error": c.Query("error"),
})
// ShowLogin displays the login page.
func (s *AdminServer) ShowLogin(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/login", http.StatusSeeOther)
}
// HandleLogin handles login form submission
func (s *AdminServer) HandleLogin(adminUser, adminPassword, readOnlyUser, readOnlyPassword string) gin.HandlerFunc {
return func(c *gin.Context) {
loginUsername := c.PostForm("username")
loginPassword := c.PostForm("password")
// HandleLogin handles login form submission.
func (s *AdminServer) HandleLogin(store sessions.Store, adminUser, adminPassword, readOnlyUser, readOnlyPassword string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseForm(); err != nil {
http.Redirect(w, r, "/login?error=Invalid form submission", http.StatusSeeOther)
return
}
session, err := store.Get(r, sessionName)
if err != nil {
http.Redirect(w, r, "/login?error=Unable to create session. Please try again or contact administrator.", http.StatusSeeOther)
return
}
if err := ValidateSessionCSRFToken(session, r); err != nil {
http.Redirect(w, r, "/login?error=Invalid CSRF token", http.StatusSeeOther)
return
}
loginUsername := r.FormValue("username")
loginPassword := r.FormValue("password")
var role string
var authenticated bool
// Check admin credentials
// Check admin credentials.
if adminPassword != "" && loginUsername == adminUser && subtle.ConstantTimeCompare([]byte(loginPassword), []byte(adminPassword)) == 1 {
role = "admin"
authenticated = true
} else if readOnlyPassword != "" && loginUsername == readOnlyUser && subtle.ConstantTimeCompare([]byte(loginPassword), []byte(readOnlyPassword)) == 1 {
// Check read-only credentials
// Check read-only credentials.
role = "readonly"
authenticated = true
}
if authenticated {
session := sessions.Default(c)
// Clear any existing invalid session data before setting new values
session.Clear()
session.Set("authenticated", true)
session.Set("username", loginUsername)
session.Set("role", role)
for key := range session.Values {
delete(session.Values, key)
}
session.Values["authenticated"] = true
session.Values["username"] = loginUsername
session.Values["role"] = role
csrfToken, err := generateCSRFToken()
if err != nil {
c.Redirect(http.StatusSeeOther, "/login?error=Unable to create session. Please try again or contact administrator.")
http.Redirect(w, r, "/login?error=Unable to create session. Please try again or contact administrator.", http.StatusSeeOther)
return
}
session.Set(sessionCSRFTokenKey, csrfToken)
if err := session.Save(); err != nil {
// Log the detailed error server-side for diagnostics
session.Values[sessionCSRFTokenKey] = csrfToken
if err := session.Save(r, w); err != nil {
// Log the detailed error server-side for diagnostics.
glog.Errorf("Failed to save session for user %s: %v", loginUsername, err)
c.Redirect(http.StatusSeeOther, "/login?error=Unable to create session. Please try again or contact administrator.")
http.Redirect(w, r, "/login?error=Unable to create session. Please try again or contact administrator.", http.StatusSeeOther)
return
}
c.Redirect(http.StatusSeeOther, "/admin")
http.Redirect(w, r, "/admin", http.StatusSeeOther)
return
}
// Authentication failed
c.Redirect(http.StatusSeeOther, "/login?error=Invalid credentials")
// Authentication failed.
http.Redirect(w, r, "/login?error=Invalid credentials", http.StatusSeeOther)
}
}
// HandleLogout handles user logout
func (s *AdminServer) HandleLogout(c *gin.Context) {
session := sessions.Default(c)
session.Clear()
if err := session.Save(); err != nil {
// HandleLogout handles user logout.
func (s *AdminServer) HandleLogout(store sessions.Store, w http.ResponseWriter, r *http.Request) {
session, err := store.Get(r, sessionName)
if err != nil {
http.Redirect(w, r, "/login", http.StatusSeeOther)
return
}
for key := range session.Values {
delete(session.Values, key)
}
session.Options.MaxAge = -1
if err := session.Save(r, w); err != nil {
glog.Warningf("Failed to save session during logout: %v", err)
}
c.Redirect(http.StatusSeeOther, "/login")
http.Redirect(w, r, "/login", http.StatusSeeOther)
}
+90 -54
View File
@@ -8,7 +8,7 @@ import (
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
@@ -29,7 +29,7 @@ type S3BucketsData struct {
}
type CreateBucketRequest struct {
Name string `json:"name" binding:"required"`
Name string `json:"name"` // validated manually in CreateBucket
Region string `json:"region"`
QuotaSize int64 `json:"quota_size"` // Quota size in bytes
QuotaUnit string `json:"quota_unit"` // Unit: MB, GB, TB
@@ -45,47 +45,51 @@ type CreateBucketRequest struct {
// S3 Bucket Management Handlers
// ShowS3Buckets displays the Object Store buckets management page
func (s *AdminServer) ShowS3Buckets(c *gin.Context) {
username := c.GetString("username")
func (s *AdminServer) ShowS3Buckets(w http.ResponseWriter, r *http.Request) {
username := UsernameFromContext(r.Context())
data, err := s.GetS3BucketsData()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get Object Store buckets: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get Object Store buckets: "+err.Error())
return
}
data.Username = username
c.JSON(http.StatusOK, data)
writeJSON(w, http.StatusOK, data)
}
// ShowBucketDetails displays detailed information about a specific bucket
func (s *AdminServer) ShowBucketDetails(c *gin.Context) {
bucketName := c.Param("bucket")
func (s *AdminServer) ShowBucketDetails(w http.ResponseWriter, r *http.Request) {
bucketName := mux.Vars(r)["bucket"]
if bucketName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Bucket name is required"})
writeJSONError(w, http.StatusBadRequest, "Bucket name is required")
return
}
details, err := s.GetBucketDetails(bucketName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get bucket details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get bucket details: "+err.Error())
return
}
c.JSON(http.StatusOK, details)
writeJSON(w, http.StatusOK, details)
}
// CreateBucket creates a new S3 bucket
func (s *AdminServer) CreateBucket(c *gin.Context) {
func (s *AdminServer) CreateBucket(w http.ResponseWriter, r *http.Request) {
var req CreateBucketRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if strings.TrimSpace(req.Name) == "" {
writeJSONError(w, http.StatusBadRequest, "Bucket name is required")
return
}
// Validate bucket name (basic validation)
if len(req.Name) < 3 || len(req.Name) > 63 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Bucket name must be between 3 and 63 characters"})
writeJSONError(w, http.StatusBadRequest, "Bucket name must be between 3 and 63 characters")
return
}
@@ -96,42 +100,47 @@ func (s *AdminServer) CreateBucket(c *gin.Context) {
// Validate object lock mode
if req.ObjectLockMode != "GOVERNANCE" && req.ObjectLockMode != "COMPLIANCE" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Object lock mode must be either GOVERNANCE or COMPLIANCE"})
writeJSONError(w, http.StatusBadRequest, "Object lock mode must be either GOVERNANCE or COMPLIANCE")
return
}
// Validate retention duration if default retention is enabled
if req.SetDefaultRetention {
if req.ObjectLockDuration <= 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Object lock duration must be greater than 0 days when default retention is enabled"})
writeJSONError(w, http.StatusBadRequest, "Object lock duration must be greater than 0 days when default retention is enabled")
return
}
}
}
// Convert quota to bytes
quotaBytes := convertQuotaToBytes(req.QuotaSize, req.QuotaUnit)
normalizedUnit, err := normalizeQuotaUnit(req.QuotaUnit)
if err != nil {
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
req.QuotaUnit = normalizedUnit
quotaBytes := convertQuotaToBytes(req.QuotaSize, normalizedUnit)
// Validate quota: if enabled, size must be greater than 0
if req.QuotaEnabled && quotaBytes <= 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Quota size must be greater than 0 when quota is enabled"})
writeJSONError(w, http.StatusBadRequest, "Quota size must be greater than 0 when quota is enabled")
return
}
// Sanitize owner: trim whitespace and enforce max length
owner := strings.TrimSpace(req.Owner)
if len(owner) > MaxOwnerNameLength {
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Owner name must be %d characters or less", MaxOwnerNameLength)})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Owner name must be %d characters or less", MaxOwnerNameLength))
return
}
err := s.CreateS3BucketWithObjectLock(req.Name, quotaBytes, req.QuotaEnabled, req.VersioningEnabled, req.ObjectLockEnabled, req.ObjectLockMode, req.SetDefaultRetention, req.ObjectLockDuration, owner)
err = s.CreateS3BucketWithObjectLock(req.Name, quotaBytes, req.QuotaEnabled, req.VersioningEnabled, req.ObjectLockEnabled, req.ObjectLockMode, req.SetDefaultRetention, req.ObjectLockDuration, owner)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create bucket: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create bucket: "+err.Error())
return
}
c.JSON(http.StatusCreated, gin.H{
writeJSON(w, http.StatusCreated, map[string]interface{}{
"message": "Bucket created successfully",
"bucket": req.Name,
"quota_size": req.QuotaSize,
@@ -146,10 +155,10 @@ func (s *AdminServer) CreateBucket(c *gin.Context) {
}
// UpdateBucketQuota updates the quota settings for a bucket
func (s *AdminServer) UpdateBucketQuota(c *gin.Context) {
bucketName := c.Param("bucket")
func (s *AdminServer) UpdateBucketQuota(w http.ResponseWriter, r *http.Request) {
bucketName := mux.Vars(r)["bucket"]
if bucketName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Bucket name is required"})
writeJSONError(w, http.StatusBadRequest, "Bucket name is required")
return
}
@@ -158,21 +167,32 @@ func (s *AdminServer) UpdateBucketQuota(c *gin.Context) {
QuotaUnit string `json:"quota_unit"`
QuotaEnabled bool `json:"quota_enabled"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Convert quota to bytes
quotaBytes := convertQuotaToBytes(req.QuotaSize, req.QuotaUnit)
if req.QuotaEnabled && req.QuotaSize <= 0 {
writeJSONError(w, http.StatusBadRequest, "quota_size must be > 0 when quota_enabled is true")
return
}
err := s.SetBucketQuota(bucketName, quotaBytes, req.QuotaEnabled)
normalizedUnit, err := normalizeQuotaUnit(req.QuotaUnit)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update bucket quota: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
req.QuotaUnit = normalizedUnit
// Convert quota to bytes
quotaBytes := convertQuotaToBytes(req.QuotaSize, normalizedUnit)
err = s.SetBucketQuota(bucketName, quotaBytes, req.QuotaEnabled)
if err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to update bucket quota: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Bucket quota updated successfully",
"bucket": bucketName,
"quota_size": req.QuotaSize,
@@ -182,30 +202,30 @@ func (s *AdminServer) UpdateBucketQuota(c *gin.Context) {
}
// DeleteBucket deletes an S3 bucket
func (s *AdminServer) DeleteBucket(c *gin.Context) {
bucketName := c.Param("bucket")
func (s *AdminServer) DeleteBucket(w http.ResponseWriter, r *http.Request) {
bucketName := mux.Vars(r)["bucket"]
if bucketName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Bucket name is required"})
writeJSONError(w, http.StatusBadRequest, "Bucket name is required")
return
}
err := s.DeleteS3Bucket(bucketName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to delete bucket: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to delete bucket: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Bucket deleted successfully",
"bucket": bucketName,
})
}
// UpdateBucketOwner updates the owner of an S3 bucket
func (s *AdminServer) UpdateBucketOwner(c *gin.Context) {
bucketName := c.Param("bucket")
func (s *AdminServer) UpdateBucketOwner(w http.ResponseWriter, r *http.Request) {
bucketName := mux.Vars(r)["bucket"]
if bucketName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Bucket name is required"})
writeJSONError(w, http.StatusBadRequest, "Bucket name is required")
return
}
@@ -213,31 +233,31 @@ func (s *AdminServer) UpdateBucketOwner(c *gin.Context) {
var req struct {
Owner *string `json:"owner"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Require owner field to be explicitly provided
if req.Owner == nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Owner field is required (use empty string to clear owner)"})
writeJSONError(w, http.StatusBadRequest, "Owner field is required (use empty string to clear owner)")
return
}
// Trim and validate owner
owner := strings.TrimSpace(*req.Owner)
if len(owner) > MaxOwnerNameLength {
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Owner name must be %d characters or less", MaxOwnerNameLength)})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Owner name must be %d characters or less", MaxOwnerNameLength))
return
}
err := s.SetBucketOwner(bucketName, owner)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update bucket owner: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to update bucket owner: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Bucket owner updated successfully",
"bucket": bucketName,
"owner": owner,
@@ -284,14 +304,14 @@ func (s *AdminServer) SetBucketOwner(bucketName string, owner string) error {
}
// ListBucketsAPI returns the list of buckets as JSON
func (s *AdminServer) ListBucketsAPI(c *gin.Context) {
func (s *AdminServer) ListBucketsAPI(w http.ResponseWriter, r *http.Request) {
buckets, err := s.GetS3Buckets()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get buckets: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get buckets: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"buckets": buckets,
"total": len(buckets),
})
@@ -303,16 +323,32 @@ func convertQuotaToBytes(size int64, unit string) int64 {
return 0
}
switch strings.ToUpper(unit) {
switch unit {
case "TB":
return size * 1024 * 1024 * 1024 * 1024
case "GB":
return size * 1024 * 1024 * 1024
case "MB":
return size * 1024 * 1024
case "KB":
return size * 1024
case "B":
return size
default:
// Default to MB if unit is not recognized
return size * 1024 * 1024
return 0
}
}
func normalizeQuotaUnit(unit string) (string, error) {
normalized := strings.ToUpper(strings.TrimSpace(unit))
if normalized == "" {
return "MB", nil
}
switch normalized {
case "B", "KB", "MB", "GB", "TB":
return normalized, nil
default:
return "", fmt.Errorf("unsupported quota unit: %s", unit)
}
}
+35 -4
View File
@@ -7,6 +7,7 @@ import (
"path/filepath"
"sort"
"strings"
"sync"
"time"
"github.com/seaweedfs/seaweedfs/weed/admin/maintenance"
@@ -87,6 +88,11 @@ func isValidTaskID(taskID string) bool {
// ConfigPersistence handles saving and loading configuration files
type ConfigPersistence struct {
dataDir string
// tasksMu serializes all filesystem operations on the tasks/ directory.
// SaveTaskState, LoadTaskState, LoadAllTaskStates, DeleteTaskState, and
// CleanupCompletedTasks are called from multiple goroutines concurrently
// after saveTaskState was moved outside mq.mutex in the maintenance queue.
tasksMu sync.Mutex
}
// NewConfigPersistence creates a new configuration persistence manager
@@ -937,6 +943,8 @@ func (cp *ConfigPersistence) ListTaskDetails() ([]string, error) {
// CleanupCompletedTasks removes old completed tasks beyond the retention limit
func (cp *ConfigPersistence) CleanupCompletedTasks() error {
cp.tasksMu.Lock()
defer cp.tasksMu.Unlock()
if cp.dataDir == "" {
return fmt.Errorf("no data directory specified, cannot cleanup completed tasks")
}
@@ -946,8 +954,8 @@ func (cp *ConfigPersistence) CleanupCompletedTasks() error {
return nil // No tasks directory, nothing to cleanup
}
// Load all tasks and find completed/failed ones
allTasks, err := cp.LoadAllTaskStates()
// Use unlocked helpers to avoid deadlock (tasksMu is already held)
allTasks, err := cp.loadAllTaskStatesLocked()
if err != nil {
return fmt.Errorf("failed to load tasks for cleanup: %w", err)
}
@@ -998,7 +1006,7 @@ func (cp *ConfigPersistence) CleanupCompletedTasks() error {
if len(completedTasks) > MaxCompletedTasks {
tasksToDelete := completedTasks[MaxCompletedTasks:]
for _, task := range tasksToDelete {
if err := cp.DeleteTaskState(task.ID); err != nil {
if err := cp.deleteTaskStateLocked(task.ID); err != nil {
glog.Warningf("Failed to delete old completed task %s: %v", task.ID, err)
} else {
glog.V(2).Infof("Cleaned up old completed task %s (completed: %v)", task.ID, task.CompletedAt)
@@ -1012,6 +1020,8 @@ func (cp *ConfigPersistence) CleanupCompletedTasks() error {
// SaveTaskState saves a task state to protobuf file
func (cp *ConfigPersistence) SaveTaskState(task *maintenance.MaintenanceTask) error {
cp.tasksMu.Lock()
defer cp.tasksMu.Unlock()
if cp.dataDir == "" {
return fmt.Errorf("no data directory specified, cannot save task state")
}
@@ -1051,6 +1061,13 @@ func (cp *ConfigPersistence) SaveTaskState(task *maintenance.MaintenanceTask) er
// LoadTaskState loads a task state from protobuf file
func (cp *ConfigPersistence) LoadTaskState(taskID string) (*maintenance.MaintenanceTask, error) {
cp.tasksMu.Lock()
defer cp.tasksMu.Unlock()
return cp.loadTaskStateLocked(taskID)
}
// loadTaskStateLocked loads a single task state. Must be called with tasksMu held.
func (cp *ConfigPersistence) loadTaskStateLocked(taskID string) (*maintenance.MaintenanceTask, error) {
if cp.dataDir == "" {
return nil, fmt.Errorf("no data directory specified, cannot load task state")
}
@@ -1084,6 +1101,13 @@ func (cp *ConfigPersistence) LoadTaskState(taskID string) (*maintenance.Maintena
// LoadAllTaskStates loads all task states from disk
func (cp *ConfigPersistence) LoadAllTaskStates() ([]*maintenance.MaintenanceTask, error) {
cp.tasksMu.Lock()
defer cp.tasksMu.Unlock()
return cp.loadAllTaskStatesLocked()
}
// loadAllTaskStatesLocked loads all task states from disk. Must be called with tasksMu held.
func (cp *ConfigPersistence) loadAllTaskStatesLocked() ([]*maintenance.MaintenanceTask, error) {
if cp.dataDir == "" {
return []*maintenance.MaintenanceTask{}, nil
}
@@ -1102,7 +1126,7 @@ func (cp *ConfigPersistence) LoadAllTaskStates() ([]*maintenance.MaintenanceTask
for _, entry := range entries {
if !entry.IsDir() && filepath.Ext(entry.Name()) == ".pb" {
taskID := entry.Name()[:len(entry.Name())-3] // Remove .pb extension
task, err := cp.LoadTaskState(taskID)
task, err := cp.loadTaskStateLocked(taskID)
if err != nil {
glog.Warningf("Failed to load task state for %s: %v", taskID, err)
continue
@@ -1117,6 +1141,13 @@ func (cp *ConfigPersistence) LoadAllTaskStates() ([]*maintenance.MaintenanceTask
// DeleteTaskState removes a task state file from disk
func (cp *ConfigPersistence) DeleteTaskState(taskID string) error {
cp.tasksMu.Lock()
defer cp.tasksMu.Unlock()
return cp.deleteTaskStateLocked(taskID)
}
// deleteTaskStateLocked removes a task state file. Must be called with tasksMu held.
func (cp *ConfigPersistence) deleteTaskStateLocked(taskID string) error {
if cp.dataDir == "" {
return fmt.Errorf("no data directory specified, cannot delete task state")
}
+58
View File
@@ -0,0 +1,58 @@
package dash
import "context"
type contextKey string
const (
contextUsernameKey contextKey = "admin.username"
contextRoleKey contextKey = "admin.role"
contextCSRFKey contextKey = "admin.csrf"
)
// WithAuthContext stores auth metadata on the request context.
func WithAuthContext(ctx context.Context, username, role, csrfToken string) context.Context {
if username != "" {
ctx = context.WithValue(ctx, contextUsernameKey, username)
}
if role != "" {
ctx = context.WithValue(ctx, contextRoleKey, role)
}
if csrfToken != "" {
ctx = context.WithValue(ctx, contextCSRFKey, csrfToken)
}
return ctx
}
// UsernameFromContext retrieves the username from context.
func UsernameFromContext(ctx context.Context) string {
if ctx == nil {
return ""
}
if value, ok := ctx.Value(contextUsernameKey).(string); ok {
return value
}
return ""
}
// RoleFromContext retrieves the role from context.
func RoleFromContext(ctx context.Context) string {
if ctx == nil {
return ""
}
if value, ok := ctx.Value(contextRoleKey).(string); ok {
return value
}
return ""
}
// CSRFTokenFromContext retrieves the CSRF token from context.
func CSRFTokenFromContext(ctx context.Context) string {
if ctx == nil {
return ""
}
if value, ok := ctx.Value(contextCSRFKey).(string); ok {
return value
}
return ""
}
+55 -19
View File
@@ -4,10 +4,10 @@ import (
"crypto/rand"
"crypto/subtle"
"encoding/hex"
"fmt"
"net/http"
"github.com/gin-contrib/sessions"
"github.com/gin-gonic/gin"
"github.com/gorilla/sessions"
)
const sessionCSRFTokenKey = "csrf_token"
@@ -20,41 +20,77 @@ func generateCSRFToken() (string, error) {
return hex.EncodeToString(tokenBytes), nil
}
func getOrCreateSessionCSRFToken(session sessions.Session) (string, error) {
if existing, ok := session.Get(sessionCSRFTokenKey).(string); ok && existing != "" {
func getOrCreateSessionCSRFToken(session *sessions.Session, r *http.Request, w http.ResponseWriter) (string, error) {
if existing, ok := session.Values[sessionCSRFTokenKey].(string); ok && existing != "" {
return existing, nil
}
token, err := generateCSRFToken()
if err != nil {
return "", err
}
session.Set(sessionCSRFTokenKey, token)
if err := session.Save(); err != nil {
session.Values[sessionCSRFTokenKey] = token
if err := session.Save(r, w); err != nil {
return "", err
}
return token, nil
}
func requireSessionCSRFToken(c *gin.Context) bool {
session := sessions.Default(c)
if session.Get("authenticated") != true {
func requireSessionCSRFToken(w http.ResponseWriter, r *http.Request) bool {
expectedToken := CSRFTokenFromContext(r.Context())
username := UsernameFromContext(r.Context())
if expectedToken == "" {
// Admin UI can run without auth; in that mode CSRF token checks are not applicable.
return true
}
expectedToken, ok := session.Get(sessionCSRFTokenKey).(string)
if !ok || expectedToken == "" {
c.JSON(http.StatusForbidden, gin.H{"error": "missing CSRF session token"})
if username == "" {
return true
}
writeJSONError(w, http.StatusForbidden, "missing CSRF session token")
return false
}
providedToken := c.GetHeader("X-CSRF-Token")
if providedToken == "" {
providedToken = c.PostForm("csrf_token")
providedToken, err := getProvidedCSRFToken(r)
if err != nil {
writeJSONError(w, http.StatusBadRequest, "Failed to parse form: "+err.Error())
return false
}
if providedToken == "" || subtle.ConstantTimeCompare([]byte(expectedToken), []byte(providedToken)) != 1 {
c.JSON(http.StatusForbidden, gin.H{"error": "invalid CSRF token"})
writeJSONError(w, http.StatusForbidden, "invalid CSRF token")
return false
}
return true
}
func getProvidedCSRFToken(r *http.Request) (string, error) {
providedToken := r.Header.Get("X-CSRF-Token")
if providedToken != "" {
return providedToken, nil
}
if err := r.ParseForm(); err != nil {
return "", err
}
return r.FormValue("csrf_token"), nil
}
func EnsureSessionCSRFToken(session *sessions.Session, r *http.Request, w http.ResponseWriter) (string, error) {
if session == nil {
return "", fmt.Errorf("session is nil")
}
return getOrCreateSessionCSRFToken(session, r, w)
}
func ValidateSessionCSRFToken(session *sessions.Session, r *http.Request) error {
if session == nil {
return fmt.Errorf("session is nil")
}
expectedToken, _ := session.Values[sessionCSRFTokenKey].(string)
providedToken, err := getProvidedCSRFToken(r)
if err != nil {
return fmt.Errorf("failed to read CSRF token: %w", err)
}
if expectedToken == "" {
return fmt.Errorf("missing session CSRF token")
}
if providedToken == "" || subtle.ConstantTimeCompare([]byte(expectedToken), []byte(providedToken)) != 1 {
return fmt.Errorf("invalid CSRF token")
}
return nil
}
+20 -76
View File
@@ -6,10 +6,7 @@ import (
"sort"
"time"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
"github.com/seaweedfs/seaweedfs/weed/pb/volume_server_pb"
"github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding"
)
@@ -333,10 +330,19 @@ func (s *AdminServer) GetClusterEcVolumes(page int, pageSize int, sortBy string,
// Process each shard this server has for this volume
shardBits := ecShardInfo.EcIndexBits
shardSizes := ecShardInfo.ShardSizes
sizeIndex := 0
for shardId := 0; shardId < erasure_coding.MaxShardCount; shardId++ {
if (shardBits & (1 << uint(shardId))) != 0 {
// Record shard location
volume.ShardLocations[shardId] = node.Id
if sizeIndex < len(shardSizes) {
size := shardSizes[sizeIndex]
if size >= 0 {
volume.ShardSizes[shardId] = size
}
}
sizeIndex++
totalShards++
}
}
@@ -354,38 +360,6 @@ func (s *AdminServer) GetClusterEcVolumes(page int, pageSize int, sortBy string,
return nil, err
}
// Collect shard size information from volume servers
for volumeId, volume := range volumeData {
// Group servers by volume to minimize gRPC calls
serverHasVolume := make(map[string]bool)
for _, server := range volume.Servers {
serverHasVolume[server] = true
}
// Query each server for shard sizes
for server := range serverHasVolume {
err := s.WithVolumeServerClient(pb.ServerAddress(server), func(client volume_server_pb.VolumeServerClient) error {
resp, err := client.VolumeEcShardsInfo(context.Background(), &volume_server_pb.VolumeEcShardsInfoRequest{
VolumeId: volumeId,
})
if err != nil {
glog.V(1).Infof("Failed to get EC shard info from %s for volume %d: %v", server, volumeId, err)
return nil // Continue with other servers, don't fail the entire request
}
// Update shard sizes
for _, shardInfo := range resp.EcShardInfos {
volume.ShardSizes[int(shardInfo.ShardId)] = shardInfo.Size
}
return nil
})
if err != nil {
glog.V(1).Infof("Failed to connect to volume server %s: %v", server, err)
}
}
}
// Calculate completeness for each volume
completeVolumes := 0
incompleteVolumes := 0
@@ -616,13 +590,23 @@ func (s *AdminServer) GetEcVolumeDetails(volumeID uint32, sortBy string, sortOrd
// Create individual shard entries for each shard this server has
shardBits := ecShardInfo.EcIndexBits
shardSizes := ecShardInfo.ShardSizes
sizeIndex := 0
for shardId := 0; shardId < erasure_coding.MaxShardCount; shardId++ {
if (shardBits & (1 << uint(shardId))) != 0 {
var shardSize uint64
if sizeIndex < len(shardSizes) {
size := shardSizes[sizeIndex]
if size >= 0 {
shardSize = uint64(size)
}
}
sizeIndex++
ecShard := EcShardWithInfo{
VolumeID: ecShardInfo.Id,
ShardID: uint32(shardId),
Collection: ecShardInfo.Collection,
Size: 0, // EC shards don't have individual size in the API response
Size: shardSize,
Server: node.Id,
DataCenter: dc.Id,
Rack: rack.Id,
@@ -653,46 +637,6 @@ func (s *AdminServer) GetEcVolumeDetails(volumeID uint32, sortBy string, sortOrd
return nil, fmt.Errorf("EC volume %d not found", volumeID)
}
// Collect shard size information from volume servers
shardSizeMap := make(map[string]map[uint32]uint64) // server -> shardId -> size
for _, shard := range shards {
server := shard.Server
if _, exists := shardSizeMap[server]; !exists {
// Query this server for shard sizes
err := s.WithVolumeServerClient(pb.ServerAddress(server), func(client volume_server_pb.VolumeServerClient) error {
resp, err := client.VolumeEcShardsInfo(context.Background(), &volume_server_pb.VolumeEcShardsInfoRequest{
VolumeId: volumeID,
})
if err != nil {
glog.V(1).Infof("Failed to get EC shard info from %s for volume %d: %v", server, volumeID, err)
return nil // Continue with other servers, don't fail the entire request
}
// Store shard sizes for this server
shardSizeMap[server] = make(map[uint32]uint64)
for _, shardInfo := range resp.EcShardInfos {
shardSizeMap[server][shardInfo.ShardId] = uint64(shardInfo.Size)
}
return nil
})
if err != nil {
glog.V(1).Infof("Failed to connect to volume server %s: %v", server, err)
}
}
}
// Update shard sizes in the shards array
for i := range shards {
server := shards[i].Server
shardId := shards[i].ShardID
if serverSizes, exists := shardSizeMap[server]; exists {
if size, exists := serverSizes[shardId]; exists {
shards[i].Size = size
}
}
}
// Calculate completeness based on unique shard IDs
foundShards := make(map[int]bool)
for _, shard := range shards {
+24
View File
@@ -0,0 +1,24 @@
package dash
import (
"io"
"net/http"
"github.com/seaweedfs/seaweedfs/weed/admin/internal/httputil"
)
func writeJSON(w http.ResponseWriter, status int, payload interface{}) {
httputil.WriteJSON(w, status, payload)
}
func writeJSONError(w http.ResponseWriter, status int, message string) {
httputil.WriteJSONError(w, status, message)
}
func decodeJSONBody(r io.Reader, v interface{}) error {
return httputil.DecodeJSONBody(r, v)
}
func newJSONMaxReader(w http.ResponseWriter, r *http.Request) io.Reader {
return httputil.NewJSONMaxReader(w, r)
}
+110 -90
View File
@@ -4,109 +4,129 @@ import (
"net/http"
"strings"
"github.com/gin-contrib/sessions"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/gorilla/sessions"
)
// setAuthContext sets username and role in context for use in handlers
func setAuthContext(c *gin.Context, username, role interface{}) {
c.Set("username", username)
if role != nil {
c.Set("role", role)
} else {
// Default to admin for backward compatibility
c.Set("role", "admin")
}
const sessionName = "admin-session"
// SessionName returns the cookie session name used by the admin UI.
func SessionName() string {
return sessionName
}
// RequireAuth checks if user is authenticated
func RequireAuth() gin.HandlerFunc {
return func(c *gin.Context) {
session := sessions.Default(c)
authenticated := session.Get("authenticated")
username := session.Get("username")
role := session.Get("role")
type sessionValidationErrorKind int
if authenticated != true || username == nil {
c.Redirect(http.StatusTemporaryRedirect, "/login")
c.Abort()
return
}
const (
sessionValidationErrorKindUnauthenticated sessionValidationErrorKind = iota
sessionValidationErrorKindSessionInit
)
csrfToken, err := getOrCreateSessionCSRFToken(session)
if err != nil {
c.Redirect(http.StatusTemporaryRedirect, "/login?error=Unable to initialize session")
c.Abort()
return
}
// Set username and role in context for use in handlers
setAuthContext(c, username, role)
c.Set("csrf_token", csrfToken)
c.Next()
}
type sessionValidationError struct {
kind sessionValidationErrorKind
err error
}
// RequireAuthAPI checks if user is authenticated for API endpoints
// Returns JSON error instead of redirecting to login page
func RequireAuthAPI() gin.HandlerFunc {
return func(c *gin.Context) {
session := sessions.Default(c)
authenticated := session.Get("authenticated")
username := session.Get("username")
role := session.Get("role")
if authenticated != true || username == nil {
c.JSON(http.StatusUnauthorized, gin.H{
"error": "Authentication required",
"message": "Please log in to access this endpoint",
})
c.Abort()
return
}
csrfToken, err := getOrCreateSessionCSRFToken(session)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{
"error": "Failed to initialize session",
"message": "Unable to initialize CSRF token",
})
c.Abort()
return
}
// Set username and role in context for use in handlers
setAuthContext(c, username, role)
c.Set("csrf_token", csrfToken)
c.Next()
func (e *sessionValidationError) Error() string {
if e.err != nil {
return e.err.Error()
}
return "session validation failed"
}
// RequireWriteAccess checks if user has admin role (write access)
// Returns JSON error for API endpoints, redirects for HTML endpoints
func RequireWriteAccess() gin.HandlerFunc {
return func(c *gin.Context) {
role, exists := c.Get("role")
if !exists {
role = "admin" // Default for backward compatibility
}
func (e *sessionValidationError) Unwrap() error {
return e.err
}
roleStr, ok := role.(string)
if !ok || roleStr != "admin" {
// Check if this is an API request (path starts with /api) or HTML request
path := c.Request.URL.Path
if strings.HasPrefix(path, "/api") {
c.JSON(http.StatusForbidden, gin.H{
"error": "Insufficient permissions",
"message": "This operation requires admin access. Read-only users can only view data.",
})
} else {
c.Redirect(http.StatusSeeOther, "/admin?error=Insufficient permissions")
func validateSession(store sessions.Store, w http.ResponseWriter, r *http.Request) (string, string, string, error) {
session, err := store.Get(r, sessionName)
if err != nil {
return "", "", "", &sessionValidationError{kind: sessionValidationErrorKindSessionInit, err: err}
}
authenticated, _ := session.Values["authenticated"].(bool)
username, _ := session.Values["username"].(string)
role, _ := session.Values["role"].(string)
if !authenticated || username == "" {
return "", "", "", &sessionValidationError{kind: sessionValidationErrorKindUnauthenticated}
}
csrfToken, err := getOrCreateSessionCSRFToken(session, r, w)
if err != nil {
return "", "", "", &sessionValidationError{kind: sessionValidationErrorKindSessionInit, err: err}
}
return username, role, csrfToken, nil
}
// RequireAuth checks if user is authenticated.
func RequireAuth(store sessions.Store) mux.MiddlewareFunc {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
username, role, csrfToken, err := validateSession(store, w, r)
if err != nil {
if verr, ok := err.(*sessionValidationError); ok && verr.kind == sessionValidationErrorKindUnauthenticated {
http.Redirect(w, r, "/login", http.StatusTemporaryRedirect)
} else {
http.Redirect(w, r, "/login?error=Unable to initialize session", http.StatusTemporaryRedirect)
}
return
}
c.Abort()
return
}
c.Next()
ctx := WithAuthContext(r.Context(), username, role, csrfToken)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
}
// RequireAuthAPI checks if user is authenticated for API endpoints.
// Returns JSON error instead of redirecting to login page.
func RequireAuthAPI(store sessions.Store) mux.MiddlewareFunc {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
username, role, csrfToken, err := validateSession(store, w, r)
if err != nil {
if verr, ok := err.(*sessionValidationError); ok && verr.kind == sessionValidationErrorKindUnauthenticated {
writeJSON(w, http.StatusUnauthorized, map[string]string{
"error": "Authentication required",
"message": "Please log in to access this endpoint",
})
} else {
writeJSON(w, http.StatusInternalServerError, map[string]string{
"error": "Failed to initialize session",
"message": "Unable to initialize session",
})
}
return
}
ctx := WithAuthContext(r.Context(), username, role, csrfToken)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
}
// RequireWriteAccess checks if user has admin role (write access).
// Returns JSON error for API endpoints, redirects for HTML endpoints.
func RequireWriteAccess() mux.MiddlewareFunc {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
role := RoleFromContext(r.Context())
if role != "admin" {
// Check if this is an API request (path starts with /api) or HTML request.
if strings.HasPrefix(r.URL.Path, "/api") {
writeJSON(w, http.StatusForbidden, map[string]string{
"error": "Insufficient permissions",
"message": "This operation requires admin access. Read-only users can only view data.",
})
} else {
http.Redirect(w, r, "/admin?error=Insufficient permissions", http.StatusSeeOther)
}
return
}
next.ServeHTTP(w, r)
})
}
}
+113 -108
View File
@@ -13,7 +13,7 @@ import (
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/plugin"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/plugin_pb"
@@ -32,17 +32,17 @@ const (
)
// GetPluginStatusAPI returns plugin status.
func (s *AdminServer) GetPluginStatusAPI(c *gin.Context) {
func (s *AdminServer) GetPluginStatusAPI(w http.ResponseWriter, r *http.Request) {
plugin := s.GetPlugin()
if plugin == nil {
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"enabled": false,
"worker_grpc_port": s.GetWorkerGrpcPort(),
})
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"enabled": true,
"configured": plugin.IsConfigured(),
"base_dir": plugin.BaseDir(),
@@ -52,101 +52,104 @@ func (s *AdminServer) GetPluginStatusAPI(c *gin.Context) {
}
// GetPluginWorkersAPI returns currently connected plugin workers.
func (s *AdminServer) GetPluginWorkersAPI(c *gin.Context) {
func (s *AdminServer) GetPluginWorkersAPI(w http.ResponseWriter, r *http.Request) {
workers := s.GetPluginWorkers()
if workers == nil {
c.JSON(http.StatusOK, []interface{}{})
writeJSON(w, http.StatusOK, []interface{}{})
return
}
c.JSON(http.StatusOK, workers)
writeJSON(w, http.StatusOK, workers)
}
// GetPluginJobTypesAPI returns known plugin job types from workers and persisted data.
func (s *AdminServer) GetPluginJobTypesAPI(c *gin.Context) {
func (s *AdminServer) GetPluginJobTypesAPI(w http.ResponseWriter, r *http.Request) {
jobTypes, err := s.ListPluginJobTypes()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
if jobTypes == nil {
c.JSON(http.StatusOK, []interface{}{})
writeJSON(w, http.StatusOK, []interface{}{})
return
}
c.JSON(http.StatusOK, jobTypes)
writeJSON(w, http.StatusOK, jobTypes)
}
// GetPluginJobsAPI returns tracked jobs for monitoring.
func (s *AdminServer) GetPluginJobsAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Query("job_type"))
state := strings.TrimSpace(c.Query("state"))
limit := parsePositiveInt(c.Query("limit"), 200)
func (s *AdminServer) GetPluginJobsAPI(w http.ResponseWriter, r *http.Request) {
query := r.URL.Query()
jobType := strings.TrimSpace(query.Get("job_type"))
state := strings.TrimSpace(query.Get("state"))
limit := parsePositiveInt(query.Get("limit"), 200)
jobs := s.ListPluginJobs(jobType, state, limit)
if jobs == nil {
c.JSON(http.StatusOK, []interface{}{})
writeJSON(w, http.StatusOK, []interface{}{})
return
}
c.JSON(http.StatusOK, jobs)
writeJSON(w, http.StatusOK, jobs)
}
// GetPluginJobAPI returns one tracked job.
func (s *AdminServer) GetPluginJobAPI(c *gin.Context) {
jobID := strings.TrimSpace(c.Param("jobId"))
func (s *AdminServer) GetPluginJobAPI(w http.ResponseWriter, r *http.Request) {
jobID := strings.TrimSpace(mux.Vars(r)["jobId"])
if jobID == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobId is required"})
writeJSONError(w, http.StatusBadRequest, "jobId is required")
return
}
job, found := s.GetPluginJob(jobID)
if !found {
c.JSON(http.StatusNotFound, gin.H{"error": "job not found"})
writeJSONError(w, http.StatusNotFound, "job not found")
return
}
c.JSON(http.StatusOK, job)
writeJSON(w, http.StatusOK, job)
}
// GetPluginJobDetailAPI returns detailed information for one tracked plugin job.
func (s *AdminServer) GetPluginJobDetailAPI(c *gin.Context) {
jobID := strings.TrimSpace(c.Param("jobId"))
func (s *AdminServer) GetPluginJobDetailAPI(w http.ResponseWriter, r *http.Request) {
jobID := strings.TrimSpace(mux.Vars(r)["jobId"])
if jobID == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobId is required"})
writeJSONError(w, http.StatusBadRequest, "jobId is required")
return
}
activityLimit := parsePositiveInt(c.Query("activity_limit"), 500)
relatedLimit := parsePositiveInt(c.Query("related_limit"), 20)
query := r.URL.Query()
activityLimit := parsePositiveInt(query.Get("activity_limit"), 500)
relatedLimit := parsePositiveInt(query.Get("related_limit"), 20)
detail, found, err := s.GetPluginJobDetail(jobID, activityLimit, relatedLimit)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
if !found || detail == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "job detail not found"})
writeJSONError(w, http.StatusNotFound, "job detail not found")
return
}
c.JSON(http.StatusOK, detail)
writeJSON(w, http.StatusOK, detail)
}
// GetPluginActivitiesAPI returns recent plugin activities.
func (s *AdminServer) GetPluginActivitiesAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Query("job_type"))
limit := parsePositiveInt(c.Query("limit"), 500)
func (s *AdminServer) GetPluginActivitiesAPI(w http.ResponseWriter, r *http.Request) {
query := r.URL.Query()
jobType := strings.TrimSpace(query.Get("job_type"))
limit := parsePositiveInt(query.Get("limit"), 500)
activities := s.ListPluginActivities(jobType, limit)
if activities == nil {
c.JSON(http.StatusOK, []interface{}{})
writeJSON(w, http.StatusOK, []interface{}{})
return
}
c.JSON(http.StatusOK, activities)
writeJSON(w, http.StatusOK, activities)
}
// GetPluginSchedulerStatesAPI returns per-job-type scheduler status for monitoring.
func (s *AdminServer) GetPluginSchedulerStatesAPI(c *gin.Context) {
jobTypeFilter := strings.TrimSpace(c.Query("job_type"))
func (s *AdminServer) GetPluginSchedulerStatesAPI(w http.ResponseWriter, r *http.Request) {
jobTypeFilter := strings.TrimSpace(r.URL.Query().Get("job_type"))
states, err := s.ListPluginSchedulerStates()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
@@ -157,71 +160,71 @@ func (s *AdminServer) GetPluginSchedulerStatesAPI(c *gin.Context) {
filtered = append(filtered, state)
}
}
c.JSON(http.StatusOK, filtered)
writeJSON(w, http.StatusOK, filtered)
return
}
if states == nil {
c.JSON(http.StatusOK, []interface{}{})
writeJSON(w, http.StatusOK, []interface{}{})
return
}
c.JSON(http.StatusOK, states)
writeJSON(w, http.StatusOK, states)
}
// RequestPluginJobTypeSchemaAPI asks a worker for one job type schema.
func (s *AdminServer) RequestPluginJobTypeSchemaAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) RequestPluginJobTypeSchemaAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
forceRefresh := c.DefaultQuery("force_refresh", "false") == "true"
forceRefresh := strings.EqualFold(r.URL.Query().Get("force_refresh"), "true")
ctx, cancel := context.WithTimeout(c.Request.Context(), defaultPluginDetectionTimeout)
ctx, cancel := context.WithTimeout(r.Context(), defaultPluginDetectionTimeout)
defer cancel()
descriptor, err := s.RequestPluginJobTypeDescriptor(ctx, jobType, forceRefresh)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
renderProtoJSON(c, http.StatusOK, descriptor)
renderProtoJSON(w, http.StatusOK, descriptor)
}
// GetPluginJobTypeDescriptorAPI returns persisted descriptor for a job type.
func (s *AdminServer) GetPluginJobTypeDescriptorAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) GetPluginJobTypeDescriptorAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
descriptor, err := s.LoadPluginJobTypeDescriptor(jobType)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
if descriptor == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "descriptor not found"})
writeJSONError(w, http.StatusNotFound, "descriptor not found")
return
}
renderProtoJSON(c, http.StatusOK, descriptor)
renderProtoJSON(w, http.StatusOK, descriptor)
}
// GetPluginJobTypeConfigAPI loads persisted config for a job type.
func (s *AdminServer) GetPluginJobTypeConfigAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) GetPluginJobTypeConfigAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
config, err := s.LoadPluginJobTypeConfig(jobType)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
if config == nil {
@@ -233,20 +236,20 @@ func (s *AdminServer) GetPluginJobTypeConfigAPI(c *gin.Context) {
}
}
renderProtoJSON(c, http.StatusOK, config)
renderProtoJSON(w, http.StatusOK, config)
}
// UpdatePluginJobTypeConfigAPI stores persisted config for a job type.
func (s *AdminServer) UpdatePluginJobTypeConfigAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) UpdatePluginJobTypeConfigAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
config := &plugin_pb.PersistedJobTypeConfig{}
if err := parseProtoJSONBody(c, config); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
if err := parseProtoJSONBody(w, r, config); err != nil {
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
@@ -264,35 +267,35 @@ func (s *AdminServer) UpdatePluginJobTypeConfigAPI(c *gin.Context) {
config.WorkerConfigValues = map[string]*plugin_pb.ConfigValue{}
}
username := c.GetString("username")
username := UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
config.UpdatedBy = username
if err := s.SavePluginJobTypeConfig(config); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
renderProtoJSON(c, http.StatusOK, config)
renderProtoJSON(w, http.StatusOK, config)
}
// GetPluginRunHistoryAPI returns bounded run history for a job type.
func (s *AdminServer) GetPluginRunHistoryAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) GetPluginRunHistoryAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
history, err := s.GetPluginRunHistory(jobType)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
if history == nil {
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"job_type": jobType,
"successful_runs": []interface{}{},
"error_runs": []interface{}{},
@@ -301,14 +304,14 @@ func (s *AdminServer) GetPluginRunHistoryAPI(c *gin.Context) {
return
}
c.JSON(http.StatusOK, history)
writeJSON(w, http.StatusOK, history)
}
// TriggerPluginDetectionAPI runs one detector for this job type and returns proposals.
func (s *AdminServer) TriggerPluginDetectionAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) TriggerPluginDetectionAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
@@ -318,19 +321,19 @@ func (s *AdminServer) TriggerPluginDetectionAPI(c *gin.Context) {
TimeoutSeconds int `json:"timeout_seconds"`
}
if err := c.ShouldBindJSON(&req); err != nil && err != io.EOF {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil && err != io.EOF {
writeJSONError(w, http.StatusBadRequest, "invalid request body: "+err.Error())
return
}
clusterContext, err := s.parseOrBuildClusterContext(req.ClusterContext)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
timeout := normalizeTimeout(req.TimeoutSeconds, defaultPluginDetectionTimeout, maxPluginDetectionTimeout)
ctx, cancel := context.WithTimeout(c.Request.Context(), timeout)
ctx, cancel := context.WithTimeout(r.Context(), timeout)
defer cancel()
report, err := s.RunPluginDetectionWithReport(ctx, jobType, clusterContext, req.MaxResults)
@@ -384,7 +387,7 @@ func (s *AdminServer) TriggerPluginDetectionAPI(c *gin.Context) {
}
}
response := gin.H{
response := map[string]interface{}{
"job_type": jobType,
"request_id": requestID,
"detector_worker_id": detectorWorkerID,
@@ -396,18 +399,18 @@ func (s *AdminServer) TriggerPluginDetectionAPI(c *gin.Context) {
if err != nil {
response["error"] = err.Error()
c.JSON(http.StatusInternalServerError, response)
writeJSON(w, http.StatusInternalServerError, response)
return
}
c.JSON(http.StatusOK, response)
writeJSON(w, http.StatusOK, response)
}
// RunPluginJobTypeAPI runs full workflow for one job type: detect then dispatch detected jobs.
func (s *AdminServer) RunPluginJobTypeAPI(c *gin.Context) {
jobType := strings.TrimSpace(c.Param("jobType"))
func (s *AdminServer) RunPluginJobTypeAPI(w http.ResponseWriter, r *http.Request) {
jobType := strings.TrimSpace(mux.Vars(r)["jobType"])
if jobType == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "jobType is required"})
writeJSONError(w, http.StatusBadRequest, "jobType is required")
return
}
@@ -418,8 +421,8 @@ func (s *AdminServer) RunPluginJobTypeAPI(c *gin.Context) {
Attempt int32 `json:"attempt"`
}
if err := c.ShouldBindJSON(&req); err != nil && err != io.EOF {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil && err != io.EOF {
writeJSONError(w, http.StatusBadRequest, "invalid request body: "+err.Error())
return
}
if req.Attempt < 1 {
@@ -428,24 +431,24 @@ func (s *AdminServer) RunPluginJobTypeAPI(c *gin.Context) {
clusterContext, err := s.parseOrBuildClusterContext(req.ClusterContext)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
timeout := normalizeTimeout(req.TimeoutSeconds, defaultPluginRunTimeout, maxPluginRunTimeout)
ctx, cancel := context.WithTimeout(c.Request.Context(), timeout)
ctx, cancel := context.WithTimeout(r.Context(), timeout)
defer cancel()
proposals, err := s.RunPluginDetection(ctx, jobType, clusterContext, req.MaxResults)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
detectedCount := len(proposals)
filteredProposals, skippedActiveCount, err := s.FilterPluginProposalsWithActiveJobs(jobType, proposals)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
@@ -485,7 +488,7 @@ func (s *AdminServer) RunPluginJobTypeAPI(c *gin.Context) {
results = append(results, result)
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"job_type": jobType,
"detected_count": detectedCount,
"ready_to_execute_count": len(filteredProposals),
@@ -498,7 +501,7 @@ func (s *AdminServer) RunPluginJobTypeAPI(c *gin.Context) {
}
// ExecutePluginJobAPI executes one job on a capable worker and waits for completion.
func (s *AdminServer) ExecutePluginJobAPI(c *gin.Context) {
func (s *AdminServer) ExecutePluginJobAPI(w http.ResponseWriter, r *http.Request) {
var req struct {
Job json.RawMessage `json:"job"`
ClusterContext json.RawMessage `json:"cluster_context"`
@@ -506,24 +509,24 @@ func (s *AdminServer) ExecutePluginJobAPI(c *gin.Context) {
TimeoutSeconds int `json:"timeout_seconds"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "invalid request body: "+err.Error())
return
}
if len(req.Job) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "job is required"})
writeJSONError(w, http.StatusBadRequest, "job is required")
return
}
job := &plugin_pb.JobSpec{}
if err := (protojson.UnmarshalOptions{DiscardUnknown: true}).Unmarshal(req.Job, job); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid job payload: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "invalid job payload: "+err.Error())
return
}
clusterContext, err := s.parseOrBuildClusterContext(req.ClusterContext)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
@@ -532,7 +535,7 @@ func (s *AdminServer) ExecutePluginJobAPI(c *gin.Context) {
}
timeout := normalizeTimeout(req.TimeoutSeconds, defaultPluginExecutionTimeout, maxPluginExecutionTimeout)
ctx, cancel := context.WithTimeout(c.Request.Context(), timeout)
ctx, cancel := context.WithTimeout(r.Context(), timeout)
defer cancel()
completed, err := s.ExecutePluginJob(ctx, job, clusterContext, req.Attempt)
@@ -540,15 +543,15 @@ func (s *AdminServer) ExecutePluginJobAPI(c *gin.Context) {
if completed != nil {
payload, marshalErr := protoMessageToMap(completed)
if marshalErr == nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error(), "completion": payload})
writeJSON(w, http.StatusInternalServerError, map[string]interface{}{"error": err.Error(), "completion": payload})
return
}
}
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
renderProtoJSON(c, http.StatusOK, completed)
renderProtoJSON(w, http.StatusOK, completed)
}
func (s *AdminServer) parseOrBuildClusterContext(raw json.RawMessage) (*plugin_pb.ClusterContext, error) {
@@ -636,8 +639,8 @@ func (s *AdminServer) buildDefaultPluginClusterContext() *plugin_pb.ClusterConte
const parseProtoJSONBodyMaxBytes = 1 << 20 // 1 MB
func parseProtoJSONBody(c *gin.Context, message proto.Message) error {
limitedBody := http.MaxBytesReader(c.Writer, c.Request.Body, parseProtoJSONBodyMaxBytes)
func parseProtoJSONBody(w http.ResponseWriter, r *http.Request, message proto.Message) error {
limitedBody := http.MaxBytesReader(w, r.Body, parseProtoJSONBodyMaxBytes)
data, err := io.ReadAll(limitedBody)
if err != nil {
return fmt.Errorf("failed to read request body: %w", err)
@@ -651,17 +654,19 @@ func parseProtoJSONBody(c *gin.Context, message proto.Message) error {
return nil
}
func renderProtoJSON(c *gin.Context, statusCode int, message proto.Message) {
func renderProtoJSON(w http.ResponseWriter, statusCode int, message proto.Message) {
payload, err := protojson.MarshalOptions{
UseProtoNames: true,
EmitUnpopulated: true,
}.Marshal(message)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to encode response: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "failed to encode response: "+err.Error())
return
}
c.Data(statusCode, "application/json", payload)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(statusCode)
_, _ = w.Write(payload)
}
func protoMessageToMap(message proto.Message) (map[string]interface{}, error) {
+175 -146
View File
@@ -12,7 +12,6 @@ import (
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
@@ -67,10 +66,10 @@ func parseNamespaceInput(namespace string) ([]string, error) {
return s3tables.ParseNamespace(namespace)
}
func (s *AdminServer) parseNamespaceFromGin(c *gin.Context, namespace string) ([]string, bool) {
func (s *AdminServer) parseNamespaceFromRequest(w http.ResponseWriter, namespace string) ([]string, bool) {
parts, err := parseNamespaceInput(namespace)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid namespace: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "Invalid namespace: "+err.Error())
return nil, false
}
return parts, true
@@ -569,58 +568,61 @@ func parseSummaryInt(summary map[string]string, keys ...string) (int64, bool) {
// API handlers
func (s *AdminServer) ListS3TablesBucketsAPI(c *gin.Context) {
data, err := s.GetS3TablesBucketsData(c.Request.Context())
func (s *AdminServer) ListS3TablesBucketsAPI(w http.ResponseWriter, r *http.Request) {
data, err := s.GetS3TablesBucketsData(r.Context())
if err != nil {
writeS3TablesError(c, err)
writeS3TablesError(w, err)
return
}
c.JSON(200, data)
writeJSON(w, http.StatusOK, data)
}
func (s *AdminServer) CreateS3TablesBucket(c *gin.Context) {
func (s *AdminServer) CreateS3TablesBucket(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
Name string `json:"name"`
Tags map[string]string `json:"tags"`
Owner string `json:"owner"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.Name == "" {
c.JSON(400, gin.H{"error": "Bucket name is required"})
writeJSONError(w, http.StatusBadRequest, "Bucket name is required")
return
}
owner := strings.TrimSpace(req.Owner)
if len(owner) > MaxOwnerNameLength {
c.JSON(400, gin.H{"error": fmt.Sprintf("Owner name must be %d characters or less", MaxOwnerNameLength)})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Owner name must be %d characters or less", MaxOwnerNameLength))
return
}
if len(req.Tags) > 0 {
if err := s3tables.ValidateTags(req.Tags); err != nil {
c.JSON(400, gin.H{"error": "Invalid tags: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "Invalid tags: "+err.Error())
return
}
}
createReq := &s3tables.CreateTableBucketRequest{Name: req.Name, Tags: req.Tags}
var resp s3tables.CreateTableBucketResponse
if err := s.executeS3TablesOperation(c.Request.Context(), "CreateTableBucket", createReq, &resp); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "CreateTableBucket", createReq, &resp); err != nil {
writeS3TablesError(w, err)
return
}
if owner != "" {
if err := s.SetTableBucketOwner(c.Request.Context(), req.Name, owner); err != nil {
if err := s.SetTableBucketOwner(r.Context(), req.Name, owner); err != nil {
deleteReq := &s3tables.DeleteTableBucketRequest{TableBucketARN: resp.ARN}
if deleteErr := s.executeS3TablesOperation(c.Request.Context(), "DeleteTableBucket", deleteReq, nil); deleteErr != nil {
c.JSON(500, gin.H{"error": fmt.Sprintf("Failed to set table bucket owner: %v; rollback delete failed: %v", err, deleteErr)})
if deleteErr := s.executeS3TablesOperation(r.Context(), "DeleteTableBucket", deleteReq, nil); deleteErr != nil {
writeJSONError(w, http.StatusInternalServerError, fmt.Sprintf("Failed to set table bucket owner: %v; rollback delete failed: %v", err, deleteErr))
return
}
writeS3TablesError(c, err)
writeS3TablesError(w, err)
return
}
}
c.JSON(201, gin.H{"arn": resp.ARN})
writeJSON(w, http.StatusCreated, map[string]interface{}{"arn": resp.ARN})
}
func (s *AdminServer) SetTableBucketOwner(ctx context.Context, bucketName, owner string) error {
@@ -663,101 +665,107 @@ func (s *AdminServer) SetTableBucketOwner(ctx context.Context, bucketName, owner
})
}
func (s *AdminServer) DeleteS3TablesBucket(c *gin.Context) {
bucketArn := c.Query("bucket")
func (s *AdminServer) DeleteS3TablesBucket(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
bucketArn := r.URL.Query().Get("bucket")
if bucketArn == "" {
c.JSON(400, gin.H{"error": "Bucket ARN is required"})
writeJSONError(w, http.StatusBadRequest, "Bucket ARN is required")
return
}
req := &s3tables.DeleteTableBucketRequest{TableBucketARN: bucketArn}
if err := s.executeS3TablesOperation(c.Request.Context(), "DeleteTableBucket", req, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "DeleteTableBucket", req, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Bucket deleted"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Bucket deleted"})
}
func (s *AdminServer) ListS3TablesNamespacesAPI(c *gin.Context) {
bucketArn := c.Query("bucket")
func (s *AdminServer) ListS3TablesNamespacesAPI(w http.ResponseWriter, r *http.Request) {
bucketArn := r.URL.Query().Get("bucket")
if bucketArn == "" {
c.JSON(400, gin.H{"error": "bucket query parameter is required"})
writeJSONError(w, http.StatusBadRequest, "bucket query parameter is required")
return
}
data, err := s.GetS3TablesNamespacesData(c.Request.Context(), bucketArn)
data, err := s.GetS3TablesNamespacesData(r.Context(), bucketArn)
if err != nil {
writeS3TablesError(c, err)
writeS3TablesError(w, err)
return
}
c.JSON(200, data)
writeJSON(w, http.StatusOK, data)
}
func (s *AdminServer) CreateS3TablesNamespace(c *gin.Context) {
if !requireSessionCSRFToken(c) {
func (s *AdminServer) CreateS3TablesNamespace(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
BucketARN string `json:"bucket_arn"`
Name string `json:"name"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.BucketARN == "" || req.Name == "" {
c.JSON(400, gin.H{"error": "bucket_arn and name are required"})
writeJSONError(w, http.StatusBadRequest, "bucket_arn and name are required")
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, req.Name)
namespaceParts, ok := s.parseNamespaceFromRequest(w, req.Name)
if !ok {
return
}
createReq := &s3tables.CreateNamespaceRequest{TableBucketARN: req.BucketARN, Namespace: namespaceParts}
var resp s3tables.CreateNamespaceResponse
if err := s.executeS3TablesOperation(c.Request.Context(), "CreateNamespace", createReq, &resp); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "CreateNamespace", createReq, &resp); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(201, gin.H{"namespace": resp.Namespace})
writeJSON(w, http.StatusCreated, map[string]interface{}{"namespace": resp.Namespace})
}
func (s *AdminServer) DeleteS3TablesNamespace(c *gin.Context) {
if !requireSessionCSRFToken(c) {
func (s *AdminServer) DeleteS3TablesNamespace(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
bucketArn := c.Query("bucket")
namespace := c.Query("name")
bucketArn := r.URL.Query().Get("bucket")
namespace := r.URL.Query().Get("name")
if bucketArn == "" || namespace == "" {
c.JSON(400, gin.H{"error": "bucket and name query parameters are required"})
writeJSONError(w, http.StatusBadRequest, "bucket and name query parameters are required")
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, namespace)
namespaceParts, ok := s.parseNamespaceFromRequest(w, namespace)
if !ok {
return
}
req := &s3tables.DeleteNamespaceRequest{TableBucketARN: bucketArn, Namespace: namespaceParts}
if err := s.executeS3TablesOperation(c.Request.Context(), "DeleteNamespace", req, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "DeleteNamespace", req, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Namespace deleted"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Namespace deleted"})
}
func (s *AdminServer) ListS3TablesTablesAPI(c *gin.Context) {
bucketArn := c.Query("bucket")
func (s *AdminServer) ListS3TablesTablesAPI(w http.ResponseWriter, r *http.Request) {
bucketArn := r.URL.Query().Get("bucket")
if bucketArn == "" {
c.JSON(400, gin.H{"error": "bucket query parameter is required"})
writeJSONError(w, http.StatusBadRequest, "bucket query parameter is required")
return
}
namespace := c.Query("namespace")
data, err := s.GetS3TablesTablesData(c.Request.Context(), bucketArn, namespace)
namespace := r.URL.Query().Get("namespace")
data, err := s.GetS3TablesTablesData(r.Context(), bucketArn, namespace)
if err != nil {
writeS3TablesError(c, err)
writeS3TablesError(w, err)
return
}
c.JSON(200, data)
writeJSON(w, http.StatusOK, data)
}
func (s *AdminServer) CreateS3TablesTable(c *gin.Context) {
func (s *AdminServer) CreateS3TablesTable(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
BucketARN string `json:"bucket_arn"`
Namespace string `json:"namespace"`
@@ -766,15 +774,15 @@ func (s *AdminServer) CreateS3TablesTable(c *gin.Context) {
Tags map[string]string `json:"tags"`
Metadata *s3tables.TableMetadata `json:"metadata"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.BucketARN == "" || req.Namespace == "" || req.Name == "" {
c.JSON(400, gin.H{"error": "bucket_arn, namespace, and name are required"})
writeJSONError(w, http.StatusBadRequest, "bucket_arn, namespace, and name are required")
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, req.Namespace)
namespaceParts, ok := s.parseNamespaceFromRequest(w, req.Namespace)
if !ok {
return
}
@@ -784,7 +792,7 @@ func (s *AdminServer) CreateS3TablesTable(c *gin.Context) {
}
if len(req.Tags) > 0 {
if err := s3tables.ValidateTags(req.Tags); err != nil {
c.JSON(400, gin.H{"error": "Invalid tags: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "Invalid tags: "+err.Error())
return
}
}
@@ -797,211 +805,232 @@ func (s *AdminServer) CreateS3TablesTable(c *gin.Context) {
Metadata: req.Metadata,
}
var resp s3tables.CreateTableResponse
if err := s.executeS3TablesOperation(c.Request.Context(), "CreateTable", createReq, &resp); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "CreateTable", createReq, &resp); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(201, gin.H{"table_arn": resp.TableARN, "version_token": resp.VersionToken})
writeJSON(w, http.StatusCreated, map[string]interface{}{"table_arn": resp.TableARN, "version_token": resp.VersionToken})
}
func (s *AdminServer) DeleteS3TablesTable(c *gin.Context) {
bucketArn := c.Query("bucket")
namespace := c.Query("namespace")
name := c.Query("name")
version := c.Query("version")
if bucketArn == "" || namespace == "" || name == "" {
c.JSON(400, gin.H{"error": "bucket, namespace, and name query parameters are required"})
func (s *AdminServer) DeleteS3TablesTable(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, namespace)
bucketArn := r.URL.Query().Get("bucket")
namespace := r.URL.Query().Get("namespace")
name := r.URL.Query().Get("name")
version := r.URL.Query().Get("version")
if bucketArn == "" || namespace == "" || name == "" {
writeJSONError(w, http.StatusBadRequest, "bucket, namespace, and name query parameters are required")
return
}
namespaceParts, ok := s.parseNamespaceFromRequest(w, namespace)
if !ok {
return
}
req := &s3tables.DeleteTableRequest{TableBucketARN: bucketArn, Namespace: namespaceParts, Name: name, VersionToken: version}
if err := s.executeS3TablesOperation(c.Request.Context(), "DeleteTable", req, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "DeleteTable", req, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Table deleted"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Table deleted"})
}
func (s *AdminServer) PutS3TablesBucketPolicy(c *gin.Context) {
func (s *AdminServer) PutS3TablesBucketPolicy(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
BucketARN string `json:"bucket_arn"`
Policy string `json:"policy"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.BucketARN == "" || req.Policy == "" {
c.JSON(400, gin.H{"error": "bucket_arn and policy are required"})
writeJSONError(w, http.StatusBadRequest, "bucket_arn and policy are required")
return
}
putReq := &s3tables.PutTableBucketPolicyRequest{TableBucketARN: req.BucketARN, ResourcePolicy: req.Policy}
if err := s.executeS3TablesOperation(c.Request.Context(), "PutTableBucketPolicy", putReq, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "PutTableBucketPolicy", putReq, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Policy updated"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Policy updated"})
}
func (s *AdminServer) GetS3TablesBucketPolicy(c *gin.Context) {
bucketArn := c.Query("bucket")
func (s *AdminServer) GetS3TablesBucketPolicy(w http.ResponseWriter, r *http.Request) {
bucketArn := r.URL.Query().Get("bucket")
if bucketArn == "" {
c.JSON(400, gin.H{"error": "bucket query parameter is required"})
writeJSONError(w, http.StatusBadRequest, "bucket query parameter is required")
return
}
getReq := &s3tables.GetTableBucketPolicyRequest{TableBucketARN: bucketArn}
var resp s3tables.GetTableBucketPolicyResponse
if err := s.executeS3TablesOperation(c.Request.Context(), "GetTableBucketPolicy", getReq, &resp); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "GetTableBucketPolicy", getReq, &resp); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"policy": resp.ResourcePolicy})
writeJSON(w, http.StatusOK, map[string]interface{}{"policy": resp.ResourcePolicy})
}
func (s *AdminServer) DeleteS3TablesBucketPolicy(c *gin.Context) {
bucketArn := c.Query("bucket")
func (s *AdminServer) DeleteS3TablesBucketPolicy(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
bucketArn := r.URL.Query().Get("bucket")
if bucketArn == "" {
c.JSON(400, gin.H{"error": "bucket query parameter is required"})
writeJSONError(w, http.StatusBadRequest, "bucket query parameter is required")
return
}
deleteReq := &s3tables.DeleteTableBucketPolicyRequest{TableBucketARN: bucketArn}
if err := s.executeS3TablesOperation(c.Request.Context(), "DeleteTableBucketPolicy", deleteReq, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "DeleteTableBucketPolicy", deleteReq, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Policy deleted"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Policy deleted"})
}
func (s *AdminServer) PutS3TablesTablePolicy(c *gin.Context) {
func (s *AdminServer) PutS3TablesTablePolicy(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
BucketARN string `json:"bucket_arn"`
Namespace string `json:"namespace"`
Name string `json:"name"`
Policy string `json:"policy"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.BucketARN == "" || req.Namespace == "" || req.Name == "" || req.Policy == "" {
c.JSON(400, gin.H{"error": "bucket_arn, namespace, name, and policy are required"})
writeJSONError(w, http.StatusBadRequest, "bucket_arn, namespace, name, and policy are required")
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, req.Namespace)
namespaceParts, ok := s.parseNamespaceFromRequest(w, req.Namespace)
if !ok {
return
}
putReq := &s3tables.PutTablePolicyRequest{TableBucketARN: req.BucketARN, Namespace: namespaceParts, Name: req.Name, ResourcePolicy: req.Policy}
if err := s.executeS3TablesOperation(c.Request.Context(), "PutTablePolicy", putReq, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "PutTablePolicy", putReq, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Policy updated"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Policy updated"})
}
func (s *AdminServer) GetS3TablesTablePolicy(c *gin.Context) {
bucketArn := c.Query("bucket")
namespace := c.Query("namespace")
name := c.Query("name")
func (s *AdminServer) GetS3TablesTablePolicy(w http.ResponseWriter, r *http.Request) {
bucketArn := r.URL.Query().Get("bucket")
namespace := r.URL.Query().Get("namespace")
name := r.URL.Query().Get("name")
if bucketArn == "" || namespace == "" || name == "" {
c.JSON(400, gin.H{"error": "bucket, namespace, and name query parameters are required"})
writeJSONError(w, http.StatusBadRequest, "bucket, namespace, and name query parameters are required")
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, namespace)
namespaceParts, ok := s.parseNamespaceFromRequest(w, namespace)
if !ok {
return
}
getReq := &s3tables.GetTablePolicyRequest{TableBucketARN: bucketArn, Namespace: namespaceParts, Name: name}
var resp s3tables.GetTablePolicyResponse
if err := s.executeS3TablesOperation(c.Request.Context(), "GetTablePolicy", getReq, &resp); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "GetTablePolicy", getReq, &resp); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"policy": resp.ResourcePolicy})
writeJSON(w, http.StatusOK, map[string]interface{}{"policy": resp.ResourcePolicy})
}
func (s *AdminServer) DeleteS3TablesTablePolicy(c *gin.Context) {
bucketArn := c.Query("bucket")
namespace := c.Query("namespace")
name := c.Query("name")
if bucketArn == "" || namespace == "" || name == "" {
c.JSON(400, gin.H{"error": "bucket, namespace, and name query parameters are required"})
func (s *AdminServer) DeleteS3TablesTablePolicy(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
namespaceParts, ok := s.parseNamespaceFromGin(c, namespace)
bucketArn := r.URL.Query().Get("bucket")
namespace := r.URL.Query().Get("namespace")
name := r.URL.Query().Get("name")
if bucketArn == "" || namespace == "" || name == "" {
writeJSONError(w, http.StatusBadRequest, "bucket, namespace, and name query parameters are required")
return
}
namespaceParts, ok := s.parseNamespaceFromRequest(w, namespace)
if !ok {
return
}
deleteReq := &s3tables.DeleteTablePolicyRequest{TableBucketARN: bucketArn, Namespace: namespaceParts, Name: name}
if err := s.executeS3TablesOperation(c.Request.Context(), "DeleteTablePolicy", deleteReq, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "DeleteTablePolicy", deleteReq, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Policy deleted"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Policy deleted"})
}
func (s *AdminServer) TagS3TablesResource(c *gin.Context) {
func (s *AdminServer) TagS3TablesResource(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
ResourceARN string `json:"resource_arn"`
Tags map[string]string `json:"tags"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.ResourceARN == "" || len(req.Tags) == 0 {
c.JSON(400, gin.H{"error": "resource_arn and tags are required"})
writeJSONError(w, http.StatusBadRequest, "resource_arn and tags are required")
return
}
if err := s3tables.ValidateTags(req.Tags); err != nil {
c.JSON(400, gin.H{"error": "Invalid tags: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "Invalid tags: "+err.Error())
return
}
tagReq := &s3tables.TagResourceRequest{ResourceARN: req.ResourceARN, Tags: req.Tags}
if err := s.executeS3TablesOperation(c.Request.Context(), "TagResource", tagReq, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "TagResource", tagReq, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Tags updated"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Tags updated"})
}
func (s *AdminServer) ListS3TablesTags(c *gin.Context) {
resourceArn := c.Query("arn")
func (s *AdminServer) ListS3TablesTags(w http.ResponseWriter, r *http.Request) {
resourceArn := r.URL.Query().Get("arn")
if resourceArn == "" {
c.JSON(400, gin.H{"error": "arn query parameter is required"})
writeJSONError(w, http.StatusBadRequest, "arn query parameter is required")
return
}
listReq := &s3tables.ListTagsForResourceRequest{ResourceARN: resourceArn}
var resp s3tables.ListTagsForResourceResponse
if err := s.executeS3TablesOperation(c.Request.Context(), "ListTagsForResource", listReq, &resp); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "ListTagsForResource", listReq, &resp); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, resp)
writeJSON(w, http.StatusOK, resp)
}
func (s *AdminServer) UntagS3TablesResource(c *gin.Context) {
func (s *AdminServer) UntagS3TablesResource(w http.ResponseWriter, r *http.Request) {
if !requireSessionCSRFToken(w, r) {
return
}
var req struct {
ResourceARN string `json:"resource_arn"`
TagKeys []string `json:"tag_keys"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(400, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.ResourceARN == "" || len(req.TagKeys) == 0 {
c.JSON(400, gin.H{"error": "resource_arn and tag_keys are required"})
writeJSONError(w, http.StatusBadRequest, "resource_arn and tag_keys are required")
return
}
untagReq := &s3tables.UntagResourceRequest{ResourceARN: req.ResourceARN, TagKeys: req.TagKeys}
if err := s.executeS3TablesOperation(c.Request.Context(), "UntagResource", untagReq, nil); err != nil {
writeS3TablesError(c, err)
if err := s.executeS3TablesOperation(r.Context(), "UntagResource", untagReq, nil); err != nil {
writeS3TablesError(w, err)
return
}
c.JSON(200, gin.H{"message": "Tags removed"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Tags removed"})
}
func parseS3TablesErrorMessage(err error) string {
@@ -1018,8 +1047,8 @@ func parseS3TablesErrorMessage(err error) string {
return err.Error()
}
func writeS3TablesError(c *gin.Context, err error) {
c.JSON(s3TablesErrorStatus(err), gin.H{"error": parseS3TablesErrorMessage(err)})
func writeS3TablesError(w http.ResponseWriter, err error) {
writeJSONError(w, s3TablesErrorStatus(err), parseS3TablesErrorMessage(err))
}
func s3TablesErrorStatus(err error) int {
+271 -446
View File
@@ -5,7 +5,8 @@ import (
"net/url"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/gorilla/sessions"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
@@ -18,6 +19,7 @@ import (
// AdminHandlers contains all the HTTP handlers for the admin interface
type AdminHandlers struct {
adminServer *dash.AdminServer
sessionStore sessions.Store
authHandlers *AuthHandlers
clusterHandlers *ClusterHandlers
fileBrowserHandlers *FileBrowserHandlers
@@ -29,8 +31,8 @@ type AdminHandlers struct {
}
// NewAdminHandlers creates a new instance of AdminHandlers
func NewAdminHandlers(adminServer *dash.AdminServer) *AdminHandlers {
authHandlers := NewAuthHandlers(adminServer)
func NewAdminHandlers(adminServer *dash.AdminServer, store sessions.Store) *AdminHandlers {
authHandlers := NewAuthHandlers(adminServer, store)
clusterHandlers := NewClusterHandlers(adminServer)
fileBrowserHandlers := NewFileBrowserHandlers(adminServer)
userHandlers := NewUserHandlers(adminServer)
@@ -40,6 +42,7 @@ func NewAdminHandlers(adminServer *dash.AdminServer) *AdminHandlers {
serviceAccountHandlers := NewServiceAccountHandlers(adminServer)
return &AdminHandlers{
adminServer: adminServer,
sessionStore: store,
authHandlers: authHandlers,
clusterHandlers: clusterHandlers,
fileBrowserHandlers: fileBrowserHandlers,
@@ -52,17 +55,17 @@ func NewAdminHandlers(adminServer *dash.AdminServer) *AdminHandlers {
}
// SetupRoutes configures all the routes for the admin interface
func (h *AdminHandlers) SetupRoutes(r *gin.Engine, authRequired bool, adminUser, adminPassword, readOnlyUser, readOnlyPassword string, enableUI bool) {
func (h *AdminHandlers) SetupRoutes(r *mux.Router, authRequired bool, adminUser, adminPassword, readOnlyUser, readOnlyPassword string, enableUI bool) {
// Health check (no auth required)
r.GET("/health", h.HealthCheck)
r.HandleFunc("/health", h.HealthCheck).Methods(http.MethodGet)
// Prometheus metrics endpoint (no auth required)
r.GET("/metrics", gin.WrapH(promhttp.HandlerFor(stats.Gather, promhttp.HandlerOpts{})))
r.Handle("/metrics", promhttp.HandlerFor(stats.Gather, promhttp.HandlerOpts{})).Methods(http.MethodGet)
// Favicon route (no auth required) - redirect to static version
r.GET("/favicon.ico", func(c *gin.Context) {
c.Redirect(http.StatusMovedPermanently, "/static/favicon.ico")
})
r.HandleFunc("/favicon.ico", func(w http.ResponseWriter, req *http.Request) {
http.Redirect(w, req, "/static/favicon.ico", http.StatusMovedPermanently)
}).Methods(http.MethodGet)
// Skip UI routes if UI is not enabled
if !enableUI {
@@ -71,499 +74,321 @@ func (h *AdminHandlers) SetupRoutes(r *gin.Engine, authRequired bool, adminUser,
if authRequired {
// Authentication routes (no auth required)
r.GET("/login", h.authHandlers.ShowLogin)
r.POST("/login", h.authHandlers.HandleLogin(adminUser, adminPassword, readOnlyUser, readOnlyPassword))
r.GET("/logout", h.authHandlers.HandleLogout)
r.HandleFunc("/login", h.authHandlers.ShowLogin).Methods(http.MethodGet)
r.Handle("/login", h.authHandlers.HandleLogin(adminUser, adminPassword, readOnlyUser, readOnlyPassword)).Methods(http.MethodPost)
r.HandleFunc("/logout", h.authHandlers.HandleLogout).Methods(http.MethodGet)
// Protected routes group
protected := r.Group("/")
protected.Use(dash.RequireAuth())
protected := r.NewRoute().Subrouter()
protected.Use(dash.RequireAuth(h.sessionStore))
h.registerUIRoutes(protected)
// Main admin interface routes
protected.GET("/", h.ShowDashboard)
protected.GET("/admin", h.ShowDashboard)
// Object Store management routes
protected.GET("/object-store/buckets", h.ShowS3Buckets)
protected.GET("/object-store/buckets/:bucket", h.ShowBucketDetails)
protected.GET("/object-store/users", h.userHandlers.ShowObjectStoreUsers)
protected.GET("/object-store/policies", h.policyHandlers.ShowPolicies)
protected.GET("/object-store/service-accounts", h.serviceAccountHandlers.ShowServiceAccounts)
protected.GET("/object-store/s3tables/buckets", h.ShowS3TablesBuckets)
protected.GET("/object-store/s3tables/buckets/:bucket/namespaces", h.ShowS3TablesNamespaces)
protected.GET("/object-store/s3tables/buckets/:bucket/namespaces/:namespace/tables", h.ShowS3TablesTables)
protected.GET("/object-store/s3tables/buckets/:bucket/namespaces/:namespace/tables/:table", h.ShowS3TablesTableDetails)
protected.GET("/object-store/iceberg", h.ShowIcebergCatalog)
protected.GET("/object-store/iceberg/:catalog/namespaces", h.ShowIcebergNamespaces)
protected.GET("/object-store/iceberg/:catalog/namespaces/:namespace/tables", h.ShowIcebergTables)
protected.GET("/object-store/iceberg/:catalog/namespaces/:namespace/tables/:table", h.ShowIcebergTableDetails)
// File browser routes
protected.GET("/files", h.fileBrowserHandlers.ShowFileBrowser)
// Cluster management routes
protected.GET("/cluster/masters", h.clusterHandlers.ShowClusterMasters)
protected.GET("/cluster/filers", h.clusterHandlers.ShowClusterFilers)
protected.GET("/cluster/volume-servers", h.clusterHandlers.ShowClusterVolumeServers)
// Storage management routes
protected.GET("/storage/volumes", h.clusterHandlers.ShowClusterVolumes)
protected.GET("/storage/volumes/:id/:server", h.clusterHandlers.ShowVolumeDetails)
protected.GET("/storage/collections", h.clusterHandlers.ShowClusterCollections)
protected.GET("/storage/collections/:name", h.clusterHandlers.ShowCollectionDetails)
protected.GET("/storage/ec-shards", h.clusterHandlers.ShowClusterEcShards)
protected.GET("/storage/ec-volumes/:id", h.clusterHandlers.ShowEcVolumeDetails)
// Message Queue management routes
protected.GET("/mq/brokers", h.mqHandlers.ShowBrokers)
protected.GET("/mq/topics", h.mqHandlers.ShowTopics)
protected.GET("/mq/topics/:namespace/:topic", h.mqHandlers.ShowTopicDetails)
protected.GET("/plugin", h.pluginHandlers.ShowPlugin)
protected.GET("/plugin/configuration", h.pluginHandlers.ShowPluginConfiguration)
protected.GET("/plugin/queue", h.pluginHandlers.ShowPluginQueue)
protected.GET("/plugin/detection", h.pluginHandlers.ShowPluginDetection)
protected.GET("/plugin/execution", h.pluginHandlers.ShowPluginExecution)
protected.GET("/plugin/monitoring", h.pluginHandlers.ShowPluginMonitoring)
// API routes for AJAX calls
api := r.Group("/api")
api.Use(dash.RequireAuthAPI()) // Use API-specific auth middleware
{
api.GET("/cluster/topology", h.clusterHandlers.GetClusterTopology)
api.GET("/cluster/masters", h.clusterHandlers.GetMasters)
api.GET("/cluster/volumes", h.clusterHandlers.GetVolumeServers)
api.GET("/admin", h.adminServer.ShowAdmin) // JSON API for admin data
api.GET("/config", h.adminServer.GetConfigInfo) // Configuration information
// S3 API routes
s3Api := api.Group("/s3")
{
s3Api.GET("/buckets", h.adminServer.ListBucketsAPI)
s3Api.POST("/buckets", dash.RequireWriteAccess(), h.adminServer.CreateBucket)
s3Api.DELETE("/buckets/:bucket", dash.RequireWriteAccess(), h.adminServer.DeleteBucket)
s3Api.GET("/buckets/:bucket", h.adminServer.ShowBucketDetails)
s3Api.PUT("/buckets/:bucket/quota", dash.RequireWriteAccess(), h.adminServer.UpdateBucketQuota)
s3Api.PUT("/buckets/:bucket/owner", dash.RequireWriteAccess(), h.adminServer.UpdateBucketOwner)
}
// User management API routes
usersApi := api.Group("/users")
{
usersApi.GET("", h.userHandlers.GetUsers)
usersApi.POST("", dash.RequireWriteAccess(), h.userHandlers.CreateUser)
usersApi.GET("/:username", h.userHandlers.GetUserDetails)
usersApi.PUT("/:username", dash.RequireWriteAccess(), h.userHandlers.UpdateUser)
usersApi.DELETE("/:username", dash.RequireWriteAccess(), h.userHandlers.DeleteUser)
usersApi.POST("/:username/access-keys", dash.RequireWriteAccess(), h.userHandlers.CreateAccessKey)
usersApi.DELETE("/:username/access-keys/:accessKeyId", dash.RequireWriteAccess(), h.userHandlers.DeleteAccessKey)
usersApi.PUT("/:username/access-keys/:accessKeyId/status", dash.RequireWriteAccess(), h.userHandlers.UpdateAccessKeyStatus)
usersApi.GET("/:username/policies", h.userHandlers.GetUserPolicies)
usersApi.PUT("/:username/policies", dash.RequireWriteAccess(), h.userHandlers.UpdateUserPolicies)
}
// Service Account management API routes
saApi := api.Group("/service-accounts")
{
saApi.GET("", h.serviceAccountHandlers.GetServiceAccounts)
saApi.POST("", dash.RequireWriteAccess(), h.serviceAccountHandlers.CreateServiceAccount)
saApi.GET("/:id", h.serviceAccountHandlers.GetServiceAccountDetails)
saApi.PUT("/:id", dash.RequireWriteAccess(), h.serviceAccountHandlers.UpdateServiceAccount)
saApi.DELETE("/:id", dash.RequireWriteAccess(), h.serviceAccountHandlers.DeleteServiceAccount)
}
// Object Store Policy management API routes
objectStorePoliciesApi := api.Group("/object-store/policies")
{
objectStorePoliciesApi.GET("", h.policyHandlers.GetPolicies)
objectStorePoliciesApi.POST("", dash.RequireWriteAccess(), h.policyHandlers.CreatePolicy)
objectStorePoliciesApi.GET("/:name", h.policyHandlers.GetPolicy)
objectStorePoliciesApi.PUT("/:name", dash.RequireWriteAccess(), h.policyHandlers.UpdatePolicy)
objectStorePoliciesApi.DELETE("/:name", dash.RequireWriteAccess(), h.policyHandlers.DeletePolicy)
objectStorePoliciesApi.POST("/validate", h.policyHandlers.ValidatePolicy)
}
// S3 Tables API routes
s3TablesApi := api.Group("/s3tables")
{
s3TablesApi.GET("/buckets", h.adminServer.ListS3TablesBucketsAPI)
s3TablesApi.POST("/buckets", dash.RequireWriteAccess(), h.adminServer.CreateS3TablesBucket)
s3TablesApi.DELETE("/buckets", dash.RequireWriteAccess(), h.adminServer.DeleteS3TablesBucket)
s3TablesApi.GET("/namespaces", h.adminServer.ListS3TablesNamespacesAPI)
s3TablesApi.POST("/namespaces", dash.RequireWriteAccess(), h.adminServer.CreateS3TablesNamespace)
s3TablesApi.DELETE("/namespaces", dash.RequireWriteAccess(), h.adminServer.DeleteS3TablesNamespace)
s3TablesApi.GET("/tables", h.adminServer.ListS3TablesTablesAPI)
s3TablesApi.POST("/tables", dash.RequireWriteAccess(), h.adminServer.CreateS3TablesTable)
s3TablesApi.DELETE("/tables", dash.RequireWriteAccess(), h.adminServer.DeleteS3TablesTable)
s3TablesApi.PUT("/bucket-policy", dash.RequireWriteAccess(), h.adminServer.PutS3TablesBucketPolicy)
s3TablesApi.GET("/bucket-policy", h.adminServer.GetS3TablesBucketPolicy)
s3TablesApi.DELETE("/bucket-policy", dash.RequireWriteAccess(), h.adminServer.DeleteS3TablesBucketPolicy)
s3TablesApi.PUT("/table-policy", dash.RequireWriteAccess(), h.adminServer.PutS3TablesTablePolicy)
s3TablesApi.GET("/table-policy", h.adminServer.GetS3TablesTablePolicy)
s3TablesApi.DELETE("/table-policy", dash.RequireWriteAccess(), h.adminServer.DeleteS3TablesTablePolicy)
s3TablesApi.PUT("/tags", dash.RequireWriteAccess(), h.adminServer.TagS3TablesResource)
s3TablesApi.GET("/tags", h.adminServer.ListS3TablesTags)
s3TablesApi.DELETE("/tags", dash.RequireWriteAccess(), h.adminServer.UntagS3TablesResource)
}
// File management API routes
filesApi := api.Group("/files")
{
filesApi.DELETE("/delete", dash.RequireWriteAccess(), h.fileBrowserHandlers.DeleteFile)
filesApi.DELETE("/delete-multiple", dash.RequireWriteAccess(), h.fileBrowserHandlers.DeleteMultipleFiles)
filesApi.POST("/create-folder", dash.RequireWriteAccess(), h.fileBrowserHandlers.CreateFolder)
filesApi.POST("/upload", dash.RequireWriteAccess(), h.fileBrowserHandlers.UploadFile)
filesApi.GET("/download", h.fileBrowserHandlers.DownloadFile)
filesApi.GET("/view", h.fileBrowserHandlers.ViewFile)
filesApi.GET("/properties", h.fileBrowserHandlers.GetFileProperties)
}
// Volume management API routes
volumeApi := api.Group("/volumes")
{
volumeApi.POST("/:id/:server/vacuum", dash.RequireWriteAccess(), h.clusterHandlers.VacuumVolume)
}
// Plugin API routes
pluginApi := api.Group("/plugin")
{
pluginApi.GET("/status", h.adminServer.GetPluginStatusAPI)
pluginApi.GET("/workers", h.adminServer.GetPluginWorkersAPI)
pluginApi.GET("/job-types", h.adminServer.GetPluginJobTypesAPI)
pluginApi.GET("/jobs", h.adminServer.GetPluginJobsAPI)
pluginApi.GET("/jobs/:jobId", h.adminServer.GetPluginJobAPI)
pluginApi.GET("/jobs/:jobId/detail", h.adminServer.GetPluginJobDetailAPI)
pluginApi.GET("/activities", h.adminServer.GetPluginActivitiesAPI)
pluginApi.GET("/scheduler-states", h.adminServer.GetPluginSchedulerStatesAPI)
pluginApi.GET("/job-types/:jobType/descriptor", h.adminServer.GetPluginJobTypeDescriptorAPI)
pluginApi.POST("/job-types/:jobType/schema", h.adminServer.RequestPluginJobTypeSchemaAPI)
pluginApi.GET("/job-types/:jobType/config", h.adminServer.GetPluginJobTypeConfigAPI)
pluginApi.PUT("/job-types/:jobType/config", dash.RequireWriteAccess(), h.adminServer.UpdatePluginJobTypeConfigAPI)
pluginApi.GET("/job-types/:jobType/runs", h.adminServer.GetPluginRunHistoryAPI)
pluginApi.POST("/job-types/:jobType/detect", dash.RequireWriteAccess(), h.adminServer.TriggerPluginDetectionAPI)
pluginApi.POST("/job-types/:jobType/run", dash.RequireWriteAccess(), h.adminServer.RunPluginJobTypeAPI)
pluginApi.POST("/jobs/execute", dash.RequireWriteAccess(), h.adminServer.ExecutePluginJobAPI)
}
// Message Queue API routes
mqApi := api.Group("/mq")
{
mqApi.GET("/topics/:namespace/:topic", h.mqHandlers.GetTopicDetailsAPI)
mqApi.POST("/topics/create", dash.RequireWriteAccess(), h.mqHandlers.CreateTopicAPI)
mqApi.POST("/topics/retention/update", dash.RequireWriteAccess(), h.mqHandlers.UpdateTopicRetentionAPI)
mqApi.POST("/retention/purge", dash.RequireWriteAccess(), h.adminServer.TriggerTopicRetentionPurgeAPI)
}
}
} else {
// No authentication required - all routes are public
r.GET("/", h.ShowDashboard)
r.GET("/admin", h.ShowDashboard)
// Object Store management routes
r.GET("/object-store/buckets", h.ShowS3Buckets)
r.GET("/object-store/buckets/:bucket", h.ShowBucketDetails)
r.GET("/object-store/users", h.userHandlers.ShowObjectStoreUsers)
r.GET("/object-store/policies", h.policyHandlers.ShowPolicies)
r.GET("/object-store/service-accounts", h.serviceAccountHandlers.ShowServiceAccounts)
r.GET("/object-store/s3tables/buckets", h.ShowS3TablesBuckets)
r.GET("/object-store/s3tables/buckets/:bucket/namespaces", h.ShowS3TablesNamespaces)
r.GET("/object-store/s3tables/buckets/:bucket/namespaces/:namespace/tables", h.ShowS3TablesTables)
r.GET("/object-store/s3tables/buckets/:bucket/namespaces/:namespace/tables/:table", h.ShowS3TablesTableDetails)
r.GET("/object-store/iceberg", h.ShowIcebergCatalog)
r.GET("/object-store/iceberg/:catalog/namespaces", h.ShowIcebergNamespaces)
r.GET("/object-store/iceberg/:catalog/namespaces/:namespace/tables", h.ShowIcebergTables)
r.GET("/object-store/iceberg/:catalog/namespaces/:namespace/tables/:table", h.ShowIcebergTableDetails)
// File browser routes
r.GET("/files", h.fileBrowserHandlers.ShowFileBrowser)
// Cluster management routes
r.GET("/cluster/masters", h.clusterHandlers.ShowClusterMasters)
r.GET("/cluster/filers", h.clusterHandlers.ShowClusterFilers)
r.GET("/cluster/volume-servers", h.clusterHandlers.ShowClusterVolumeServers)
// Storage management routes
r.GET("/storage/volumes", h.clusterHandlers.ShowClusterVolumes)
r.GET("/storage/volumes/:id/:server", h.clusterHandlers.ShowVolumeDetails)
r.GET("/storage/collections", h.clusterHandlers.ShowClusterCollections)
r.GET("/storage/collections/:name", h.clusterHandlers.ShowCollectionDetails)
r.GET("/storage/ec-shards", h.clusterHandlers.ShowClusterEcShards)
r.GET("/storage/ec-volumes/:id", h.clusterHandlers.ShowEcVolumeDetails)
// Message Queue management routes
r.GET("/mq/brokers", h.mqHandlers.ShowBrokers)
r.GET("/mq/topics", h.mqHandlers.ShowTopics)
r.GET("/mq/topics/:namespace/:topic", h.mqHandlers.ShowTopicDetails)
r.GET("/plugin", h.pluginHandlers.ShowPlugin)
r.GET("/plugin/configuration", h.pluginHandlers.ShowPluginConfiguration)
r.GET("/plugin/queue", h.pluginHandlers.ShowPluginQueue)
r.GET("/plugin/detection", h.pluginHandlers.ShowPluginDetection)
r.GET("/plugin/execution", h.pluginHandlers.ShowPluginExecution)
r.GET("/plugin/monitoring", h.pluginHandlers.ShowPluginMonitoring)
// API routes for AJAX calls
api := r.Group("/api")
{
api.GET("/cluster/topology", h.clusterHandlers.GetClusterTopology)
api.GET("/cluster/masters", h.clusterHandlers.GetMasters)
api.GET("/cluster/volumes", h.clusterHandlers.GetVolumeServers)
api.GET("/admin", h.adminServer.ShowAdmin) // JSON API for admin data
api.GET("/config", h.adminServer.GetConfigInfo) // Configuration information
// S3 API routes
s3Api := api.Group("/s3")
{
s3Api.GET("/buckets", h.adminServer.ListBucketsAPI)
s3Api.POST("/buckets", h.adminServer.CreateBucket)
s3Api.DELETE("/buckets/:bucket", h.adminServer.DeleteBucket)
s3Api.GET("/buckets/:bucket", h.adminServer.ShowBucketDetails)
s3Api.PUT("/buckets/:bucket/quota", h.adminServer.UpdateBucketQuota)
s3Api.PUT("/buckets/:bucket/owner", h.adminServer.UpdateBucketOwner)
}
// User management API routes
usersApi := api.Group("/users")
{
usersApi.GET("", h.userHandlers.GetUsers)
usersApi.POST("", h.userHandlers.CreateUser)
usersApi.GET("/:username", h.userHandlers.GetUserDetails)
usersApi.PUT("/:username", h.userHandlers.UpdateUser)
usersApi.DELETE("/:username", h.userHandlers.DeleteUser)
usersApi.POST("/:username/access-keys", h.userHandlers.CreateAccessKey)
usersApi.DELETE("/:username/access-keys/:accessKeyId", h.userHandlers.DeleteAccessKey)
usersApi.PUT("/:username/access-keys/:accessKeyId/status", h.userHandlers.UpdateAccessKeyStatus)
usersApi.GET("/:username/policies", h.userHandlers.GetUserPolicies)
usersApi.PUT("/:username/policies", h.userHandlers.UpdateUserPolicies)
}
// Service Account management API routes
saApi := api.Group("/service-accounts")
{
saApi.GET("", h.serviceAccountHandlers.GetServiceAccounts)
saApi.POST("", h.serviceAccountHandlers.CreateServiceAccount)
saApi.GET("/:id", h.serviceAccountHandlers.GetServiceAccountDetails)
saApi.PUT("/:id", h.serviceAccountHandlers.UpdateServiceAccount)
saApi.DELETE("/:id", h.serviceAccountHandlers.DeleteServiceAccount)
}
// Object Store Policy management API routes
objectStorePoliciesApi := api.Group("/object-store/policies")
{
objectStorePoliciesApi.GET("", h.policyHandlers.GetPolicies)
objectStorePoliciesApi.POST("", h.policyHandlers.CreatePolicy)
objectStorePoliciesApi.GET("/:name", h.policyHandlers.GetPolicy)
objectStorePoliciesApi.PUT("/:name", h.policyHandlers.UpdatePolicy)
objectStorePoliciesApi.DELETE("/:name", h.policyHandlers.DeletePolicy)
objectStorePoliciesApi.POST("/validate", h.policyHandlers.ValidatePolicy)
}
// S3 Tables API routes
s3TablesApi := api.Group("/s3tables")
{
s3TablesApi.GET("/buckets", h.adminServer.ListS3TablesBucketsAPI)
s3TablesApi.POST("/buckets", h.adminServer.CreateS3TablesBucket)
s3TablesApi.DELETE("/buckets", h.adminServer.DeleteS3TablesBucket)
s3TablesApi.GET("/namespaces", h.adminServer.ListS3TablesNamespacesAPI)
s3TablesApi.POST("/namespaces", h.adminServer.CreateS3TablesNamespace)
s3TablesApi.DELETE("/namespaces", h.adminServer.DeleteS3TablesNamespace)
s3TablesApi.GET("/tables", h.adminServer.ListS3TablesTablesAPI)
s3TablesApi.POST("/tables", h.adminServer.CreateS3TablesTable)
s3TablesApi.DELETE("/tables", h.adminServer.DeleteS3TablesTable)
s3TablesApi.PUT("/bucket-policy", h.adminServer.PutS3TablesBucketPolicy)
s3TablesApi.GET("/bucket-policy", h.adminServer.GetS3TablesBucketPolicy)
s3TablesApi.DELETE("/bucket-policy", h.adminServer.DeleteS3TablesBucketPolicy)
s3TablesApi.PUT("/table-policy", h.adminServer.PutS3TablesTablePolicy)
s3TablesApi.GET("/table-policy", h.adminServer.GetS3TablesTablePolicy)
s3TablesApi.DELETE("/table-policy", h.adminServer.DeleteS3TablesTablePolicy)
s3TablesApi.PUT("/tags", h.adminServer.TagS3TablesResource)
s3TablesApi.GET("/tags", h.adminServer.ListS3TablesTags)
s3TablesApi.DELETE("/tags", h.adminServer.UntagS3TablesResource)
}
// File management API routes
filesApi := api.Group("/files")
{
filesApi.DELETE("/delete", h.fileBrowserHandlers.DeleteFile)
filesApi.DELETE("/delete-multiple", h.fileBrowserHandlers.DeleteMultipleFiles)
filesApi.POST("/create-folder", h.fileBrowserHandlers.CreateFolder)
filesApi.POST("/upload", h.fileBrowserHandlers.UploadFile)
filesApi.GET("/download", h.fileBrowserHandlers.DownloadFile)
filesApi.GET("/view", h.fileBrowserHandlers.ViewFile)
filesApi.GET("/properties", h.fileBrowserHandlers.GetFileProperties)
}
// Volume management API routes
volumeApi := api.Group("/volumes")
{
volumeApi.POST("/:id/:server/vacuum", h.clusterHandlers.VacuumVolume)
}
// Plugin API routes
pluginApi := api.Group("/plugin")
{
pluginApi.GET("/status", h.adminServer.GetPluginStatusAPI)
pluginApi.GET("/workers", h.adminServer.GetPluginWorkersAPI)
pluginApi.GET("/job-types", h.adminServer.GetPluginJobTypesAPI)
pluginApi.GET("/jobs", h.adminServer.GetPluginJobsAPI)
pluginApi.GET("/jobs/:jobId", h.adminServer.GetPluginJobAPI)
pluginApi.GET("/jobs/:jobId/detail", h.adminServer.GetPluginJobDetailAPI)
pluginApi.GET("/activities", h.adminServer.GetPluginActivitiesAPI)
pluginApi.GET("/scheduler-states", h.adminServer.GetPluginSchedulerStatesAPI)
pluginApi.GET("/job-types/:jobType/descriptor", h.adminServer.GetPluginJobTypeDescriptorAPI)
pluginApi.POST("/job-types/:jobType/schema", h.adminServer.RequestPluginJobTypeSchemaAPI)
pluginApi.GET("/job-types/:jobType/config", h.adminServer.GetPluginJobTypeConfigAPI)
pluginApi.PUT("/job-types/:jobType/config", h.adminServer.UpdatePluginJobTypeConfigAPI)
pluginApi.GET("/job-types/:jobType/runs", h.adminServer.GetPluginRunHistoryAPI)
pluginApi.POST("/job-types/:jobType/detect", h.adminServer.TriggerPluginDetectionAPI)
pluginApi.POST("/job-types/:jobType/run", h.adminServer.RunPluginJobTypeAPI)
pluginApi.POST("/jobs/execute", h.adminServer.ExecutePluginJobAPI)
}
// Message Queue API routes
mqApi := api.Group("/mq")
{
mqApi.GET("/topics/:namespace/:topic", h.mqHandlers.GetTopicDetailsAPI)
mqApi.POST("/topics/create", h.mqHandlers.CreateTopicAPI)
mqApi.POST("/topics/retention/update", h.mqHandlers.UpdateTopicRetentionAPI)
mqApi.POST("/retention/purge", h.adminServer.TriggerTopicRetentionPurgeAPI)
}
}
api := r.PathPrefix("/api").Subrouter()
api.Use(dash.RequireAuthAPI(h.sessionStore))
h.registerAPIRoutes(api, true)
return
}
// No authentication required - all routes are public
h.registerUIRoutes(r)
api := r.PathPrefix("/api").Subrouter()
h.registerAPIRoutes(api, false)
}
func (h *AdminHandlers) registerUIRoutes(r *mux.Router) {
// Main admin interface routes
r.HandleFunc("/", h.ShowDashboard).Methods(http.MethodGet)
r.HandleFunc("/admin", h.ShowDashboard).Methods(http.MethodGet)
// Object Store management routes
r.HandleFunc("/object-store/buckets", h.ShowS3Buckets).Methods(http.MethodGet)
r.HandleFunc("/object-store/buckets/{bucket}", h.ShowBucketDetails).Methods(http.MethodGet)
r.HandleFunc("/object-store/users", h.userHandlers.ShowObjectStoreUsers).Methods(http.MethodGet)
r.HandleFunc("/object-store/policies", h.policyHandlers.ShowPolicies).Methods(http.MethodGet)
r.HandleFunc("/object-store/service-accounts", h.serviceAccountHandlers.ShowServiceAccounts).Methods(http.MethodGet)
r.HandleFunc("/object-store/s3tables/buckets", h.ShowS3TablesBuckets).Methods(http.MethodGet)
r.HandleFunc("/object-store/s3tables/buckets/{bucket}/namespaces", h.ShowS3TablesNamespaces).Methods(http.MethodGet)
r.HandleFunc("/object-store/s3tables/buckets/{bucket}/namespaces/{namespace}/tables", h.ShowS3TablesTables).Methods(http.MethodGet)
r.HandleFunc("/object-store/s3tables/buckets/{bucket}/namespaces/{namespace}/tables/{table}", h.ShowS3TablesTableDetails).Methods(http.MethodGet)
r.HandleFunc("/object-store/iceberg", h.ShowIcebergCatalog).Methods(http.MethodGet)
r.HandleFunc("/object-store/iceberg/{catalog}/namespaces", h.ShowIcebergNamespaces).Methods(http.MethodGet)
r.HandleFunc("/object-store/iceberg/{catalog}/namespaces/{namespace}/tables", h.ShowIcebergTables).Methods(http.MethodGet)
r.HandleFunc("/object-store/iceberg/{catalog}/namespaces/{namespace}/tables/{table}", h.ShowIcebergTableDetails).Methods(http.MethodGet)
// File browser routes
r.HandleFunc("/files", h.fileBrowserHandlers.ShowFileBrowser).Methods(http.MethodGet)
// Cluster management routes
r.HandleFunc("/cluster/masters", h.clusterHandlers.ShowClusterMasters).Methods(http.MethodGet)
r.HandleFunc("/cluster/filers", h.clusterHandlers.ShowClusterFilers).Methods(http.MethodGet)
r.HandleFunc("/cluster/volume-servers", h.clusterHandlers.ShowClusterVolumeServers).Methods(http.MethodGet)
// Storage management routes
r.HandleFunc("/storage/volumes", h.clusterHandlers.ShowClusterVolumes).Methods(http.MethodGet)
r.HandleFunc("/storage/volumes/{id}/{server}", h.clusterHandlers.ShowVolumeDetails).Methods(http.MethodGet)
r.HandleFunc("/storage/collections", h.clusterHandlers.ShowClusterCollections).Methods(http.MethodGet)
r.HandleFunc("/storage/collections/{name}", h.clusterHandlers.ShowCollectionDetails).Methods(http.MethodGet)
r.HandleFunc("/storage/ec-shards", h.clusterHandlers.ShowClusterEcShards).Methods(http.MethodGet)
r.HandleFunc("/storage/ec-volumes/{id}", h.clusterHandlers.ShowEcVolumeDetails).Methods(http.MethodGet)
// Message Queue management routes
r.HandleFunc("/mq/brokers", h.mqHandlers.ShowBrokers).Methods(http.MethodGet)
r.HandleFunc("/mq/topics", h.mqHandlers.ShowTopics).Methods(http.MethodGet)
r.HandleFunc("/mq/topics/{namespace}/{topic}", h.mqHandlers.ShowTopicDetails).Methods(http.MethodGet)
// Plugin pages
r.HandleFunc("/plugin", h.pluginHandlers.ShowPlugin).Methods(http.MethodGet)
r.HandleFunc("/plugin/configuration", h.pluginHandlers.ShowPluginConfiguration).Methods(http.MethodGet)
r.HandleFunc("/plugin/queue", h.pluginHandlers.ShowPluginQueue).Methods(http.MethodGet)
r.HandleFunc("/plugin/detection", h.pluginHandlers.ShowPluginDetection).Methods(http.MethodGet)
r.HandleFunc("/plugin/execution", h.pluginHandlers.ShowPluginExecution).Methods(http.MethodGet)
r.HandleFunc("/plugin/monitoring", h.pluginHandlers.ShowPluginMonitoring).Methods(http.MethodGet)
}
func (h *AdminHandlers) registerAPIRoutes(api *mux.Router, enforceWrite bool) {
wrapWrite := func(handler http.HandlerFunc) http.Handler {
if !enforceWrite {
return handler
}
return dash.RequireWriteAccess()(handler)
}
api.HandleFunc("/cluster/topology", h.clusterHandlers.GetClusterTopology).Methods(http.MethodGet)
api.HandleFunc("/cluster/masters", h.clusterHandlers.GetMasters).Methods(http.MethodGet)
api.HandleFunc("/cluster/volumes", h.clusterHandlers.GetVolumeServers).Methods(http.MethodGet)
api.HandleFunc("/admin", h.adminServer.ShowAdmin).Methods(http.MethodGet)
api.HandleFunc("/config", h.adminServer.GetConfigInfo).Methods(http.MethodGet)
s3Api := api.PathPrefix("/s3").Subrouter()
s3Api.HandleFunc("/buckets", h.adminServer.ListBucketsAPI).Methods(http.MethodGet)
s3Api.Handle("/buckets", wrapWrite(h.adminServer.CreateBucket)).Methods(http.MethodPost)
s3Api.Handle("/buckets/{bucket}", wrapWrite(h.adminServer.DeleteBucket)).Methods(http.MethodDelete)
s3Api.HandleFunc("/buckets/{bucket}", h.adminServer.ShowBucketDetails).Methods(http.MethodGet)
s3Api.Handle("/buckets/{bucket}/quota", wrapWrite(h.adminServer.UpdateBucketQuota)).Methods(http.MethodPut)
s3Api.Handle("/buckets/{bucket}/owner", wrapWrite(h.adminServer.UpdateBucketOwner)).Methods(http.MethodPut)
usersApi := api.PathPrefix("/users").Subrouter()
usersApi.HandleFunc("", h.userHandlers.GetUsers).Methods(http.MethodGet)
usersApi.Handle("", wrapWrite(h.userHandlers.CreateUser)).Methods(http.MethodPost)
usersApi.HandleFunc("/{username}", h.userHandlers.GetUserDetails).Methods(http.MethodGet)
usersApi.Handle("/{username}", wrapWrite(h.userHandlers.UpdateUser)).Methods(http.MethodPut)
usersApi.Handle("/{username}", wrapWrite(h.userHandlers.DeleteUser)).Methods(http.MethodDelete)
usersApi.Handle("/{username}/access-keys", wrapWrite(h.userHandlers.CreateAccessKey)).Methods(http.MethodPost)
usersApi.Handle("/{username}/access-keys/{accessKeyId}", wrapWrite(h.userHandlers.DeleteAccessKey)).Methods(http.MethodDelete)
usersApi.Handle("/{username}/access-keys/{accessKeyId}/status", wrapWrite(h.userHandlers.UpdateAccessKeyStatus)).Methods(http.MethodPut)
usersApi.HandleFunc("/{username}/policies", h.userHandlers.GetUserPolicies).Methods(http.MethodGet)
usersApi.Handle("/{username}/policies", wrapWrite(h.userHandlers.UpdateUserPolicies)).Methods(http.MethodPut)
saApi := api.PathPrefix("/service-accounts").Subrouter()
saApi.HandleFunc("", h.serviceAccountHandlers.GetServiceAccounts).Methods(http.MethodGet)
saApi.Handle("", wrapWrite(h.serviceAccountHandlers.CreateServiceAccount)).Methods(http.MethodPost)
saApi.HandleFunc("/{id}", h.serviceAccountHandlers.GetServiceAccountDetails).Methods(http.MethodGet)
saApi.Handle("/{id}", wrapWrite(h.serviceAccountHandlers.UpdateServiceAccount)).Methods(http.MethodPut)
saApi.Handle("/{id}", wrapWrite(h.serviceAccountHandlers.DeleteServiceAccount)).Methods(http.MethodDelete)
policyApi := api.PathPrefix("/object-store/policies").Subrouter()
policyApi.HandleFunc("", h.policyHandlers.GetPolicies).Methods(http.MethodGet)
policyApi.Handle("", wrapWrite(h.policyHandlers.CreatePolicy)).Methods(http.MethodPost)
policyApi.HandleFunc("/{name}", h.policyHandlers.GetPolicy).Methods(http.MethodGet)
policyApi.Handle("/{name}", wrapWrite(h.policyHandlers.UpdatePolicy)).Methods(http.MethodPut)
policyApi.Handle("/{name}", wrapWrite(h.policyHandlers.DeletePolicy)).Methods(http.MethodDelete)
policyApi.HandleFunc("/validate", h.policyHandlers.ValidatePolicy).Methods(http.MethodPost)
s3TablesApi := api.PathPrefix("/s3tables").Subrouter()
s3TablesApi.HandleFunc("/buckets", h.adminServer.ListS3TablesBucketsAPI).Methods(http.MethodGet)
s3TablesApi.Handle("/buckets", wrapWrite(h.adminServer.CreateS3TablesBucket)).Methods(http.MethodPost)
s3TablesApi.Handle("/buckets", wrapWrite(h.adminServer.DeleteS3TablesBucket)).Methods(http.MethodDelete)
s3TablesApi.HandleFunc("/namespaces", h.adminServer.ListS3TablesNamespacesAPI).Methods(http.MethodGet)
s3TablesApi.Handle("/namespaces", wrapWrite(h.adminServer.CreateS3TablesNamespace)).Methods(http.MethodPost)
s3TablesApi.Handle("/namespaces", wrapWrite(h.adminServer.DeleteS3TablesNamespace)).Methods(http.MethodDelete)
s3TablesApi.HandleFunc("/tables", h.adminServer.ListS3TablesTablesAPI).Methods(http.MethodGet)
s3TablesApi.Handle("/tables", wrapWrite(h.adminServer.CreateS3TablesTable)).Methods(http.MethodPost)
s3TablesApi.Handle("/tables", wrapWrite(h.adminServer.DeleteS3TablesTable)).Methods(http.MethodDelete)
s3TablesApi.Handle("/bucket-policy", wrapWrite(h.adminServer.PutS3TablesBucketPolicy)).Methods(http.MethodPut)
s3TablesApi.HandleFunc("/bucket-policy", h.adminServer.GetS3TablesBucketPolicy).Methods(http.MethodGet)
s3TablesApi.Handle("/bucket-policy", wrapWrite(h.adminServer.DeleteS3TablesBucketPolicy)).Methods(http.MethodDelete)
s3TablesApi.Handle("/table-policy", wrapWrite(h.adminServer.PutS3TablesTablePolicy)).Methods(http.MethodPut)
s3TablesApi.HandleFunc("/table-policy", h.adminServer.GetS3TablesTablePolicy).Methods(http.MethodGet)
s3TablesApi.Handle("/table-policy", wrapWrite(h.adminServer.DeleteS3TablesTablePolicy)).Methods(http.MethodDelete)
s3TablesApi.Handle("/tags", wrapWrite(h.adminServer.TagS3TablesResource)).Methods(http.MethodPut)
s3TablesApi.HandleFunc("/tags", h.adminServer.ListS3TablesTags).Methods(http.MethodGet)
s3TablesApi.Handle("/tags", wrapWrite(h.adminServer.UntagS3TablesResource)).Methods(http.MethodDelete)
filesApi := api.PathPrefix("/files").Subrouter()
filesApi.Handle("/delete", wrapWrite(h.fileBrowserHandlers.DeleteFile)).Methods(http.MethodDelete)
filesApi.Handle("/delete-multiple", wrapWrite(h.fileBrowserHandlers.DeleteMultipleFiles)).Methods(http.MethodDelete)
filesApi.Handle("/create-folder", wrapWrite(h.fileBrowserHandlers.CreateFolder)).Methods(http.MethodPost)
filesApi.Handle("/upload", wrapWrite(h.fileBrowserHandlers.UploadFile)).Methods(http.MethodPost)
filesApi.HandleFunc("/download", h.fileBrowserHandlers.DownloadFile).Methods(http.MethodGet)
filesApi.HandleFunc("/view", h.fileBrowserHandlers.ViewFile).Methods(http.MethodGet)
filesApi.HandleFunc("/properties", h.fileBrowserHandlers.GetFileProperties).Methods(http.MethodGet)
volumeApi := api.PathPrefix("/volumes").Subrouter()
volumeApi.Handle("/{id}/{server}/vacuum", wrapWrite(h.clusterHandlers.VacuumVolume)).Methods(http.MethodPost)
pluginApi := api.PathPrefix("/plugin").Subrouter()
pluginApi.HandleFunc("/status", h.adminServer.GetPluginStatusAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/workers", h.adminServer.GetPluginWorkersAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/job-types", h.adminServer.GetPluginJobTypesAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/jobs", h.adminServer.GetPluginJobsAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/jobs/{jobId}", h.adminServer.GetPluginJobAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/jobs/{jobId}/detail", h.adminServer.GetPluginJobDetailAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/activities", h.adminServer.GetPluginActivitiesAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/scheduler-states", h.adminServer.GetPluginSchedulerStatesAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/job-types/{jobType}/descriptor", h.adminServer.GetPluginJobTypeDescriptorAPI).Methods(http.MethodGet)
pluginApi.HandleFunc("/job-types/{jobType}/schema", h.adminServer.RequestPluginJobTypeSchemaAPI).Methods(http.MethodPost)
pluginApi.HandleFunc("/job-types/{jobType}/config", h.adminServer.GetPluginJobTypeConfigAPI).Methods(http.MethodGet)
pluginApi.Handle("/job-types/{jobType}/config", wrapWrite(h.adminServer.UpdatePluginJobTypeConfigAPI)).Methods(http.MethodPut)
pluginApi.HandleFunc("/job-types/{jobType}/runs", h.adminServer.GetPluginRunHistoryAPI).Methods(http.MethodGet)
pluginApi.Handle("/job-types/{jobType}/detect", wrapWrite(h.adminServer.TriggerPluginDetectionAPI)).Methods(http.MethodPost)
pluginApi.Handle("/job-types/{jobType}/run", wrapWrite(h.adminServer.RunPluginJobTypeAPI)).Methods(http.MethodPost)
pluginApi.Handle("/jobs/execute", wrapWrite(h.adminServer.ExecutePluginJobAPI)).Methods(http.MethodPost)
mqApi := api.PathPrefix("/mq").Subrouter()
mqApi.HandleFunc("/topics/{namespace}/{topic}", h.mqHandlers.GetTopicDetailsAPI).Methods(http.MethodGet)
mqApi.Handle("/topics/create", wrapWrite(h.mqHandlers.CreateTopicAPI)).Methods(http.MethodPost)
mqApi.Handle("/topics/retention/update", wrapWrite(h.mqHandlers.UpdateTopicRetentionAPI)).Methods(http.MethodPost)
mqApi.Handle("/retention/purge", wrapWrite(h.adminServer.TriggerTopicRetentionPurgeAPI)).Methods(http.MethodPost)
}
// HealthCheck returns the health status of the admin interface
func (h *AdminHandlers) HealthCheck(c *gin.Context) {
c.JSON(200, gin.H{"health": "ok"})
func (h *AdminHandlers) HealthCheck(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"health": "ok"})
}
// ShowDashboard renders the main admin dashboard
func (h *AdminHandlers) ShowDashboard(c *gin.Context) {
func (h *AdminHandlers) ShowDashboard(w http.ResponseWriter, r *http.Request) {
// Get admin data from the server
adminData := h.getAdminData(c)
adminData := h.getAdminData(r)
username := h.getUsername(r)
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
adminComponent := app.Admin(adminData)
layoutComponent := layout.Layout(c, adminComponent)
err := layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, adminComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowS3Buckets renders the Object Store buckets management page
func (h *AdminHandlers) ShowS3Buckets(c *gin.Context) {
func (h *AdminHandlers) ShowS3Buckets(w http.ResponseWriter, r *http.Request) {
// Get Object Store buckets data from the server
s3Data := h.getS3BucketsData(c)
s3Data := h.getS3BucketsData(r)
username := h.getUsername(r)
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
s3Component := app.S3Buckets(s3Data)
layoutComponent := layout.Layout(c, s3Component)
err := layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, s3Component)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowS3TablesBuckets renders the S3 Tables buckets page
func (h *AdminHandlers) ShowS3TablesBuckets(c *gin.Context) {
username := h.getUsername(c)
func (h *AdminHandlers) ShowS3TablesBuckets(w http.ResponseWriter, r *http.Request) {
username := h.getUsername(r)
data, err := h.adminServer.GetS3TablesBucketsData(c.Request.Context())
data, err := h.adminServer.GetS3TablesBucketsData(r.Context())
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get S3 Tables buckets: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get S3 Tables buckets: "+err.Error())
return
}
data.Username = username
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
component := app.S3TablesBuckets(data)
layoutComponent := layout.Layout(c, component)
if err := layoutComponent.Render(c.Request.Context(), c.Writer); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
}
}
// ShowS3TablesNamespaces renders namespaces for a table bucket
func (h *AdminHandlers) ShowS3TablesNamespaces(c *gin.Context) {
username := h.getUsername(c)
func (h *AdminHandlers) ShowS3TablesNamespaces(w http.ResponseWriter, r *http.Request) {
username := h.getUsername(r)
bucketName := c.Param("bucket")
bucketName := mux.Vars(r)["bucket"]
arn, err := buildS3TablesBucketArn(bucketName)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
data, err := h.adminServer.GetS3TablesNamespacesData(c.Request.Context(), arn)
data, err := h.adminServer.GetS3TablesNamespacesData(r.Context(), arn)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get S3 Tables namespaces: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get S3 Tables namespaces: "+err.Error())
return
}
data.Username = username
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
component := app.S3TablesNamespaces(data)
layoutComponent := layout.Layout(c, component)
if err := layoutComponent.Render(c.Request.Context(), c.Writer); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
}
}
// ShowS3TablesTables renders tables for a namespace
func (h *AdminHandlers) ShowS3TablesTables(c *gin.Context) {
username := h.getUsername(c)
func (h *AdminHandlers) ShowS3TablesTables(w http.ResponseWriter, r *http.Request) {
username := h.getUsername(r)
bucketName := c.Param("bucket")
namespace := c.Param("namespace")
bucketName := mux.Vars(r)["bucket"]
namespace := mux.Vars(r)["namespace"]
arn, err := buildS3TablesBucketArn(bucketName)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
data, err := h.adminServer.GetS3TablesTablesData(c.Request.Context(), arn, namespace)
data, err := h.adminServer.GetS3TablesTablesData(r.Context(), arn, namespace)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get S3 Tables tables: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get S3 Tables tables: "+err.Error())
return
}
data.Username = username
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
component := app.S3TablesTables(data)
layoutComponent := layout.Layout(c, component)
if err := layoutComponent.Render(c.Request.Context(), c.Writer); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
}
}
// ShowS3TablesTableDetails renders Iceberg table metadata and snapshot details on the merged S3 Tables path.
func (h *AdminHandlers) ShowS3TablesTableDetails(c *gin.Context) {
bucketName := c.Param("bucket")
namespace := c.Param("namespace")
tableName := c.Param("table")
func (h *AdminHandlers) ShowS3TablesTableDetails(w http.ResponseWriter, r *http.Request) {
bucketName := mux.Vars(r)["bucket"]
namespace := mux.Vars(r)["namespace"]
tableName := mux.Vars(r)["table"]
arn, err := buildS3TablesBucketArn(bucketName)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
data, err := h.adminServer.GetIcebergTableDetailsData(c.Request.Context(), bucketName, arn, namespace, tableName)
username := h.getUsername(r)
data, err := h.adminServer.GetIcebergTableDetailsData(r.Context(), bucketName, arn, namespace, tableName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get table details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get table details: "+err.Error())
return
}
data.Username = h.getUsername(c)
data.Username = username
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
component := app.IcebergTableDetails(data)
layoutComponent := layout.Layout(c, component)
if err := layoutComponent.Render(c.Request.Context(), c.Writer); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
}
}
@@ -572,8 +397,8 @@ func buildS3TablesBucketArn(bucketName string) (string, error) {
}
// getUsername returns the username from context, defaulting to "admin" if not set
func (h *AdminHandlers) getUsername(c *gin.Context) string {
username := c.GetString("username")
func (h *AdminHandlers) getUsername(r *http.Request) string {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
@@ -581,45 +406,45 @@ func (h *AdminHandlers) getUsername(c *gin.Context) string {
}
// ShowIcebergCatalog redirects legacy Iceberg catalog URL to the merged S3 Tables buckets page.
func (h *AdminHandlers) ShowIcebergCatalog(c *gin.Context) {
c.Redirect(http.StatusMovedPermanently, "/object-store/s3tables/buckets")
func (h *AdminHandlers) ShowIcebergCatalog(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/object-store/s3tables/buckets", http.StatusMovedPermanently)
}
// ShowIcebergNamespaces redirects legacy Iceberg namespaces URL to the merged S3 Tables namespaces page.
func (h *AdminHandlers) ShowIcebergNamespaces(c *gin.Context) {
catalogName := c.Param("catalog")
c.Redirect(http.StatusMovedPermanently, "/object-store/s3tables/buckets/"+url.PathEscape(catalogName)+"/namespaces")
func (h *AdminHandlers) ShowIcebergNamespaces(w http.ResponseWriter, r *http.Request) {
catalogName := mux.Vars(r)["catalog"]
http.Redirect(w, r, "/object-store/s3tables/buckets/"+url.PathEscape(catalogName)+"/namespaces", http.StatusMovedPermanently)
}
// ShowIcebergTables redirects legacy Iceberg tables URL to the merged S3 Tables tables page.
func (h *AdminHandlers) ShowIcebergTables(c *gin.Context) {
catalogName := c.Param("catalog")
namespace := c.Param("namespace")
c.Redirect(http.StatusMovedPermanently, "/object-store/s3tables/buckets/"+url.PathEscape(catalogName)+"/namespaces/"+url.PathEscape(namespace)+"/tables")
func (h *AdminHandlers) ShowIcebergTables(w http.ResponseWriter, r *http.Request) {
catalogName := mux.Vars(r)["catalog"]
namespace := mux.Vars(r)["namespace"]
http.Redirect(w, r, "/object-store/s3tables/buckets/"+url.PathEscape(catalogName)+"/namespaces/"+url.PathEscape(namespace)+"/tables", http.StatusMovedPermanently)
}
// ShowIcebergTableDetails redirects legacy Iceberg table details URL to the merged S3 Tables details page.
func (h *AdminHandlers) ShowIcebergTableDetails(c *gin.Context) {
catalogName := c.Param("catalog")
namespace := c.Param("namespace")
tableName := c.Param("table")
c.Redirect(http.StatusMovedPermanently, "/object-store/s3tables/buckets/"+url.PathEscape(catalogName)+"/namespaces/"+url.PathEscape(namespace)+"/tables/"+url.PathEscape(tableName))
func (h *AdminHandlers) ShowIcebergTableDetails(w http.ResponseWriter, r *http.Request) {
catalogName := mux.Vars(r)["catalog"]
namespace := mux.Vars(r)["namespace"]
tableName := mux.Vars(r)["table"]
http.Redirect(w, r, "/object-store/s3tables/buckets/"+url.PathEscape(catalogName)+"/namespaces/"+url.PathEscape(namespace)+"/tables/"+url.PathEscape(tableName), http.StatusMovedPermanently)
}
// ShowBucketDetails returns detailed information about a specific bucket
func (h *AdminHandlers) ShowBucketDetails(c *gin.Context) {
bucketName := c.Param("bucket")
func (h *AdminHandlers) ShowBucketDetails(w http.ResponseWriter, r *http.Request) {
bucketName := mux.Vars(r)["bucket"]
details, err := h.adminServer.GetBucketDetails(bucketName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get bucket details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get bucket details: "+err.Error())
return
}
c.JSON(http.StatusOK, details)
writeJSON(w, http.StatusOK, details)
}
// getS3BucketsData retrieves Object Store buckets data from the server
func (h *AdminHandlers) getS3BucketsData(c *gin.Context) dash.S3BucketsData {
username := c.GetString("username")
func (h *AdminHandlers) getS3BucketsData(r *http.Request) dash.S3BucketsData {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
@@ -642,8 +467,8 @@ func (h *AdminHandlers) getS3BucketsData(c *gin.Context) dash.S3BucketsData {
}
// getAdminData retrieves admin data from the server (now uses consolidated method)
func (h *AdminHandlers) getAdminData(c *gin.Context) dash.AdminData {
username := c.GetString("username")
func (h *AdminHandlers) getAdminData(r *http.Request) dash.AdminData {
username := dash.UsernameFromContext(r.Context())
// Use the consolidated GetAdminData method from AdminServer
adminData, err := h.adminServer.GetAdminData(username)
@@ -1,73 +1,74 @@
package handlers
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/gorilla/sessions"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
)
func TestSetupRoutes_RegistersPluginSchedulerStatesAPI_NoAuth(t *testing.T) {
gin.SetMode(gin.TestMode)
router := gin.New()
router := mux.NewRouter()
newRouteTestAdminHandlers().SetupRoutes(router, false, "", "", "", "", true)
if !hasRoute(router, "GET", "/api/plugin/scheduler-states") {
if !hasRoute(router, http.MethodGet, "/api/plugin/scheduler-states") {
t.Fatalf("expected GET /api/plugin/scheduler-states to be registered in no-auth mode")
}
if !hasRoute(router, "GET", "/api/plugin/jobs/:jobId/detail") {
if !hasRoute(router, http.MethodGet, "/api/plugin/jobs/example/detail") {
t.Fatalf("expected GET /api/plugin/jobs/:jobId/detail to be registered in no-auth mode")
}
}
func TestSetupRoutes_RegistersPluginSchedulerStatesAPI_WithAuth(t *testing.T) {
gin.SetMode(gin.TestMode)
router := gin.New()
router := mux.NewRouter()
newRouteTestAdminHandlers().SetupRoutes(router, true, "admin", "password", "", "", true)
if !hasRoute(router, "GET", "/api/plugin/scheduler-states") {
if !hasRoute(router, http.MethodGet, "/api/plugin/scheduler-states") {
t.Fatalf("expected GET /api/plugin/scheduler-states to be registered in auth mode")
}
if !hasRoute(router, "GET", "/api/plugin/jobs/:jobId/detail") {
if !hasRoute(router, http.MethodGet, "/api/plugin/jobs/example/detail") {
t.Fatalf("expected GET /api/plugin/jobs/:jobId/detail to be registered in auth mode")
}
}
func TestSetupRoutes_RegistersPluginPages_NoAuth(t *testing.T) {
gin.SetMode(gin.TestMode)
router := gin.New()
router := mux.NewRouter()
newRouteTestAdminHandlers().SetupRoutes(router, false, "", "", "", "", true)
assertHasRoute(t, router, "GET", "/plugin")
assertHasRoute(t, router, "GET", "/plugin/configuration")
assertHasRoute(t, router, "GET", "/plugin/queue")
assertHasRoute(t, router, "GET", "/plugin/detection")
assertHasRoute(t, router, "GET", "/plugin/execution")
assertHasRoute(t, router, "GET", "/plugin/monitoring")
assertHasRoute(t, router, http.MethodGet, "/plugin")
assertHasRoute(t, router, http.MethodGet, "/plugin/configuration")
assertHasRoute(t, router, http.MethodGet, "/plugin/queue")
assertHasRoute(t, router, http.MethodGet, "/plugin/detection")
assertHasRoute(t, router, http.MethodGet, "/plugin/execution")
assertHasRoute(t, router, http.MethodGet, "/plugin/monitoring")
}
func TestSetupRoutes_RegistersPluginPages_WithAuth(t *testing.T) {
gin.SetMode(gin.TestMode)
router := gin.New()
router := mux.NewRouter()
newRouteTestAdminHandlers().SetupRoutes(router, true, "admin", "password", "", "", true)
assertHasRoute(t, router, "GET", "/plugin")
assertHasRoute(t, router, "GET", "/plugin/configuration")
assertHasRoute(t, router, "GET", "/plugin/queue")
assertHasRoute(t, router, "GET", "/plugin/detection")
assertHasRoute(t, router, "GET", "/plugin/execution")
assertHasRoute(t, router, "GET", "/plugin/monitoring")
assertHasRoute(t, router, http.MethodGet, "/plugin")
assertHasRoute(t, router, http.MethodGet, "/plugin/configuration")
assertHasRoute(t, router, http.MethodGet, "/plugin/queue")
assertHasRoute(t, router, http.MethodGet, "/plugin/detection")
assertHasRoute(t, router, http.MethodGet, "/plugin/execution")
assertHasRoute(t, router, http.MethodGet, "/plugin/monitoring")
}
func newRouteTestAdminHandlers() *AdminHandlers {
adminServer := &dash.AdminServer{}
store := sessions.NewCookieStore([]byte("test-session-key"))
return &AdminHandlers{
adminServer: adminServer,
authHandlers: &AuthHandlers{adminServer: adminServer},
sessionStore: store,
authHandlers: &AuthHandlers{adminServer: adminServer, sessionStore: store},
clusterHandlers: &ClusterHandlers{adminServer: adminServer},
fileBrowserHandlers: &FileBrowserHandlers{adminServer: adminServer},
userHandlers: &UserHandlers{adminServer: adminServer},
@@ -78,16 +79,13 @@ func newRouteTestAdminHandlers() *AdminHandlers {
}
}
func hasRoute(router *gin.Engine, method string, path string) bool {
for _, route := range router.Routes() {
if route.Method == method && route.Path == path {
return true
}
}
return false
func hasRoute(router *mux.Router, method string, path string) bool {
req := httptest.NewRequest(method, path, nil)
var match mux.RouteMatch
return router.Match(req, &match)
}
func assertHasRoute(t *testing.T, router *gin.Engine, method string, path string) {
func assertHasRoute(t *testing.T, router *mux.Router, method string, path string) {
t.Helper()
if !hasRoute(router, method, path) {
t.Fatalf("expected %s %s to be registered", method, path)
+35 -22
View File
@@ -3,52 +3,65 @@ package handlers
import (
"net/http"
"github.com/gin-contrib/sessions"
"github.com/gin-gonic/gin"
"github.com/gorilla/sessions"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
// AuthHandlers contains authentication-related HTTP handlers
type AuthHandlers struct {
adminServer *dash.AdminServer
adminServer *dash.AdminServer
sessionStore sessions.Store
}
// NewAuthHandlers creates a new instance of AuthHandlers
func NewAuthHandlers(adminServer *dash.AdminServer) *AuthHandlers {
func NewAuthHandlers(adminServer *dash.AdminServer, store sessions.Store) *AuthHandlers {
return &AuthHandlers{
adminServer: adminServer,
adminServer: adminServer,
sessionStore: store,
}
}
// ShowLogin displays the login page
func (a *AuthHandlers) ShowLogin(c *gin.Context) {
session := sessions.Default(c)
// If already authenticated, redirect to admin
if session.Get("authenticated") == true {
c.Redirect(http.StatusSeeOther, "/admin")
return
func (a *AuthHandlers) ShowLogin(w http.ResponseWriter, r *http.Request) {
session, err := a.sessionStore.Get(r, dash.SessionName())
var csrfToken string
if err == nil {
if authenticated, _ := session.Values["authenticated"].(bool); authenticated {
http.Redirect(w, r, "/admin", http.StatusSeeOther)
return
}
} else {
glog.V(1).Infof("Failed to load session for login page: %v", err)
}
errorMessage := c.Query("error")
if session != nil {
token, tokenErr := dash.EnsureSessionCSRFToken(session, r, w)
if tokenErr != nil {
glog.V(1).Infof("Failed to ensure CSRF token for login page: %v", tokenErr)
} else {
csrfToken = token
}
}
errorMessage := r.URL.Query().Get("error")
// Render login template
c.Header("Content-Type", "text/html")
loginComponent := layout.LoginForm(c, "SeaweedFS Admin", errorMessage)
err := loginComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render login template: " + err.Error()})
w.Header().Set("Content-Type", "text/html")
loginComponent := layout.LoginForm("SeaweedFS Admin", errorMessage, csrfToken)
if err := loginComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render login template: "+err.Error())
return
}
}
// HandleLogin handles login form submission
func (a *AuthHandlers) HandleLogin(adminUser, adminPassword, readOnlyUser, readOnlyPassword string) gin.HandlerFunc {
return a.adminServer.HandleLogin(adminUser, adminPassword, readOnlyUser, readOnlyPassword)
func (a *AuthHandlers) HandleLogin(adminUser, adminPassword, readOnlyUser, readOnlyPassword string) http.HandlerFunc {
return a.adminServer.HandleLogin(a.sessionStore, adminUser, adminPassword, readOnlyUser, readOnlyPassword)
}
// HandleLogout handles user logout
func (a *AuthHandlers) HandleLogout(c *gin.Context) {
a.adminServer.HandleLogout(c)
func (a *AuthHandlers) HandleLogout(w http.ResponseWriter, r *http.Request) {
a.adminServer.HandleLogout(a.sessionStore, w, r)
}
+146 -161
View File
@@ -5,7 +5,7 @@ import (
"net/http"
"strconv"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -24,402 +24,387 @@ func NewClusterHandlers(adminServer *dash.AdminServer) *ClusterHandlers {
}
// ShowClusterVolumeServers renders the cluster volume servers page
func (h *ClusterHandlers) ShowClusterVolumeServers(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterVolumeServers(w http.ResponseWriter, r *http.Request) {
// Get cluster volume servers data
volumeServersData, err := h.adminServer.GetClusterVolumeServers()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster volume servers: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster volume servers: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
volumeServersData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
volumeServersComponent := app.ClusterVolumeServers(*volumeServersData)
layoutComponent := layout.Layout(c, volumeServersComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, volumeServersComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowClusterVolumes renders the cluster volumes page
func (h *ClusterHandlers) ShowClusterVolumes(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterVolumes(w http.ResponseWriter, r *http.Request) {
// Get pagination and sorting parameters from query string
page := 1
if p := c.Query("page"); p != "" {
if p := r.URL.Query().Get("page"); p != "" {
if parsed, err := strconv.Atoi(p); err == nil && parsed > 0 {
page = parsed
}
}
pageSize := 100
if ps := c.Query("pageSize"); ps != "" {
if ps := r.URL.Query().Get("pageSize"); ps != "" {
if parsed, err := strconv.Atoi(ps); err == nil && parsed > 0 && parsed <= 1000 {
pageSize = parsed
}
}
sortBy := c.DefaultQuery("sortBy", "id")
sortOrder := c.DefaultQuery("sortOrder", "asc")
collection := c.Query("collection") // Optional collection filter
sortBy := defaultQuery(r.URL.Query().Get("sortBy"), "id")
sortOrder := defaultQuery(r.URL.Query().Get("sortOrder"), "asc")
collection := r.URL.Query().Get("collection") // Optional collection filter
// Get cluster volumes data
volumesData, err := h.adminServer.GetClusterVolumes(page, pageSize, sortBy, sortOrder, collection)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster volumes: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster volumes: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
volumesData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
volumesComponent := app.ClusterVolumes(*volumesData)
layoutComponent := layout.Layout(c, volumesComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, volumesComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowVolumeDetails renders the volume details page
func (h *ClusterHandlers) ShowVolumeDetails(c *gin.Context) {
volumeIDStr := c.Param("id")
server := c.Param("server")
func (h *ClusterHandlers) ShowVolumeDetails(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
volumeIDStr := vars["id"]
server := vars["server"]
if volumeIDStr == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Volume ID is required"})
writeJSONError(w, http.StatusBadRequest, "Volume ID is required")
return
}
if server == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Server is required"})
writeJSONError(w, http.StatusBadRequest, "Server is required")
return
}
volumeID, err := strconv.Atoi(volumeIDStr)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid volume ID"})
writeJSONError(w, http.StatusBadRequest, "Invalid volume ID")
return
}
// Get volume details
volumeDetails, err := h.adminServer.GetVolumeDetails(volumeID, server)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get volume details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get volume details: "+err.Error())
return
}
username := usernameOrDefault(r)
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
volumeDetailsComponent := app.VolumeDetails(*volumeDetails)
layoutComponent := layout.Layout(c, volumeDetailsComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, volumeDetailsComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowClusterCollections renders the cluster collections page
func (h *ClusterHandlers) ShowClusterCollections(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterCollections(w http.ResponseWriter, r *http.Request) {
// Get cluster collections data
collectionsData, err := h.adminServer.GetClusterCollections()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster collections: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster collections: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
collectionsData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
collectionsComponent := app.ClusterCollections(*collectionsData)
layoutComponent := layout.Layout(c, collectionsComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, collectionsComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowCollectionDetails renders the collection detail page
func (h *ClusterHandlers) ShowCollectionDetails(c *gin.Context) {
collectionName := c.Param("name")
func (h *ClusterHandlers) ShowCollectionDetails(w http.ResponseWriter, r *http.Request) {
collectionName := mux.Vars(r)["name"]
if collectionName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Collection name is required"})
writeJSONError(w, http.StatusBadRequest, "Collection name is required")
return
}
// Parse query parameters
page, _ := strconv.Atoi(c.DefaultQuery("page", "1"))
pageSize, _ := strconv.Atoi(c.DefaultQuery("page_size", "25"))
sortBy := c.DefaultQuery("sort_by", "volume_id")
sortOrder := c.DefaultQuery("sort_order", "asc")
query := r.URL.Query()
page, _ := strconv.Atoi(defaultQuery(query.Get("page"), "1"))
pageSize, _ := strconv.Atoi(defaultQuery(query.Get("page_size"), "25"))
sortBy := defaultQuery(query.Get("sort_by"), "volume_id")
sortOrder := defaultQuery(query.Get("sort_order"), "asc")
// Get collection details data (volumes and EC volumes)
collectionDetailsData, err := h.adminServer.GetCollectionDetails(collectionName, page, pageSize, sortBy, sortOrder)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get collection details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get collection details: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
collectionDetailsData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
collectionDetailsComponent := app.CollectionDetails(*collectionDetailsData)
layoutComponent := layout.Layout(c, collectionDetailsComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, collectionDetailsComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowClusterEcShards handles the cluster EC shards page (individual shards view)
func (h *ClusterHandlers) ShowClusterEcShards(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterEcShards(w http.ResponseWriter, r *http.Request) {
// Parse query parameters
page, _ := strconv.Atoi(c.DefaultQuery("page", "1"))
pageSize, _ := strconv.Atoi(c.DefaultQuery("page_size", "100"))
sortBy := c.DefaultQuery("sort_by", "volume_id")
sortOrder := c.DefaultQuery("sort_order", "asc")
collection := c.DefaultQuery("collection", "")
query := r.URL.Query()
page, _ := strconv.Atoi(defaultQuery(query.Get("page"), "1"))
pageSize, _ := strconv.Atoi(defaultQuery(query.Get("page_size"), "100"))
sortBy := defaultQuery(query.Get("sort_by"), "volume_id")
sortOrder := defaultQuery(query.Get("sort_order"), "asc")
collection := defaultQuery(query.Get("collection"), "")
// Get data from admin server
data, err := h.adminServer.GetClusterEcVolumes(page, pageSize, sortBy, sortOrder, collection)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
data.Username = username
// Render template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
ecVolumesComponent := app.ClusterEcVolumes(*data)
layoutComponent := layout.Layout(c, ecVolumesComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, ecVolumesComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
}
// ShowEcVolumeDetails renders the EC volume details page
func (h *ClusterHandlers) ShowEcVolumeDetails(c *gin.Context) {
volumeIDStr := c.Param("id")
func (h *ClusterHandlers) ShowEcVolumeDetails(w http.ResponseWriter, r *http.Request) {
volumeIDStr := mux.Vars(r)["id"]
if volumeIDStr == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Volume ID is required"})
writeJSONError(w, http.StatusBadRequest, "Volume ID is required")
return
}
volumeID, err := strconv.Atoi(volumeIDStr)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid volume ID"})
writeJSONError(w, http.StatusBadRequest, "Invalid volume ID")
return
}
// Check that volumeID is within uint32 range
if volumeID < 0 || uint64(volumeID) > math.MaxUint32 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Volume ID out of range"})
writeJSONError(w, http.StatusBadRequest, "Volume ID out of range")
return
}
// Parse sorting parameters
sortBy := c.DefaultQuery("sort_by", "shard_id")
sortOrder := c.DefaultQuery("sort_order", "asc")
query := r.URL.Query()
sortBy := defaultQuery(query.Get("sort_by"), "shard_id")
sortOrder := defaultQuery(query.Get("sort_order"), "asc")
// Get EC volume details
ecVolumeDetails, err := h.adminServer.GetEcVolumeDetails(uint32(volumeID), sortBy, sortOrder)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get EC volume details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get EC volume details: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
ecVolumeDetails.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
ecVolumeDetailsComponent := app.EcVolumeDetails(*ecVolumeDetails)
layoutComponent := layout.Layout(c, ecVolumeDetailsComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, ecVolumeDetailsComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowClusterMasters renders the cluster masters page
func (h *ClusterHandlers) ShowClusterMasters(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterMasters(w http.ResponseWriter, r *http.Request) {
// Get cluster masters data
mastersData, err := h.adminServer.GetClusterMasters()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster masters: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster masters: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
mastersData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
mastersComponent := app.ClusterMasters(*mastersData)
layoutComponent := layout.Layout(c, mastersComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, mastersComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowClusterFilers renders the cluster filers page
func (h *ClusterHandlers) ShowClusterFilers(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterFilers(w http.ResponseWriter, r *http.Request) {
// Get cluster filers data
filersData, err := h.adminServer.GetClusterFilers()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster filers: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster filers: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
filersData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
filersComponent := app.ClusterFilers(*filersData)
layoutComponent := layout.Layout(c, filersComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, filersComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowClusterBrokers renders the cluster message brokers page
func (h *ClusterHandlers) ShowClusterBrokers(c *gin.Context) {
func (h *ClusterHandlers) ShowClusterBrokers(w http.ResponseWriter, r *http.Request) {
// Get cluster brokers data
brokersData, err := h.adminServer.GetClusterBrokers()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster brokers: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster brokers: "+err.Error())
return
}
// Set username
username := c.GetString("username")
if username == "" {
username = "admin"
}
username := usernameOrDefault(r)
brokersData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
brokersComponent := app.ClusterBrokers(*brokersData)
layoutComponent := layout.Layout(c, brokersComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, brokersComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// GetClusterTopology returns the cluster topology as JSON
func (h *ClusterHandlers) GetClusterTopology(c *gin.Context) {
func (h *ClusterHandlers) GetClusterTopology(w http.ResponseWriter, r *http.Request) {
topology, err := h.adminServer.GetClusterTopology()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
c.JSON(http.StatusOK, topology)
writeJSON(w, http.StatusOK, topology)
}
// GetMasters returns master node information
func (h *ClusterHandlers) GetMasters(c *gin.Context) {
// Simple master info
c.JSON(http.StatusOK, gin.H{"masters": []gin.H{{"address": "localhost:9333"}}})
func (h *ClusterHandlers) GetMasters(w http.ResponseWriter, r *http.Request) {
mastersData, err := h.adminServer.GetClusterMasters()
if err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster masters: "+err.Error())
return
}
writeJSON(w, http.StatusOK, mastersData)
}
// GetVolumeServers returns volume server information
func (h *ClusterHandlers) GetVolumeServers(c *gin.Context) {
func (h *ClusterHandlers) GetVolumeServers(w http.ResponseWriter, r *http.Request) {
topology, err := h.adminServer.GetClusterTopology()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"volume_servers": topology.VolumeServers})
writeJSON(w, http.StatusOK, map[string]interface{}{"volume_servers": topology.VolumeServers})
}
// VacuumVolume handles volume vacuum requests via API
func (h *ClusterHandlers) VacuumVolume(c *gin.Context) {
volumeIDStr := c.Param("id")
server := c.Param("server")
func (h *ClusterHandlers) VacuumVolume(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
volumeIDStr := vars["id"]
server := vars["server"]
if volumeIDStr == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Volume ID is required"})
writeJSONError(w, http.StatusBadRequest, "Volume ID is required")
return
}
volumeID, err := strconv.Atoi(volumeIDStr)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid volume ID"})
writeJSONError(w, http.StatusBadRequest, "Invalid volume ID")
return
}
if server == "" {
writeJSONError(w, http.StatusBadRequest, "Server is required")
return
}
// Perform vacuum operation
err = h.adminServer.VacuumVolume(volumeID, server)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{
"error": "Failed to vacuum volume: " + err.Error(),
})
writeJSONError(w, http.StatusInternalServerError, "Failed to vacuum volume: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Volume vacuum started successfully",
"volume_id": volumeID,
"server": server,
})
}
func usernameOrDefault(r *http.Request) string {
username := dash.UsernameFromContext(r.Context())
if username == "" {
return "admin"
}
return username
}
+82 -75
View File
@@ -16,7 +16,6 @@ import (
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -59,16 +58,16 @@ func (h *FileBrowserHandlers) newClientWithTimeout(timeout time.Duration) http.C
}
// ShowFileBrowser renders the file browser page
func (h *FileBrowserHandlers) ShowFileBrowser(c *gin.Context) {
func (h *FileBrowserHandlers) ShowFileBrowser(w http.ResponseWriter, r *http.Request) {
// Get path from query parameter, default to root
path := c.DefaultQuery("path", "/")
path := defaultQuery(r.URL.Query().Get("path"), "/")
// Normalize Windows-style paths for consistency
path = util.CleanWindowsPath(path)
// Get pagination parameters
lastFileName := c.DefaultQuery("lastFileName", "")
lastFileName := r.URL.Query().Get("lastFileName")
pageSize, err := strconv.Atoi(c.DefaultQuery("limit", "20"))
pageSize, err := strconv.Atoi(defaultQuery(r.URL.Query().Get("limit"), "20"))
if err != nil || pageSize < 1 {
pageSize = 20
}
@@ -79,36 +78,42 @@ func (h *FileBrowserHandlers) ShowFileBrowser(c *gin.Context) {
// Get file browser data with cursor-based pagination
browserData, err := h.adminServer.GetFileBrowser(path, lastFileName, pageSize)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get file browser data: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get file browser data: "+err.Error())
return
}
// Set username
username := c.GetString("username")
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
browserData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
browserComponent := app.FileBrowser(*browserData)
layoutComponent := layout.Layout(c, browserComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, browserComponent)
err = layoutComponent.Render(r.Context(), w)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// DeleteFile handles file deletion API requests
func (h *FileBrowserHandlers) DeleteFile(c *gin.Context) {
func (h *FileBrowserHandlers) DeleteFile(w http.ResponseWriter, r *http.Request) {
var request struct {
Path string `json:"path" binding:"required"`
}
if err := c.ShouldBindJSON(&request); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &request); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if strings.TrimSpace(request.Path) == "" {
writeJSONError(w, http.StatusBadRequest, "path is required")
return
}
@@ -124,29 +129,36 @@ func (h *FileBrowserHandlers) DeleteFile(c *gin.Context) {
return err
})
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to delete file: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to delete file: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"message": "File deleted successfully"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "File deleted successfully"})
}
// DeleteMultipleFiles handles multiple file deletion API requests
func (h *FileBrowserHandlers) DeleteMultipleFiles(c *gin.Context) {
func (h *FileBrowserHandlers) DeleteMultipleFiles(w http.ResponseWriter, r *http.Request) {
var request struct {
Paths []string `json:"paths" binding:"required"`
}
if err := c.ShouldBindJSON(&request); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &request); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if len(request.Paths) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "No paths provided"})
writeJSONError(w, http.StatusBadRequest, "No paths provided")
return
}
for _, path := range request.Paths {
if strings.TrimSpace(path) == "" {
writeJSONError(w, http.StatusBadRequest, "path is required")
return
}
}
var deletedCount int
var failedCount int
var errors []string
@@ -189,37 +201,40 @@ func (h *FileBrowserHandlers) DeleteMultipleFiles(c *gin.Context) {
} else {
response["message"] = fmt.Sprintf("Deleted %d item(s), failed to delete %d item(s)", deletedCount, failedCount)
}
c.JSON(http.StatusOK, response)
writeJSON(w, http.StatusOK, response)
} else {
response["message"] = "Failed to delete all selected items"
c.JSON(http.StatusInternalServerError, response)
writeJSON(w, http.StatusInternalServerError, response)
}
}
// CreateFolder handles folder creation requests
func (h *FileBrowserHandlers) CreateFolder(c *gin.Context) {
func (h *FileBrowserHandlers) CreateFolder(w http.ResponseWriter, r *http.Request) {
var request struct {
Path string `json:"path" binding:"required"`
FolderName string `json:"folder_name" binding:"required"`
}
if err := c.ShouldBindJSON(&request); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &request); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if strings.TrimSpace(request.Path) == "" {
writeJSONError(w, http.StatusBadRequest, "path is required")
return
}
// Clean and validate folder name
folderName := strings.TrimSpace(request.FolderName)
if folderName == "" || strings.Contains(folderName, "/") || strings.Contains(folderName, "\\") {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid folder name"})
writeJSONError(w, http.StatusBadRequest, "Invalid folder name")
return
}
// Create full path for new folder
fullPath := filepath.Join(request.Path, folderName)
if !strings.HasPrefix(fullPath, "/") {
fullPath = "/" + fullPath
}
base := "/" + strings.TrimPrefix(request.Path, "/")
fullPath := path.Join(base, folderName)
// Create folder via filer
err := h.adminServer.WithFilerClient(func(client filer_pb.SeaweedFilerClient) error {
@@ -241,32 +256,32 @@ func (h *FileBrowserHandlers) CreateFolder(c *gin.Context) {
return err
})
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create folder: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create folder: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"message": "Folder created successfully"})
writeJSON(w, http.StatusOK, map[string]interface{}{"message": "Folder created successfully"})
}
// UploadFile handles file upload requests
func (h *FileBrowserHandlers) UploadFile(c *gin.Context) {
func (h *FileBrowserHandlers) UploadFile(w http.ResponseWriter, r *http.Request) {
// Get the current path
currentPath := c.PostForm("path")
currentPath := r.FormValue("path")
if currentPath == "" {
currentPath = "/"
}
// Parse multipart form
err := c.Request.ParseMultipartForm(1 << 30) // 1GB max memory for large file uploads
err := r.ParseMultipartForm(1 << 30) // 1GB max memory for large file uploads
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Failed to parse multipart form: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "Failed to parse multipart form: "+err.Error())
return
}
// Get uploaded files (supports multiple files)
files := c.Request.MultipartForm.File["files"]
files := r.MultipartForm.File["files"]
if len(files) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "No files uploaded"})
writeJSONError(w, http.StatusBadRequest, "No files uploaded")
return
}
@@ -292,16 +307,8 @@ func (h *FileBrowserHandlers) UploadFile(c *gin.Context) {
fullPath = "/" + fullPath
}
// Open the file
file, err := fileHeader.Open()
if err != nil {
failedUploads = append(failedUploads, fmt.Sprintf("%s: %v", fileName, err))
continue
}
// Upload file to filer
err = h.uploadFileToFiler(fullPath, fileHeader)
file.Close()
if err != nil {
failedUploads = append(failedUploads, fmt.Sprintf("%s: %v", fileName, err))
@@ -331,10 +338,10 @@ func (h *FileBrowserHandlers) UploadFile(c *gin.Context) {
} else {
response["message"] = fmt.Sprintf("Uploaded %d file(s), %d failed", len(uploadResults), len(failedUploads))
}
c.JSON(http.StatusOK, response)
writeJSON(w, http.StatusOK, response)
} else {
response["message"] = "All file uploads failed"
c.JSON(http.StatusInternalServerError, response)
writeJSON(w, http.StatusInternalServerError, response)
}
}
@@ -561,23 +568,23 @@ func (h *FileBrowserHandlers) fetchFileContent(filePath string, timeout time.Dur
// DownloadFile handles file download requests by proxying through the Admin UI server
// This ensures mTLS works correctly since the Admin UI server has the client certificates
func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
filePath := c.Query("path")
func (h *FileBrowserHandlers) DownloadFile(w http.ResponseWriter, r *http.Request) {
filePath := r.URL.Query().Get("path")
if filePath == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "File path is required"})
writeJSONError(w, http.StatusBadRequest, "File path is required")
return
}
// Get filer address
filerAddress := h.adminServer.GetFilerAddress()
if filerAddress == "" {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Filer address not configured"})
writeJSONError(w, http.StatusInternalServerError, "Filer address not configured")
return
}
// Validate filer address to prevent SSRF
if err := h.validateFilerAddress(filerAddress); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Invalid filer address configuration"})
writeJSONError(w, http.StatusInternalServerError, "Invalid filer address configuration")
return
}
filerHttpAddress := pb.ServerAddress(filerAddress).ToHttpAddress()
@@ -585,7 +592,7 @@ func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
// Validate and sanitize the file path
cleanFilePath, err := h.validateAndCleanFilePath(filePath)
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid file path: " + err.Error()})
writeJSONError(w, http.StatusBadRequest, "Invalid file path: "+err.Error())
return
}
@@ -593,7 +600,7 @@ func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
downloadURL := fmt.Sprintf("%s%s", filerHttpAddress, cleanFilePath)
downloadURL, err = h.httpClient.NormalizeHttpScheme(downloadURL)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to construct download URL: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to construct download URL: "+err.Error())
return
}
@@ -602,9 +609,9 @@ func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
// Safe: filerAddress validated by validateFilerAddress() to match configured filer
// Safe: cleanFilePath validated and cleaned by validateAndCleanFilePath() to prevent path traversal
// Use request context so download is cancelled when client disconnects
req, err := http.NewRequestWithContext(c.Request.Context(), "GET", downloadURL, nil)
req, err := http.NewRequestWithContext(r.Context(), "GET", downloadURL, nil)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create request: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create request: "+err.Error())
return
}
client := h.newClientWithTimeout(5 * time.Minute) // Longer timeout for large file downloads
@@ -613,7 +620,7 @@ func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
resp, err := client.Do(req)
if err != nil {
c.JSON(http.StatusBadGateway, gin.H{"error": "Failed to fetch file from filer: " + err.Error()})
writeJSONError(w, http.StatusBadGateway, "Failed to fetch file from filer: "+err.Error())
return
}
defer resp.Body.Close()
@@ -621,10 +628,10 @@ func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
if resp.StatusCode != http.StatusOK {
body, err := io.ReadAll(resp.Body)
if err != nil {
c.JSON(resp.StatusCode, gin.H{"error": fmt.Sprintf("Filer returned status %d but failed to read response body: %v", resp.StatusCode, err)})
writeJSONError(w, resp.StatusCode, fmt.Sprintf("Filer returned status %d but failed to read response body: %v", resp.StatusCode, err))
return
}
c.JSON(resp.StatusCode, gin.H{"error": fmt.Sprintf("Filer returned status %d: %s", resp.StatusCode, string(body))})
writeJSONError(w, resp.StatusCode, fmt.Sprintf("Filer returned status %d: %s", resp.StatusCode, string(body)))
return
}
@@ -632,33 +639,33 @@ func (h *FileBrowserHandlers) DownloadFile(c *gin.Context) {
fileName := filepath.Base(cleanFilePath)
// Use mime.FormatMediaType for RFC 6266 compliant Content-Disposition,
// properly handling non-ASCII characters and special characters
c.Header("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": fileName}))
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": fileName}))
// Use content type from filer response, or default to octet-stream
contentType := resp.Header.Get("Content-Type")
if contentType == "" {
contentType = "application/octet-stream"
}
c.Header("Content-Type", contentType)
w.Header().Set("Content-Type", contentType)
// Set content length if available
if resp.ContentLength > 0 {
c.Header("Content-Length", fmt.Sprintf("%d", resp.ContentLength))
w.Header().Set("Content-Length", fmt.Sprintf("%d", resp.ContentLength))
}
// Stream the response body to the client
c.Status(http.StatusOK)
_, err = io.Copy(c.Writer, resp.Body)
w.WriteHeader(http.StatusOK)
_, err = io.Copy(w, resp.Body)
if err != nil {
glog.Errorf("Error streaming file download: %v", err)
}
}
// ViewFile handles file viewing requests (for text files, images, etc.)
func (h *FileBrowserHandlers) ViewFile(c *gin.Context) {
filePath := c.Query("path")
func (h *FileBrowserHandlers) ViewFile(w http.ResponseWriter, r *http.Request) {
filePath := r.URL.Query().Get("path")
if filePath == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "File path is required"})
writeJSONError(w, http.StatusBadRequest, "File path is required")
return
}
@@ -704,7 +711,7 @@ func (h *FileBrowserHandlers) ViewFile(c *gin.Context) {
return nil
})
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get file metadata: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get file metadata: "+err.Error())
return
}
@@ -752,7 +759,7 @@ func (h *FileBrowserHandlers) ViewFile(c *gin.Context) {
}
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"file": fileEntry,
"content": content,
"viewable": viewable,
@@ -761,10 +768,10 @@ func (h *FileBrowserHandlers) ViewFile(c *gin.Context) {
}
// GetFileProperties handles file properties requests
func (h *FileBrowserHandlers) GetFileProperties(c *gin.Context) {
filePath := c.Query("path")
func (h *FileBrowserHandlers) GetFileProperties(w http.ResponseWriter, r *http.Request) {
filePath := r.URL.Query().Get("path")
if filePath == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "File path is required"})
writeJSONError(w, http.StatusBadRequest, "File path is required")
return
}
@@ -853,11 +860,11 @@ func (h *FileBrowserHandlers) GetFileProperties(c *gin.Context) {
return nil
})
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get file properties: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get file properties: "+err.Error())
return
}
c.JSON(http.StatusOK, properties)
writeJSON(w, http.StatusOK, properties)
}
// Helper function to format bytes
+28
View File
@@ -0,0 +1,28 @@
package handlers
import (
"io"
"net/http"
"github.com/seaweedfs/seaweedfs/weed/admin/internal/httputil"
)
func writeJSON(w http.ResponseWriter, status int, payload interface{}) {
httputil.WriteJSON(w, status, payload)
}
func writeJSONError(w http.ResponseWriter, status int, message string) {
httputil.WriteJSONError(w, status, message)
}
func decodeJSONBody(r io.Reader, v interface{}) error {
return httputil.DecodeJSONBody(r, v)
}
func newJSONMaxReader(w http.ResponseWriter, r *http.Request) io.Reader {
return httputil.NewJSONMaxReader(w, r)
}
func defaultQuery(value, fallback string) string {
return httputil.DefaultQuery(value, fallback)
}
+57 -51
View File
@@ -4,7 +4,7 @@ import (
"fmt"
"net/http"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -23,146 +23,152 @@ func NewMessageQueueHandlers(adminServer *dash.AdminServer) *MessageQueueHandler
}
// ShowBrokers renders the message queue brokers page
func (h *MessageQueueHandlers) ShowBrokers(c *gin.Context) {
func (h *MessageQueueHandlers) ShowBrokers(w http.ResponseWriter, r *http.Request) {
// Get cluster brokers data
brokersData, err := h.adminServer.GetClusterBrokers()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get cluster brokers: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get cluster brokers: "+err.Error())
return
}
// Set username
username := c.GetString("username")
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
brokersData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
brokersComponent := app.ClusterBrokers(*brokersData)
layoutComponent := layout.Layout(c, brokersComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, brokersComponent)
err = layoutComponent.Render(r.Context(), w)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowTopics renders the message queue topics page
func (h *MessageQueueHandlers) ShowTopics(c *gin.Context) {
func (h *MessageQueueHandlers) ShowTopics(w http.ResponseWriter, r *http.Request) {
// Get topics data
topicsData, err := h.adminServer.GetTopics()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get topics: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get topics: "+err.Error())
return
}
// Set username
username := c.GetString("username")
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
topicsData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
topicsComponent := app.Topics(*topicsData)
layoutComponent := layout.Layout(c, topicsComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, topicsComponent)
err = layoutComponent.Render(r.Context(), w)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowSubscribers renders the message queue subscribers page
func (h *MessageQueueHandlers) ShowSubscribers(c *gin.Context) {
func (h *MessageQueueHandlers) ShowSubscribers(w http.ResponseWriter, r *http.Request) {
// Get subscribers data
subscribersData, err := h.adminServer.GetSubscribers()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get subscribers: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get subscribers: "+err.Error())
return
}
// Set username
username := c.GetString("username")
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
subscribersData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
subscribersComponent := app.Subscribers(*subscribersData)
layoutComponent := layout.Layout(c, subscribersComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, subscribersComponent)
err = layoutComponent.Render(r.Context(), w)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// ShowTopicDetails renders the topic details page
func (h *MessageQueueHandlers) ShowTopicDetails(c *gin.Context) {
func (h *MessageQueueHandlers) ShowTopicDetails(w http.ResponseWriter, r *http.Request) {
// Get topic parameters from URL
namespace := c.Param("namespace")
topicName := c.Param("topic")
vars := mux.Vars(r)
namespace := vars["namespace"]
topicName := vars["topic"]
if namespace == "" || topicName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Missing namespace or topic name"})
writeJSONError(w, http.StatusBadRequest, "Missing namespace or topic name")
return
}
// Get topic details data
topicDetailsData, err := h.adminServer.GetTopicDetails(namespace, topicName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get topic details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get topic details: "+err.Error())
return
}
// Set username
username := c.GetString("username")
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
topicDetailsData.Username = username
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
topicDetailsComponent := app.TopicDetails(*topicDetailsData)
layoutComponent := layout.Layout(c, topicDetailsComponent)
err = layoutComponent.Render(c.Request.Context(), c.Writer)
viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, topicDetailsComponent)
err = layoutComponent.Render(r.Context(), w)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// GetTopicDetailsAPI returns topic details as JSON for AJAX calls
func (h *MessageQueueHandlers) GetTopicDetailsAPI(c *gin.Context) {
func (h *MessageQueueHandlers) GetTopicDetailsAPI(w http.ResponseWriter, r *http.Request) {
// Get topic parameters from URL
namespace := c.Param("namespace")
topicName := c.Param("topic")
vars := mux.Vars(r)
namespace := vars["namespace"]
topicName := vars["topic"]
if namespace == "" || topicName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Missing namespace or topic name"})
writeJSONError(w, http.StatusBadRequest, "Missing namespace or topic name")
return
}
// Get topic details data
topicDetailsData, err := h.adminServer.GetTopicDetails(namespace, topicName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get topic details: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get topic details: "+err.Error())
return
}
// Return JSON data
c.JSON(http.StatusOK, topicDetailsData)
writeJSON(w, http.StatusOK, topicDetailsData)
}
// CreateTopicAPI creates a new topic with retention configuration
func (h *MessageQueueHandlers) CreateTopicAPI(c *gin.Context) {
func (h *MessageQueueHandlers) CreateTopicAPI(w http.ResponseWriter, r *http.Request) {
var req struct {
Namespace string `json:"namespace" binding:"required"`
Name string `json:"name" binding:"required"`
@@ -173,30 +179,30 @@ func (h *MessageQueueHandlers) CreateTopicAPI(c *gin.Context) {
} `json:"retention"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Validate inputs
if req.PartitionCount < 1 || req.PartitionCount > 100 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Partition count must be between 1 and 100"})
writeJSONError(w, http.StatusBadRequest, "Partition count must be between 1 and 100")
return
}
if req.Retention.Enabled && req.Retention.RetentionSeconds <= 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Retention seconds must be positive when retention is enabled"})
writeJSONError(w, http.StatusBadRequest, "Retention seconds must be positive when retention is enabled")
return
}
// Create the topic via admin server
err := h.adminServer.CreateTopicWithRetention(req.Namespace, req.Name, req.PartitionCount, req.Retention.Enabled, req.Retention.RetentionSeconds)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create topic: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create topic: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Topic created successfully",
"topic": fmt.Sprintf("%s.%s", req.Namespace, req.Name),
})
@@ -211,27 +217,27 @@ type UpdateTopicRetentionRequest struct {
} `json:"retention"`
}
func (h *MessageQueueHandlers) UpdateTopicRetentionAPI(c *gin.Context) {
func (h *MessageQueueHandlers) UpdateTopicRetentionAPI(w http.ResponseWriter, r *http.Request) {
var request UpdateTopicRetentionRequest
if err := c.ShouldBindJSON(&request); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &request); err != nil {
writeJSONError(w, http.StatusBadRequest, err.Error())
return
}
// Validate required fields
if request.Namespace == "" || request.Name == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "namespace and name are required"})
writeJSONError(w, http.StatusBadRequest, "namespace and name are required")
return
}
// Update the topic retention
err := h.adminServer.UpdateTopicRetention(request.Namespace, request.Name, request.Retention.Enabled, request.Retention.RetentionSeconds)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
writeJSONError(w, http.StatusInternalServerError, err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Topic retention updated successfully",
"topic": request.Namespace + "." + request.Name,
})
+20 -18
View File
@@ -4,7 +4,6 @@ import (
"bytes"
"net/http"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -23,45 +22,48 @@ func NewPluginHandlers(adminServer *dash.AdminServer) *PluginHandlers {
}
// ShowPlugin displays plugin overview page.
func (h *PluginHandlers) ShowPlugin(c *gin.Context) {
h.renderPluginPage(c, "overview")
func (h *PluginHandlers) ShowPlugin(w http.ResponseWriter, r *http.Request) {
h.renderPluginPage(w, r, "overview")
}
// ShowPluginConfiguration displays plugin configuration page.
func (h *PluginHandlers) ShowPluginConfiguration(c *gin.Context) {
h.renderPluginPage(c, "configuration")
func (h *PluginHandlers) ShowPluginConfiguration(w http.ResponseWriter, r *http.Request) {
h.renderPluginPage(w, r, "configuration")
}
// ShowPluginDetection displays plugin detection jobs page.
func (h *PluginHandlers) ShowPluginDetection(c *gin.Context) {
h.renderPluginPage(c, "detection")
func (h *PluginHandlers) ShowPluginDetection(w http.ResponseWriter, r *http.Request) {
h.renderPluginPage(w, r, "detection")
}
// ShowPluginQueue displays plugin job queue page.
func (h *PluginHandlers) ShowPluginQueue(c *gin.Context) {
h.renderPluginPage(c, "queue")
func (h *PluginHandlers) ShowPluginQueue(w http.ResponseWriter, r *http.Request) {
h.renderPluginPage(w, r, "queue")
}
// ShowPluginExecution displays plugin execution jobs page.
func (h *PluginHandlers) ShowPluginExecution(c *gin.Context) {
h.renderPluginPage(c, "execution")
func (h *PluginHandlers) ShowPluginExecution(w http.ResponseWriter, r *http.Request) {
h.renderPluginPage(w, r, "execution")
}
// ShowPluginMonitoring displays plugin monitoring page.
func (h *PluginHandlers) ShowPluginMonitoring(c *gin.Context) {
func (h *PluginHandlers) ShowPluginMonitoring(w http.ResponseWriter, r *http.Request) {
// Backward-compatible alias for the old monitoring URL.
h.renderPluginPage(c, "detection")
h.renderPluginPage(w, r, "detection")
}
func (h *PluginHandlers) renderPluginPage(c *gin.Context, page string) {
func (h *PluginHandlers) renderPluginPage(w http.ResponseWriter, r *http.Request, page string) {
component := app.Plugin(page)
layoutComponent := layout.Layout(c, component)
viewCtx := layout.NewViewContext(r, dash.UsernameFromContext(r.Context()), dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
var buf bytes.Buffer
if err := layoutComponent.Render(c.Request.Context(), &buf); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
if err := layoutComponent.Render(r.Context(), &buf); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
c.Data(http.StatusOK, "text/html; charset=utf-8", buf.Bytes())
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.WriteHeader(http.StatusOK)
_, _ = w.Write(buf.Bytes())
}
+53 -59
View File
@@ -5,7 +5,7 @@ import (
"net/http"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -26,53 +26,53 @@ func NewPolicyHandlers(adminServer *dash.AdminServer) *PolicyHandlers {
}
// ShowPolicies renders the policies management page
func (h *PolicyHandlers) ShowPolicies(c *gin.Context) {
func (h *PolicyHandlers) ShowPolicies(w http.ResponseWriter, r *http.Request) {
// Get policies data from the server
policiesData := h.getPoliciesData(c)
policiesData := h.getPoliciesData(r)
// Render HTML template
c.Header("Content-Type", "text/html")
w.Header().Set("Content-Type", "text/html")
policiesComponent := app.Policies(policiesData)
layoutComponent := layout.Layout(c, policiesComponent)
err := layoutComponent.Render(c.Request.Context(), c.Writer)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
viewCtx := layout.NewViewContext(r, dash.UsernameFromContext(r.Context()), dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, policiesComponent)
if err := layoutComponent.Render(r.Context(), w); err != nil {
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// GetPolicies returns the list of policies as JSON
func (h *PolicyHandlers) GetPolicies(c *gin.Context) {
func (h *PolicyHandlers) GetPolicies(w http.ResponseWriter, r *http.Request) {
policies, err := h.adminServer.GetPolicies()
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get policies: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get policies: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"policies": policies})
writeJSON(w, http.StatusOK, map[string]interface{}{"policies": policies})
}
// CreatePolicy handles policy creation
func (h *PolicyHandlers) CreatePolicy(c *gin.Context) {
func (h *PolicyHandlers) CreatePolicy(w http.ResponseWriter, r *http.Request) {
var req dash.CreatePolicyRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Validate policy name
if req.Name == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Policy name is required"})
writeJSONError(w, http.StatusBadRequest, "Policy name is required")
return
}
// Check if policy already exists
existingPolicy, err := h.adminServer.GetPolicy(req.Name)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to check existing policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to check existing policy: "+err.Error())
return
}
if existingPolicy != nil {
c.JSON(http.StatusConflict, gin.H{"error": "Policy with this name already exists"})
writeJSONError(w, http.StatusConflict, "Policy with this name already exists")
return
}
@@ -80,11 +80,11 @@ func (h *PolicyHandlers) CreatePolicy(c *gin.Context) {
err = h.adminServer.CreatePolicy(req.Name, req.Document)
if err != nil {
glog.Errorf("Failed to create policy %s: %v", req.Name, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create policy: "+err.Error())
return
}
c.JSON(http.StatusCreated, gin.H{
writeJSON(w, http.StatusCreated, map[string]interface{}{
"success": true,
"message": "Policy created successfully",
"policy": req.Name,
@@ -92,49 +92,49 @@ func (h *PolicyHandlers) CreatePolicy(c *gin.Context) {
}
// GetPolicy returns a specific policy
func (h *PolicyHandlers) GetPolicy(c *gin.Context) {
policyName := c.Param("name")
func (h *PolicyHandlers) GetPolicy(w http.ResponseWriter, r *http.Request) {
policyName := mux.Vars(r)["name"]
if policyName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Policy name is required"})
writeJSONError(w, http.StatusBadRequest, "Policy name is required")
return
}
policy, err := h.adminServer.GetPolicy(policyName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get policy: "+err.Error())
return
}
if policy == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "Policy not found"})
writeJSONError(w, http.StatusNotFound, "Policy not found")
return
}
c.JSON(http.StatusOK, policy)
writeJSON(w, http.StatusOK, policy)
}
// UpdatePolicy handles policy updates
func (h *PolicyHandlers) UpdatePolicy(c *gin.Context) {
policyName := c.Param("name")
func (h *PolicyHandlers) UpdatePolicy(w http.ResponseWriter, r *http.Request) {
policyName := mux.Vars(r)["name"]
if policyName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Policy name is required"})
writeJSONError(w, http.StatusBadRequest, "Policy name is required")
return
}
var req dash.UpdatePolicyRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Check if policy exists
existingPolicy, err := h.adminServer.GetPolicy(policyName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to check existing policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to check existing policy: "+err.Error())
return
}
if existingPolicy == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "Policy not found"})
writeJSONError(w, http.StatusNotFound, "Policy not found")
return
}
@@ -142,11 +142,11 @@ func (h *PolicyHandlers) UpdatePolicy(c *gin.Context) {
err = h.adminServer.UpdatePolicy(policyName, req.Document)
if err != nil {
glog.Errorf("Failed to update policy %s: %v", policyName, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to update policy: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"success": true,
"message": "Policy updated successfully",
"policy": policyName,
@@ -154,21 +154,21 @@ func (h *PolicyHandlers) UpdatePolicy(c *gin.Context) {
}
// DeletePolicy handles policy deletion
func (h *PolicyHandlers) DeletePolicy(c *gin.Context) {
policyName := c.Param("name")
func (h *PolicyHandlers) DeletePolicy(w http.ResponseWriter, r *http.Request) {
policyName := mux.Vars(r)["name"]
if policyName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Policy name is required"})
writeJSONError(w, http.StatusBadRequest, "Policy name is required")
return
}
// Check if policy exists
existingPolicy, err := h.adminServer.GetPolicy(policyName)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to check existing policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to check existing policy: "+err.Error())
return
}
if existingPolicy == nil {
c.JSON(http.StatusNotFound, gin.H{"error": "Policy not found"})
writeJSONError(w, http.StatusNotFound, "Policy not found")
return
}
@@ -176,11 +176,11 @@ func (h *PolicyHandlers) DeletePolicy(c *gin.Context) {
err = h.adminServer.DeletePolicy(policyName)
if err != nil {
glog.Errorf("Failed to delete policy %s: %v", policyName, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to delete policy: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to delete policy: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"success": true,
"message": "Policy deleted successfully",
"policy": policyName,
@@ -188,60 +188,54 @@ func (h *PolicyHandlers) DeletePolicy(c *gin.Context) {
}
// ValidatePolicy validates a policy document without saving it
func (h *PolicyHandlers) ValidatePolicy(c *gin.Context) {
func (h *PolicyHandlers) ValidatePolicy(w http.ResponseWriter, r *http.Request) {
var req struct {
Document policy_engine.PolicyDocument `json:"document" binding:"required"`
Document policy_engine.PolicyDocument `json:"document"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Basic validation
if req.Document.Version == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Policy version is required"})
writeJSONError(w, http.StatusBadRequest, "Policy version is required")
return
}
if len(req.Document.Statement) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "Policy must have at least one statement"})
writeJSONError(w, http.StatusBadRequest, "Policy must have at least one statement")
return
}
// Validate each statement
for i, statement := range req.Document.Statement {
if statement.Effect != "Allow" && statement.Effect != "Deny" {
c.JSON(http.StatusBadRequest, gin.H{
"error": fmt.Sprintf("Statement %d: Effect must be 'Allow' or 'Deny'", i+1),
})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Statement %d: Effect must be 'Allow' or 'Deny'", i+1))
return
}
if len(statement.Action.Strings()) == 0 {
c.JSON(http.StatusBadRequest, gin.H{
"error": fmt.Sprintf("Statement %d: Action is required", i+1),
})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Statement %d: Action is required", i+1))
return
}
if len(statement.Resource.Strings()) == 0 {
c.JSON(http.StatusBadRequest, gin.H{
"error": fmt.Sprintf("Statement %d: Resource is required", i+1),
})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Statement %d: Resource is required", i+1))
return
}
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"valid": true,
"message": "Policy document is valid",
})
}
// getPoliciesData retrieves policies data from the server
func (h *PolicyHandlers) getPoliciesData(c *gin.Context) dash.PoliciesData {
username := c.GetString("username")
func (h *PolicyHandlers) getPoliciesData(r *http.Request) dash.PoliciesData {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
+48 -47
View File
@@ -6,7 +6,7 @@ import (
"net/http"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -26,153 +26,154 @@ func NewServiceAccountHandlers(adminServer *dash.AdminServer) *ServiceAccountHan
}
// ShowServiceAccounts renders the service accounts management page
func (h *ServiceAccountHandlers) ShowServiceAccounts(c *gin.Context) {
data := h.getServiceAccountsData(c)
func (h *ServiceAccountHandlers) ShowServiceAccounts(w http.ResponseWriter, r *http.Request) {
data := h.getServiceAccountsData(r)
// Render to buffer first to avoid partial writes on error
var buf bytes.Buffer
component := app.ServiceAccounts(data)
layoutComponent := layout.Layout(c, component)
err := layoutComponent.Render(c.Request.Context(), &buf)
viewCtx := layout.NewViewContext(r, dash.UsernameFromContext(r.Context()), dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, component)
err := layoutComponent.Render(r.Context(), &buf)
if err != nil {
glog.Errorf("Failed to render service accounts template: %v", err)
c.AbortWithStatus(http.StatusInternalServerError)
w.WriteHeader(http.StatusInternalServerError)
return
}
// Only write to response if render succeeded
c.Header("Content-Type", "text/html")
c.Writer.Write(buf.Bytes())
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write(buf.Bytes())
}
// GetServiceAccounts returns the list of service accounts as JSON
func (h *ServiceAccountHandlers) GetServiceAccounts(c *gin.Context) {
parentUser := c.Query("parent_user")
func (h *ServiceAccountHandlers) GetServiceAccounts(w http.ResponseWriter, r *http.Request) {
parentUser := r.URL.Query().Get("parent_user")
accounts, err := h.adminServer.GetServiceAccounts(c.Request.Context(), parentUser)
accounts, err := h.adminServer.GetServiceAccounts(r.Context(), parentUser)
if err != nil {
glog.Errorf("Failed to get service accounts: %v", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get service accounts"})
writeJSONError(w, http.StatusInternalServerError, "Failed to get service accounts")
return
}
c.JSON(http.StatusOK, gin.H{"service_accounts": accounts})
writeJSON(w, http.StatusOK, map[string]interface{}{"service_accounts": accounts})
}
// CreateServiceAccount handles service account creation
func (h *ServiceAccountHandlers) CreateServiceAccount(c *gin.Context) {
func (h *ServiceAccountHandlers) CreateServiceAccount(w http.ResponseWriter, r *http.Request) {
var req dash.CreateServiceAccountRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
if req.ParentUser == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "ParentUser is required"})
writeJSONError(w, http.StatusBadRequest, "ParentUser is required")
return
}
sa, err := h.adminServer.CreateServiceAccount(c.Request.Context(), req)
sa, err := h.adminServer.CreateServiceAccount(r.Context(), req)
if err != nil {
glog.Errorf("Failed to create service account for user %s: %v", req.ParentUser, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create service account"})
writeJSONError(w, http.StatusInternalServerError, "Failed to create service account")
return
}
c.JSON(http.StatusCreated, gin.H{
writeJSON(w, http.StatusCreated, map[string]interface{}{
"message": "Service account created successfully",
"service_account": sa,
})
}
// GetServiceAccountDetails returns detailed information about a service account
func (h *ServiceAccountHandlers) GetServiceAccountDetails(c *gin.Context) {
id := c.Param("id")
func (h *ServiceAccountHandlers) GetServiceAccountDetails(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
if id == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Service account ID is required"})
writeJSONError(w, http.StatusBadRequest, "Service account ID is required")
return
}
sa, err := h.adminServer.GetServiceAccountDetails(c.Request.Context(), id)
sa, err := h.adminServer.GetServiceAccountDetails(r.Context(), id)
if err != nil {
// Distinguish not-found errors from internal errors
if errors.Is(err, dash.ErrServiceAccountNotFound) {
c.JSON(http.StatusNotFound, gin.H{"error": "Service account not found: " + err.Error()})
writeJSONError(w, http.StatusNotFound, "Service account not found: "+err.Error())
} else {
glog.Errorf("Failed to get service account details for %s: %v", id, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get service account details"})
writeJSONError(w, http.StatusInternalServerError, "Failed to get service account details")
}
return
}
c.JSON(http.StatusOK, sa)
writeJSON(w, http.StatusOK, sa)
}
// UpdateServiceAccount handles service account updates
func (h *ServiceAccountHandlers) UpdateServiceAccount(c *gin.Context) {
id := c.Param("id")
func (h *ServiceAccountHandlers) UpdateServiceAccount(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
if id == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Service account ID is required"})
writeJSONError(w, http.StatusBadRequest, "Service account ID is required")
return
}
var req dash.UpdateServiceAccountRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
sa, err := h.adminServer.UpdateServiceAccount(c.Request.Context(), id, req)
sa, err := h.adminServer.UpdateServiceAccount(r.Context(), id, req)
if err != nil {
// Distinguish not-found errors from internal errors
if errors.Is(err, dash.ErrServiceAccountNotFound) {
c.JSON(http.StatusNotFound, gin.H{"error": "Service account not found"})
writeJSONError(w, http.StatusNotFound, "Service account not found")
} else {
glog.Errorf("Failed to update service account %s: %v", id, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update service account"})
writeJSONError(w, http.StatusInternalServerError, "Failed to update service account")
}
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Service account updated successfully",
"service_account": sa,
})
}
// DeleteServiceAccount handles service account deletion
func (h *ServiceAccountHandlers) DeleteServiceAccount(c *gin.Context) {
id := c.Param("id")
func (h *ServiceAccountHandlers) DeleteServiceAccount(w http.ResponseWriter, r *http.Request) {
id := mux.Vars(r)["id"]
if id == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Service account ID is required"})
writeJSONError(w, http.StatusBadRequest, "Service account ID is required")
return
}
err := h.adminServer.DeleteServiceAccount(c.Request.Context(), id)
err := h.adminServer.DeleteServiceAccount(r.Context(), id)
if err != nil {
// Distinguish not-found errors from internal errors
if errors.Is(err, dash.ErrServiceAccountNotFound) {
c.JSON(http.StatusNotFound, gin.H{"error": "Service account not found"})
writeJSONError(w, http.StatusNotFound, "Service account not found")
} else {
glog.Errorf("Failed to delete service account %s: %v", id, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to delete service account"})
writeJSONError(w, http.StatusInternalServerError, "Failed to delete service account")
}
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Service account deleted successfully",
})
}
// getServiceAccountsData retrieves service accounts data for the template
func (h *ServiceAccountHandlers) getServiceAccountsData(c *gin.Context) dash.ServiceAccountsData {
username := c.GetString("username")
func (h *ServiceAccountHandlers) getServiceAccountsData(r *http.Request) dash.ServiceAccountsData {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
// Get all service accounts
accounts, err := h.adminServer.GetServiceAccounts(c.Request.Context(), "")
accounts, err := h.adminServer.GetServiceAccounts(r.Context(), "")
if err != nil {
glog.Errorf("Failed to get service accounts: %v", err)
return dash.ServiceAccountsData{
@@ -193,7 +194,7 @@ func (h *ServiceAccountHandlers) getServiceAccountsData(c *gin.Context) dash.Ser
// Get available users for dropdown
var availableUsers []string
users, err := h.adminServer.GetObjectStoreUsers(c.Request.Context())
users, err := h.adminServer.GetObjectStoreUsers(r.Context())
if err != nil {
glog.Errorf("Failed to get users for dropdown: %v", err)
} else {
+76 -73
View File
@@ -5,7 +5,7 @@ import (
"net/http"
"time"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/seaweedfs/seaweedfs/weed/admin/dash"
"github.com/seaweedfs/seaweedfs/weed/admin/view/app"
"github.com/seaweedfs/seaweedfs/weed/admin/view/layout"
@@ -25,256 +25,259 @@ func NewUserHandlers(adminServer *dash.AdminServer) *UserHandlers {
}
// ShowObjectStoreUsers renders the object store users management page
func (h *UserHandlers) ShowObjectStoreUsers(c *gin.Context) {
func (h *UserHandlers) ShowObjectStoreUsers(w http.ResponseWriter, r *http.Request) {
// Get object store users data from the server
usersData := h.getObjectStoreUsersData(c)
usersData := h.getObjectStoreUsersData(r)
// Render HTML template
// Add cache-control headers to prevent browser caching of inline JavaScript
c.Header("Content-Type", "text/html")
c.Header("Cache-Control", "no-cache, no-store, must-revalidate")
c.Header("Pragma", "no-cache")
c.Header("Expires", "0")
c.Header("ETag", fmt.Sprintf("\"%d\"", time.Now().Unix()))
w.Header().Set("Content-Type", "text/html")
w.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate")
w.Header().Set("Pragma", "no-cache")
w.Header().Set("Expires", "0")
w.Header().Set("ETag", fmt.Sprintf("\"%d\"", time.Now().Unix()))
usersComponent := app.ObjectStoreUsers(usersData)
layoutComponent := layout.Layout(c, usersComponent)
err := layoutComponent.Render(c.Request.Context(), c.Writer)
viewCtx := layout.NewViewContext(r, dash.UsernameFromContext(r.Context()), dash.CSRFTokenFromContext(r.Context()))
layoutComponent := layout.Layout(viewCtx, usersComponent)
err := layoutComponent.Render(r.Context(), w)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to render template: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
return
}
}
// GetUsers returns the list of users as JSON
func (h *UserHandlers) GetUsers(c *gin.Context) {
users, err := h.adminServer.GetObjectStoreUsers(c.Request.Context())
func (h *UserHandlers) GetUsers(w http.ResponseWriter, r *http.Request) {
users, err := h.adminServer.GetObjectStoreUsers(r.Context())
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get users: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get users: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"users": users})
writeJSON(w, http.StatusOK, map[string]interface{}{"users": users})
}
// CreateUser handles user creation
func (h *UserHandlers) CreateUser(c *gin.Context) {
func (h *UserHandlers) CreateUser(w http.ResponseWriter, r *http.Request) {
var req dash.CreateUserRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Validate required fields
if req.Username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
user, err := h.adminServer.CreateObjectStoreUser(req)
if err != nil {
glog.Errorf("Failed to create user %s: %v", req.Username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create user: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create user: "+err.Error())
return
}
c.JSON(http.StatusCreated, gin.H{
writeJSON(w, http.StatusCreated, map[string]interface{}{
"message": "User created successfully",
"user": user,
})
}
// UpdateUser handles user updates
func (h *UserHandlers) UpdateUser(c *gin.Context) {
username := c.Param("username")
func (h *UserHandlers) UpdateUser(w http.ResponseWriter, r *http.Request) {
username := mux.Vars(r)["username"]
if username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
var req dash.UpdateUserRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
user, err := h.adminServer.UpdateObjectStoreUser(username, req)
if err != nil {
glog.Errorf("Failed to update user %s: %v", username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update user: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to update user: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "User updated successfully",
"user": user,
})
}
// DeleteUser handles user deletion
func (h *UserHandlers) DeleteUser(c *gin.Context) {
username := c.Param("username")
func (h *UserHandlers) DeleteUser(w http.ResponseWriter, r *http.Request) {
username := mux.Vars(r)["username"]
if username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
err := h.adminServer.DeleteObjectStoreUser(username)
if err != nil {
glog.Errorf("Failed to delete user %s: %v", username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to delete user: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to delete user: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "User deleted successfully",
})
}
// GetUserDetails returns detailed information about a specific user
func (h *UserHandlers) GetUserDetails(c *gin.Context) {
username := c.Param("username")
func (h *UserHandlers) GetUserDetails(w http.ResponseWriter, r *http.Request) {
username := mux.Vars(r)["username"]
if username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
user, err := h.adminServer.GetObjectStoreUserDetails(username)
if err != nil {
c.JSON(http.StatusNotFound, gin.H{"error": "User not found: " + err.Error()})
writeJSONError(w, http.StatusNotFound, "User not found: "+err.Error())
return
}
c.JSON(http.StatusOK, user)
writeJSON(w, http.StatusOK, user)
}
// CreateAccessKey creates a new access key for a user
func (h *UserHandlers) CreateAccessKey(c *gin.Context) {
username := c.Param("username")
func (h *UserHandlers) CreateAccessKey(w http.ResponseWriter, r *http.Request) {
username := mux.Vars(r)["username"]
if username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
accessKey, err := h.adminServer.CreateAccessKey(username)
if err != nil {
glog.Errorf("Failed to create access key for user %s: %v", username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to create access key: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to create access key: "+err.Error())
return
}
c.JSON(http.StatusCreated, gin.H{
writeJSON(w, http.StatusCreated, map[string]interface{}{
"message": "Access key created successfully",
"access_key": accessKey,
})
}
// DeleteAccessKey deletes an access key for a user
func (h *UserHandlers) DeleteAccessKey(c *gin.Context) {
username := c.Param("username")
accessKeyId := c.Param("accessKeyId")
func (h *UserHandlers) DeleteAccessKey(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
username := vars["username"]
accessKeyId := vars["accessKeyId"]
if username == "" || accessKeyId == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username and access key ID are required"})
writeJSONError(w, http.StatusBadRequest, "Username and access key ID are required")
return
}
err := h.adminServer.DeleteAccessKey(username, accessKeyId)
if err != nil {
glog.Errorf("Failed to delete access key %s for user %s: %v", accessKeyId, username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to delete access key: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to delete access key: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Access key deleted successfully",
})
}
// UpdateAccessKeyStatus updates the status of an access key for a user
func (h *UserHandlers) UpdateAccessKeyStatus(c *gin.Context) {
username := c.Param("username")
accessKeyId := c.Param("accessKeyId")
func (h *UserHandlers) UpdateAccessKeyStatus(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
username := vars["username"]
accessKeyId := vars["accessKeyId"]
if username == "" || accessKeyId == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username and access key ID are required"})
writeJSONError(w, http.StatusBadRequest, "Username and access key ID are required")
return
}
var req dash.UpdateAccessKeyStatusRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
// Validate status
if req.Status != dash.AccessKeyStatusActive && req.Status != dash.AccessKeyStatusInactive {
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("Status must be '%s' or '%s'", dash.AccessKeyStatusActive, dash.AccessKeyStatusInactive)})
writeJSONError(w, http.StatusBadRequest, fmt.Sprintf("Status must be '%s' or '%s'", dash.AccessKeyStatusActive, dash.AccessKeyStatusInactive))
return
}
err := h.adminServer.UpdateAccessKeyStatus(username, accessKeyId, req.Status)
if err != nil {
glog.Errorf("Failed to update access key status %s for user %s: %v", accessKeyId, username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update access key status: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to update access key status: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "Access key updated successfully",
})
}
// GetUserPolicies returns the policies for a user
func (h *UserHandlers) GetUserPolicies(c *gin.Context) {
username := c.Param("username")
func (h *UserHandlers) GetUserPolicies(w http.ResponseWriter, r *http.Request) {
username := mux.Vars(r)["username"]
if username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
policies, err := h.adminServer.GetUserPolicies(username)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to get user policies: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to get user policies: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{"policies": policies})
writeJSON(w, http.StatusOK, map[string]interface{}{"policies": policies})
}
// UpdateUserPolicies updates the policies for a user
func (h *UserHandlers) UpdateUserPolicies(c *gin.Context) {
username := c.Param("username")
func (h *UserHandlers) UpdateUserPolicies(w http.ResponseWriter, r *http.Request) {
username := mux.Vars(r)["username"]
if username == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "Username is required"})
writeJSONError(w, http.StatusBadRequest, "Username is required")
return
}
var req dash.UpdateUserPoliciesRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "Invalid request: " + err.Error()})
if err := decodeJSONBody(newJSONMaxReader(w, r), &req); err != nil {
writeJSONError(w, http.StatusBadRequest, "Invalid request: "+err.Error())
return
}
err := h.adminServer.UpdateUserPolicies(username, req.Actions)
if err != nil {
glog.Errorf("Failed to update policies for user %s: %v", username, err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to update user policies: " + err.Error()})
writeJSONError(w, http.StatusInternalServerError, "Failed to update user policies: "+err.Error())
return
}
c.JSON(http.StatusOK, gin.H{
writeJSON(w, http.StatusOK, map[string]interface{}{
"message": "User policies updated successfully",
})
}
// getObjectStoreUsersData retrieves object store users data from the server
func (h *UserHandlers) getObjectStoreUsersData(c *gin.Context) dash.ObjectStoreUsersData {
username := c.GetString("username")
func (h *UserHandlers) getObjectStoreUsersData(r *http.Request) dash.ObjectStoreUsersData {
username := dash.UsernameFromContext(r.Context())
if username == "" {
username = "admin"
}
// Get object store users
users, err := h.adminServer.GetObjectStoreUsers(c.Request.Context())
users, err := h.adminServer.GetObjectStoreUsers(r.Context())
if err != nil {
glog.Errorf("Failed to get object store users: %v", err)
// Return empty data on error
+42
View File
@@ -0,0 +1,42 @@
package httputil
import (
"encoding/json"
"io"
"net/http"
"github.com/seaweedfs/seaweedfs/weed/glog"
)
const MaxJSONBodyBytes = 1 << 20
func NewJSONMaxReader(w http.ResponseWriter, r *http.Request) io.Reader {
return http.MaxBytesReader(w, r.Body, MaxJSONBodyBytes)
}
func DecodeJSONBody(r io.Reader, v interface{}) error {
decoder := json.NewDecoder(r)
return decoder.Decode(v)
}
func WriteJSON(w http.ResponseWriter, status int, payload interface{}) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
if payload == nil {
return
}
if err := json.NewEncoder(w).Encode(payload); err != nil {
glog.Errorf("failed to encode JSON response (status=%d, payload=%T): %v", status, payload, err)
}
}
func WriteJSONError(w http.ResponseWriter, status int, message string) {
WriteJSON(w, status, map[string]string{"error": message})
}
func DefaultQuery(value, fallback string) string {
if value == "" {
return fallback
}
return value
}
+151 -59
View File
@@ -130,10 +130,10 @@ func (mq *MaintenanceQueue) cleanupCompletedTasks() {
// AddTask adds a new maintenance task to the queue with deduplication
func (mq *MaintenanceQueue) AddTask(task *MaintenanceTask) {
mq.mutex.Lock()
defer mq.mutex.Unlock()
// Check for duplicate tasks (same type + volume + not completed)
if mq.hasDuplicateTask(task) {
mq.mutex.Unlock()
glog.V(1).Infof("Task skipped (duplicate): %s for volume %d on %s (already queued or running)",
task.Type, task.VolumeID, task.Server)
return
@@ -169,16 +169,23 @@ func (mq *MaintenanceQueue) AddTask(task *MaintenanceTask) {
return mq.pendingTasks[i].ScheduledAt.Before(mq.pendingTasks[j].ScheduledAt)
})
// Save task state to persistence
mq.saveTaskState(task)
scheduleInfo := ""
if !task.ScheduledAt.IsZero() && time.Until(task.ScheduledAt) > time.Minute {
scheduleInfo = fmt.Sprintf(", scheduled for %v", task.ScheduledAt.Format("15:04:05"))
}
// Snapshot task state while lock is still held to avoid data race;
// also capture log fields from the snapshot so the live task pointer
// is not accessed after mq.mutex is released.
taskSnapshot := snapshotTask(task)
mq.mutex.Unlock()
// Save task state to persistence outside the lock to avoid blocking
// RegisterWorker and HTTP handlers (GetTasks) during disk I/O
mq.saveTaskState(taskSnapshot)
glog.Infof("Task queued: %s (%s) volume %d on %s, priority %d%s, reason: %s",
task.ID, task.Type, task.VolumeID, task.Server, task.Priority, scheduleInfo, task.Reason)
taskSnapshot.ID, taskSnapshot.Type, taskSnapshot.VolumeID, taskSnapshot.Server, taskSnapshot.Priority, scheduleInfo, taskSnapshot.Reason)
}
// hasDuplicateTask checks if a similar task already exists (same type, volume, and not completed)
@@ -286,11 +293,14 @@ func (mq *MaintenanceQueue) GetNextTask(workerID string, capabilities []Maintena
// Now acquire write lock to actually assign the task
mq.mutex.Lock()
defer mq.mutex.Unlock()
// Capture ID before the re-check so it is available for logging after unlock.
selectedTaskID := selectedTask.ID
// Re-check that the task is still available (it might have been assigned to another worker)
if selectedIndex >= len(mq.pendingTasks) || mq.pendingTasks[selectedIndex].ID != selectedTask.ID {
glog.V(2).Infof("Task %s no longer available for worker %s: assigned to another worker", selectedTask.ID, workerID)
if selectedIndex >= len(mq.pendingTasks) || mq.pendingTasks[selectedIndex].ID != selectedTaskID {
mq.mutex.Unlock()
glog.V(2).Infof("Task %s no longer available for worker %s: assigned to another worker", selectedTaskID, workerID)
return nil
}
@@ -331,6 +341,7 @@ func (mq *MaintenanceQueue) GetNextTask(workerID string, capabilities []Maintena
if len(selectedTask.AssignmentHistory) > 0 {
selectedTask.AssignmentHistory = selectedTask.AssignmentHistory[:len(selectedTask.AssignmentHistory)-1]
}
mq.mutex.Unlock()
// Return nil so the task is not removed from pendingTasks and not returned to the worker
return nil
}
@@ -348,11 +359,15 @@ func (mq *MaintenanceQueue) GetNextTask(workerID string, capabilities []Maintena
// Track pending operation
mq.trackPendingOperation(selectedTask)
// Save task state after assignment
mq.saveTaskState(selectedTask)
// Snapshot task state while lock is still held to avoid data race
selectedSnapshot := snapshotTask(selectedTask)
mq.mutex.Unlock()
// Save task state to persistence outside the lock
mq.saveTaskState(selectedSnapshot)
glog.Infof("Task assigned: %s (%s) → worker %s (volume %d, server %s)",
selectedTask.ID, selectedTask.Type, workerID, selectedTask.VolumeID, selectedTask.Server)
selectedSnapshot.ID, selectedSnapshot.Type, workerID, selectedSnapshot.VolumeID, selectedSnapshot.Server)
return selectedTask
}
@@ -360,10 +375,10 @@ func (mq *MaintenanceQueue) GetNextTask(workerID string, capabilities []Maintena
// CompleteTask marks a task as completed
func (mq *MaintenanceQueue) CompleteTask(taskID string, error string) {
mq.mutex.Lock()
defer mq.mutex.Unlock()
task, exists := mq.tasks[taskID]
if !exists {
mq.mutex.Unlock()
glog.Warningf("Attempted to complete non-existent task: %s", taskID)
return
}
@@ -388,6 +403,12 @@ func (mq *MaintenanceQueue) CompleteTask(taskID string, error string) {
duration = completedTime.Sub(*task.StartedAt)
}
// Capture workerID before it may be cleared during retry
originalWorkerID := task.WorkerID
var taskToSave *MaintenanceTask
var logFn func()
if error != "" {
task.Status = TaskStatusFailed
task.Error = error
@@ -420,10 +441,12 @@ func (mq *MaintenanceQueue) CompleteTask(taskID string, error string) {
mq.integration.SyncTask(task)
}
// Save task state after retry setup
mq.saveTaskState(task)
glog.Warningf("Task failed, scheduling retry: %s (%s) attempt %d/%d, worker %s, duration %v, error: %s",
taskID, task.Type, task.RetryCount, task.MaxRetries, task.WorkerID, duration, error)
taskToSave = task
retryCount, maxRetries := task.RetryCount, task.MaxRetries
logFn = func() {
glog.Warningf("Task failed, scheduling retry: %s (%s) attempt %d/%d, worker %s, duration %v, error: %s",
taskID, task.Type, retryCount, maxRetries, originalWorkerID, duration, error)
}
} else {
// Record unassignment due to permanent failure
if task.WorkerID != "" && len(task.AssignmentHistory) > 0 {
@@ -435,23 +458,27 @@ func (mq *MaintenanceQueue) CompleteTask(taskID string, error string) {
}
}
// Save task state after permanent failure
mq.saveTaskState(task)
glog.Errorf("Task failed permanently: %s (%s) worker %s, duration %v, after %d retries: %s",
taskID, task.Type, task.WorkerID, duration, task.MaxRetries, error)
taskToSave = task
maxRetries := task.MaxRetries
logFn = func() {
glog.Errorf("Task failed permanently: %s (%s) worker %s, duration %v, after %d retries: %s",
taskID, task.Type, originalWorkerID, duration, maxRetries, error)
}
}
} else {
task.Status = TaskStatusCompleted
task.Progress = 100
// Save task state after successful completion
mq.saveTaskState(task)
glog.Infof("Task completed: %s (%s) worker %s, duration %v, volume %d",
taskID, task.Type, task.WorkerID, duration, task.VolumeID)
taskToSave = task
volumeID := task.VolumeID
logFn = func() {
glog.Infof("Task completed: %s (%s) worker %s, duration %v, volume %d",
taskID, task.Type, originalWorkerID, duration, volumeID)
}
}
// Update worker
if task.WorkerID != "" {
if worker, exists := mq.workers[task.WorkerID]; exists {
// Update worker load and capture state before releasing lock
if originalWorkerID != "" {
if worker, exists := mq.workers[originalWorkerID]; exists {
worker.CurrentTask = nil
worker.CurrentLoad--
if worker.CurrentLoad == 0 {
@@ -459,16 +486,32 @@ func (mq *MaintenanceQueue) CompleteTask(taskID string, error string) {
}
}
}
taskStatus := task.Status
taskCount := len(mq.tasks)
// Snapshot task state while lock is still held to avoid data race
var taskToSaveSnapshot *MaintenanceTask
if taskToSave != nil {
taskToSaveSnapshot = snapshotTask(taskToSave)
}
mq.mutex.Unlock()
// Save task state to persistence outside the lock
if taskToSaveSnapshot != nil {
mq.saveTaskState(taskToSaveSnapshot)
}
if logFn != nil {
logFn()
}
// Remove pending operation (unless it's being retried)
if task.Status != TaskStatusPending {
if taskStatus != TaskStatusPending {
mq.removePendingOperation(taskID)
}
// Periodically cleanup old completed tasks (every 10th completion)
if task.Status == TaskStatusCompleted {
// Simple counter-based trigger for cleanup
if len(mq.tasks)%10 == 0 {
// Periodically cleanup old completed tasks (when total task count is a multiple of 10)
if taskStatus == TaskStatusCompleted {
if taskCount%10 == 0 {
go mq.cleanupCompletedTasks()
}
}
@@ -476,35 +519,46 @@ func (mq *MaintenanceQueue) CompleteTask(taskID string, error string) {
// UpdateTaskProgress updates the progress of a running task
func (mq *MaintenanceQueue) UpdateTaskProgress(taskID string, progress float64) {
mq.mutex.RLock()
defer mq.mutex.RUnlock()
mq.mutex.Lock()
if task, exists := mq.tasks[taskID]; exists {
oldProgress := task.Progress
task.Progress = progress
task.Status = TaskStatusInProgress
// Update pending operation status
mq.updatePendingOperationStatus(taskID, "in_progress")
// Log progress at significant milestones or changes
if progress == 0 {
glog.V(1).Infof("Task started: %s (%s) worker %s, volume %d",
taskID, task.Type, task.WorkerID, task.VolumeID)
} else if progress >= 100 {
glog.V(1).Infof("Task progress: %s (%s) worker %s, %.1f%% complete",
taskID, task.Type, task.WorkerID, progress)
} else if progress-oldProgress >= 25 { // Log every 25% increment
glog.V(1).Infof("Task progress: %s (%s) worker %s, %.1f%% complete",
taskID, task.Type, task.WorkerID, progress)
}
// Save task state after progress update
if progress == 0 || progress >= 100 || progress-oldProgress >= 10 {
mq.saveTaskState(task)
}
} else {
task, exists := mq.tasks[taskID]
if !exists {
mq.mutex.Unlock()
glog.V(2).Infof("Progress update for unknown task: %s (%.1f%%)", taskID, progress)
return
}
oldProgress := task.Progress
task.Progress = progress
task.Status = TaskStatusInProgress
// Update pending operation status while lock is held
mq.updatePendingOperationStatus(taskID, "in_progress")
// Determine whether to persist and capture log fields before unlocking
shouldSave := progress == 0 || progress >= 100 || progress-oldProgress >= 10
var taskSnapshot *MaintenanceTask
if shouldSave {
taskSnapshot = snapshotTask(task)
}
taskType, workerID, volumeID := task.Type, task.WorkerID, task.VolumeID
mq.mutex.Unlock()
// Log progress at significant milestones or changes
if progress == 0 {
glog.V(1).Infof("Task started: %s (%s) worker %s, volume %d",
taskID, taskType, workerID, volumeID)
} else if progress >= 100 {
glog.V(1).Infof("Task progress: %s (%s) worker %s, %.1f%% complete",
taskID, taskType, workerID, progress)
} else if progress-oldProgress >= 25 { // Log every 25% increment
glog.V(1).Infof("Task progress: %s (%s) worker %s, %.1f%% complete",
taskID, taskType, workerID, progress)
}
// Save task state outside the lock to avoid blocking readers
if taskSnapshot != nil {
mq.saveTaskState(taskSnapshot)
}
}
@@ -1004,3 +1058,41 @@ func (mq *MaintenanceQueue) updatePendingOperationStatus(taskID string, status s
pendingOps.UpdateOperationStatus(taskID, status)
}
// snapshotTask returns a shallow copy of t with slice and map fields deep-copied
// so that the snapshot can be safely passed to saveTaskState after mq.mutex is
// released without racing against concurrent mutations of the live task struct.
// Must be called with mq.mutex held.
func snapshotTask(t *MaintenanceTask) *MaintenanceTask {
cp := *t // copy all scalar / pointer-sized fields
// Deep-copy AssignmentHistory: the slice header and each record pointer.
// Records themselves are never mutated after being appended, so copying
// the pointers is sufficient.
if t.AssignmentHistory != nil {
cp.AssignmentHistory = make([]*TaskAssignmentRecord, len(t.AssignmentHistory))
copy(cp.AssignmentHistory, t.AssignmentHistory)
}
// Deep-copy Tags map to avoid concurrent map read/write.
if t.Tags != nil {
cp.Tags = make(map[string]string, len(t.Tags))
for k, v := range t.Tags {
cp.Tags[k] = v
}
}
// Copy optional time pointers so a concurrent nil-assignment (e.g. retry
// path clearing StartedAt) does not race with maintenanceTaskToProtobuf
// reading the pointed-to value.
if t.StartedAt != nil {
ts := *t.StartedAt
cp.StartedAt = &ts
}
if t.CompletedAt != nil {
tc := *t.CompletedAt
cp.CompletedAt = &tc
}
return &cp
}
+19
View File
@@ -0,0 +1,19 @@
package layout
import "net/http"
// ViewContext contains per-request metadata needed by layout templates.
type ViewContext struct {
Request *http.Request
Username string
CSRFToken string
}
// NewViewContext builds a ViewContext from request metadata.
func NewViewContext(r *http.Request, username, csrfToken string) ViewContext {
return ViewContext{
Request: r,
Username: username,
CSRFToken: csrfToken,
}
}
+10 -7
View File
@@ -4,19 +4,21 @@ import (
"fmt"
"strings"
"time"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/util/version"
)
templ Layout(c *gin.Context, content templ.Component) {
templ Layout(view ViewContext, content templ.Component) {
{{
username := c.GetString("username")
username := view.Username
if username == "" {
username = "admin"
username = "unknown"
}
csrfToken := c.GetString("csrf_token")
csrfToken := view.CSRFToken
currentPath := c.Request.URL.Path
currentPath := ""
if view.Request != nil {
currentPath = view.Request.URL.Path
}
// Detect if we're on a message queue page to keep submenu expanded
isMQPage := strings.HasPrefix(currentPath, "/mq/")
@@ -357,7 +359,7 @@ templ Layout(c *gin.Context, content templ.Component) {
</html>
}
templ LoginForm(c *gin.Context, title string, errorMessage string) {
templ LoginForm(title string, errorMessage string, csrfToken string) {
<!DOCTYPE html>
<html lang="en">
<head>
@@ -388,6 +390,7 @@ templ LoginForm(c *gin.Context, title string, errorMessage string) {
}
<form method="POST" action="/login">
<input type="hidden" name="csrf_token" value={ csrfToken }>
<div class="mb-3">
<label for="username" class="form-label">Username</label>
<div class="input-group">
+32 -17
View File
@@ -10,13 +10,12 @@ import templruntime "github.com/a-h/templ/runtime"
import (
"fmt"
"github.com/gin-gonic/gin"
"github.com/seaweedfs/seaweedfs/weed/util/version"
"strings"
"time"
)
func Layout(c *gin.Context, content templ.Component) templ.Component {
func Layout(view ViewContext, content templ.Component) templ.Component {
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
@@ -37,13 +36,16 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
templ_7745c5c3_Var1 = templ.NopComponent
}
ctx = templ.ClearChildren(ctx)
username := c.GetString("username")
username := view.Username
if username == "" {
username = "admin"
username = "unknown"
}
csrfToken := c.GetString("csrf_token")
csrfToken := view.CSRFToken
currentPath := c.Request.URL.Path
currentPath := ""
if view.Request != nil {
currentPath = view.Request.URL.Path
}
// Detect if we're on a message queue page to keep submenu expanded
isMQPage := strings.HasPrefix(currentPath, "/mq/")
@@ -63,7 +65,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
var templ_7745c5c3_Var2 string
templ_7745c5c3_Var2, templ_7745c5c3_Err = templ.JoinStringErrs(csrfToken)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 39, Col: 47}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 41, Col: 47}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var2))
if templ_7745c5c3_Err != nil {
@@ -76,7 +78,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
var templ_7745c5c3_Var3 string
templ_7745c5c3_Var3, templ_7745c5c3_Err = templ.JoinStringErrs(username)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 70, Col: 73}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 72, Col: 73}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var3))
if templ_7745c5c3_Err != nil {
@@ -111,7 +113,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
var templ_7745c5c3_Var6 string
templ_7745c5c3_Var6, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%t", isClusterPage))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 97, Col: 207}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 99, Col: 207}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var6))
if templ_7745c5c3_Err != nil {
@@ -168,7 +170,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
var templ_7745c5c3_Var11 string
templ_7745c5c3_Var11, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%t", isStoragePage))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 122, Col: 207}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 124, Col: 207}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var11))
if templ_7745c5c3_Err != nil {
@@ -342,7 +344,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
var templ_7745c5c3_Var14 string
templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", time.Now().Year()))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 337, Col: 60}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 339, Col: 60}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14))
if templ_7745c5c3_Err != nil {
@@ -355,7 +357,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
var templ_7745c5c3_Var15 string
templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(version.VERSION_NUMBER)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 337, Col: 102}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 339, Col: 102}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
if templ_7745c5c3_Err != nil {
@@ -379,7 +381,7 @@ func Layout(c *gin.Context, content templ.Component) templ.Component {
})
}
func LoginForm(c *gin.Context, title string, errorMessage string) templ.Component {
func LoginForm(title string, errorMessage string, csrfToken string) templ.Component {
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
@@ -407,7 +409,7 @@ func LoginForm(c *gin.Context, title string, errorMessage string) templ.Componen
var templ_7745c5c3_Var17 string
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(title)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 365, Col: 17}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 367, Col: 17}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
if templ_7745c5c3_Err != nil {
@@ -420,7 +422,7 @@ func LoginForm(c *gin.Context, title string, errorMessage string) templ.Componen
var templ_7745c5c3_Var18 string
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinStringErrs(title)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 379, Col: 57}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 381, Col: 57}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18))
if templ_7745c5c3_Err != nil {
@@ -438,7 +440,7 @@ func LoginForm(c *gin.Context, title string, errorMessage string) templ.Componen
var templ_7745c5c3_Var19 string
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinStringErrs(errorMessage)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 386, Col: 45}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 388, Col: 45}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19))
if templ_7745c5c3_Err != nil {
@@ -449,7 +451,20 @@ func LoginForm(c *gin.Context, title string, errorMessage string) templ.Componen
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "<form method=\"POST\" action=\"/login\"><div class=\"mb-3\"><label for=\"username\" class=\"form-label\">Username</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-user\"></i></span> <input type=\"text\" class=\"form-control\" id=\"username\" name=\"username\" required></div></div><div class=\"mb-4\"><label for=\"password\" class=\"form-label\">Password</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-lock\"></i></span> <input type=\"password\" class=\"form-control\" id=\"password\" name=\"password\" required></div></div><button type=\"submit\" class=\"btn btn-primary w-100\"><i class=\"fas fa-sign-in-alt me-2\"></i>Sign In</button></form></div></div></div></div></div><script src=\"/static/js/bootstrap.bundle.min.js\"></script></body></html>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "<form method=\"POST\" action=\"/login\"><input type=\"hidden\" name=\"csrf_token\" value=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var20 string
templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(csrfToken)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/layout/layout.templ`, Line: 393, Col: 84}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "\"><div class=\"mb-3\"><label for=\"username\" class=\"form-label\">Username</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-user\"></i></span> <input type=\"text\" class=\"form-control\" id=\"username\" name=\"username\" required></div></div><div class=\"mb-4\"><label for=\"password\" class=\"form-label\">Password</label><div class=\"input-group\"><span class=\"input-group-text\"><i class=\"fas fa-lock\"></i></span> <input type=\"password\" class=\"form-control\" id=\"password\" name=\"password\" required></div></div><button type=\"submit\" class=\"btn btn-primary w-100\"><i class=\"fas fa-sign-in-alt me-2\"></i>Sign In</button></form></div></div></div></div></div><script src=\"/static/js/bootstrap.bundle.min.js\"></script></body></html>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
+2 -1
View File
@@ -3,6 +3,7 @@ package cluster
import (
"context"
"github.com/dustin/go-humanize/english"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/pb"
"github.com/seaweedfs/seaweedfs/weed/pb/master_pb"
@@ -20,7 +21,7 @@ func ListExistingPeerUpdates(master pb.ServerAddress, grpcDialOption grpc.DialOp
return err
}
glog.V(0).Infof("the cluster has %d %s\n", len(resp.ClusterNodes), clientType)
glog.V(0).Infof("the cluster has %d %s\n", len(resp.ClusterNodes), english.PluralWord(len(resp.ClusterNodes), clientType, ""))
for _, node := range resp.ClusterNodes {
existingNodes = append(existingNodes, &master_pb.ClusterNodeUpdate{
NodeType: FilerType,
+150 -60
View File
@@ -1,22 +1,24 @@
package command
import (
"bufio"
"context"
"crypto/rand"
"fmt"
"log"
"net"
"net/http"
"os"
"os/signal"
"os/user"
"path/filepath"
"runtime/debug"
"strings"
"syscall"
"time"
"github.com/gin-contrib/sessions"
"github.com/gin-contrib/sessions/cookie"
"github.com/gin-gonic/gin"
"github.com/gorilla/mux"
"github.com/gorilla/sessions"
"github.com/spf13/viper"
"github.com/seaweedfs/seaweedfs/weed/admin"
@@ -232,25 +234,10 @@ func runAdmin(cmd *Command, args []string) bool {
// startAdminServer starts the actual admin server
func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool, icebergPort int) error {
// Set Gin mode
gin.SetMode(gin.ReleaseMode)
// Create router
r := gin.New()
r.Use(gin.LoggerWithFormatter(func(param gin.LogFormatterParams) string {
if param.StatusCode == 200 {
return ""
}
return fmt.Sprintf("[GIN] %v | %3d | %13v | %15s | %-7s %s\n%s",
param.TimeStamp.Format("2006/01/02 - 15:04:05"),
param.StatusCode,
param.Latency,
param.ClientIP,
param.Method,
param.Path,
param.ErrorMessage,
)
}), gin.Recovery())
r := mux.NewRouter()
r.Use(loggingMiddleware)
r.Use(recoveryMiddleware)
// Create data directory first if specified (needed for session key storage)
var dataDir string
@@ -276,30 +263,30 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
// Detect TLS configuration to set Secure cookie flag
cookieSecure := viper.GetString("https.admin.key") != ""
// Session store - load or generate session key
sessionKeyBytes, err := loadOrGenerateSessionKey(dataDir)
// Session store - load or generate session keys
authKey, encKey, err := loadOrGenerateSessionKeys(dataDir)
if err != nil {
return fmt.Errorf("failed to get session key: %w", err)
}
store := cookie.NewStore(sessionKeyBytes)
store := sessions.NewCookieStore(authKey, encKey)
// Configure session options to ensure cookies are properly saved
store.Options(sessions.Options{
store.Options = &sessions.Options{
Path: "/",
MaxAge: 3600 * 24, // 24 hours
HttpOnly: true, // Prevent JavaScript access
Secure: cookieSecure, // Set based on actual TLS configuration
SameSite: http.SameSiteLaxMode,
})
r.Use(sessions.Sessions("admin-session", store))
}
// Static files - serve from embedded filesystem
staticFS, err := admin.GetStaticFS()
if err != nil {
log.Printf("Warning: Failed to load embedded static files: %v", err)
} else {
r.StaticFS("/static", http.FS(staticFS))
staticHandler := http.FileServer(http.FS(staticFS))
r.Handle("/static", http.RedirectHandler("/static/", http.StatusMovedPermanently))
r.PathPrefix("/static/").Handler(http.StripPrefix("/static/", staticHandler))
}
// Create admin server (plugin is always enabled)
@@ -328,7 +315,7 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
// Create handlers and setup routes
authRequired := *options.adminPassword != ""
adminHandlers := handlers.NewAdminHandlers(adminServer)
adminHandlers := handlers.NewAdminHandlers(adminServer, store)
adminHandlers.SetupRoutes(r, authRequired, *options.adminUser, *options.adminPassword, *options.readOnlyUser, *options.readOnlyPassword, enableUI)
// Server configuration
@@ -395,49 +382,156 @@ func startAdminServer(ctx context.Context, options AdminOptions, enableUI bool,
return nil
}
type statusRecorder struct {
http.ResponseWriter
status int
}
func (r *statusRecorder) WriteHeader(status int) {
r.status = status
r.ResponseWriter.WriteHeader(status)
}
func (r *statusRecorder) Write(b []byte) (int, error) {
if r.status == 0 {
r.status = http.StatusOK
}
return r.ResponseWriter.Write(b)
}
func (r *statusRecorder) Flush() {
if f, ok := r.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
func (r *statusRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) {
if h, ok := r.ResponseWriter.(http.Hijacker); ok {
return h.Hijack()
}
return nil, nil, http.ErrNotSupported
}
func (r *statusRecorder) Push(target string, opts *http.PushOptions) error {
if p, ok := r.ResponseWriter.(http.Pusher); ok {
return p.Push(target, opts)
}
return http.ErrNotSupported
}
func (r *statusRecorder) Unwrap() http.ResponseWriter {
return r.ResponseWriter
}
func loggingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
recorder := &statusRecorder{ResponseWriter: w}
next.ServeHTTP(recorder, r)
status := recorder.status
if status == 0 {
status = http.StatusOK
}
if status >= 200 && status < 300 {
return
}
log.Printf("[HTTP] %v | %3d | %13v | %15s | %-7s %s",
time.Now().Format("2006/01/02 - 15:04:05"),
status,
time.Since(start),
r.RemoteAddr,
r.Method,
r.URL.Path,
)
})
}
func recoveryMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer func() {
if err := recover(); err != nil {
log.Printf("panic: %v\n%s", err, debug.Stack())
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
}()
next.ServeHTTP(w, r)
})
}
// GetAdminOptions returns the admin command options for testing
func GetAdminOptions() *AdminOptions {
return &AdminOptions{}
}
// loadOrGenerateSessionKey loads an existing session key from dataDir or generates a new one
func loadOrGenerateSessionKey(dataDir string) ([]byte, error) {
const sessionKeyLength = 32
// loadOrGenerateSessionKeys loads or creates authentication/encryption keys for session cookies.
func loadOrGenerateSessionKeys(dataDir string) ([]byte, []byte, error) {
const keyLen = 32
if dataDir == "" {
// No persistence, generate random key
log.Println("No dataDir specified, generating ephemeral session key")
key := make([]byte, sessionKeyLength)
_, err := rand.Read(key)
return key, err
// No persistence, generate ephemeral keys
log.Println("No dataDir specified, generating ephemeral session keys")
authKey := make([]byte, keyLen)
encKey := make([]byte, keyLen)
if _, err := rand.Read(authKey); err != nil {
return nil, nil, err
}
if _, err := rand.Read(encKey); err != nil {
return nil, nil, err
}
return authKey, encKey, nil
}
sessionKeyPath := filepath.Join(dataDir, ".session_key")
// Try to load existing key
if data, err := os.ReadFile(sessionKeyPath); err == nil {
if len(data) == sessionKeyLength {
switch len(data) {
case keyLen:
authKey := make([]byte, keyLen)
copy(authKey, data)
encKey := make([]byte, keyLen)
if _, err := rand.Read(encKey); err != nil {
return nil, nil, err
}
log.Printf("Warning: Upgrading session key at %s by adding an encryption key; existing cookies will be invalidated", sessionKeyPath)
combined := append(authKey, encKey...)
if err := os.WriteFile(sessionKeyPath, combined, 0600); err != nil {
log.Printf("Warning: Failed to persist upgraded session key: %v", err)
} else {
log.Printf("Upgraded session key file to include encryption key: %s", sessionKeyPath)
}
return authKey, encKey, nil
case 2 * keyLen:
authKey := make([]byte, keyLen)
encKey := make([]byte, keyLen)
copy(authKey, data[:keyLen])
copy(encKey, data[keyLen:])
log.Printf("Loaded persisted session key from %s", sessionKeyPath)
return data, nil
return authKey, encKey, nil
default:
log.Printf("Warning: Invalid session key file (expected %d or %d bytes, got %d), generating new key", keyLen, 2*keyLen, len(data))
}
log.Printf("Warning: Invalid session key file (expected %d bytes, got %d), generating new key", sessionKeyLength, len(data))
} else if !os.IsNotExist(err) {
log.Printf("Warning: Failed to read session key from %s: %v. A new key will be generated.", sessionKeyPath, err)
}
// Generate new key
key := make([]byte, sessionKeyLength)
key := make([]byte, 2*keyLen)
if _, err := rand.Read(key); err != nil {
return nil, err
return nil, nil, err
}
// Save key for future use
if err := os.WriteFile(sessionKeyPath, key, 0600); err != nil {
log.Printf("Warning: Failed to persist session key: %v", err)
} else {
log.Printf("Generated and persisted new session key to %s", sessionKeyPath)
}
return key, nil
return key[:keyLen], key[keyLen:], nil
}
// expandHomeDir expands the tilde (~) in a path to the user's home directory
@@ -466,20 +560,16 @@ func expandHomeDir(path string) (string, error) {
}
// Handle ~username/ patterns
if strings.HasPrefix(path, "~") {
parts := strings.SplitN(path[1:], "/", 2)
username := parts[0]
parts := strings.SplitN(path[1:], "/", 2)
username := parts[0]
targetUser, err := user.Lookup(username)
if err != nil {
return "", fmt.Errorf("user %s not found: %v", username, err)
}
if len(parts) == 1 {
return targetUser.HomeDir, nil
}
return filepath.Join(targetUser.HomeDir, parts[1]), nil
targetUser, err := user.Lookup(username)
if err != nil {
return "", fmt.Errorf("user %s not found: %v", username, err)
}
return path, nil
if len(parts) == 1 {
return targetUser.HomeDir, nil
}
return filepath.Join(targetUser.HomeDir, parts[1]), nil
}
-2
View File
@@ -37,12 +37,10 @@ var Commands = []*Command{
cmdMqAgent,
cmdMqBroker,
cmdMqKafkaGateway,
cmdDB,
cmdS3,
cmdScaffold,
cmdServer,
cmdShell,
cmdSql,
cmdUpdate,
cmdUpload,
cmdVersion,
+45 -35
View File
@@ -25,8 +25,8 @@ import (
)
var (
copy CopyOptions
waitGroup sync.WaitGroup
copyOptions CopyOptions
waitGroup sync.WaitGroup
)
type CopyOptions struct {
@@ -50,17 +50,17 @@ type CopyOptions struct {
func init() {
cmdFilerCopy.Run = runCopy // break init cycle
cmdFilerCopy.IsDebug = cmdFilerCopy.Flag.Bool("debug", false, "verbose debug information")
copy.include = cmdFilerCopy.Flag.String("include", "", "pattens of files to copy, e.g., *.pdf, *.html, ab?d.txt, works together with -dir")
copy.replication = cmdFilerCopy.Flag.String("replication", "", "replication type")
copy.collection = cmdFilerCopy.Flag.String("collection", "", "optional collection name")
copy.ttl = cmdFilerCopy.Flag.String("ttl", "", "time to live, e.g.: 1m, 1h, 1d, 1M, 1y")
copy.diskType = cmdFilerCopy.Flag.String("disk", "", "[hdd|ssd|<tag>] hard drive or solid state drive or any tag")
copy.maxMB = cmdFilerCopy.Flag.Int("maxMB", 4, "split files larger than the limit")
copy.concurrentFiles = cmdFilerCopy.Flag.Int("c", 8, "concurrent file copy goroutines")
copy.concurrentChunks = cmdFilerCopy.Flag.Int("concurrentChunks", 8, "concurrent chunk copy goroutines for each file")
copy.checkSize = cmdFilerCopy.Flag.Bool("check.size", false, "copy when the target file size is different from the source file")
copy.verbose = cmdFilerCopy.Flag.Bool("verbose", false, "print out details during copying")
copy.volumeServerAccess = cmdFilerCopy.Flag.String("volumeServerAccess", "direct", "access volume servers by [direct|publicUrl]")
copyOptions.include = cmdFilerCopy.Flag.String("include", "", "patterns of files to copy, e.g., *.pdf, *.html, ab?d.txt, works together with -dir")
copyOptions.replication = cmdFilerCopy.Flag.String("replication", "", "replication type")
copyOptions.collection = cmdFilerCopy.Flag.String("collection", "", "optional collection name")
copyOptions.ttl = cmdFilerCopy.Flag.String("ttl", "", "time to live, e.g.: 1m, 1h, 1d, 1M, 1y")
copyOptions.diskType = cmdFilerCopy.Flag.String("disk", "", "[hdd|ssd|<tag>] hard drive or solid state drive or any tag")
copyOptions.maxMB = cmdFilerCopy.Flag.Int("maxMB", 4, "split files larger than the limit")
copyOptions.concurrentFiles = cmdFilerCopy.Flag.Int("c", 8, "concurrent file copy goroutines")
copyOptions.concurrentChunks = cmdFilerCopy.Flag.Int("concurrentChunks", 8, "concurrent chunk copy goroutines for each file")
copyOptions.checkSize = cmdFilerCopy.Flag.Bool("check.size", false, "copy when the target file size is different from the source file")
copyOptions.verbose = cmdFilerCopy.Flag.Bool("verbose", false, "print out details during copying")
copyOptions.volumeServerAccess = cmdFilerCopy.Flag.String("volumeServerAccess", "direct", "access volume servers by [direct|publicUrl]")
}
var cmdFilerCopy = &Command{
@@ -99,9 +99,9 @@ func runCopy(cmd *Command, args []string) bool {
return false
}
copy.grpcDialOption = security.LoadClientTLS(util.GetViper(), "grpc.client")
copyOptions.grpcDialOption = security.LoadClientTLS(util.GetViper(), "grpc.client")
masters, collection, replication, dirBuckets, maxMB, cipher, err := readFilerConfiguration(copy.grpcDialOption, filerAddress)
masters, collection, replication, dirBuckets, maxMB, cipher, err := readFilerConfiguration(copyOptions.grpcDialOption, filerAddress)
if err != nil {
fmt.Printf("read from filer %s: %v\n", filerAddress, err)
return false
@@ -110,38 +110,45 @@ func runCopy(cmd *Command, args []string) bool {
restPath := urlPath[len(dirBuckets)+1:]
if strings.Index(restPath, "/") > 0 {
expectedBucket := restPath[:strings.Index(restPath, "/")]
if *copy.collection == "" {
*copy.collection = expectedBucket
} else if *copy.collection != expectedBucket {
fmt.Printf("destination %s uses collection \"%s\": unexpected collection \"%v\"\n", urlPath, expectedBucket, *copy.collection)
if *copyOptions.collection == "" {
*copyOptions.collection = expectedBucket
} else if *copyOptions.collection != expectedBucket {
fmt.Printf("destination %s uses collection \"%s\": unexpected collection \"%v\"\n", urlPath, expectedBucket, *copyOptions.collection)
return true
}
}
}
if *copy.collection == "" {
*copy.collection = collection
if *copyOptions.collection == "" {
*copyOptions.collection = collection
}
if *copy.replication == "" {
*copy.replication = replication
if *copyOptions.replication == "" {
*copyOptions.replication = replication
}
if *copy.maxMB == 0 {
*copy.maxMB = int(maxMB)
if *copyOptions.maxMB == 0 {
*copyOptions.maxMB = int(maxMB)
}
copy.masters = masters
copy.cipher = cipher
copyOptions.masters = masters
copyOptions.cipher = cipher
ttl, err := needle.ReadTTL(*copy.ttl)
ttl, err := needle.ReadTTL(*copyOptions.ttl)
if err != nil {
fmt.Printf("parsing ttl %s: %v\n", *copy.ttl, err)
fmt.Printf("parsing ttl %s: %v\n", *copyOptions.ttl, err)
return false
}
copy.ttlSec = int32(ttl.Minutes()) * 60
copyOptions.ttlSec = int32(ttl.Minutes()) * 60
if *cmdFilerCopy.IsDebug {
grace.SetupProfiling("filer.copy.cpu.pprof", "filer.copy.mem.pprof")
grace.SetupProfiling("filer.copyOptions.cpu.pprof", "filer.copyOptions.mem.pprof")
}
fileCopyTaskChan := make(chan FileCopyTask, *copy.concurrentFiles)
concurrentFiles := *copyOptions.concurrentFiles
if concurrentFiles <= 0 {
fmt.Fprintf(os.Stderr, "Invalid concurrency %d; using at least 1 worker\n", concurrentFiles)
concurrentFiles = 1
}
*copyOptions.concurrentFiles = concurrentFiles
fileCopyTaskChan := make(chan FileCopyTask, concurrentFiles)
go func() {
defer close(fileCopyTaskChan)
@@ -152,12 +159,12 @@ func runCopy(cmd *Command, args []string) bool {
}
}
}()
for i := 0; i < *copy.concurrentFiles; i++ {
for i := 0; i < concurrentFiles; i++ {
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
worker := FileCopyWorker{
options: &copy,
options: &copyOptions,
filerAddress: filerAddress,
signature: util.RandomInt32(),
}
@@ -481,8 +488,11 @@ func (worker *FileCopyWorker) uploadFileInChunks(task FileCopyTask, f *os.File,
for _, chunk := range chunks {
fileIds = append(fileIds, chunk.FileId)
}
if len(worker.options.masters) == 0 {
return fmt.Errorf("upload data %v: %w (cleanup skipped: no masters configured)", fileName, uploadError)
}
operation.DeleteFileIds(func(_ context.Context) pb.ServerAddress {
return pb.ServerAddress(copy.masters[0])
return pb.ServerAddress(worker.options.masters[0])
}, false, worker.options.grpcDialOption, fileIds)
return uploadError
}
+12
View File
@@ -248,6 +248,18 @@ func (m *IAMManager) CreateRole(ctx context.Context, filerAddress string, roleNa
return m.roleStore.StoreRole(ctx, "", roleName, roleDef)
}
// GetRole retrieves a role definition by name.
func (m *IAMManager) GetRole(ctx context.Context, roleName string) (*RoleDefinition, error) {
if !m.initialized {
return nil, fmt.Errorf("IAM manager not initialized")
}
if roleName == "" {
return nil, fmt.Errorf("role name cannot be empty")
}
return m.roleStore.GetRole(ctx, m.getFilerAddress(), roleName)
}
// UpdateBucketPolicy updates the policy for a bucket
func (m *IAMManager) UpdateBucketPolicy(ctx context.Context, bucketName string, policyJSON []byte) error {
if !m.initialized {
+12 -38
View File
@@ -788,25 +788,9 @@ func extractSignedHeaders(signedHeaders []string, r *http.Request) (http.Header,
func extractHostHeader(r *http.Request) string {
forwardedHost := r.Header.Get("X-Forwarded-Host")
forwardedPort := r.Header.Get("X-Forwarded-Port")
forwardedProto := r.Header.Get("X-Forwarded-Proto")
// Determine the effective scheme with correct order of precedence:
// 1. X-Forwarded-Proto (most authoritative, reflects client's original protocol)
// 2. r.TLS (authoritative for direct connection to server)
// 3. r.URL.Scheme (fallback, may not always be set correctly)
// 4. Default to "http"
scheme := "http"
if r.URL.Scheme != "" {
scheme = r.URL.Scheme
}
if r.TLS != nil {
scheme = "https"
}
if forwardedProto != "" {
scheme = forwardedProto
}
var host, port string
explicitPort := false
if forwardedHost != "" {
// X-Forwarded-Host can be a comma-separated list of hosts when there are multiple proxies.
// Use only the first host in the list and trim spaces for robustness.
@@ -815,12 +799,16 @@ func extractHostHeader(r *http.Request) string {
} else {
host = strings.TrimSpace(forwardedHost)
}
// Baseline port from forwarded port if available
if forwardedPort != "" {
port = forwardedPort
explicitPort = true
}
// If the host itself contains a port, it should take precedence
if h, p, err := net.SplitHostPort(host); err == nil {
host = h
port = p
}
if forwardedPort != "" && isDefaultPort(scheme, port) {
port = forwardedPort
explicitPort = true
}
} else {
host = r.Host
@@ -830,12 +818,13 @@ func extractHostHeader(r *http.Request) string {
if h, p, err := net.SplitHostPort(host); err == nil {
host = h
port = p
explicitPort = true
}
}
// If we have a non-default port, join it with the host.
// If a port was explicitly provided, join it with the host.
// net.JoinHostPort will handle bracketing for IPv6.
if port != "" && !isDefaultPort(scheme, port) {
if explicitPort && port != "" {
// Strip existing brackets before calling JoinHostPort, which automatically adds
// brackets for IPv6 addresses. This prevents double-bracketing like [[::1]]:8080.
// Using Trim handles both well-formed and malformed bracketed hosts.
@@ -843,7 +832,7 @@ func extractHostHeader(r *http.Request) string {
return net.JoinHostPort(host, port)
}
// No port or default port was stripped. According to AWS SDK behavior (aws-sdk-go-v2),
// No explicit port was provided (or port was empty). According to AWS SDK behavior (aws-sdk-go-v2),
// when a default port is removed from an IPv6 address, the brackets should also be removed.
// This matches AWS S3 signature calculation requirements.
// Reference: https://github.com/aws/aws-sdk-go-v2/blob/main/aws/signer/internal/v4/host.go
@@ -855,21 +844,6 @@ func extractHostHeader(r *http.Request) string {
return host
}
func isDefaultPort(scheme, port string) bool {
if port == "" {
return true
}
switch port {
case "80":
return strings.EqualFold(scheme, "http")
case "443":
return strings.EqualFold(scheme, "https")
default:
return false
}
}
// getScope generate a string of a specific date, an AWS region, and a service.
func getScope(t time.Time, region string, service string) string {
scope := strings.Join([]string{
+16 -8
View File
@@ -208,7 +208,7 @@ func TestExtractHostHeader(t *testing.T) {
forwardedHost: "example.com",
forwardedPort: "80",
forwardedProto: "http",
expected: "example.com",
expected: "example.com:80",
},
{
name: "X-Forwarded-Host with X-Forwarded-Port (HTTPS standard port 443)",
@@ -216,7 +216,7 @@ func TestExtractHostHeader(t *testing.T) {
forwardedHost: "example.com",
forwardedPort: "443",
forwardedProto: "https",
expected: "example.com",
expected: "example.com:443",
},
// Issue #6649: X-Forwarded-Host already contains port (Traefik/HAProxy style)
{
@@ -227,6 +227,14 @@ func TestExtractHostHeader(t *testing.T) {
forwardedProto: "https",
expected: "127.0.0.1:8433",
},
{
name: "X-Forwarded-Host with standard port already included (HTTPS 443)",
hostHeader: "backend:8333",
forwardedHost: "example.com:443",
forwardedPort: "443",
forwardedProto: "https",
expected: "example.com:443",
},
{
name: "X-Forwarded-Host with port, no X-Forwarded-Port header",
hostHeader: "backend:8333",
@@ -253,20 +261,20 @@ func TestExtractHostHeader(t *testing.T) {
expected: "[::1]:8080",
},
{
name: "IPv6 address without brackets and standard port, should strip brackets per AWS SDK",
name: "IPv6 address without brackets and standard port, should include brackets and port when explicit",
hostHeader: "backend:8333",
forwardedHost: "::1",
forwardedPort: "80",
forwardedProto: "http",
expected: "::1",
expected: "[::1]:80",
},
{
name: "IPv6 address without brackets and standard HTTPS port, should strip brackets per AWS SDK",
name: "IPv6 address without brackets and standard HTTPS port, should include brackets and port when explicit",
hostHeader: "backend:8333",
forwardedHost: "2001:db8::1",
forwardedPort: "443",
forwardedProto: "https",
expected: "2001:db8::1",
expected: "[2001:db8::1]:443",
},
{
name: "IPv6 address with brackets but no port, should add port",
@@ -277,12 +285,12 @@ func TestExtractHostHeader(t *testing.T) {
expected: "[2001:db8::1]:8080",
},
{
name: "IPv6 full address with brackets and default port (should strip port and brackets)",
name: "IPv6 full address with brackets and default port (should preserve port if explicit)",
hostHeader: "backend:8333",
forwardedHost: "[2001:db8:85a3::8a2e:370:7334]:443",
forwardedPort: "443",
forwardedProto: "https",
expected: "2001:db8:85a3::8a2e:370:7334",
expected: "[2001:db8:85a3::8a2e:370:7334]:443",
},
{
name: "IPv4-mapped IPv6 address without brackets, should add brackets with port",
+4 -4
View File
@@ -415,13 +415,13 @@ func TestSignatureV4WithoutProxy(t *testing.T) {
name: "HTTP with standard port",
host: "backend:80",
proto: "http",
expectedHost: "backend",
expectedHost: "backend:80",
},
{
name: "HTTPS with standard port",
host: "backend:443",
proto: "https",
expectedHost: "backend",
expectedHost: "backend:443",
},
{
name: "HTTP without port",
@@ -451,13 +451,13 @@ func TestSignatureV4WithoutProxy(t *testing.T) {
name: "IPv6 HTTP with standard port",
host: "[::1]:80",
proto: "http",
expectedHost: "::1",
expectedHost: "[::1]:80",
},
{
name: "IPv6 HTTPS with standard port",
host: "[::1]:443",
proto: "https",
expectedHost: "::1",
expectedHost: "[::1]:443",
},
{
name: "IPv6 HTTP without port",
+10
View File
@@ -47,6 +47,11 @@ type IAMIntegration interface {
DefaultAllow() bool
}
// IAMManagerProvider exposes the IAMManager backing an IAM integration.
type IAMManagerProvider interface {
GetIAMManager() *integration.IAMManager
}
// S3IAMIntegration provides IAM integration for S3 API
type S3IAMIntegration struct {
iamManager *integration.IAMManager
@@ -70,6 +75,11 @@ func NewS3IAMIntegration(iamManager *integration.IAMManager, filerAddress string
}
}
// GetIAMManager returns the IAMManager backing this integration.
func (s3iam *S3IAMIntegration) GetIAMManager() *integration.IAMManager {
return s3iam.iamManager
}
// AuthenticateJWT authenticates JWT tokens using our STS service
func (s3iam *S3IAMIntegration) AuthenticateJWT(ctx context.Context, r *http.Request) (*IAMIdentity, s3err.ErrorCode) {
+79 -1
View File
@@ -47,6 +47,37 @@ type ListBucketResultV2 struct {
StartAfter string `xml:"StartAfter,omitempty"`
}
type listBucketResultV1 struct {
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ ListBucketResult"`
Metadata []MetadataEntry `xml:"Metadata,omitempty"`
Name string `xml:"Name"`
Prefix string `xml:"Prefix"`
Marker string `xml:"Marker"`
NextMarker string `xml:"NextMarker,omitempty"`
MaxKeys int `xml:"MaxKeys"`
Delimiter string `xml:"Delimiter,omitempty"`
IsTruncated bool `xml:"IsTruncated"`
Contents []ListEntry `xml:"Contents,omitempty"`
CommonPrefixes []PrefixEntry `xml:"CommonPrefixes,omitempty"`
EncodingType string `xml:"EncodingType,omitempty"`
}
func toListBucketResultV1(in ListBucketResult) listBucketResultV1 {
return listBucketResultV1{
Metadata: in.Metadata,
Name: in.Name,
Prefix: in.Prefix,
Marker: in.Marker,
NextMarker: in.NextMarker,
MaxKeys: in.MaxKeys,
Delimiter: in.Delimiter,
IsTruncated: in.IsTruncated,
Contents: in.Contents,
CommonPrefixes: in.CommonPrefixes,
EncodingType: in.EncodingType,
}
}
func (s3a *S3ApiServer) ListObjectsV2Handler(w http.ResponseWriter, r *http.Request) {
// https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
@@ -148,6 +179,7 @@ func (s3a *S3ApiServer) ListObjectsV1Handler(w http.ResponseWriter, r *http.Requ
s3err.WriteErrorResponse(w, r, s3err.ErrInternalError)
return
}
sanitizeV1MarkerEcho(&response, marker, encodingTypeUrl)
if len(response.Contents) == 0 {
if exists, existErr := s3a.bucketExists(bucket); existErr == nil && !exists {
@@ -157,7 +189,47 @@ func (s3a *S3ApiServer) ListObjectsV1Handler(w http.ResponseWriter, r *http.Requ
}
glog.V(3).Infof("ListObjectsV1Handler response: %+v", response)
writeSuccessResponseXML(w, r, response)
writeSuccessResponseXML(w, r, toListBucketResultV1(response))
}
func sanitizeV1MarkerEcho(response *ListBucketResult, marker string, encodingTypeUrl bool) {
if marker == "" {
return
}
markerCandidates := map[string]struct{}{
marker: {},
strings.TrimPrefix(marker, "/"): {},
}
if encodingTypeUrl {
escapedMarker := urlPathEscape(strings.TrimPrefix(marker, "/"))
markerCandidates[escapedMarker] = struct{}{}
}
matchesMarker := func(v string) bool {
if _, ok := markerCandidates[v]; ok {
return true
}
_, ok := markerCandidates[strings.TrimPrefix(v, "/")]
return ok
}
if len(response.Contents) > 0 {
filtered := response.Contents[:0]
for _, content := range response.Contents {
if matchesMarker(content.Key) {
continue
}
filtered = append(filtered, content)
}
response.Contents = filtered
}
// doListFilerEntries advances nextMarker to the last emitted entry and skips
// the marker in exclusive mode. So NextMarker==marker indicates no progress.
if matchesMarker(response.NextMarker) && len(response.Contents) == 0 && len(response.CommonPrefixes) == 0 {
response.NextMarker = ""
response.IsTruncated = false
}
}
func (s3a *S3ApiServer) listFilerEntries(bucket string, originalPrefix string, maxKeys uint16, originalMarker string, delimiter string, encodingTypeUrl bool, fetchOwner bool) (response ListBucketResult, err error) {
@@ -543,6 +615,12 @@ func (s3a *S3ApiServer) doListFilerEntries(client filer_pb.SeaweedFilerClient, d
}
}
entry := resp.Entry
// listFilerEntries always calls doListFilerEntries with inclusiveStartFrom=false
// (S3 marker semantics are exclusive), but keep the guard explicit to preserve
// behavior if inclusive callers are introduced in the future.
if !inclusiveStartFrom && marker != "" && entry.Name == marker {
continue
}
if cursor.maxKeys <= 0 {
cursor.isTruncated = true
@@ -64,6 +64,48 @@ func (c *testFilerClient) ListEntries(ctx context.Context, in *filer_pb.ListEntr
return &testListEntriesStream{entries: entries}, nil
}
type markerEchoFilerClient struct {
filer_pb.SeaweedFilerClient
entriesByDir map[string][]*filer_pb.Entry
returnFollowing bool
}
// markerEchoFilerClient intentionally ignores request Limit/InclusiveStartFrom
// and simulates a backend that may echo StartFromFileName. entriesByDir controls
// returned entries; returnFollowing controls whether ListEntries returns only the
// echoed marker or the echoed marker plus following entries.
func (c *markerEchoFilerClient) ListEntries(ctx context.Context, in *filer_pb.ListEntriesRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[filer_pb.ListEntriesResponse], error) {
entries := c.entriesByDir[in.Directory]
ensureEntryAttributes(entries)
if in.StartFromFileName == "" {
return &testListEntriesStream{entries: entries}, nil
}
for i, e := range entries {
if e.Name == in.StartFromFileName {
// Emulate buggy backend behavior: return marker again even when exclusive.
if c.returnFollowing {
echoAndFollowing := entries[i:]
return &testListEntriesStream{entries: echoAndFollowing}, nil
}
return &testListEntriesStream{entries: []*filer_pb.Entry{e}}, nil
}
}
return &testListEntriesStream{entries: nil}, nil
}
func ensureEntryAttributes(entries []*filer_pb.Entry) {
for _, entry := range entries {
if entry == nil {
continue
}
if entry.Attributes == nil {
entry.Attributes = &filer_pb.FuseAttributes{}
}
}
}
func TestListObjectsHandler(t *testing.T) {
// https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
@@ -96,6 +138,20 @@ func TestListObjectsHandler(t *testing.T) {
}
}
func TestListObjectsV1NamespaceResponse(t *testing.T) {
response := ListBucketResult{
Name: "test_container",
Prefix: "",
Marker: "",
NextMarker: "",
MaxKeys: 1000,
IsTruncated: false,
}
encoded := string(s3err.EncodeXMLResponse(toListBucketResultV1(response)))
assert.Contains(t, encoded, `<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">`)
}
func Test_normalizePrefixMarker(t *testing.T) {
type args struct {
prefix string
@@ -253,6 +309,52 @@ func TestDoListFilerEntries_BucketRootPrefixSlashDelimiterSlash_ListsDirectories
assert.Contains(t, seen, "Veeam")
}
func TestDoListFilerEntries_ExclusiveStartSkipsMarkerEcho(t *testing.T) {
s3a := &S3ApiServer{}
client := &markerEchoFilerClient{
entriesByDir: map[string][]*filer_pb.Entry{
"/buckets/test-bucket": {
{Name: "file.txt", Attributes: &filer_pb.FuseAttributes{}},
{Name: "test.txt", Attributes: &filer_pb.FuseAttributes{}},
},
},
}
cursor := &ListingCursor{maxKeys: 1000}
var seen []string
nextMarker, err := s3a.doListFilerEntries(client, "/buckets/test-bucket", "", cursor, "test.txt", "", false, "test-bucket", func(dir string, entry *filer_pb.Entry) {
seen = append(seen, entry.Name)
})
assert.NoError(t, err)
assert.Empty(t, seen, "marker entry should not be returned in exclusive mode")
assert.Equal(t, "", nextMarker, "next marker should be empty when only marker echo is returned")
}
func TestDoListFilerEntries_ExclusiveStartSkipsMarkerEchoWithSubsequentEntries(t *testing.T) {
s3a := &S3ApiServer{}
client := &markerEchoFilerClient{
entriesByDir: map[string][]*filer_pb.Entry{
"/buckets/test-bucket": {
{Name: "file.txt", Attributes: &filer_pb.FuseAttributes{}},
{Name: "test.txt", Attributes: &filer_pb.FuseAttributes{}},
{Name: "zebra.txt", Attributes: &filer_pb.FuseAttributes{}},
},
},
returnFollowing: true,
}
cursor := &ListingCursor{maxKeys: 1000}
var seen []string
nextMarker, err := s3a.doListFilerEntries(client, "/buckets/test-bucket", "", cursor, "test.txt", "", false, "test-bucket", func(dir string, entry *filer_pb.Entry) {
seen = append(seen, entry.Name)
})
assert.NoError(t, err)
assert.Equal(t, []string{"zebra.txt"}, seen, "marker should be skipped while subsequent entries are returned")
assert.Equal(t, "zebra.txt", nextMarker)
}
func TestAllowUnorderedWithDelimiterValidation(t *testing.T) {
t.Run("should return error when allow-unordered=true and delimiter are both present", func(t *testing.T) {
// Create a request with both allow-unordered=true and delimiter
@@ -329,6 +431,38 @@ func TestAllowUnorderedWithDelimiterValidation(t *testing.T) {
})
}
func TestSanitizeV1MarkerEcho_NoProgressGuard(t *testing.T) {
response := ListBucketResult{
Marker: "test.txt",
NextMarker: "test.txt",
IsTruncated: true,
Contents: []ListEntry{
{Key: "test.txt"},
},
}
sanitizeV1MarkerEcho(&response, "test.txt", false)
assert.Empty(t, response.Contents)
assert.Equal(t, "", response.NextMarker)
assert.False(t, response.IsTruncated)
response2 := ListBucketResult{
Marker: "test file.txt",
NextMarker: "test%20file.txt",
IsTruncated: true,
Contents: []ListEntry{
{Key: "test%20file.txt"},
},
}
sanitizeV1MarkerEcho(&response2, "test file.txt", true)
assert.Empty(t, response2.Contents)
assert.Equal(t, "", response2.NextMarker)
assert.False(t, response2.IsTruncated)
}
// TestMaxKeysParameterValidation tests the validation of max-keys parameter
func TestMaxKeysParameterValidation(t *testing.T) {
t.Run("valid max-keys values should work", func(t *testing.T) {
+238 -14
View File
@@ -5,6 +5,7 @@ package s3api
import (
"bytes"
"container/heap"
"encoding/hex"
"encoding/xml"
"errors"
@@ -201,10 +202,11 @@ func (s3a *S3ApiServer) createDeleteMarker(bucket, object string) (string, error
// versionListItem represents an item in the unified version/prefix list
type versionListItem struct {
key string
versionId string
isPrefix bool
versionData interface{} // *VersionEntry or *DeleteMarkerEntry
key string
versionId string
isPrefix bool
lastModified time.Time
versionData interface{} // *VersionEntry or *DeleteMarkerEntry
}
// listObjectVersions lists all versions of an object
@@ -213,6 +215,13 @@ func (s3a *S3ApiServer) listObjectVersions(bucket, prefix, keyMarker, versionIdM
if maxKeys > 1000 {
maxKeys = 1000
}
// For ListObjectVersions without delimiter, provide newest-first ordering across keys.
// This aligns with object-lock retention refresh workflows that expect monotonically
// decreasing LastModified values across pages.
if delimiter == "" {
return s3a.listObjectVersionsNewestFirst(bucket, prefix, keyMarker, versionIdMarker, maxKeys)
}
// Pre-allocate with capacity for maxKeys+1 to reduce reallocations
// The extra 1 is for truncation detection
allVersions := make([]interface{}, 0, maxKeys+1)
@@ -241,7 +250,7 @@ func (s3a *S3ApiServer) listObjectVersions(bucket, prefix, keyMarker, versionIdM
// - The alternative (collecting all) causes memory issues for buckets with many versions
// - Pagination continues correctly; users can page through to see all versions
maxCollect := maxKeys + 1 // +1 to detect truncation
err := s3a.findVersionsRecursively(bucketPath, "", &allVersions, processedObjects, seenVersionIds, bucket, prefix, keyMarker, versionIdMarker, delimiter, commonPrefixes, maxCollect)
err := s3a.findVersionsRecursively(bucketPath, "", &allVersions, processedObjects, seenVersionIds, bucket, prefix, keyMarker, versionIdMarker, delimiter, commonPrefixes, maxCollect, nil)
if err != nil {
glog.Errorf("listObjectVersions: findVersionsRecursively failed: %v", err)
return nil, err
@@ -265,6 +274,56 @@ func (s3a *S3ApiServer) listObjectVersions(bucket, prefix, keyMarker, versionIdM
return result, nil
}
// listObjectVersionsNewestFirst returns object versions ordered by LastModified (newest-first)
// across all keys. This mode scans the full keyspace to ensure globally consistent ordering.
func (s3a *S3ApiServer) listObjectVersionsNewestFirst(bucket, prefix, keyMarker, versionIdMarker string, maxKeys int) (*S3ListObjectVersionsResult, error) {
// Marker handling: when version-id-marker is empty, skip the entire key.
skipKeyMarkerOnly := keyMarker != "" && versionIdMarker == ""
var markerItem *versionListItem
if keyMarker != "" && versionIdMarker != "" {
lastModified, err := s3a.getVersionLastModified(bucket, keyMarker, versionIdMarker)
if err != nil {
glog.V(2).Infof("listObjectVersionsNewestFirst: marker lookup failed for %s/%s@%s: %v", bucket, keyMarker, versionIdMarker, err)
lastModified = versionIdToTime(versionIdMarker)
}
if !lastModified.IsZero() {
markerItem = &versionListItem{
key: keyMarker,
versionId: versionIdMarker,
lastModified: lastModified,
}
}
}
pager := newVersionListPager(maxKeys, markerItem, skipKeyMarkerOnly, keyMarker)
// Collect all versions without key-marker filtering to ensure global ordering.
allVersions := make([]interface{}, 0)
processedObjects := make(map[string]bool)
seenVersionIds := make(map[string]bool)
bucketPath := s3a.bucketDir(bucket)
err := s3a.findVersionsRecursively(bucketPath, "", &allVersions, processedObjects, seenVersionIds, bucket, prefix, "", "", "", nil, 0, pager.consider)
if err != nil {
glog.Errorf("listObjectVersionsNewestFirst: findVersionsRecursively failed: %v", err)
return nil, err
}
clear(processedObjects)
clear(seenVersionIds)
combinedList := pager.itemsSorted()
// Apply MaxKeys truncation and determine pagination markers
truncatedList, nextKeyMarker, nextVersionIdMarker, isTruncated := s3a.truncateAndSetMarkers(combinedList, maxKeys)
// Build the final response by splitting items back into their respective fields
result := s3a.splitIntoResult(truncatedList, bucket, prefix, keyMarker, versionIdMarker, "", maxKeys, isTruncated, nextKeyMarker, nextVersionIdMarker)
return result, nil
}
// buildSortedCombinedList merges versions and common prefixes into a single list
// sorted lexicographically by key, with versions preceding prefixes for the same key.
func (s3a *S3ApiServer) buildSortedCombinedList(allVersions []interface{}, commonPrefixes map[string]bool) []versionListItem {
@@ -273,19 +332,23 @@ func (s3a *S3ApiServer) buildSortedCombinedList(allVersions []interface{}, commo
// Add versions
for _, version := range allVersions {
var key, versionId string
var lastModified time.Time
switch v := version.(type) {
case *VersionEntry:
key = v.Key
versionId = v.VersionId
lastModified = v.LastModified
case *DeleteMarkerEntry:
key = v.Key
versionId = v.VersionId
lastModified = v.LastModified
}
combinedList = append(combinedList, versionListItem{
key: key,
versionId: versionId,
isPrefix: false,
versionData: version,
key: key,
versionId: versionId,
isPrefix: false,
lastModified: lastModified,
versionData: version,
})
}
@@ -313,6 +376,133 @@ func (s3a *S3ApiServer) buildSortedCombinedList(allVersions []interface{}, commo
return combinedList
}
// compareVersionListItemsByLastModified compares two list items using newest-first ordering.
// Returns negative if a should come before b, positive if b should come before a, 0 if equal.
func compareVersionListItemsByLastModified(a, b versionListItem) int {
if a.isPrefix != b.isPrefix {
if a.isPrefix {
return 1
}
return -1
}
if !a.isPrefix {
if !a.lastModified.Equal(b.lastModified) {
if a.lastModified.After(b.lastModified) {
return -1
}
return 1
}
if a.key != b.key {
if a.key < b.key {
return -1
}
return 1
}
if a.versionId != b.versionId {
return compareVersionIds(a.versionId, b.versionId)
}
return 0
}
if a.key != b.key {
if a.key < b.key {
return -1
}
return 1
}
return 0
}
type versionListItemHeap []versionListItem
func (h versionListItemHeap) Len() int { return len(h) }
func (h versionListItemHeap) Less(i, j int) bool {
// Oldest first so we can evict the oldest when we exceed maxKeys+1.
return compareVersionListItemsByLastModified(h[i], h[j]) > 0
}
func (h versionListItemHeap) Swap(i, j int) { h[i], h[j] = h[j], h[i] }
func (h *versionListItemHeap) Push(x interface{}) {
*h = append(*h, x.(versionListItem))
}
func (h *versionListItemHeap) Pop() interface{} {
old := *h
n := len(old)
item := old[n-1]
*h = old[:n-1]
return item
}
type versionListPager struct {
maxKeys int
markerItem *versionListItem
skipKeyMarkerOnly bool
keyMarker string
items versionListItemHeap
}
func newVersionListPager(maxKeys int, markerItem *versionListItem, skipKeyMarkerOnly bool, keyMarker string) *versionListPager {
return &versionListPager{
maxKeys: maxKeys,
markerItem: markerItem,
skipKeyMarkerOnly: skipKeyMarkerOnly,
keyMarker: keyMarker,
items: make(versionListItemHeap, 0, maxKeys+1),
}
}
func (p *versionListPager) consider(item versionListItem) {
if p.skipKeyMarkerOnly && item.key == p.keyMarker {
return
}
if p.markerItem != nil {
if compareVersionListItemsByLastModified(item, *p.markerItem) <= 0 {
return
}
}
// Keep only the newest maxKeys+1 items.
if p.maxKeys <= 0 {
return
}
if len(p.items) < p.maxKeys+1 {
heap.Push(&p.items, item)
return
}
if compareVersionListItemsByLastModified(item, p.items[0]) < 0 {
heap.Pop(&p.items)
heap.Push(&p.items, item)
}
}
func (p *versionListPager) itemsSorted() []versionListItem {
items := make([]versionListItem, len(p.items))
copy(items, p.items)
sort.Slice(items, func(i, j int) bool {
return compareVersionListItemsByLastModified(items[i], items[j]) < 0
})
return items
}
func versionIdToTime(versionId string) time.Time {
ts := getVersionTimestamp(versionId)
if ts == 0 {
return time.Time{}
}
return time.Unix(0, ts)
}
func (s3a *S3ApiServer) getVersionLastModified(bucket, object, versionId string) (time.Time, error) {
entry, err := s3a.getSpecificObjectVersion(bucket, object, versionId)
if err != nil {
return time.Time{}, err
}
if entry == nil || entry.Attributes == nil {
return time.Time{}, fmt.Errorf("missing entry attributes for %s/%s@%s", bucket, object, versionId)
}
return time.Unix(entry.Attributes.Mtime, 0), nil
}
// truncateAndSetMarkers applies MaxKeys limit and determines pagination markers
func (s3a *S3ApiServer) truncateAndSetMarkers(combinedList []versionListItem, maxKeys int) (truncated []versionListItem, nextKeyMarker, nextVersionIdMarker string, isTruncated bool) {
isTruncated = len(combinedList) > maxKeys
@@ -374,6 +564,7 @@ type versionCollector struct {
seenVersionIds map[string]bool
delimiter string
commonPrefixes map[string]bool
emitItem func(item versionListItem)
}
// isFull returns true if we've collected enough versions
@@ -447,7 +638,13 @@ func (vc *versionCollector) addVersion(version *ObjectVersion, objectKey string)
LastModified: version.LastModified,
Owner: vc.s3a.getObjectOwnerFromVersion(version, vc.bucket, objectKey),
}
*vc.allVersions = append(*vc.allVersions, deleteMarker)
vc.emitVersion(versionListItem{
key: objectKey,
versionId: version.VersionId,
isPrefix: false,
lastModified: version.LastModified,
versionData: deleteMarker,
})
} else {
versionEntry := &VersionEntry{
Key: objectKey,
@@ -459,10 +656,24 @@ func (vc *versionCollector) addVersion(version *ObjectVersion, objectKey string)
Owner: vc.s3a.getObjectOwnerFromVersion(version, vc.bucket, objectKey),
StorageClass: StorageClass(vc.s3a.getStorageClassFromExtended(entryExtended(version))),
}
*vc.allVersions = append(*vc.allVersions, versionEntry)
vc.emitVersion(versionListItem{
key: objectKey,
versionId: version.VersionId,
isPrefix: false,
lastModified: version.LastModified,
versionData: versionEntry,
})
}
}
func (vc *versionCollector) emitVersion(item versionListItem) {
if vc.emitItem != nil {
vc.emitItem(item)
return
}
*vc.allVersions = append(*vc.allVersions, item.versionData)
}
// processVersionsDirectory handles a .versions directory entry
func (vc *versionCollector) processVersionsDirectory(entryPath string) error {
objectKey := strings.TrimSuffix(entryPath, s3_constants.VersionsFolder)
@@ -530,7 +741,13 @@ func (vc *versionCollector) processExplicitDirectory(entryPath string, entry *fi
Owner: vc.s3a.getObjectOwnerFromEntry(entry),
StorageClass: StorageClass(vc.s3a.getStorageClassFromExtended(entry.Extended)),
}
*vc.allVersions = append(*vc.allVersions, versionEntry)
vc.emitVersion(versionListItem{
key: directoryKey,
versionId: "null",
isPrefix: false,
lastModified: versionEntry.LastModified,
versionData: versionEntry,
})
}
// processRegularFile handles a regular file entry (pre-versioning or suspended-versioning object)
@@ -591,14 +808,20 @@ func (vc *versionCollector) processRegularFile(currentPath, entryPath string, en
Owner: vc.s3a.getObjectOwnerFromEntry(entry),
StorageClass: StorageClass(vc.s3a.getStorageClassFromExtended(entry.Extended)),
}
*vc.allVersions = append(*vc.allVersions, versionEntry)
vc.emitVersion(versionListItem{
key: normalizedObjectKey,
versionId: "null",
isPrefix: false,
lastModified: versionEntry.LastModified,
versionData: versionEntry,
})
}
// findVersionsRecursively searches for .versions directories and regular files recursively
// with efficient pagination support. It skips objects before keyMarker and applies versionIdMarker filtering.
// maxCollect limits the number of versions to collect for memory efficiency (must be > 0)
// delimiter and commonPrefixes are used to group keys that share a common prefix
func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string, allVersions *[]interface{}, processedObjects map[string]bool, seenVersionIds map[string]bool, bucket, prefix, keyMarker, versionIdMarker, delimiter string, commonPrefixes map[string]bool, maxCollect int) error {
func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string, allVersions *[]interface{}, processedObjects map[string]bool, seenVersionIds map[string]bool, bucket, prefix, keyMarker, versionIdMarker, delimiter string, commonPrefixes map[string]bool, maxCollect int, emitItem func(item versionListItem)) error {
vc := &versionCollector{
s3a: s3a,
bucket: bucket,
@@ -611,6 +834,7 @@ func (s3a *S3ApiServer) findVersionsRecursively(currentPath, relativePath string
seenVersionIds: seenVersionIds,
delimiter: delimiter,
commonPrefixes: commonPrefixes,
emitItem: emitItem,
}
return vc.collectVersions(currentPath, relativePath)
@@ -0,0 +1,64 @@
package s3api
import (
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestVersionListPagerNewestFirstOrderingAndPagination(t *testing.T) {
base := time.Date(2026, 2, 6, 11, 3, 16, 0, time.UTC)
items := []versionListItem{
{
key: "k10/repo/log_20260206110316",
versionId: "v1",
lastModified: base,
},
{
key: "k10/repo/log_20260206110317",
versionId: "v2",
lastModified: base.Add(1 * time.Second),
},
{
key: "k10/repo/log_20260206110315",
versionId: "v0",
lastModified: base.Add(-1 * time.Second),
},
}
// Page 1: newest-first
pager := newVersionListPager(1, nil, false, "")
for _, item := range []versionListItem{items[0], items[2], items[1]} {
pager.consider(item)
}
sorted := pager.itemsSorted()
s3a := &S3ApiServer{}
truncated, nextKeyMarker, nextVersionIdMarker, isTruncated := s3a.truncateAndSetMarkers(sorted, 1)
require.Len(t, truncated, 1)
require.Equal(t, items[1].key, truncated[0].key)
require.Equal(t, items[1].versionId, truncated[0].versionId)
require.True(t, isTruncated)
require.Equal(t, items[1].key, nextKeyMarker)
require.Equal(t, items[1].versionId, nextVersionIdMarker)
// Page 2: start after marker (should return the next newest item)
marker := &versionListItem{
key: items[1].key,
versionId: items[1].versionId,
lastModified: items[1].lastModified,
}
pager2 := newVersionListPager(1, marker, false, "")
for _, item := range items {
pager2.consider(item)
}
sorted2 := pager2.itemsSorted()
truncated2, _, _, _ := s3a.truncateAndSetMarkers(sorted2, 1)
require.Len(t, truncated2, 1)
require.Equal(t, items[0].key, truncated2[0].key)
require.Equal(t, items[0].versionId, truncated2[0].versionId)
}
+32 -3
View File
@@ -5,6 +5,7 @@ package s3api
// AWS SDKs to obtain temporary credentials using OIDC/JWT tokens.
import (
"context"
"encoding/xml"
"errors"
"fmt"
@@ -13,6 +14,7 @@ import (
"time"
"github.com/seaweedfs/seaweedfs/weed/glog"
"github.com/seaweedfs/seaweedfs/weed/iam/integration"
"github.com/seaweedfs/seaweedfs/weed/iam/ldap"
"github.com/seaweedfs/seaweedfs/weed/iam/sts"
"github.com/seaweedfs/seaweedfs/weed/iam/utils"
@@ -339,7 +341,7 @@ func (h *STSHandlers) handleAssumeRole(w http.ResponseWriter, r *http.Request) {
}
// Generate common STS components
stsCreds, assumedUser, err := h.prepareSTSCredentials(roleArn, roleSessionName, durationSeconds, sessionPolicyJSON, modifyClaims)
stsCreds, assumedUser, err := h.prepareSTSCredentials(r.Context(), roleArn, roleSessionName, durationSeconds, sessionPolicyJSON, modifyClaims)
if err != nil {
h.writeSTSErrorResponse(w, r, STSErrInternalError, err)
return
@@ -480,7 +482,7 @@ func (h *STSHandlers) handleAssumeRoleWithLDAPIdentity(w http.ResponseWriter, r
claims.WithIdentityProvider("ldap", identity.UserID, identity.Provider)
}
stsCreds, assumedUser, err := h.prepareSTSCredentials(roleArn, roleSessionName, durationSeconds, sessionPolicyJSON, modifyClaims)
stsCreds, assumedUser, err := h.prepareSTSCredentials(r.Context(), roleArn, roleSessionName, durationSeconds, sessionPolicyJSON, modifyClaims)
if err != nil {
h.writeSTSErrorResponse(w, r, STSErrInternalError, err)
return
@@ -499,7 +501,7 @@ func (h *STSHandlers) handleAssumeRoleWithLDAPIdentity(w http.ResponseWriter, r
}
// prepareSTSCredentials extracts common shared logic for credential generation
func (h *STSHandlers) prepareSTSCredentials(roleArn, roleSessionName string,
func (h *STSHandlers) prepareSTSCredentials(ctx context.Context, roleArn, roleSessionName string,
durationSeconds *int64, sessionPolicy string, modifyClaims func(*sts.STSSessionClaims)) (STSCredentials, *AssumedRoleUser, error) {
// Calculate duration
@@ -546,6 +548,33 @@ func (h *STSHandlers) prepareSTSCredentials(roleArn, roleSessionName string,
WithSessionName(roleSessionName).
WithRoleInfo(effectiveRoleArn, fmt.Sprintf("%s:%s", roleName, roleSessionName), assumedRoleArn)
// If IAM integration is available, embed the role's attached policies into the session token.
// This makes the token self-sufficient for authorization even when role lookup is unavailable.
var policyManager *integration.IAMManager
if h.iam != nil && h.iam.iamIntegration != nil {
if provider, ok := h.iam.iamIntegration.(IAMManagerProvider); ok {
policyManager = provider.GetIAMManager()
}
}
if policyManager != nil {
roleNameForPolicies := utils.ExtractRoleNameFromArn(roleArn)
if roleNameForPolicies == "" {
roleNameForPolicies = utils.ExtractRoleNameFromPrincipal(roleArn)
}
if roleNameForPolicies != "" && len(claims.Policies) == 0 {
roleDef, err := policyManager.GetRole(ctx, roleNameForPolicies)
if err != nil {
glog.V(2).Infof("Failed to load role %q for policy embedding: %v", roleNameForPolicies, err)
} else if roleDef == nil {
glog.V(2).Infof("Role definition %q was missing for policy embedding", roleNameForPolicies)
} else if len(roleDef.AttachedPolicies) > 0 {
claims.WithPolicies(roleDef.AttachedPolicies)
}
}
}
if sessionPolicy != "" {
claims.WithSessionPolicy(sessionPolicy)
}
+94 -4
View File
@@ -6,7 +6,10 @@ import (
"net/http"
"net/url"
"testing"
"time"
"github.com/seaweedfs/seaweedfs/weed/iam/integration"
"github.com/seaweedfs/seaweedfs/weed/iam/policy"
"github.com/seaweedfs/seaweedfs/weed/iam/sts"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3_constants"
"github.com/seaweedfs/seaweedfs/weed/s3api/s3err"
@@ -75,7 +78,7 @@ func TestAssumeRole_CallerIdentityFallback(t *testing.T) {
}
}
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(fallbackRoleArn, "test-session", nil, "", modifyClaims)
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(context.Background(), fallbackRoleArn, "test-session", nil, "", modifyClaims)
require.NoError(t, err)
// Assertions
@@ -107,7 +110,7 @@ func TestAssumeRole_CallerIdentityFallback(t *testing.T) {
fallbackRoleArn := callerIdentity.PrincipalArn
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(fallbackRoleArn, "nested-session", nil, "", nil)
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(context.Background(), fallbackRoleArn, "nested-session", nil, "", nil)
require.NoError(t, err)
// The role name should be extracted from the assumed role ARN ("admin")
@@ -124,7 +127,7 @@ func TestAssumeRole_CallerIdentityFallback(t *testing.T) {
t.Run("Explicit RoleArn Provided", func(t *testing.T) {
explicitRoleArn := "arn:aws:iam::111122223333:role/TargetRole"
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(explicitRoleArn, "explicit-session", nil, "", nil)
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(context.Background(), explicitRoleArn, "explicit-session", nil, "", nil)
require.NoError(t, err)
// Role name should be "TargetRole"
@@ -140,7 +143,7 @@ func TestAssumeRole_CallerIdentityFallback(t *testing.T) {
t.Run("Malformed ARN", func(t *testing.T) {
malformedArn := "invalid-arn"
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(malformedArn, "bad-session", nil, "", nil)
stsCreds, assumedUser, err := stsHandlers.prepareSTSCredentials(context.Background(), malformedArn, "bad-session", nil, "", nil)
require.NoError(t, err)
// Fallback behavior: use full string as role name if extraction fails
@@ -151,3 +154,90 @@ func TestAssumeRole_CallerIdentityFallback(t *testing.T) {
assert.Equal(t, malformedArn, sessionInfo.RoleArn)
})
}
func TestAssumeRole_EmbedsRolePolicies(t *testing.T) {
t.Run("RoleWithAttachedPolicies", func(t *testing.T) {
ctx := context.Background()
manager := newTestSTSIntegrationManager(t)
writePolicy := &policy.PolicyDocument{
Version: "2012-10-17",
Statement: []policy.Statement{
{
Effect: "Allow",
Action: []string{"s3:*"},
Resource: []string{
"arn:aws:s3:::*",
"arn:aws:s3:::*/*",
},
},
},
}
require.NoError(t, manager.CreatePolicy(ctx, "", "S3WritePolicy", writePolicy))
roleName := "LakekeeperVendedRole"
require.NoError(t, manager.CreateRole(ctx, "", roleName, &integration.RoleDefinition{
RoleName: roleName,
AttachedPolicies: []string{"S3WritePolicy"},
}))
iam := &IdentityAccessManagement{
iamIntegration: NewS3IAMIntegration(manager, ""),
}
stsHandlers := NewSTSHandlers(manager.GetSTSService(), iam)
roleArn := fmt.Sprintf("arn:aws:iam::%s:role/%s", defaultAccountID, roleName)
stsCreds, _, err := stsHandlers.prepareSTSCredentials(ctx, roleArn, "test-session", nil, "", nil)
require.NoError(t, err)
sessionInfo, err := manager.GetSTSService().ValidateSessionToken(ctx, stsCreds.SessionToken)
require.NoError(t, err)
require.NotNil(t, sessionInfo)
assert.Equal(t, []string{"S3WritePolicy"}, sessionInfo.Policies)
})
t.Run("RoleWithoutAttachedPolicies", func(t *testing.T) {
ctx := context.Background()
manager := newTestSTSIntegrationManager(t)
roleName := "LakekeeperEmptyRole"
require.NoError(t, manager.CreateRole(ctx, "", roleName, &integration.RoleDefinition{
RoleName: roleName,
}))
iam := &IdentityAccessManagement{
iamIntegration: NewS3IAMIntegration(manager, ""),
}
stsHandlers := NewSTSHandlers(manager.GetSTSService(), iam)
roleArn := fmt.Sprintf("arn:aws:iam::%s:role/%s", defaultAccountID, roleName)
stsCreds, _, err := stsHandlers.prepareSTSCredentials(ctx, roleArn, "test-session", nil, "", nil)
require.NoError(t, err)
sessionInfo, err := manager.GetSTSService().ValidateSessionToken(ctx, stsCreds.SessionToken)
require.NoError(t, err)
assert.Empty(t, sessionInfo.Policies)
})
}
func newTestSTSIntegrationManager(t *testing.T) *integration.IAMManager {
t.Helper()
manager := integration.NewIAMManager()
config := &integration.IAMConfig{
STS: &sts.STSConfig{
TokenDuration: sts.FlexibleDuration{Duration: time.Hour},
MaxSessionLength: sts.FlexibleDuration{Duration: 12 * time.Hour},
Issuer: "test-issuer",
SigningKey: []byte("test-signing-key-at-least-32-bytes-long-for-security"),
},
Policy: &policy.PolicyEngineConfig{
DefaultEffect: "Deny",
StoreType: "memory",
},
Roles: &integration.RoleStoreConfig{
StoreType: "memory",
},
}
require.NoError(t, manager.Initialize(config, func() string { return "" }))
return manager
}
+3 -3
View File
@@ -296,9 +296,9 @@ jdbc:postgresql://localhost:5432/default?user=seaweedfs&password=secret
```bash
# Start PostgreSQL protocol server
weed db -port=5432 -auth=trust
weed db -port=5432 -auth=password -users="admin:secret;readonly:pass"
weed db -port=5432 -tls-cert=server.crt -tls-key=server.key
weed-db -port=5432 -auth=trust
weed-db -port=5432 -auth=password -users="admin:secret;readonly:pass"
weed-db -port=5432 -tls-cert=server.crt -tls-key=server.key
# Configuration options
-host=localhost # Listen host
+2 -2
View File
@@ -43,7 +43,7 @@ The PostgreSQL server now directly integrates with SeaweedFS Message Queue topic
- **Real Schema Information**: Reads actual topic schemas from broker configuration
- **Actual Data Access**: Queries real MQ data stored in Parquet and log files
- **Dynamic Updates**: Reflects topic additions and schema changes automatically
- **Consistent SQL Engine**: Uses the same SQL engine as `weed sql` command
- **Consistent SQL Engine**: Uses the same SQL engine as `weed-sql` command
### Database Context Management
- **Session Isolation**: Each PostgreSQL connection has its own database context
@@ -232,7 +232,7 @@ psql -h localhost -p 5432 -U seaweedfs -d default
- **DESIGN.md**: Complete architecture and design overview
- **IMPLEMENTATION.md**: Detailed implementation guide
- **postgres-examples/**: Client examples and test scripts
- **Command Documentation**: `weed db -help`
- **Command Documentation**: `weed-db -help`
## Security Considerations
+1
View File
@@ -468,6 +468,7 @@ func lazyLoadingRoutine(m *LevelDbNeedleMap) (err error) {
glog.V(1).Infof("reset accessRecord %s", m.dbFileName)
// reset accessRecord
accessRecord = 0
m.accessFlag = 0
}
continue
}