rdma-engine: ranged RDMA READ (offset+size) in rdma-client

Match the real read API shape (file_id + offset + size): rdma-client takes optional [offset] [size] and RDMA-READs only that sub-range from the remote MR (remote_addr + offset). Verified over rxe0: reading [3 MiB, 4 MiB) of an 8 MiB target file yields bytes whose md5 matches the source slice.
rdma-engine: standalone cross-process RDMA-READ target + client
2026-10-06 14:45:51 +00:00 · 2026-06-14 18:52:54 -07:00 · 2026-06-14 18:50:06 -07:00 · 2026-06-14 17:54:42 -07:00 · 2026-06-14 17:22:05 -07:00
45 changed files with 5278 additions and 731 deletions
@@ -78,6 +78,11 @@ service SeaweedFiler {
    rpc SubscribeLocalMetadata (SubscribeMetadataRequest) returns (stream SubscribeMetadataResponse) {
    }

+    // List the metadata subscribers currently connected to this filer
+    // (FUSE mounts, S3, filer.sync, peer filers, ...).
+    rpc ListMetadataSubscribers (ListMetadataSubscribersRequest) returns (ListMetadataSubscribersResponse) {
+    }
+
    rpc KvGet (KvGetRequest) returns (KvGetResponse) {
    }

@@ -388,7 +393,13 @@ message PosixLockRequest {
    bool is_moved = 2;
    PosixLockOp op = 3;
    PosixLockRange lock = 4;
+    // locks carries the full set a mount holds on key for a KEEP_ALIVE
+    // re-assertion, so the current owner filer can rebuild its in-memory state
+    // after an ownership change or restart. lock.sid identifies the session.
    repeated PosixLockRange locks = 5;
+    // cooling_probe marks a dual-read a new owner sends to the previous owner
+    // during a ring change, so the previous owner answers from local state
+    // without itself cooling-off (no recursion).
    bool cooling_probe = 6;
 }

@@ -615,6 +626,22 @@ message SubscribeMetadataResponse {
    repeated SubscribeMetadataResponse events = 4; // batch of additional events (backlog catch-up)
    repeated LogFileChunkRef log_file_refs = 5;    // log file chunk refs for client direct-read
 }
+message ListMetadataSubscribersRequest {
+    repeated string client_types = 1; // optional filter by client type, e.g. "mount"; empty = all
+}
+message ListMetadataSubscribersResponse {
+    repeated MetadataSubscriber subscribers = 1;
+}
+message MetadataSubscriber {
+    string client_name = 1;     // "<type>@<address>"
+    string client_type = 2;     // e.g. "mount", "sw-vfs", "s3", "filer:<addr>"
+    string address = 3;         // client peer address
+    string path_prefix = 4;     // subscribed path prefix
+    int32 client_id = 5;
+    int32 client_epoch = 6;
+    int64 connected_at_ns = 7;
+    string filer_address = 8;   // the filer this subscriber is connected to
+}
 // A persisted log file that the client can read directly from volume servers.
 // The file format is: [4-byte size | protobuf LogEntry] repeated.
 // Each LogEntry.Data contains a marshaled SubscribeMetadataResponse.
@@ -161,6 +161,29 @@ dependencies = [
 "serde",
 ]

+[[package]]
+name = "bindgen"
+version = "0.69.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088"
+dependencies = [
+ "bitflags 2.9.2",
+ "cexpr",
+ "clang-sys",
+ "itertools",
+ "lazy_static",
+ "lazycell",
+ "log",
+ "prettyplease",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "rustc-hash",
+ "shlex",
+ "syn",
+ "which",
+]
+
 [[package]]
 name = "bit-set"
 version = "0.8.0"
@@ -230,6 +253,15 @@ dependencies = [
 "shlex",
 ]

+[[package]]
+name = "cexpr"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
+dependencies = [
+ "nom",
+]
+
 [[package]]
 name = "cfg-if"
 version = "1.0.1"
@@ -248,7 +280,7 @@ dependencies = [
 "num-traits",
 "serde",
 "wasm-bindgen",
- "windows-link",
+ "windows-link 0.1.3",
 ]

 [[package]]
@@ -278,6 +310,17 @@ dependencies = [
 "half",
 ]

+[[package]]
+name = "clang-sys"
+version = "1.8.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4"
+dependencies = [
+ "glob",
+ "libc",
+ "libloading",
+]
+
 [[package]]
 name = "clap"
 version = "4.5.45"
@@ -530,6 +573,12 @@ version = "0.31.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f"

+[[package]]
+name = "glob"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
+
 [[package]]
 name = "half"
 version = "2.6.0"
@@ -561,6 +610,15 @@ version = "0.5.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"

+[[package]]
+name = "home"
+version = "0.5.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc627f471c528ff0c4a49e1d5e60450c8f6461dd6d10ba9dcd3a61d3dff7728d"
+dependencies = [
+ "windows-sys 0.61.2",
+]
+
 [[package]]
 name = "iana-time-zone"
 version = "0.1.63"
@@ -655,6 +713,12 @@ version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"

+[[package]]
+name = "lazycell"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55"
+
 [[package]]
 name = "libc"
 version = "0.2.175"
@@ -677,6 +741,12 @@ version = "0.5.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f"

+[[package]]
+name = "linux-raw-sys"
+version = "0.4.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d26c52dbd32dccf2d10cac7725f8eae5296885fb5703b261f7d0a0739ec807ab"
+
 [[package]]
 name = "linux-raw-sys"
 version = "0.9.4"
@@ -723,6 +793,15 @@ dependencies = [
 "libc",
 ]

+[[package]]
+name = "memoffset"
+version = "0.6.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5aa361d4faea93603064a027415f07bd8e1d5c88c9fbf68bf56a285428fd79ce"
+dependencies = [
+ "autocfg",
+]
+
 [[package]]
 name = "minimal-lexical"
 version = "0.2.1"
@@ -910,6 +989,12 @@ version = "0.2.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b"

+[[package]]
+name = "pkg-config"
+version = "0.3.33"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
+
 [[package]]
 name = "plotters"
 version = "0.3.7"
@@ -947,6 +1032,16 @@ dependencies = [
 "zerocopy",
 ]

+[[package]]
+name = "prettyplease"
+version = "0.2.37"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
+dependencies = [
+ "proc-macro2",
+ "syn",
+]
+
 [[package]]
 name = "proc-macro2"
 version = "1.0.100"
@@ -1073,6 +1168,7 @@ dependencies = [
 "nix",
 "parking_lot",
 "proptest",
+ "rdma-sys",
 "rmp-serde",
 "serde",
 "tempfile",
@@ -1084,6 +1180,17 @@ dependencies = [
 "uuid",
 ]

+[[package]]
+name = "rdma-sys"
+version = "0.3.0"
+dependencies = [
+ "bindgen",
+ "libc",
+ "memoffset",
+ "paste",
+ "pkg-config",
+]
+
 [[package]]
 name = "redox_syscall"
 version = "0.5.17"
@@ -1171,6 +1278,25 @@ version = "0.1.26"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "56f7d92ca342cea22a06f2121d944b4fd82af56988c270852495420f961d4ace"

+[[package]]
+name = "rustc-hash"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
+
+[[package]]
+name = "rustix"
+version = "0.38.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fdb5bc1ae2baa591800df16c9ca78619bf65c0488b41b96ccec5d11220d8c154"
+dependencies = [
+ "bitflags 2.9.2",
+ "errno",
+ "libc",
+ "linux-raw-sys 0.4.15",
+ "windows-sys 0.59.0",
+]
+
 [[package]]
 name = "rustix"
 version = "1.0.8"
@@ -1180,7 +1306,7 @@ dependencies = [
 "bitflags 2.9.2",
 "errno",
 "libc",
- "linux-raw-sys",
+ "linux-raw-sys 0.9.4",
 "windows-sys 0.60.2",
 ]

@@ -1338,7 +1464,7 @@ dependencies = [
 "fastrand",
 "getrandom 0.3.3",
 "once_cell",
- "rustix",
+ "rustix 1.0.8",
 "windows-sys 0.59.0",
 ]

@@ -1671,6 +1797,18 @@ dependencies = [
 "wasm-bindgen",
 ]

+[[package]]
+name = "which"
+version = "4.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "87ba24419a2078cd2b0f2ede2691b6c66d8e47836da3b6db8265ebad47afbfc7"
+dependencies = [
+ "either",
+ "home",
+ "once_cell",
+ "rustix 0.38.44",
+]
+
 [[package]]
 name = "winapi-util"
 version = "0.1.9"
@@ -1688,7 +1826,7 @@ checksum = "c0fdd3ddb90610c7638aa2b3a3ab2904fb9e5cdbecc643ddb3647212781c4ae3"
 dependencies = [
 "windows-implement",
 "windows-interface",
- "windows-link",
+ "windows-link 0.1.3",
 "windows-result",
 "windows-strings",
 ]
@@ -1721,13 +1859,19 @@ version = "0.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "5e6ad25900d524eaabdbbb96d20b4311e1e7ae1699af4fb28c17ae66c80d798a"

+[[package]]
+name = "windows-link"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
+
 [[package]]
 name = "windows-result"
 version = "0.3.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "56f42bd332cc6c8eac5af113fc0c1fd6a8fd2aa08a0119358686e5160d0586c6"
 dependencies = [
- "windows-link",
+ "windows-link 0.1.3",
 ]

 [[package]]
@@ -1736,7 +1880,7 @@ version = "0.4.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "56e6c93f3a0c3b36176cb1327a4958a0353d5d166c2a35cb268ace15e91d3b57"
 dependencies = [
- "windows-link",
+ "windows-link 0.1.3",
 ]

 [[package]]
@@ -1766,6 +1910,15 @@ dependencies = [
 "windows-targets 0.53.3",
 ]

+[[package]]
+name = "windows-sys"
+version = "0.61.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
+dependencies = [
+ "windows-link 0.2.1",
+]
+
 [[package]]
 name = "windows-targets"
 version = "0.52.6"
@@ -1788,7 +1941,7 @@ version = "0.53.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d5fe6031c4041849d7c496a8ded650796e7b6ecc19df1a431c1a363342e5dc91"
 dependencies = [
- "windows-link",
+ "windows-link 0.1.3",
 "windows_aarch64_gnullvm 0.53.0",
 "windows_aarch64_msvc 0.53.0",
 "windows_i686_gnu 0.53.0",
@@ -10,6 +10,24 @@ license = "Apache-2.0"
 name = "rdma-engine-server"
 path = "src/main.rs"

+# Phase-1 proof: a real IBV_WR_RDMA_READ over SoftRoCE loopback (requires Linux +
+# libibverbs/librdmacm). Build with: cargo build --features real-rdma --bin rdma-loopback
+[[bin]]
+name = "rdma-loopback"
+path = "src/bin/rdma_loopback.rs"
+required-features = ["real-rdma"]
+
+# Standalone cross-process RDMA target + client (real file data over rxe).
+[[bin]]
+name = "rdma-target"
+path = "src/bin/rdma_target.rs"
+required-features = ["real-rdma"]
+
+[[bin]]
+name = "rdma-client"
+path = "src/bin/rdma_client.rs"
+required-features = ["real-rdma"]
+
 [lib]
 name = "rdma_engine"
 path = "src/lib.rs"
@@ -48,6 +66,10 @@ parking_lot = "0.12"  # Fast mutexes
 nix = { version = "0.27", features = ["mman"] }  # Unix domain sockets and system calls
 async-trait = "0.1"  # Async traits

+# Real RDMA via libibverbs + librdmacm (raw FFI). Optional, Linux-only; pulled in
+# by the `real-rdma` feature. Used by src/rdma_real.rs (RealRdmaContext).
+rdma-sys = { version = "0.3", optional = true }
+
 # Configuration
 clap = { version = "4.0", features = ["derive"] }
 config = "0.13"
@@ -61,6 +83,7 @@ tempfile = "3.0"
 default = ["mock-ucx"]
 mock-ucx = []
 real-ucx = []  # UCX integration for production RDMA
+real-rdma = ["dep:rdma-sys"]  # real libibverbs/rdma_cm datapath (SoftRoCE/real NIC)

 [profile.release]
 opt-level = 3
@@ -72,3 +95,6 @@ panic = "abort"

 [package.metadata.docs.rs]
 features = ["real-rdma"]
+
+[patch.crates-io]
+rdma-sys = { path = "vendor/rdma-sys" }
@@ -0,0 +1,58 @@
+# Real RDMA datapath (`real-rdma` feature) — proof of plumbing
+
+The engine's default backend is a **mock** (`src/rdma.rs` `MockRdmaContext`): its
+`post_read` fabricates `byte = i % 256` and never touches a wire (the Go sidecar
+in `../pkg/rdma/client.go` even discards the engine's buffer and re-fakes the
+same pattern). This module replaces that with a **real** libibverbs / `rdma_cm`
+datapath, validated over SoftRoCE (rxe).
+
+## What's here
+
+- `src/rdma_real.rs`
+  - `RealRdmaContext` — the real client backend: `connect` (via `rdma_cm` /
+    `rdma_create_ep`), `register_memory`, `post_read` (a genuine
+    `IBV_WR_RDMA_READ` via `rdma_post_read`), `poll_completion`, `device_info`.
+  - `RealResponder` — a throwaway loopback target: registers a buffer for
+    `REMOTE_READ` (`rdma_reg_read`) and hands the client its `{addr, rkey, len}`.
+- `src/bin/rdma_loopback.rs` — the proof: a responder registers a buffer with a
+  verifiable pattern (a `REALRDMA` magic header — deliberately **not** the mock's
+  `i%256`), the client RDMA-READs it, and asserts the bytes match.
+
+Connection setup and QP state transitions use `rdma_cm` (much simpler than a
+manual GID/QPN/LID handshake for RoCEv2 addressing).
+
+## Build & run (Linux + SoftRoCE only)
+
+Requires `libibverbs-dev librdmacm-dev ibverbs-providers ibverbs-utils
+rdma-core clang libclang-dev`, and a SoftRoCE link:
+
+```sh
+sudo modprobe rdma_rxe rdma_ucm
+sudo rdma link add rxe0 type rxe netdev eth0     # eth0 = your primary netdev
+ibv_rc_pingpong -d rxe0 -g 1                      # sanity-check the fabric
+
+cargo build --features real-rdma --bin rdma-loopback
+sudo ./target/debug/rdma-loopback <rxe_ip> 7471 1048576   # rxe_ip = eth0's IP
+# => REAL-RDMA-READ-SUCCESS, dest head "REALRDMA", bytes match
+```
+
+(rxe on a low-vCPU VM is a **correctness** proof, not a representative benchmark.)
+
+## Why `vendor/rdma-sys`
+
+`rdma-sys 0.3.0` (crates.io) pins **bindgen 0.59.2**, which cannot parse the
+`rdma-core 56` headers (`ib_uverbs_flow_action_esp_encap_union_… is not a valid
+Ident`). `vendor/rdma-sys` is rdma-sys 0.3.0 with bindgen bumped to 0.69.4 (and
+the two methods removed in modern bindgen — `rustfmt_bindings` / `size_t_is_usize`
+— dropped from its `build.rs`); `Cargo.toml` redirects to it via
+`[patch.crates-io]`. This is a temporary workaround — upstreaming a bindgen bump
+to rdma-sys, or moving to a maintained bindings crate, is the cleaner long-term
+fix.
+
+## Not done (the real RDMA *win* is multi-week)
+
+This is Phase-1 proof-of-plumbing only. A demonstrable zero-copy read of real
+volume data still needs: a needle-backed RDMA **target** in the volume server
+(none exists today), a control-plane handshake to publish `{remote_addr, rkey}`,
+a breaking IPC-contract change (Rust + Go, in lockstep), mount wiring, and then
+the zero-copy READ path. See the scoping notes for the phased plan.
@@ -0,0 +1,58 @@
+//! Standalone RDMA-READ client: connect to an `rdma-target`, RDMA-READ its whole
+//! registered buffer into a fresh local buffer (separate process / address
+//! space), and write it out. md5 of the output vs the target's source file
+//! proves a real cross-process RDMA read of real data.
+//!
+//!   cargo run --features real-rdma --bin rdma-client -- <target_ip> <port> <out_file> [offset] [size]
+//!
+//! With [offset]/[size] it RDMA-READs only that byte range (the real read API
+//! shape: file_id + offset + size); without them it reads the whole MR.
+
+use std::fs;
+
+use rdma_engine::rdma_real::RealRdmaContext;
+
+fn main() {
+    let args: Vec<String> = std::env::args().collect();
+    if args.len() < 4 {
+        eprintln!("usage: rdma-client <target_ip> <port> <out_file> [offset] [size]");
+        std::process::exit(2);
+    }
+    let ip = args[1].clone();
+    let port: u16 = args[2].parse().expect("port");
+    let out = &args[3];
+    let offset: u64 = args.get(4).and_then(|s| s.parse().ok()).unwrap_or(0);
+
+    let (ctx, raddr, rkey, rlen) = match RealRdmaContext::connect(&ip, port) {
+        Ok(v) => v,
+        Err(e) => {
+            eprintln!("client connect failed: {e}");
+            std::process::exit(1);
+        }
+    };
+    // Default to the rest of the MR from `offset`.
+    let size: usize = args
+        .get(5)
+        .and_then(|s| s.parse().ok())
+        .unwrap_or(rlen - offset.min(rlen as u64) as usize);
+    eprintln!(
+        "client: connected to {ip}:{port}, remote MR len={rlen} rkey=0x{rkey:x}; reading [{offset}, {})",
+        offset as usize + size
+    );
+    assert!(offset as usize + size <= rlen, "range exceeds remote MR");
+
+    let mut dest = vec![0u8; size];
+    let (_mi, mr) = ctx.register_memory(&mut dest).expect("register dest");
+    // RDMA-READ the sub-range: remote source is the MR base + offset.
+    ctx.post_read(&mut dest, mr, raddr + offset, rkey, 1)
+        .expect("post_read");
+    let wc = ctx.poll_completion().expect("poll completion");
+    ctx.finish();
+    eprintln!(
+        "client: RDMA READ status={:?} byte_len={}",
+        wc.status, wc.byte_len
+    );
+
+    fs::write(out, &dest).expect("write output");
+    eprintln!("client: wrote {} bytes to {out}", dest.len());
+}
@@ -0,0 +1,95 @@
+//! Phase-1 proof: a real `IBV_WR_RDMA_READ` over SoftRoCE (rxe) loopback.
+//!
+//! Spawns a [`RealResponder`] that registers a buffer holding a verifiable
+//! pattern (with a `REALRDMA` magic header — deliberately NOT the mock's `i%256`),
+//! then a [`RealRdmaContext`] client connects, RDMA-READs the buffer, and asserts
+//! the bytes match. Proves the engine moves real bytes over a genuine RDMA verb.
+//!
+//!   rmmod-free run (after SoftRoCE is up):
+//!     cargo run --features real-rdma --bin rdma-loopback -- <rxe_ip> [port] [len]
+//!
+//! `<rxe_ip>` must be an IP on the netdev backing the rxe link (e.g. eth0's IP),
+//! not 127.0.0.1 (rxe is bound to that netdev, not loopback).
+
+use std::sync::mpsc;
+use std::time::Instant;
+
+use rdma_engine::rdma_real::{RealRdmaContext, RealResponder};
+
+fn main() {
+    let args: Vec<String> = std::env::args().collect();
+    let ip = args.get(1).cloned().unwrap_or_else(|| "127.0.0.1".into());
+    let port: u16 = args.get(2).and_then(|s| s.parse().ok()).unwrap_or(7471);
+    let len: usize = args.get(3).and_then(|s| s.parse().ok()).unwrap_or(1 << 20);
+
+    // A verifiable pattern, distinct from the mock's i%256, with a magic header.
+    let mut src = vec![0u8; len];
+    for (i, b) in src.iter_mut().enumerate() {
+        *b = (i.wrapping_mul(131).wrapping_add(17) & 0xff) as u8;
+    }
+    let magic = b"REALRDMA";
+    src[..magic.len()].copy_from_slice(magic);
+    let expected = src.clone();
+
+    let (ready_tx, ready_rx) = mpsc::channel::<()>();
+    let sip = ip.clone();
+    let server = std::thread::spawn(move || {
+        let mut data = src;
+        if let Err(e) = RealResponder::serve_once(&sip, port, &mut data, move || {
+            let _ = ready_tx.send(());
+        }) {
+            eprintln!("responder error: {e}");
+        }
+    });
+    // Wait until the responder is listening before connecting.
+    let _ = ready_rx.recv();
+
+    let t0 = Instant::now();
+    let (ctx, raddr, rkey, rlen) = match RealRdmaContext::connect(&ip, port) {
+        Ok(v) => v,
+        Err(e) => {
+            eprintln!("client connect failed: {e}");
+            let _ = server.join();
+            std::process::exit(1);
+        }
+    };
+    println!(
+        "connected: device={:?} remote MR addr=0x{raddr:x} rkey=0x{rkey:x} len={rlen}",
+        ctx.device_info().name
+    );
+
+    let mut dest = vec![0u8; rlen];
+    let (_mrinfo, mr) = ctx.register_memory(&mut dest).expect("register dest");
+    ctx.post_read(&mut dest, mr, raddr, rkey, 1).expect("post_read");
+    let wc = ctx.poll_completion().expect("poll completion");
+    let dt = t0.elapsed();
+    ctx.finish();
+    let _ = server.join();
+
+    let head: String = dest[..16.min(dest.len())]
+        .iter()
+        .map(|b| format!("{b:02x}"))
+        .collect::<Vec<_>>()
+        .join(" ");
+    println!(
+        "RDMA READ done: status={:?} byte_len={}  dest head: {head}  (\"{}\")",
+        wc.status,
+        wc.byte_len,
+        String::from_utf8_lossy(&dest[..magic.len().min(dest.len())])
+    );
+
+    if dest == expected {
+        let mibps = (rlen as f64 / (1024.0 * 1024.0)) / dt.as_secs_f64();
+        println!("VERIFY OK: {rlen} bytes match the responder's registered buffer");
+        println!("({mibps:.1} MiB/s incl. connection setup; rxe on a 2-vCPU VM is a correctness proof, not a benchmark)");
+        println!("REAL-RDMA-READ-SUCCESS");
+    } else {
+        let mismatch = dest
+            .iter()
+            .zip(expected.iter())
+            .position(|(a, b)| a != b)
+            .unwrap_or(0);
+        eprintln!("VERIFY FAILED: first mismatch at byte {mismatch}");
+        std::process::exit(1);
+    }
+}
@@ -0,0 +1,38 @@
+//! Standalone RDMA-READ target: load a file, register it for `REMOTE_READ`, and
+//! serve one client over SoftRoCE (rxe) — a real, separate-process precursor to
+//! a volume-side RDMA target. Pairs with `rdma-client`.
+//!
+//!   cargo run --features real-rdma --bin rdma-target -- <bind_ip> <port> <file>
+//!
+//! `<bind_ip>` must be an IP on the rxe-backing netdev (e.g. eth0's IP).
+
+use std::fs;
+
+use rdma_engine::rdma_real::RealResponder;
+
+fn main() {
+    let args: Vec<String> = std::env::args().collect();
+    if args.len() < 4 {
+        eprintln!("usage: rdma-target <bind_ip> <port> <file>");
+        std::process::exit(2);
+    }
+    let ip = args[1].clone();
+    let port: u16 = args[2].parse().expect("port");
+    let file = &args[3];
+
+    let mut data = fs::read(file).expect("read source file");
+    eprintln!(
+        "target: serving {} bytes of {file} on {ip}:{port} (REMOTE_READ)",
+        data.len()
+    );
+
+    match RealResponder::serve_once(&ip, port, &mut data, || {
+        eprintln!("target: listening; waiting for a client");
+    }) {
+        Ok(()) => eprintln!("target: served one client; done"),
+        Err(e) => {
+            eprintln!("target error: {e}");
+            std::process::exit(1);
+        }
+    }
+}
@@ -27,6 +27,8 @@ use anyhow::Result;

 pub mod ucx;
 pub mod rdma;
+#[cfg(feature = "real-rdma")]
+pub mod rdma_real; // real libibverbs/rdma_cm RDMA datapath (SoftRoCE/real NIC)
 pub mod ipc;
 pub mod session;
 pub mod memory;
@@ -0,0 +1,360 @@
+//! Real RDMA datapath over libibverbs + librdmacm (`rdma_cm`).
+//!
+//! Phase-1 proof-of-plumbing: a genuine `IBV_WR_RDMA_READ` (via the librdmacm
+//! convenience verb `rdma_post_read`) that pulls *real* bytes out of a remote
+//! peer's registered memory region over SoftRoCE (rxe) — replacing the mock
+//! engine that fabricated `byte = i % 256` and never touched a wire.
+//!
+//! [`RealRdmaContext`] is the client (the engine's real backend: connect /
+//! register_memory / post_read / poll_completion / device_info); [`RealResponder`]
+//! is a throwaway loopback target that registers a buffer for `REMOTE_READ` and
+//! hands the client its `{addr, rkey, len}`. Connection setup and QP state
+//! transitions are handled by `rdma_cm` (much simpler than a manual GID/QPN/LID
+//! handshake for RoCEv2). Linux + libibverbs/librdmacm only (`real-rdma` feature).
+
+#![allow(clippy::missing_safety_doc)]
+
+use std::ffi::{CStr, CString};
+use std::os::raw::{c_int, c_void};
+use std::ptr::null_mut;
+
+use rdma_sys::*;
+
+use crate::error::{RdmaError, RdmaResult};
+use crate::rdma::{CompletionStatus, MemoryRegion, RdmaDeviceInfo, RdmaOp, WorkCompletion};
+
+/// Metadata the responder hands the client so it can RDMA-READ the source MR.
+#[repr(C)]
+#[derive(Clone, Copy, Default)]
+struct RemoteInfo {
+    addr: u64,
+    rkey: u32,
+    len: u32,
+}
+
+const META_LEN: usize = std::mem::size_of::<RemoteInfo>();
+
+fn op_failed(op: &str) -> RdmaError {
+    RdmaError::OperationFailed {
+        operation: format!("{op}: {}", std::io::Error::last_os_error()),
+        status: -1,
+    }
+}
+
+unsafe fn qp_init_attr() -> ibv_qp_init_attr {
+    let mut a: ibv_qp_init_attr = std::mem::zeroed();
+    a.cap.max_send_wr = 8;
+    a.cap.max_recv_wr = 8;
+    a.cap.max_send_sge = 1;
+    a.cap.max_recv_sge = 1;
+    a.sq_sig_all = 1;
+    a
+}
+
+// Allow at least one outstanding RDMA READ in each direction.
+unsafe fn conn_param() -> rdma_conn_param {
+    let mut cp: rdma_conn_param = std::mem::zeroed();
+    cp.responder_resources = 1;
+    cp.initiator_depth = 1;
+    cp.retry_count = 7;
+    cp.rnr_retry_count = 7;
+    cp
+}
+
+unsafe fn query_device(id: *mut rdma_cm_id) -> RdmaDeviceInfo {
+    let ctx = (*id).verbs;
+    let mut attr: ibv_device_attr = std::mem::zeroed();
+    let mut name = String::from("unknown");
+    if !ctx.is_null() {
+        ibv_query_device(ctx, &mut attr);
+        let dev = (*ctx).device;
+        if !dev.is_null() {
+            name = CStr::from_ptr((*dev).name.as_ptr())
+                .to_string_lossy()
+                .into_owned();
+        }
+    }
+    RdmaDeviceInfo {
+        name,
+        vendor_id: attr.vendor_id,
+        vendor_part_id: attr.vendor_part_id,
+        hw_ver: attr.hw_ver,
+        max_mr: attr.max_mr as u32,
+        max_qp: attr.max_qp as u32,
+        max_cq: attr.max_cq as u32,
+        max_mr_size: attr.max_mr_size,
+        port_gid: String::new(),
+        port_lid: 0,
+    }
+}
+
+/// Busy-poll a librdmacm completion helper (`rdma_get_send_comp` /
+/// `rdma_get_recv_comp`) until it returns a completion or an error.
+unsafe fn wait_comp(
+    f: unsafe fn(*mut rdma_cm_id, *mut ibv_wc) -> c_int,
+    id: *mut rdma_cm_id,
+    what: &str,
+) -> RdmaResult<ibv_wc> {
+    let mut wc: ibv_wc = std::mem::zeroed();
+    loop {
+        let n = f(id, &mut wc);
+        if n < 0 {
+            return Err(op_failed(what));
+        }
+        if n > 0 {
+            break;
+        }
+    }
+    if wc.status != ibv_wc_status::IBV_WC_SUCCESS {
+        return Err(RdmaError::OperationFailed {
+            operation: format!("{what}: wc status {}", wc.status),
+            status: wc.status as i32,
+        });
+    }
+    Ok(wc)
+}
+
+/// The real RDMA client backend.
+pub struct RealRdmaContext {
+    id: *mut rdma_cm_id,
+    device: RdmaDeviceInfo,
+}
+
+impl RealRdmaContext {
+    /// Connect to a responder at `ip:port` and receive the `{addr, rkey, len}`
+    /// of its RDMA-READable source memory region.
+    pub fn connect(ip: &str, port: u16) -> RdmaResult<(Self, u64, u32, usize)> {
+        unsafe {
+            let node = CString::new(ip).map_err(|_| op_failed("bad ip"))?;
+            let svc = CString::new(port.to_string()).unwrap();
+            let mut hints: rdma_addrinfo = std::mem::zeroed();
+            hints.ai_port_space = rdma_port_space::RDMA_PS_TCP as c_int;
+            let mut res: *mut rdma_addrinfo = null_mut();
+            if rdma_getaddrinfo(node.as_ptr(), svc.as_ptr(), &hints, &mut res) != 0 {
+                return Err(op_failed("rdma_getaddrinfo(client)"));
+            }
+            let mut attr = qp_init_attr();
+            let mut id: *mut rdma_cm_id = null_mut();
+            let r = rdma_create_ep(&mut id, res, null_mut(), &mut attr);
+            rdma_freeaddrinfo(res);
+            if r != 0 {
+                return Err(op_failed("rdma_create_ep(client)"));
+            }
+
+            // Post a receive for the 16-byte RemoteInfo before connecting.
+            let mut meta = RemoteInfo::default();
+            let meta_ptr = (&mut meta as *mut RemoteInfo).cast::<c_void>();
+            let meta_mr = rdma_reg_msgs(id, meta_ptr, META_LEN);
+            if meta_mr.is_null() {
+                rdma_destroy_ep(id);
+                return Err(op_failed("rdma_reg_msgs(meta)"));
+            }
+            if rdma_post_recv(id, null_mut(), meta_ptr, META_LEN, meta_mr) != 0 {
+                rdma_destroy_ep(id);
+                return Err(op_failed("rdma_post_recv(meta)"));
+            }
+
+            let mut cp = conn_param();
+            if rdma_connect(id, &mut cp) != 0 {
+                rdma_destroy_ep(id);
+                return Err(op_failed("rdma_connect"));
+            }
+            wait_comp(rdma_get_recv_comp, id, "recv(meta)")?;
+            rdma_dereg_mr(meta_mr);
+
+            let device = query_device(id);
+            Ok((
+                RealRdmaContext { id, device },
+                meta.addr,
+                meta.rkey,
+                meta.len as usize,
+            ))
+        }
+    }
+
+    /// Register a local buffer (LOCAL_WRITE) as the destination of an RDMA READ.
+    /// Returns the engine's `MemoryRegion` plus the raw `ibv_mr` needed to post.
+    pub fn register_memory(&self, buf: &mut [u8]) -> RdmaResult<(MemoryRegion, *mut ibv_mr)> {
+        unsafe {
+            let mr = rdma_reg_msgs(self.id, buf.as_mut_ptr().cast(), buf.len());
+            if mr.is_null() {
+                return Err(op_failed("rdma_reg_msgs(dest)"));
+            }
+            Ok((
+                MemoryRegion {
+                    addr: buf.as_ptr() as u64,
+                    rkey: (*mr).rkey,
+                    lkey: (*mr).lkey,
+                    size: buf.len(),
+                    registered: true,
+                },
+                mr,
+            ))
+        }
+    }
+
+    /// Post a genuine `IBV_WR_RDMA_READ`: pull `buf.len()` bytes from the remote
+    /// `{remote_addr, rkey}` into the local buffer described by `mr`.
+    pub fn post_read(
+        &self,
+        buf: &mut [u8],
+        mr: *mut ibv_mr,
+        remote_addr: u64,
+        rkey: u32,
+        wr_id: u64,
+    ) -> RdmaResult<()> {
+        unsafe {
+            let flags = ibv_send_flags::IBV_SEND_SIGNALED.0 as c_int;
+            if rdma_post_read(
+                self.id,
+                wr_id as *mut c_void,
+                buf.as_mut_ptr().cast(),
+                buf.len(),
+                mr,
+                flags,
+                remote_addr,
+                rkey,
+            ) != 0
+            {
+                return Err(op_failed("rdma_post_read"));
+            }
+            Ok(())
+        }
+    }
+
+    /// Wait for the RDMA READ completion (it lands on the send queue).
+    pub fn poll_completion(&self) -> RdmaResult<WorkCompletion> {
+        unsafe {
+            let wc = wait_comp(rdma_get_send_comp, self.id, "send_comp(read)")?;
+            Ok(WorkCompletion {
+                wr_id: wc.wr_id,
+                status: CompletionStatus::from(wc.status as u32),
+                opcode: RdmaOp::Read,
+                byte_len: wc.byte_len,
+                imm_data: None,
+            })
+        }
+    }
+
+    pub fn device_info(&self) -> &RdmaDeviceInfo {
+        &self.device
+    }
+
+    /// Signal the responder we are done (1-byte send) so it can tear down, then
+    /// disconnect.
+    pub fn finish(&self) {
+        unsafe {
+            let mut done = [1u8; 1];
+            let mr = rdma_reg_msgs(self.id, done.as_mut_ptr().cast(), 1);
+            if !mr.is_null() {
+                if rdma_post_send(self.id, null_mut(), done.as_mut_ptr().cast(), 1, mr, 0) == 0 {
+                    let _ = wait_comp(rdma_get_send_comp, self.id, "send_comp(done)");
+                }
+                rdma_dereg_mr(mr);
+            }
+            rdma_disconnect(self.id);
+        }
+    }
+}
+
+impl Drop for RealRdmaContext {
+    fn drop(&mut self) {
+        unsafe {
+            if !self.id.is_null() {
+                rdma_destroy_ep(self.id);
+            }
+        }
+    }
+}
+
+/// A throwaway loopback RDMA-READ target: registers `data` for `REMOTE_READ`,
+/// accepts one client, hands it `{addr, rkey, len}`, and serves until the client
+/// signals done. Blocking — run on a thread; `data` must outlive the call so the
+/// client's reads see real bytes.
+pub struct RealResponder;
+
+impl RealResponder {
+    pub fn serve_once(
+        ip: &str,
+        port: u16,
+        data: &mut [u8],
+        ready: impl FnOnce(),
+    ) -> RdmaResult<()> {
+        unsafe {
+            let node = CString::new(ip).map_err(|_| op_failed("bad ip"))?;
+            let svc = CString::new(port.to_string()).unwrap();
+            let mut hints: rdma_addrinfo = std::mem::zeroed();
+            hints.ai_flags = RAI_PASSIVE as c_int;
+            hints.ai_port_space = rdma_port_space::RDMA_PS_TCP as c_int;
+            let mut res: *mut rdma_addrinfo = null_mut();
+            if rdma_getaddrinfo(node.as_ptr(), svc.as_ptr(), &hints, &mut res) != 0 {
+                return Err(op_failed("rdma_getaddrinfo(server)"));
+            }
+            let mut attr = qp_init_attr();
+            let mut listen_id: *mut rdma_cm_id = null_mut();
+            let r = rdma_create_ep(&mut listen_id, res, null_mut(), &mut attr);
+            rdma_freeaddrinfo(res);
+            if r != 0 {
+                return Err(op_failed("rdma_create_ep(server)"));
+            }
+            if rdma_listen(listen_id, 1) != 0 {
+                rdma_destroy_ep(listen_id);
+                return Err(op_failed("rdma_listen"));
+            }
+            ready(); // the listener is up; let the client connect
+
+            let mut id: *mut rdma_cm_id = null_mut();
+            if rdma_get_request(listen_id, &mut id) != 0 {
+                rdma_destroy_ep(listen_id);
+                return Err(op_failed("rdma_get_request"));
+            }
+
+            // Expose the source buffer for REMOTE_READ.
+            let src_mr = rdma_reg_read(id, data.as_mut_ptr().cast(), data.len());
+            if src_mr.is_null() {
+                rdma_destroy_ep(id);
+                rdma_destroy_ep(listen_id);
+                return Err(op_failed("rdma_reg_read"));
+            }
+
+            let mut meta = RemoteInfo {
+                addr: data.as_ptr() as u64,
+                rkey: (*src_mr).rkey,
+                len: data.len() as u32,
+            };
+            let meta_ptr = (&mut meta as *mut RemoteInfo).cast::<c_void>();
+            let meta_mr = rdma_reg_msgs(id, meta_ptr, META_LEN);
+            let mut done = [0u8; 1];
+            let done_mr = rdma_reg_msgs(id, done.as_mut_ptr().cast(), 1);
+            if meta_mr.is_null() || done_mr.is_null() {
+                return Err(op_failed("rdma_reg_msgs(server)"));
+            }
+            // Post the done-signal receive before accepting.
+            if rdma_post_recv(id, null_mut(), done.as_mut_ptr().cast(), 1, done_mr) != 0 {
+                return Err(op_failed("rdma_post_recv(done)"));
+            }
+
+            let mut cp = conn_param();
+            if rdma_accept(id, &mut cp) != 0 {
+                return Err(op_failed("rdma_accept"));
+            }
+
+            // Hand the client the source MR coordinates.
+            if rdma_post_send(id, null_mut(), meta_ptr, META_LEN, meta_mr, 0) != 0 {
+                return Err(op_failed("rdma_post_send(meta)"));
+            }
+            wait_comp(rdma_get_send_comp, id, "send_comp(meta)")?;
+
+            // Keep src_mr/data alive until the client finishes its read.
+            wait_comp(rdma_get_recv_comp, id, "recv_comp(done)")?;
+
+            rdma_disconnect(id);
+            rdma_dereg_mr(src_mr);
+            rdma_dereg_mr(meta_mr);
+            rdma_dereg_mr(done_mr);
+            rdma_destroy_ep(id);
+            rdma_destroy_ep(listen_id);
+            Ok(())
+        }
+    }
+}
@@ -0,0 +1 @@
+{"v":1}
@@ -0,0 +1,6 @@
+{
+  "git": {
+    "sha1": "ad39d0cba38a5ffe57c3596cfca7a4476c41b61c"
+  },
+  "path_in_vcs": ""
+}
@@ -0,0 +1,54 @@
+name: RDMA-SYS-CI
+on:
+  pull_request:
+    branches: [master]
+  push:
+    branches: [master]
+  schedule: [cron: "0 */24 * * *"]
+
+env:
+  CI_RUST_TOOLCHAIN: 1.61.0
+
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Git Sumbodule Update
+        run: |
+          git submodule update --remote --recursive
+      - name: Install dependencies
+        run: |
+          # Install libibverbs-dev as dependency automatically
+          sudo apt-get install -y librdmacm-dev
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: ${{ env.CI_RUST_TOOLCHAIN }}
+          override: true
+      - name: Test
+        run: |
+          ./rdma-env-setup/scripts/setup.sh
+          ./scripts/run.sh
+      - name: Setup tmate session
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3
+
+  fmt:
+    name: Fmt
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: ${{ env.CI_RUST_TOOLCHAIN }}
+          override: true
+      - run: rustup component add rustfmt
+      - uses: actions-rs/cargo@v1
+        with:
+          command: fmt
+          args: --all -- --check
@@ -0,0 +1,10 @@
+# Generated by Cargo
+# will have compiled files and executables
+/target/
+
+# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
+# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
+Cargo.lock
+
+# These are backup files generated by rustfmt
+**/*.rs.bk
@@ -0,0 +1,4 @@
+[submodule "rdma-env-setup"]
+	path = rdma-env-setup
+	url = https://github.com/datenlord/rdma-env-setup.git
+	branch = main
@@ -0,0 +1,382 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 3
+
+[[package]]
+name = "aho-corasick"
+version = "0.7.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "ansi_term"
+version = "0.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d52a9bb7ec0cf484c551830a7ce27bd20d67eac647e1befb56b0be4ee39a55d2"
+dependencies = [
+ "winapi",
+]
+
+[[package]]
+name = "atty"
+version = "0.2.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
+dependencies = [
+ "hermit-abi",
+ "libc",
+ "winapi",
+]
+
+[[package]]
+name = "autocfg"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
+
+[[package]]
+name = "bindgen"
+version = "0.59.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2bd2a9a458e8f4304c52c43ebb0cfbd520289f8379a52e329a38afda99bf8eb8"
+dependencies = [
+ "bitflags",
+ "cexpr",
+ "clang-sys",
+ "clap",
+ "env_logger",
+ "lazy_static",
+ "lazycell",
+ "log",
+ "peeking_take_while",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "rustc-hash",
+ "shlex",
+ "which",
+]
+
+[[package]]
+name = "bitflags"
+version = "1.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+
+[[package]]
+name = "cexpr"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
+dependencies = [
+ "nom",
+]
+
+[[package]]
+name = "cfg-if"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+
+[[package]]
+name = "clang-sys"
+version = "1.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fa2e27ae6ab525c3d369ded447057bca5438d86dc3a68f6faafb8269ba82ebf3"
+dependencies = [
+ "glob",
+ "libc",
+ "libloading",
+]
+
+[[package]]
+name = "clap"
+version = "2.34.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a0610544180c38b88101fecf2dd634b174a62eef6946f84dfc6a7127512b381c"
+dependencies = [
+ "ansi_term",
+ "atty",
+ "bitflags",
+ "strsim",
+ "textwrap",
+ "unicode-width",
+ "vec_map",
+]
+
+[[package]]
+name = "either"
+version = "1.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "90e5c1c8368803113bf0c9584fc495a58b86dc8a29edbf8fe877d21d9507e797"
+
+[[package]]
+name = "env_logger"
+version = "0.9.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a12e6657c4c97ebab115a42dcee77225f7f482cdd841cf7088c657a42e9e00e7"
+dependencies = [
+ "atty",
+ "humantime",
+ "log",
+ "regex",
+ "termcolor",
+]
+
+[[package]]
+name = "glob"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9b919933a397b79c37e33b77bb2aa3dc8eb6e165ad809e58ff75bc7db2e34574"
+
+[[package]]
+name = "hermit-abi"
+version = "0.1.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33"
+dependencies = [
+ "libc",
+]
+
+[[package]]
+name = "humantime"
+version = "2.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4"
+
+[[package]]
+name = "lazy_static"
+version = "1.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
+
+[[package]]
+name = "lazycell"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55"
+
+[[package]]
+name = "libc"
+version = "0.2.139"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "201de327520df007757c1f0adce6e827fe8562fbc28bfd9c15571c66ca1f5f79"
+
+[[package]]
+name = "libloading"
+version = "0.7.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b67380fd3b2fbe7527a606e18729d21c6f3951633d0500574c4dc22d2d638b9f"
+dependencies = [
+ "cfg-if",
+ "winapi",
+]
+
+[[package]]
+name = "log"
+version = "0.4.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
+dependencies = [
+ "cfg-if",
+]
+
+[[package]]
+name = "memchr"
+version = "2.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2dffe52ecf27772e601905b7522cb4ef790d2cc203488bbd0e2fe85fcb74566d"
+
+[[package]]
+name = "memoffset"
+version = "0.6.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5aa361d4faea93603064a027415f07bd8e1d5c88c9fbf68bf56a285428fd79ce"
+dependencies = [
+ "autocfg",
+]
+
+[[package]]
+name = "minimal-lexical"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
+
+[[package]]
+name = "nom"
+version = "7.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a8903e5a29a317527874d0402f867152a3d21c908bb0b933e416c65e301d4c36"
+dependencies = [
+ "memchr",
+ "minimal-lexical",
+]
+
+[[package]]
+name = "once_cell"
+version = "1.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "86f0b0d4bf799edbc74508c1e8bf170ff5f41238e5f8225603ca7caaae2b7860"
+
+[[package]]
+name = "paste"
+version = "1.0.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d01a5bd0424d00070b0098dd17ebca6f961a959dead1dbcbbbc1d1cd8d3deeba"
+
+[[package]]
+name = "peeking_take_while"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "19b17cddbe7ec3f8bc800887bab5e717348c95ea2ca0b1bf0837fb964dc67099"
+
+[[package]]
+name = "pkg-config"
+version = "0.3.26"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ac9a59f73473f1b8d852421e59e64809f025994837ef743615c6d0c5b305160"
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.49"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "57a8eca9f9c4ffde41714334dee777596264c7825420f521abc92b5b5deb63a5"
+dependencies = [
+ "unicode-ident",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8856d8364d252a14d474036ea1358d63c9e6965c8e5c1885c18f73d70bff9c7b"
+dependencies = [
+ "proc-macro2",
+]
+
+[[package]]
+name = "rdma-sys"
+version = "0.3.0"
+dependencies = [
+ "bindgen",
+ "libc",
+ "memoffset",
+ "paste",
+ "pkg-config",
+]
+
+[[package]]
+name = "regex"
+version = "1.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e076559ef8e241f2ae3479e36f97bd5741c0330689e217ad51ce2c76808b868a"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.6.28"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "456c603be3e8d448b072f410900c09faf164fbce2d480456f50eea6e25f9c848"
+
+[[package]]
+name = "rustc-hash"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
+
+[[package]]
+name = "shlex"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "43b2853a4d09f215c24cc5489c992ce46052d359b5109343cbafbf26bc62f8a3"
+
+[[package]]
+name = "strsim"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a"
+
+[[package]]
+name = "termcolor"
+version = "1.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bab24d30b911b2376f3a13cc2cd443142f0c81dda04c118693e35b3835757755"
+dependencies = [
+ "winapi-util",
+]
+
+[[package]]
+name = "textwrap"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060"
+dependencies = [
+ "unicode-width",
+]
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "84a22b9f218b40614adcb3f4ff08b703773ad44fa9423e4e0d346d5db86e4ebc"
+
+[[package]]
+name = "unicode-width"
+version = "0.1.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c0edd1e5b14653f783770bce4a4dabb4a5108a5370a5f5d8cfe8710c361f6c8b"
+
+[[package]]
+name = "vec_map"
+version = "0.8.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f1bddf1187be692e79c5ffeab891132dfb0f236ed36a43c7ed39f1165ee20191"
+
+[[package]]
+name = "which"
+version = "4.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1c831fbbee9e129a8cf93e7747a82da9d95ba8e16621cae60ec2cdc849bacb7b"
+dependencies = [
+ "either",
+ "libc",
+ "once_cell",
+]
+
+[[package]]
+name = "winapi"
+version = "0.3.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
+dependencies = [
+ "winapi-i686-pc-windows-gnu",
+ "winapi-x86_64-pc-windows-gnu",
+]
+
+[[package]]
+name = "winapi-i686-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
+
+[[package]]
+name = "winapi-util"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178"
+dependencies = [
+ "winapi",
+]
+
+[[package]]
+name = "winapi-x86_64-pc-windows-gnu"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
@@ -0,0 +1,35 @@
+# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
+#
+# When uploading crates to the registry Cargo will automatically
+# "normalize" Cargo.toml files for maximal compatibility
+# with all versions of Cargo and also rewrite `path` dependencies
+# to registry (e.g., crates.io) dependencies.
+#
+# If you are reading this file be aware that the original Cargo.toml
+# will likely look very different (and much more reasonable).
+# See Cargo.toml.orig for the original contents.
+
+[package]
+edition = "2021"
+name = "rdma-sys"
+version = "0.3.0"
+authors = ["Pu Wang <nicolas.weeks@gmail.com>"]
+description = "Rdma ibverbs lib Rust binding"
+license-file = "LICENSE"
+repository = "https://github.com/datenlord/rdma-sys"
+resolver = "2"
+
+[dependencies.libc]
+version = "0.2"
+
+[dependencies.memoffset]
+version = "0.6"
+
+[dependencies.paste]
+version = "1.0"
+
+[build-dependencies.bindgen]
+version = "0.69.4"
+
+[build-dependencies.pkg-config]
+version = "0.3"
@@ -0,0 +1,19 @@
+[package]
+name = "rdma-sys"
+version = "0.3.0"
+authors = ["Pu Wang <nicolas.weeks@gmail.com>"]
+edition = "2021"
+description = "Rdma ibverbs lib Rust binding"
+license-file = "LICENSE"
+repository = "https://github.com/datenlord/rdma-sys"
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+libc = "0.2"
+memoffset = "0.6"
+paste = "1.0"
+# intrusive-collections = "0.9"
+
+[build-dependencies]
+bindgen = "0.59.2"
+pkg-config = "0.3"
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 datenlord
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
@@ -0,0 +1,3 @@
+# Rdma ibverbs lib Rust binding
+
+This lib is the ibverbs low-level Rust binding. As inline function and nested structure are not handled properly in the automatic bind generator, we deal with them case by case manually.
@@ -0,0 +1,161 @@
+use std::env;
+use std::path::Path;
+
+fn link_rdma_core(lib_name: &str, pkg_name: &str, version: &str, include_paths: &mut Vec<String>) {
+    let result: _ = pkg_config::Config::new()
+        .atleast_version(version)
+        .statik(false)
+        .probe(lib_name);
+
+    let lib = result.unwrap_or_else(|_| panic!("please install {pkg_name} {version})"));
+    println!("found {pkg_name} {}", lib.version);
+
+    for path in lib.include_paths {
+        let path = path.to_str().expect("non-utf8 path");
+        include_paths.push(path.to_owned());
+    }
+}
+
+fn main() {
+    let mut include_paths: Vec<String> = Vec::new();
+
+    {
+        let lib_name = "libibverbs";
+        let pkg_name = "libibverbs-dev";
+        let version = "1.8.28";
+        link_rdma_core(lib_name, pkg_name, version, &mut include_paths);
+    }
+
+    {
+        let lib_name = "librdmacm";
+        let pkg_name = "librdmacm-dev";
+        let version = "1.2.28";
+        link_rdma_core(lib_name, pkg_name, version, &mut include_paths);
+    }
+
+    {
+        include_paths.sort_unstable();
+        include_paths.dedup_by(|x, first| x == first);
+        include_paths.push("/usr/include".into());
+        println!("include paths: {:?}", include_paths);
+    }
+
+    let include_args = include_paths.iter().map(|p| format!("-I{}", p));
+
+    let bindings = bindgen::Builder::default()
+        .clang_args(include_args)
+        .header("src/bindings.h")
+        .allowlist_function("ibv_.*")
+        .allowlist_type("ibv_.*")
+        .allowlist_function("rdma_.*")
+        .allowlist_type("rdma_.*")
+        .allowlist_type("verbs_.*")
+        .allowlist_type("ib_uverbs_access_flags")
+        //.allowlist_type("verbs_devices_ops")
+        //.allowlist_var("verbs_provider_.*")
+        .blocklist_type("in6_addr")
+        .opaque_type("pthread_.*")
+        .blocklist_type("sockaddr.*")
+        .blocklist_type("timespec")
+        .blocklist_type("ibv_ah_attr")
+        .blocklist_type("ibv_async_event")
+        .blocklist_type("ibv_flow_spec")
+        .blocklist_type("ibv_gid")
+        .blocklist_type("ibv_global_route")
+        .blocklist_type("ibv_mw_bind_info")
+        .blocklist_type("ibv_ops_wr")
+        .blocklist_type("ibv_send_wr")
+        .blocklist_type("ibv_wc")
+        .blocklist_type("rdma_addr")
+        .blocklist_type("rdma_cm_event")
+        .blocklist_type("rdma_ib_addr")
+        .blocklist_type("rdma_ud_param")
+        // Following ENUM will used with bitwise-or
+        // including flags, mask, caps, bits, fields, size
+        .bitfield_enum("ibv_device_cap_flags")
+        .bitfield_enum("ibv_odp_transport_cap_bits")
+        .bitfield_enum("ibv_odp_general_caps")
+        .bitfield_enum("ibv_rx_hash_function_flags")
+        .bitfield_enum("ibv_rx_hash_fields")
+        .bitfield_enum("ibv_raw_packet_caps")
+        .bitfield_enum("ibv_tm_cap_flags")
+        .bitfield_enum("ibv_pci_atomic_op_size")
+        .bitfield_enum("ibv_port_cap_flags")
+        .bitfield_enum("ibv_port_cap_flags2")
+        .bitfield_enum("ibv_create_cq_wc_flags")
+        .bitfield_enum("ibv_wc_flags")
+        .bitfield_enum("ibv_access_flags")
+        .bitfield_enum("ibv_xrcd_init_attr_mask")
+        .bitfield_enum("ibv_rereg_mr_flags")
+        .bitfield_enum("ibv_srq_attr_mask")
+        .bitfield_enum("ibv_srq_init_attr_mask") // TODO: need to be bitfield?
+        .bitfield_enum("ibv_wq_init_attr_mask")
+        .bitfield_enum("ibv_wq_flags")
+        .bitfield_enum("ibv_wq_attr_mask")
+        .bitfield_enum("ibv_ind_table_init_attr_mask")
+        .bitfield_enum("ibv_qp_init_attr_mask") // TODO: need to be bitfield?
+        .bitfield_enum("ibv_qp_create_flags")
+        .bitfield_enum("ibv_qp_create_send_ops_flags")
+        .bitfield_enum("ibv_qp_open_attr_mask")
+        .bitfield_enum("ibv_qp_attr_mask")
+        .bitfield_enum("ibv_send_flags")
+        .bitfield_enum("ibv_ops_flags")
+        .bitfield_enum("ibv_cq_attr_mask")
+        .bitfield_enum("ibv_flow_flags")
+        .bitfield_enum("ibv_flow_action_esp_mask")
+        .bitfield_enum("ibv_cq_init_attr_mask")
+        .bitfield_enum("ibv_create_cq_attr_flags")
+        .bitfield_enum("ibv_parent_domain_init_attr_mask")
+        .bitfield_enum("ibv_read_counters_flags")
+        .bitfield_enum("ibv_values_mask")
+        .bitfield_enum("ib_uverbs_access_flags")
+        .bitfield_enum("rdma_cm_join_mc_attr_mask")
+        .bitfield_enum("rdma_cm_mc_join_flags")
+        // Following ENUM will be const in a sub-mod
+        .constified_enum_module("ibv_node_type")
+        .constified_enum_module("ibv_transport_type")
+        .constified_enum_module("ibv_atomic_cap")
+        .constified_enum_module("ibv_mtu")
+        .constified_enum_module("ibv_port_state")
+        .constified_enum_module("ibv_wc_status")
+        .constified_enum_module("ibv_wc_opcode")
+        .constified_enum_module("ibv_mw_type")
+        .constified_enum_module("ibv_rate")
+        .constified_enum_module("ibv_srq_type")
+        .constified_enum_module("ibv_wq_type")
+        .constified_enum_module("ibv_wq_state")
+        .constified_enum_module("ibv_qp_type")
+        .constified_enum_module("ibv_qp_state")
+        .constified_enum_module("ibv_mig_state")
+        .constified_enum_module("ibv_wr_opcode")
+        .constified_enum_module("ibv_ops_wr_opcode")
+        .constified_enum_module("ibv_flow_attr_type")
+        .constified_enum_module("ibv_flow_spec_type")
+        .constified_enum_module("ibv_counter_description")
+        .constified_enum_module("ibv_rereg_mr_err_code")
+        .constified_enum_module("ib_uverbs_advise_mr_advice")
+        .constified_enum_module("rdma_cm_event_type")
+        .constified_enum_module("rdma_driver_id")
+        .constified_enum_module("rdma_port_space")
+        .rustified_enum("ibv_event_type")
+        // unions with non-`Copy` fields other than `ManuallyDrop<T>` are unstable
+        // for example: `pub eth: ibv_flow_spec_eth`
+        // note: see issue #55149 <https://github.com/rust-lang/rust/issues/55149> for more information
+        .derive_copy(true)
+        .derive_debug(false)
+        .derive_default(false)
+        .generate_comments(false)
+        //.generate_inline_functions(true)
+        //.default_macro_constant_type(bindgen::MacroTypeVariation::Unsigned)
+        .prepend_enum_name(false)
+        .disable_untagged_union()
+        .generate()
+        .expect("Unable to generate bindings");
+
+    let out_dir = env::var_os("OUT_DIR").unwrap();
+    let dest_path = Path::new(&out_dir).join("bindings.rs");
+
+    bindings
+        .write_to_file(dest_path)
+        .expect("Could not write bindings");
+}
@@ -0,0 +1,168 @@
+//! This demo shows how to establish a connection between server and client
+//! and send msg to the other end.
+//!
+//! You can try this example by running:
+//!
+//!     cargo run --example server
+//!
+//! And then start client in another terminal by running:
+//!
+//!     cargo run --example client <server_ip> <port>
+//!
+//! The default port is 7471.
+
+use rdma_sys::*;
+use std::{env, process::exit, ptr::null_mut};
+
+fn run(ip: &str, port: &str) -> i32 {
+    let mut send_msg = vec![1_u8; 16];
+    let mut recv_msg = vec![0_u8; 16];
+    let mut hints = unsafe { std::mem::zeroed::<rdma_addrinfo>() };
+    let mut res: *mut rdma_addrinfo = null_mut();
+
+    hints.ai_port_space = rdma_port_space::RDMA_PS_TCP as i32;
+    let mut ret =
+        unsafe { rdma_getaddrinfo(ip.as_ptr().cast(), port.as_ptr().cast(), &hints, &mut res) };
+
+    if ret != 0 {
+        println!("rdma_getaddrinfo");
+        return ret;
+    }
+
+    let mut attr = unsafe { std::mem::zeroed::<ibv_qp_init_attr>() };
+    let mut id: *mut rdma_cm_id = null_mut();
+    attr.cap.max_send_wr = 1;
+    attr.cap.max_recv_wr = 1;
+    attr.cap.max_send_sge = 1;
+    attr.cap.max_recv_sge = 1;
+    attr.cap.max_inline_data = 16;
+    attr.qp_context = id.cast();
+    attr.sq_sig_all = 1;
+    ret = unsafe { rdma_create_ep(&mut id, res, null_mut(), &mut attr) };
+    // Check to see if we got inline data allowed or not
+    let mut send_flags = 0_u32;
+    if attr.cap.max_inline_data >= 16 {
+        send_flags = ibv_send_flags::IBV_SEND_INLINE.0;
+    } else {
+        println!("rdma_client: device doesn't support IBV_SEND_INLINE, using sge sends");
+    }
+
+    if ret != 0 {
+        println!("rdma_create_ep");
+        unsafe {
+            rdma_freeaddrinfo(res);
+        }
+        return ret;
+    }
+
+    let mr = unsafe { rdma_reg_msgs(id, recv_msg.as_mut_ptr().cast(), 16) };
+    if mr.is_null() {
+        println!("rdma_reg_msgs for recv_msg");
+        unsafe {
+            rdma_destroy_ep(id);
+        }
+        return -1;
+    }
+
+    let mut send_mr = null_mut();
+    if (send_flags & ibv_send_flags::IBV_SEND_INLINE.0) as u32 == 0 {
+        println!("flags {:?}", send_flags);
+        send_mr = unsafe { rdma_reg_msgs(id, send_msg.as_mut_ptr().cast(), 16) };
+        if send_mr.is_null() {
+            println!("rdma_reg_msgs for send_msg");
+            unsafe {
+                rdma_dereg_mr(mr);
+            }
+            return -1;
+        }
+    }
+
+    ret = unsafe { rdma_post_recv(id, null_mut(), recv_msg.as_mut_ptr().cast(), 16, mr) };
+    if ret != 0 {
+        println!("rdma_post_recv");
+        if (send_flags & ibv_send_flags::IBV_SEND_INLINE.0) as u32 == 0 {
+            unsafe { rdma_dereg_mr(send_mr) };
+        }
+        return ret;
+    }
+
+    ret = unsafe { rdma_connect(id, null_mut()) };
+    if ret != 0 {
+        println!("rdma_connect");
+        unsafe {
+            rdma_disconnect(id);
+        }
+        return ret;
+    }
+
+    ret = unsafe {
+        rdma_post_send(
+            id,
+            null_mut(),
+            send_msg.as_mut_ptr().cast(),
+            16,
+            send_mr,
+            send_flags.try_into().unwrap(),
+        )
+    };
+    if ret != 0 {
+        println!("rdma_post_send");
+        unsafe {
+            rdma_disconnect(id);
+        }
+        return ret;
+    }
+
+    let mut wc = unsafe { std::mem::zeroed::<ibv_wc>() };
+    while ret == 0 {
+        ret = unsafe { rdma_get_send_comp(id, &mut wc) };
+    }
+    if ret < 0 {
+        println!("rdma_get_send_comp");
+        unsafe {
+            rdma_disconnect(id);
+        }
+        return ret;
+    }
+
+    ret = 0;
+    while ret == 0 {
+        ret = unsafe { rdma_get_recv_comp(id, &mut wc) };
+    }
+    println!("rdma_client: recv msg : {:?}", recv_msg);
+    if ret < 0 {
+        println!("rdma_get_recv_comp");
+    } else {
+        ret = 0;
+    }
+
+    ret
+}
+
+fn main() {
+    println!("rdma_client: start");
+    let args: Vec<String> = env::args().collect();
+    if args.len() != 3 {
+        println!("usage : cargo run --example client <server_ip> <port>");
+        println!("input : {:?}", args);
+        exit(-1);
+    }
+    let ip = args.get(1).unwrap().as_str();
+    let port = args.get(2).unwrap().as_str();
+
+    let ret = run(ip, port);
+
+    if ret != 0 {
+        println!(
+            "rdma_client: ret error {:?}",
+            std::io::Error::from_raw_os_error(-ret)
+        );
+        if ret == -1 {
+            println!(
+                "rdma_client: last os error {:?}",
+                std::io::Error::last_os_error()
+            );
+        }
+    }
+    println!("rdma_client: end");
+}
@@ -0,0 +1,201 @@
+//! This demo shows how to establish a connection between server and client
+//! and send msg to the other end.
+//!
+//! You can try this example by running:
+//!
+//!     cargo run --example server
+//!
+//! And then start client in another terminal by running:
+//!
+//!     cargo run --example client <server_ip> <port>
+//!
+//! The default port is 7471.
+
+use rdma_sys::*;
+use std::ptr::null_mut;
+
+static SERVER: &str = "0.0.0.0\0";
+static PORT: &str = "7471\0";
+
+fn run() -> i32 {
+    let mut send_msg = vec![1_u8; 16];
+    let mut recv_msg = vec![0_u8; 16];
+    let mut hints = unsafe { std::mem::zeroed::<rdma_addrinfo>() };
+    let mut res: *mut rdma_addrinfo = null_mut();
+    hints.ai_flags = RAI_PASSIVE.try_into().unwrap();
+    hints.ai_port_space = rdma_port_space::RDMA_PS_TCP.try_into().unwrap();
+    let mut ret = unsafe {
+        rdma_getaddrinfo(
+            SERVER.as_ptr().cast(),
+            PORT.as_ptr().cast(),
+            &hints,
+            &mut res,
+        )
+    };
+
+    if ret != 0 {
+        println!("rdma_getaddrinfo");
+        return ret;
+    }
+
+    let mut listen_id = null_mut();
+    let mut id = null_mut();
+
+    let mut init_attr = unsafe { std::mem::zeroed::<ibv_qp_init_attr>() };
+    init_attr.cap.max_send_wr = 1;
+    init_attr.cap.max_recv_wr = 1;
+    init_attr.cap.max_send_sge = 1;
+    init_attr.cap.max_recv_sge = 1;
+    init_attr.cap.max_inline_data = 16;
+    init_attr.sq_sig_all = 1;
+    ret = unsafe { rdma_create_ep(&mut listen_id, res, null_mut(), &mut init_attr) };
+    // Check to see if we got inline data allowed or not
+    if ret != 0 {
+        println!("rdma_create_ep");
+        unsafe {
+            rdma_freeaddrinfo(res);
+        }
+        return ret;
+    }
+    ret = unsafe { rdma_listen(listen_id, 0) };
+    if ret != 0 {
+        println!("rdma_listen");
+        unsafe {
+            rdma_destroy_ep(listen_id);
+        }
+        return ret;
+    }
+
+    ret = unsafe { rdma_get_request(listen_id, &mut id) };
+    if ret != 0 {
+        println!("rdma_get_request");
+        unsafe {
+            rdma_destroy_ep(listen_id);
+        }
+        return ret;
+    }
+
+    let mut qp_attr = unsafe { std::mem::zeroed::<ibv_qp_attr>() };
+    ret = unsafe {
+        ibv_query_qp(
+            (*id).qp,
+            &mut qp_attr,
+            ibv_qp_attr_mask::IBV_QP_CAP.0.try_into().unwrap(),
+            &mut init_attr,
+        )
+    };
+
+    if ret != 0 {
+        println!("ibv_query_qp");
+        unsafe {
+            rdma_destroy_ep(id);
+        }
+        return ret;
+    }
+
+    let mut send_flags = 0_u32;
+    if init_attr.cap.max_inline_data >= 16 {
+        send_flags = ibv_send_flags::IBV_SEND_INLINE.0;
+    } else {
+        println!("rdma_server: device doesn't support IBV_SEND_INLINE, using sge sends");
+    }
+
+    let recv_mr = unsafe { rdma_reg_msgs(id, recv_msg.as_mut_ptr().cast(), 16) };
+    if recv_mr.is_null() {
+        ret = -1;
+        println!("rdma_reg_msgs for recv_msg");
+        unsafe {
+            rdma_dereg_mr(recv_mr);
+        }
+        return ret;
+    }
+
+    let mut send_mr = null_mut();
+    if (send_flags & ibv_send_flags::IBV_SEND_INLINE.0) == 0 {
+        send_mr = unsafe { rdma_reg_msgs(id, send_msg.as_mut_ptr().cast(), 16) };
+        if send_mr.is_null() {
+            ret = -1;
+            println!("rdma_reg_msgs for send_msg");
+            unsafe {
+                rdma_dereg_mr(recv_mr);
+            }
+            return ret;
+        }
+    }
+    ret = unsafe { rdma_post_recv(id, null_mut(), recv_msg.as_mut_ptr().cast(), 16, recv_mr) };
+
+    if ret != 0 {
+        println!("rdma_post_recv");
+        unsafe {
+            rdma_dereg_mr(recv_mr);
+        }
+        return ret;
+    }
+
+    ret = unsafe { rdma_accept(id, null_mut()) };
+    if ret != 0 {
+        println!("rdma_accept");
+        if (send_flags & ibv_send_flags::IBV_SEND_INLINE.0) == 0 {
+            unsafe { rdma_dereg_mr(send_mr) };
+        }
+        return ret;
+    }
+
+    let mut wc = unsafe { std::mem::zeroed::<ibv_wc>() };
+    while ret == 0 {
+        ret = unsafe { rdma_get_recv_comp(id, &mut wc) };
+    }
+    if ret < 0 {
+        println!("rdma_get_recv_comp");
+        unsafe {
+            rdma_disconnect(id);
+        }
+        return ret;
+    }
+    println!("rdma_server: recv msg : {:?}", recv_msg);
+    ret = unsafe {
+        rdma_post_send(
+            id,
+            null_mut(),
+            send_msg.as_mut_ptr().cast(),
+            16,
+            send_mr,
+            send_flags.try_into().unwrap(),
+        )
+    };
+    if ret != 0 {
+        println!("rdma_post_send");
+        unsafe {
+            rdma_disconnect(id);
+        }
+        return ret;
+    }
+
+    while ret == 0 {
+        ret = unsafe { rdma_get_send_comp(id, &mut wc) };
+    }
+    if ret < 0 {
+        println!("rdma_get_send_comp");
+    } else {
+        ret = 0;
+    }
+    ret
+}
+
+fn main() {
+    println!("rdma_server: start");
+    let ret = run();
+    if ret != 0 {
+        println!(
+            "rdma_server: ret error {:?}",
+            std::io::Error::from_raw_os_error(-ret)
+        );
+        if ret == -1 {
+            println!(
+                "rdma_server: last os error {:?}",
+                std::io::Error::last_os_error()
+            );
+        }
+    }
+    println!("rdma_server: end");
+}
@@ -0,0 +1,30 @@
+#! /bin/sh
+
+RXE_DEV=rxe_eth0
+
+# Remove existing devices if any
+sudo rdma link delete $RXE_DEV
+
+set -o errexit
+set -o nounset
+set -o xtrace
+
+if [ `ifconfig -s | grep -c '^e'` -eq 0 ]; then
+    echo "no eth device"
+    exit 1
+elif [ `ifconfig -s | grep -c '^e'` -gt 1 ]; then
+    echo "multiple eth devices, select the first one"
+    ifconfig -s | grep '^e'
+fi
+
+ETH_DEV=`ifconfig -s | grep '^e' | cut -d ' ' -f 1 | head -n 1`
+ETH_IP=`ifconfig $ETH_DEV | grep inet | grep -v inet6 | awk '{print $2}' | tr -d "addr:"`
+CM_PORT=7471
+# Setup soft-roce device
+sudo rdma link add $RXE_DEV type rxe netdev $ETH_DEV
+rdma link | grep $RXE_DEV
+
+cargo test --all
+cargo run --example server &
+sleep 1 &&
+cargo run --example client $ETH_IP $CM_PORT
@@ -0,0 +1,3 @@
+#include <infiniband/verbs.h>
+#include <rdma/rdma_cma.h>
+#include <rdma/rdma_verbs.h>
@@ -0,0 +1,16 @@
+#![deny(warnings)]
+#![allow(non_snake_case, non_camel_case_types, non_upper_case_globals)]
+#![allow(deref_nullptr)] // TODO(fxbug.dev/74605): Remove once bindgen is fixed.
+#![allow(clippy::missing_safety_doc, clippy::too_many_arguments)]
+
+use libc::*;
+
+include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
+
+mod opcode;
+mod types;
+mod verbs;
+
+pub use self::opcode::*;
+pub use self::types::*;
+pub use self::verbs::*;
@@ -0,0 +1,103 @@
+pub mod ibv_opcode {
+    use paste::paste;
+    pub type Type = std::os::raw::c_int;
+    macro_rules! concat_ibv_opcode {
+        ($transport: expr, $op : expr ) => {
+            paste! {
+                pub const [<IBV_OPCODE_ $transport _ $op>]: Type = [<IBV_OPCODE_ $transport>] + [<IBV_OPCODE_ $op>] ;
+            }
+        };
+    }
+
+    /* transport types -- just used to define real constants */
+    pub const IBV_OPCODE_RC: Type = 0x00;
+    pub const IBV_OPCODE_UC: Type = 0x20;
+    pub const IBV_OPCODE_RD: Type = 0x40;
+    pub const IBV_OPCODE_UD: Type = 0x60;
+    /* operations -- just used to define real constants */
+    pub const IBV_OPCODE_SEND_FIRST: Type = 0x00;
+    pub const IBV_OPCODE_SEND_MIDDLE: Type = 0x01;
+    pub const IBV_OPCODE_SEND_LAST: Type = 0x02;
+    pub const IBV_OPCODE_SEND_LAST_WITH_IMMEDIATE: Type = 0x03;
+    pub const IBV_OPCODE_SEND_ONLY: Type = 0x04;
+    pub const IBV_OPCODE_SEND_ONLY_WITH_IMMEDIATE: Type = 0x05;
+    pub const IBV_OPCODE_RDMA_WRITE_FIRST: Type = 0x06;
+    pub const IBV_OPCODE_RDMA_WRITE_MIDDLE: Type = 0x07;
+    pub const IBV_OPCODE_RDMA_WRITE_LAST: Type = 0x08;
+    pub const IBV_OPCODE_RDMA_WRITE_LAST_WITH_IMMEDIATE: Type = 0x09;
+    pub const IBV_OPCODE_RDMA_WRITE_ONLY: Type = 0x0a;
+    pub const IBV_OPCODE_RDMA_WRITE_ONLY_WITH_IMMEDIATE: Type = 0x0b;
+    pub const IBV_OPCODE_RDMA_READ_REQUEST: Type = 0x0c;
+    pub const IBV_OPCODE_RDMA_READ_RESPONSE_FIRST: Type = 0x0d;
+    pub const IBV_OPCODE_RDMA_READ_RESPONSE_MIDDLE: Type = 0x0e;
+    pub const IBV_OPCODE_RDMA_READ_RESPONSE_LAST: Type = 0x0f;
+    pub const IBV_OPCODE_RDMA_READ_RESPONSE_ONLY: Type = 0x10;
+    pub const IBV_OPCODE_ACKNOWLEDGE: Type = 0x11;
+    pub const IBV_OPCODE_ATOMIC_ACKNOWLEDGE: Type = 0x12;
+    pub const IBV_OPCODE_COMPARE_SWAP: Type = 0x13;
+    pub const IBV_OPCODE_FETCH_ADD: Type = 0x14;
+
+    /* RC */
+    concat_ibv_opcode!(RC, SEND_FIRST);
+    concat_ibv_opcode!(RC, SEND_MIDDLE);
+    concat_ibv_opcode!(RC, SEND_LAST);
+    concat_ibv_opcode!(RC, SEND_LAST_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RC, SEND_ONLY);
+    concat_ibv_opcode!(RC, SEND_ONLY_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RC, RDMA_WRITE_FIRST);
+    concat_ibv_opcode!(RC, RDMA_WRITE_MIDDLE);
+    concat_ibv_opcode!(RC, RDMA_WRITE_LAST);
+    concat_ibv_opcode!(RC, RDMA_WRITE_LAST_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RC, RDMA_WRITE_ONLY);
+    concat_ibv_opcode!(RC, RDMA_WRITE_ONLY_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RC, RDMA_READ_REQUEST);
+    concat_ibv_opcode!(RC, RDMA_READ_RESPONSE_FIRST);
+    concat_ibv_opcode!(RC, RDMA_READ_RESPONSE_MIDDLE);
+    concat_ibv_opcode!(RC, RDMA_READ_RESPONSE_LAST);
+    concat_ibv_opcode!(RC, RDMA_READ_RESPONSE_ONLY);
+    concat_ibv_opcode!(RC, ACKNOWLEDGE);
+    concat_ibv_opcode!(RC, ATOMIC_ACKNOWLEDGE);
+    concat_ibv_opcode!(RC, COMPARE_SWAP);
+    concat_ibv_opcode!(RC, FETCH_ADD);
+
+    /* UC */
+    concat_ibv_opcode!(UC, SEND_FIRST);
+    concat_ibv_opcode!(UC, SEND_MIDDLE);
+    concat_ibv_opcode!(UC, SEND_LAST);
+    concat_ibv_opcode!(UC, SEND_LAST_WITH_IMMEDIATE);
+    concat_ibv_opcode!(UC, SEND_ONLY);
+    concat_ibv_opcode!(UC, SEND_ONLY_WITH_IMMEDIATE);
+    concat_ibv_opcode!(UC, RDMA_WRITE_FIRST);
+    concat_ibv_opcode!(UC, RDMA_WRITE_MIDDLE);
+    concat_ibv_opcode!(UC, RDMA_WRITE_LAST);
+    concat_ibv_opcode!(UC, RDMA_WRITE_LAST_WITH_IMMEDIATE);
+    concat_ibv_opcode!(UC, RDMA_WRITE_ONLY);
+    concat_ibv_opcode!(UC, RDMA_WRITE_ONLY_WITH_IMMEDIATE);
+
+    /* RD */
+    concat_ibv_opcode!(RD, SEND_FIRST);
+    concat_ibv_opcode!(RD, SEND_MIDDLE);
+    concat_ibv_opcode!(RD, SEND_LAST);
+    concat_ibv_opcode!(RD, SEND_LAST_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RD, SEND_ONLY);
+    concat_ibv_opcode!(RD, SEND_ONLY_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RD, RDMA_WRITE_FIRST);
+    concat_ibv_opcode!(RD, RDMA_WRITE_MIDDLE);
+    concat_ibv_opcode!(RD, RDMA_WRITE_LAST);
+    concat_ibv_opcode!(RD, RDMA_WRITE_LAST_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RD, RDMA_WRITE_ONLY);
+    concat_ibv_opcode!(RD, RDMA_WRITE_ONLY_WITH_IMMEDIATE);
+    concat_ibv_opcode!(RD, RDMA_READ_REQUEST);
+    concat_ibv_opcode!(RD, RDMA_READ_RESPONSE_FIRST);
+    concat_ibv_opcode!(RD, RDMA_READ_RESPONSE_MIDDLE);
+    concat_ibv_opcode!(RD, RDMA_READ_RESPONSE_LAST);
+    concat_ibv_opcode!(RD, RDMA_READ_RESPONSE_ONLY);
+    concat_ibv_opcode!(RD, ACKNOWLEDGE);
+    concat_ibv_opcode!(RD, ATOMIC_ACKNOWLEDGE);
+    concat_ibv_opcode!(RD, COMPARE_SWAP);
+    concat_ibv_opcode!(RD, FETCH_ADD);
+
+    /* UD */
+    concat_ibv_opcode!(UD, SEND_ONLY);
+    concat_ibv_opcode!(UD, SEND_ONLY_WITH_IMMEDIATE);
+}
@@ -0,0 +1,299 @@
+use crate::*;
+
+/// This file defines the types directly or indirectly involving union,
+/// in that BindGen cannot handle union very well, so mannually define them.
+
+/// Struct types involve union in <infiniband/verbs.h>
+
+// ibv_gid related union and struct types
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct ibv_gid_global_t {
+    pub subnet_prefix: __be64,
+    pub interface_id: __be64,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub union ibv_gid {
+    pub raw: [u8; 16],
+    pub global: ibv_gid_global_t,
+}
+
+// ibv_async_event related union and struct type
+#[repr(C)]
+pub union ibv_async_event_element_t {
+    pub cq: *mut ibv_cq,
+    pub qp: *mut ibv_qp,
+    pub srq: *mut ibv_srq,
+    pub wq: *mut ibv_wq,
+    pub port_num: c_int,
+}
+
+#[repr(C)]
+pub struct ibv_async_event {
+    pub element: ibv_async_event_element_t,
+    pub event_type: ibv_event_type,
+}
+
+// ibv_wc related union and struct types
+#[repr(C)]
+pub union imm_data_invalidated_rkey_union_t {
+    /// When (wc_flags & IBV_WC_WITH_IMM): Immediate data in network byte order.
+    pub imm_data: __be32,
+    /// When (wc_flags & IBV_WC_WITH_INV): Stores the invalidated rkey.
+    pub invalidated_rkey: u32,
+}
+
+#[repr(C)]
+pub struct ibv_wc {
+    pub wr_id: u64,
+    pub status: ibv_wc_status::Type,
+    pub opcode: ibv_wc_opcode::Type,
+    pub vendor_err: u32,
+    pub byte_len: u32,
+    pub imm_data_invalidated_rkey_union: imm_data_invalidated_rkey_union_t,
+    pub qp_num: u32,
+    pub src_qp: u32,
+    pub wc_flags: c_uint,
+    pub pkey_index: u16,
+    pub slid: u16,
+    pub sl: u8,
+    pub dlid_path_bits: u8,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct ibv_global_route {
+    pub dgid: ibv_gid,
+    pub flow_label: u32,
+    pub sgid_index: u8,
+    pub hop_limit: u8,
+    pub traffic_class: u8,
+}
+
+// ibv_send_wr related union and struct types
+#[repr(C)]
+#[derive(Copy, Clone)]
+pub struct ibv_mw_bind_info {
+    pub mr: *mut ibv_mr,
+    pub addr: u64,
+    pub length: u64,
+    pub mw_access_flags: ::std::os::raw::c_uint,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct rdma_t {
+    pub remote_addr: u64,
+    pub rkey: u32,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct atomic_t {
+    pub remote_addr: u64,
+    pub compare_add: u64,
+    pub swap: u64,
+    pub rkey: u32,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct ud_t {
+    pub ah: *mut ibv_ah,
+    pub remote_qpn: u32,
+    pub remote_qkey: u32,
+}
+
+#[repr(C)]
+pub union wr_t {
+    pub rdma: rdma_t,
+    pub atomic: atomic_t,
+    pub ud: ud_t,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct xrc_t {
+    pub remote_srqn: u32,
+}
+
+#[repr(C)]
+pub union qp_type_t {
+    pub xrc: xrc_t,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct bind_mw_t {
+    pub mw: *mut ibv_mw,
+    pub rkey: u32,
+    pub bind_info: ibv_mw_bind_info,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct tso_t {
+    pub hdr: *mut c_void,
+    pub hdr_sz: u16,
+    pub mss: u16,
+}
+
+#[repr(C)]
+pub union bind_mw_tso_union_t {
+    pub bind_mw: bind_mw_t,
+    pub tso: tso_t,
+}
+
+#[repr(C)]
+pub struct ibv_send_wr {
+    pub wr_id: u64,
+    pub next: *mut Self,
+    pub sg_list: *mut ibv_sge,
+    pub num_sge: c_int,
+    pub opcode: ibv_wr_opcode::Type,
+    pub send_flags: c_uint,
+    /// When opcode is *_WITH_IMM: Immediate data in network byte order.
+    /// When opcode is *_INV: Stores the rkey to invalidate
+    pub imm_data_invalidated_rkey_union: imm_data_invalidated_rkey_union_t,
+    pub wr: wr_t,
+    pub qp_type: qp_type_t,
+    pub bind_mw_tso_union: bind_mw_tso_union_t,
+}
+
+#[repr(C)]
+#[derive(Copy, Clone)]
+pub struct add_t {
+    pub recv_wr_id: u64,
+    pub sg_list: *mut ibv_sge,
+    pub num_sge: c_int,
+    pub tag: u64,
+    pub mask: u64,
+}
+
+#[repr(C)]
+#[derive(Copy, Clone)]
+pub struct tm_t {
+    pub unexpected_cnt: u32,
+    pub handle: u32,
+    pub add: add_t,
+}
+
+#[repr(C)]
+pub struct ibv_ops_wr {
+    wr_id: u64,
+    next: *mut Self,
+    opcode: ibv_ops_wr_opcode::Type,
+    flags: c_int,
+    tm: tm_t,
+}
+
+// ibv_flow_spec related union and struct types
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct hdr_t {
+    pub type_: ibv_flow_spec_type::Type,
+    pub size: u16,
+}
+
+#[repr(C)]
+pub union ibv_flow_spec_union_t {
+    pub hdr: hdr_t,
+    pub eth: ibv_flow_spec_eth,
+    pub ipv4: ibv_flow_spec_ipv4,
+    pub tcp_udp: ibv_flow_spec_tcp_udp,
+    pub ipv4_ext: ibv_flow_spec_ipv4_ext,
+    pub ipv6: ibv_flow_spec_ipv6,
+    pub esp: ibv_flow_spec_esp,
+    pub tunnel: ibv_flow_spec_tunnel,
+    pub gre: ibv_flow_spec_gre,
+    pub mpls: ibv_flow_spec_mpls,
+    pub flow_tag: ibv_flow_spec_action_tag,
+    pub drop: ibv_flow_spec_action_drop,
+    pub handle: ibv_flow_spec_action_handle,
+    pub flow_count: ibv_flow_spec_counter_action,
+}
+
+#[repr(C)]
+pub struct ibv_flow_spec {
+    pub ibv_flow_spec_union: ibv_flow_spec_union_t,
+}
+
+/// Struct types involve union in <rdma/rdma_cma.h>
+
+// rdma_addr related union and struct types
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct rdma_ib_addr {
+    pub sgid: ibv_gid,
+    pub dgid: ibv_gid,
+    pub pkey: __be16,
+}
+
+#[repr(C)]
+pub union src_addr_union_t {
+    pub src_addr: libc::sockaddr,
+    pub src_sin: libc::sockaddr_in,
+    pub src_sin6: libc::sockaddr_in6,
+    pub src_storage: libc::sockaddr_storage,
+}
+
+#[repr(C)]
+pub union dst_addr_union_t {
+    pub dst_addr: libc::sockaddr,
+    pub dst_sin: libc::sockaddr_in,
+    pub dst_sin6: libc::sockaddr_in6,
+    pub dst_storage: libc::sockaddr_storage,
+}
+
+#[repr(C)]
+pub union addr_union_t {
+    pub ibaddr: rdma_ib_addr,
+}
+
+#[repr(C)]
+pub struct rdma_addr {
+    pub src_addr_union: src_addr_union_t,
+    pub dst_addr_union: dst_addr_union_t,
+    pub addr: addr_union_t,
+}
+
+/// rdma_cm_event related union and struct types
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct ibv_ah_attr {
+    pub grh: ibv_global_route,
+    pub dlid: u16,
+    pub sl: u8,
+    pub src_path_bits: u8,
+    pub static_rate: u8,
+    pub is_global: u8,
+    pub port_num: u8,
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct rdma_ud_param {
+    pub private_data: *const ::std::os::raw::c_void,
+    pub private_data_len: u8,
+    pub ah_attr: ibv_ah_attr,
+    pub qp_num: u32,
+    pub qkey: u32,
+}
+
+#[repr(C)]
+pub union param_t {
+    pub conn: rdma_conn_param,
+    pub ud: rdma_ud_param,
+}
+
+#[repr(C)]
+pub struct rdma_cm_event {
+    pub id: *mut rdma_cm_id,
+    pub listen_id: *mut rdma_cm_id,
+    pub event: rdma_cm_event_type::Type,
+    pub status: c_int,
+    pub param: param_t,
+}
@@ -38,6 +38,9 @@ type AdminData struct {
 	TotalEcVolumes int `json:"total_ec_volumes"` // Total number of EC volumes across all servers
 	TotalEcShards  int `json:"total_ec_shards"`  // Total number of EC shards across all servers

+	// TotalMountClients is the number of connected FUSE/VFS mount clients across filers.
+	TotalMountClients int `json:"total_mount_clients"`
+
 	// Trends holds at-a-glance sparklines built from the admin's own recent
 	// cluster snapshots (no Prometheus required).
 	Trends DashboardTrends `json:"trends"`
@@ -186,6 +189,12 @@ func (s *AdminServer) GetAdminData(username string) (AdminData, error) {
 	}
 	totalEcVolumes = len(ecVolumeSet)

+	// Count connected FUSE/VFS mount clients (best-effort; don't fail the dashboard)
+	totalMountClients := 0
+	if mountData, mountErr := s.GetMountClients(); mountErr == nil {
+		totalMountClients = mountData.TotalMountClients
+	}
+
 	// Prepare admin data
 	adminData := AdminData{
 		Username:          username,
@@ -202,6 +211,7 @@ func (s *AdminServer) GetAdminData(username string) (AdminData, error) {
 		LastUpdated:       topology.UpdatedAt,
 		TotalEcVolumes:    totalEcVolumes,
 		TotalEcShards:     totalEcShards,
+		TotalMountClients: totalMountClients,
 		Trends:            s.GetDashboardTrends(),
 	}

@@ -0,0 +1,62 @@
+package dash
+
+import (
+	"context"
+	"sort"
+	"time"
+
+	"github.com/seaweedfs/seaweedfs/weed/glog"
+	"github.com/seaweedfs/seaweedfs/weed/pb"
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+)
+
+// mountClientTypes are the metadata-subscriber client types that represent a
+// FUSE/VFS mount: the Go `weed mount` ("mount") and the Rust VFS ("sw-vfs").
+var mountClientTypes = []string{"mount", "sw-vfs"}
+
+// GetMountClients queries every filer for its connected mount subscribers and
+// aggregates them. Each filer only knows the clients connected to itself, so a
+// cluster-wide view requires fanning out. An unreachable filer is skipped.
+func (s *AdminServer) GetMountClients() (*MountClientsData, error) {
+	var clients []MountClient
+
+	for _, filerAddr := range s.GetAllFilers() {
+		err := pb.WithGrpcFilerClient(false, 0, pb.ServerAddress(filerAddr), s.grpcDialOption, func(client filer_pb.SeaweedFilerClient) error {
+			resp, err := client.ListMetadataSubscribers(context.Background(), &filer_pb.ListMetadataSubscribersRequest{
+				ClientTypes: mountClientTypes,
+			})
+			if err != nil {
+				return err
+			}
+			for _, sub := range resp.Subscribers {
+				clients = append(clients, MountClient{
+					ClientName:   sub.ClientName,
+					ClientType:   sub.ClientType,
+					Address:      sub.Address,
+					PathPrefix:   sub.PathPrefix,
+					ClientId:     sub.ClientId,
+					ConnectedAt:  time.Unix(0, sub.ConnectedAtNs),
+					FilerAddress: sub.FilerAddress,
+				})
+			}
+			return nil
+		})
+		if err != nil {
+			glog.Warningf("list mount clients from filer %s: %v", filerAddr, err)
+			continue
+		}
+	}
+
+	sort.Slice(clients, func(i, j int) bool {
+		if clients[i].Address != clients[j].Address {
+			return clients[i].Address < clients[j].Address
+		}
+		return clients[i].PathPrefix < clients[j].PathPrefix
+	})
+
+	return &MountClientsData{
+		MountClients:      clients,
+		TotalMountClients: len(clients),
+		LastUpdated:       time.Now(),
+	}, nil
+}
@@ -295,6 +295,25 @@ type ClusterFilersData struct {
 	LastUpdated time.Time   `json:"last_updated"`
 }

+// MountClient is one connected FUSE/VFS mount as reported by a filer's
+// metadata-subscriber registry.
+type MountClient struct {
+	ClientName   string    `json:"client_name"`
+	ClientType   string    `json:"client_type"` // "mount" (Go) or "sw-vfs" (Rust VFS)
+	Address      string    `json:"address"`
+	PathPrefix   string    `json:"path_prefix"`
+	ClientId     int32     `json:"client_id"`
+	ConnectedAt  time.Time `json:"connected_at"`
+	FilerAddress string    `json:"filer_address"`
+}
+
+type MountClientsData struct {
+	Username          string        `json:"username"`
+	MountClients      []MountClient `json:"mount_clients"`
+	TotalMountClients int           `json:"total_mount_clients"`
+	LastUpdated       time.Time     `json:"last_updated"`
+}
+
 type MessageBrokerInfo struct {
 	Address    string    `json:"address"`
 	DataCenter string    `json:"datacenter"`
@@ -127,6 +127,7 @@ func (h *AdminHandlers) registerUIRoutes(r *mux.Router) {
 	r.HandleFunc("/cluster/filers", h.clusterHandlers.ShowClusterFilers).Methods(http.MethodGet)
 	r.HandleFunc("/cluster/s3", h.clusterHandlers.ShowClusterS3Servers).Methods(http.MethodGet)
 	r.HandleFunc("/cluster/volume-servers", h.clusterHandlers.ShowClusterVolumeServers).Methods(http.MethodGet)
+	r.HandleFunc("/cluster/mount-clients", h.clusterHandlers.ShowMountClients).Methods(http.MethodGet)

 	// Storage management routes
 	r.HandleFunc("/storage/volumes", h.clusterHandlers.ShowClusterVolumes).Methods(http.MethodGet)
@@ -51,6 +51,28 @@ func (h *ClusterHandlers) ShowClusterVolumeServers(w http.ResponseWriter, r *htt
 	}
 }

+// ShowMountClients renders the connected FUSE/VFS mount clients page
+func (h *ClusterHandlers) ShowMountClients(w http.ResponseWriter, r *http.Request) {
+	data, err := h.adminServer.GetMountClients()
+	if err != nil {
+		writeJSONError(w, http.StatusInternalServerError, "Failed to get mount clients: "+err.Error())
+		return
+	}
+
+	username := usernameOrDefault(r)
+	data.Username = username
+
+	// Render HTML template
+	w.Header().Set("Content-Type", "text/html")
+	mountClientsComponent := app.MountClients(*data)
+	viewCtx := layout.NewViewContext(r, username, dash.CSRFTokenFromContext(r.Context()))
+	layoutComponent := layout.Layout(viewCtx, mountClientsComponent)
+	if err := layoutComponent.Render(r.Context(), w); err != nil {
+		writeJSONError(w, http.StatusInternalServerError, "Failed to render template: "+err.Error())
+		return
+	}
+}
+
 // ShowClusterVolumes renders the cluster volumes page
 func (h *ClusterHandlers) ShowClusterVolumes(w http.ResponseWriter, r *http.Request) {
 	// Get pagination and sorting parameters from query string
@@ -194,6 +194,34 @@ templ Admin(data dash.AdminData) {
            </div>
        </div>

+        <!-- Mount Clients -->
+        <div class="row mb-4">
+            <div class="col-12">
+                <div class="card shadow mb-4">
+                    <div class="card-header py-3 d-flex justify-content-between align-items-center">
+                        <h6 class="m-0 font-weight-bold text-primary">
+                            <i class="fas fa-plug me-2"></i>Mount Clients
+                        </h6>
+                        <a id="mountClientsDetailsLink" href="/cluster/mount-clients" class="btn btn-sm btn-outline-primary">
+                            View Details <i class="fas fa-arrow-right ms-1"></i>
+                        </a>
+                    </div>
+                    <div class="card-body">
+                        <div class="d-flex align-items-center">
+                            <i class="fas fa-plug fa-2x text-gray-300 me-3"></i>
+                            <div>
+                                <div class="h4 mb-0 font-weight-bold text-gray-800">{fmt.Sprintf("%d", data.TotalMountClients)}</div>
+                                <div class="text-xs text-muted text-uppercase">Connected FUSE/VFS mount clients</div>
+                            </div>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </div>
+        <script>
+        (function(){var l=document.getElementById('mountClientsDetailsLink'); if(l){l.href=(window.__BASE_PATH__||'')+'/cluster/mount-clients';}})();
+        </script>
+
        <!-- Master Nodes Status -->
        <div class="row mb-4">
            <div class="col-lg-4">
@@ -199,406 +199,406 @@ func Admin(data dash.AdminData) templ.Component {
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "</div></div></div></div><!-- Master Nodes Status --><div class=\"row mb-4\"><div class=\"col-lg-4\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-server me-2\"></i>Master Nodes</h6></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-bordered\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>Address</th><th>Role</th></tr></thead> <tbody>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 16, "</div></div></div></div><!-- Mount Clients --><div class=\"row mb-4\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex justify-content-between align-items-center\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-plug me-2\"></i>Mount Clients</h6><a id=\"mountClientsDetailsLink\" href=\"/cluster/mount-clients\" class=\"btn btn-sm btn-outline-primary\">View Details <i class=\"fas fa-arrow-right ms-1\"></i></a></div><div class=\"card-body\"><div class=\"d-flex align-items-center\"><i class=\"fas fa-plug fa-2x text-gray-300 me-3\"></i><div><div class=\"h4 mb-0 font-weight-bold text-gray-800\">")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		var templ_7745c5c3_Var11 string
+		templ_7745c5c3_Var11, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.TotalMountClients))
+		if templ_7745c5c3_Err != nil {
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 213, Col: 126}
+		}
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var11))
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "</div><div class=\"text-xs text-muted text-uppercase\">Connected FUSE/VFS mount clients</div></div></div></div></div></div></div><script>\n        (function(){var l=document.getElementById('mountClientsDetailsLink'); if(l){l.href=(window.__BASE_PATH__||'')+'/cluster/mount-clients';}})();\n        </script><!-- Master Nodes Status --><div class=\"row mb-4\"><div class=\"col-lg-4\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-server me-2\"></i>Master Nodes</h6></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-bordered\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>Address</th><th>Role</th></tr></thead> <tbody>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		for _, master := range data.MasterNodes {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 17, "<tr><td><a href=\"")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "<tr><td><a href=\"")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			var templ_7745c5c3_Var11 templ.SafeURL
-			templ_7745c5c3_Var11, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(fmt.Sprintf("//%s/ui/index.html", master.Address)))
+			var templ_7745c5c3_Var12 templ.SafeURL
+			templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(fmt.Sprintf("//%s/ui/index.html", master.Address)))
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 219, Col: 121}
-			}
-			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var11))
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 18, "\" target=\"_blank\">")
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			var templ_7745c5c3_Var12 string
-			templ_7745c5c3_Var12, templ_7745c5c3_Err = templ.JoinStringErrs(master.Address)
-			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 220, Col: 67}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 247, Col: 121}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var12))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, " <i class=\"fas fa-external-link-alt ms-1 text-muted\"></i></a></td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 19, "\" target=\"_blank\">")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			var templ_7745c5c3_Var13 string
+			templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(master.Address)
+			if templ_7745c5c3_Err != nil {
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 248, Col: 67}
+			}
+			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13))
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, " <i class=\"fas fa-external-link-alt ms-1 text-muted\"></i></a></td><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			if master.IsLeader {
-				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 20, "<span class=\"badge bg-primary\">Leader</span>")
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "<span class=\"badge bg-primary\">Leader</span>")
 				if templ_7745c5c3_Err != nil {
 					return templ_7745c5c3_Err
 				}
 			} else {
-				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "<span class=\"badge bg-secondary\">Follower</span>")
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "<span class=\"badge bg-secondary\">Follower</span>")
 				if templ_7745c5c3_Err != nil {
 					return templ_7745c5c3_Err
 				}
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "</tbody></table></div></div></div></div><!-- System Health --><div class=\"col-lg-8\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-chart-pie me-2\"></i>Cluster</h6></div><div class=\"card-body text-center\"><div class=\"row\"><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		var templ_7745c5c3_Var13 string
-		templ_7745c5c3_Var13, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.MasterNodes)))
-		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 253, Col: 85}
-		}
-		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var13))
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "</h5><small class=\"text-muted\">Masters</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "</tbody></table></div></div></div></div><!-- System Health --><div class=\"col-lg-8\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-chart-pie me-2\"></i>Cluster</h6></div><div class=\"card-body text-center\"><div class=\"row\"><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var14 string
-		templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.VolumeServers)))
+		templ_7745c5c3_Var14, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.MasterNodes)))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 261, Col: 87}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 281, Col: 85}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var14))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "</h5><small class=\"text-muted\">Volume Servers</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "</h5><small class=\"text-muted\">Masters</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var15 string
-		templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.FilerNodes)))
+		templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.VolumeServers)))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 269, Col: 84}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 289, Col: 87}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "</h5><small class=\"text-muted\">Filers</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "</h5><small class=\"text-muted\">Volume Servers</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var16 string
-		templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.S3Nodes)))
+		templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.FilerNodes)))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 277, Col: 81}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 297, Col: 84}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</h5><small class=\"text-muted\">S3 Servers</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</h5><small class=\"text-muted\">Filers</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var17 string
-		templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.MessageBrokers)))
+		templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.S3Nodes)))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 285, Col: 88}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 305, Col: 81}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</h5><small class=\"text-muted\">Message Brokers</small></div></div></div></div></div></div></div></div><!-- Volume Servers --><div class=\"row\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex flex-row align-items-center justify-content-between\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-database me-2\"></i>Volume Servers</h6><div class=\"dropdown no-arrow\"><a class=\"dropdown-toggle\" href=\"#\" role=\"button\" data-bs-toggle=\"dropdown\"><i class=\"fas fa-ellipsis-v fa-sm fa-fw text-gray-400\"></i></a><div class=\"dropdown-menu dropdown-menu-right shadow animated--fade-in\"><div class=\"dropdown-header\">Actions:</div><a class=\"dropdown-item\" href=\"")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</h5><small class=\"text-muted\">S3 Servers</small></div></div></div><div class=\"col\"><div class=\"card bg-light\"><div class=\"card-body\"><h5>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		var templ_7745c5c3_Var18 templ.SafeURL
-		templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/volumes"))
+		var templ_7745c5c3_Var18 string
+		templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", len(data.MessageBrokers)))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 310, Col: 90}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 313, Col: 88}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var18))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "\">View Details</a> <a class=\"dropdown-item\" href=\"")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "</h5><small class=\"text-muted\">Message Brokers</small></div></div></div></div></div></div></div></div><!-- Volume Servers --><div class=\"row\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex flex-row align-items-center justify-content-between\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-database me-2\"></i>Volume Servers</h6><div class=\"dropdown no-arrow\"><a class=\"dropdown-toggle\" href=\"#\" role=\"button\" data-bs-toggle=\"dropdown\"><i class=\"fas fa-ellipsis-v fa-sm fa-fw text-gray-400\"></i></a><div class=\"dropdown-menu dropdown-menu-right shadow animated--fade-in\"><div class=\"dropdown-header\">Actions:</div><a class=\"dropdown-item\" href=\"")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var19 templ.SafeURL
-		templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/cluster"))
+		templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/volumes"))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 311, Col: 90}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 338, Col: 90}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var19))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "\">Topology View</a></div></div></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-hover\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>ID</th><th>Address</th><th>Data Center</th><th>Rack</th><th>Volumes</th><th>EC Shards</th><th>Capacity</th></tr></thead> <tbody>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "\">View Details</a> <a class=\"dropdown-item\" href=\"")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		var templ_7745c5c3_Var20 templ.SafeURL
+		templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/cluster"))
+		if templ_7745c5c3_Err != nil {
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 339, Col: 90}
+		}
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "\">Topology View</a></div></div></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-hover\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>ID</th><th>Address</th><th>Data Center</th><th>Rack</th><th>Volumes</th><th>EC Shards</th><th>Capacity</th></tr></thead> <tbody>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		for _, vs := range data.VolumeServers {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "<tr><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "<tr><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			var templ_7745c5c3_Var20 string
-			templ_7745c5c3_Var20, templ_7745c5c3_Err = templ.JoinStringErrs(vs.ID)
+			var templ_7745c5c3_Var21 string
+			templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinStringErrs(vs.ID)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 332, Col: 54}
-			}
-			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var20))
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "</td><td><a href=\"")
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			var templ_7745c5c3_Var21 templ.SafeURL
-			templ_7745c5c3_Var21, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(fmt.Sprintf("//%s/ui/index.html", vs.GetDisplayAddress())))
-			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 334, Col: 129}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 360, Col: 54}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var21))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "\" target=\"_blank\">")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "</td><td><a href=\"")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			var templ_7745c5c3_Var22 string
-			templ_7745c5c3_Var22, templ_7745c5c3_Err = templ.JoinStringErrs(vs.GetDisplayAddress())
+			var templ_7745c5c3_Var22 templ.SafeURL
+			templ_7745c5c3_Var22, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(fmt.Sprintf("//%s/ui/index.html", vs.GetDisplayAddress())))
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 335, Col: 75}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 362, Col: 129}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var22))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, " <i class=\"fas fa-external-link-alt ms-1 text-muted\"></i></a></td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "\" target=\"_blank\">")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var23 string
-			templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinStringErrs(vs.DataCenter)
+			templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinStringErrs(vs.GetDisplayAddress())
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 339, Col: 62}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 363, Col: 75}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var23))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "</td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, " <i class=\"fas fa-external-link-alt ms-1 text-muted\"></i></a></td><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var24 string
-			templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.JoinStringErrs(vs.Rack)
+			templ_7745c5c3_Var24, templ_7745c5c3_Err = templ.JoinStringErrs(vs.DataCenter)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 340, Col: 56}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 367, Col: 62}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var24))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "</td><td><div class=\"progress\" style=\"height: 20px;\"><div class=\"progress-bar\" role=\"progressbar\" style=\"")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "</td><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var25 string
-			templ_7745c5c3_Var25, templ_7745c5c3_Err = templruntime.SanitizeStyleAttributeValues(fmt.Sprintf("width: %d%%", calculatePercent(vs.Volumes, vs.MaxVolumes)))
+			templ_7745c5c3_Var25, templ_7745c5c3_Err = templ.JoinStringErrs(vs.Rack)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 344, Col: 135}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 368, Col: 56}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var25))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "\">")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "</td><td><div class=\"progress\" style=\"height: 20px;\"><div class=\"progress-bar\" role=\"progressbar\" style=\"")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var26 string
-			templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d/%d", vs.Volumes, vs.MaxVolumes))
+			templ_7745c5c3_Var26, templ_7745c5c3_Err = templruntime.SanitizeStyleAttributeValues(fmt.Sprintf("width: %d%%", calculatePercent(vs.Volumes, vs.MaxVolumes)))
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 345, Col: 104}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 372, Col: 135}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var26))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "</div></div></td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "\">")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			var templ_7745c5c3_Var27 string
+			templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d/%d", vs.Volumes, vs.MaxVolumes))
+			if templ_7745c5c3_Err != nil {
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 373, Col: 104}
+			}
+			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27))
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "</div></div></td><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			if vs.EcShards > 0 {
-				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "<span class=\"badge bg-info text-white me-1\">")
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "<span class=\"badge bg-info text-white me-1\">")
 				if templ_7745c5c3_Err != nil {
 					return templ_7745c5c3_Err
 				}
-				var templ_7745c5c3_Var27 string
-				templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", vs.EcShards))
+				var templ_7745c5c3_Var28 string
+				templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", vs.EcShards))
 				if templ_7745c5c3_Err != nil {
-					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 351, Col: 127}
+					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 379, Col: 127}
 				}
-				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27))
+				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28))
 				if templ_7745c5c3_Err != nil {
 					return templ_7745c5c3_Err
 				}
-				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "</span> ")
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "</span> ")
 				if templ_7745c5c3_Err != nil {
 					return templ_7745c5c3_Err
 				}
 				if vs.EcVolumes > 0 {
-					templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "<small class=\"text-muted\">(")
+					templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, "<small class=\"text-muted\">(")
 					if templ_7745c5c3_Err != nil {
 						return templ_7745c5c3_Err
 					}
-					var templ_7745c5c3_Var28 string
-					templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d vol", vs.EcVolumes))
+					var templ_7745c5c3_Var29 string
+					templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d vol", vs.EcVolumes))
 					if templ_7745c5c3_Err != nil {
-						return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 353, Col: 119}
+						return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 381, Col: 119}
 					}
-					_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28))
+					_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29))
 					if templ_7745c5c3_Err != nil {
 						return templ_7745c5c3_Err
 					}
-					templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, ")</small>")
+					templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, ")</small>")
 					if templ_7745c5c3_Err != nil {
 						return templ_7745c5c3_Err
 					}
 				}
 			} else {
-				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "<span class=\"text-muted\">-</span>")
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "<span class=\"text-muted\">-</span>")
 				if templ_7745c5c3_Err != nil {
 					return templ_7745c5c3_Err
 				}
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "</td><td>")
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			var templ_7745c5c3_Var29 string
-			templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(vs.DiskUsage))
-			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 359, Col: 74}
-			}
-			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29))
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, " / ")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "</td><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var30 string
-			templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(vs.DiskCapacity))
+			templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(vs.DiskUsage))
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 359, Col: 107}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 387, Col: 74}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, " / ")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			var templ_7745c5c3_Var31 string
+			templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(formatBytes(vs.DiskCapacity))
+			if templ_7745c5c3_Err != nil {
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 387, Col: 107}
+			}
+			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
 		if len(data.VolumeServers) == 0 {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "<tr><td colspan=\"7\" class=\"text-center text-muted py-4\"><i class=\"fas fa-info-circle me-2\"></i> No volume servers found</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "<tr><td colspan=\"7\" class=\"text-center text-muted py-4\"><i class=\"fas fa-info-circle me-2\"></i> No volume servers found</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "</tbody></table></div></div></div></div></div><!-- Filer Nodes --><div class=\"row mb-4\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex flex-row align-items-center justify-content-between\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-folder me-2\"></i>Filer Nodes</h6><div class=\"dropdown no-arrow\"><a class=\"dropdown-toggle\" href=\"#\" role=\"button\" data-bs-toggle=\"dropdown\"><i class=\"fas fa-ellipsis-v fa-sm fa-fw text-gray-400\"></i></a><div class=\"dropdown-menu dropdown-menu-right shadow animated--fade-in\"><div class=\"dropdown-header\">Actions:</div><a class=\"dropdown-item\" href=\"")
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		var templ_7745c5c3_Var31 templ.SafeURL
-		templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/filer"))
-		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 392, Col: 88}
-		}
-		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
-		if templ_7745c5c3_Err != nil {
-			return templ_7745c5c3_Err
-		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "\">File Browser</a> <a class=\"dropdown-item\" href=\"")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "</tbody></table></div></div></div></div></div><!-- Filer Nodes --><div class=\"row mb-4\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex flex-row align-items-center justify-content-between\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-folder me-2\"></i>Filer Nodes</h6><div class=\"dropdown no-arrow\"><a class=\"dropdown-toggle\" href=\"#\" role=\"button\" data-bs-toggle=\"dropdown\"><i class=\"fas fa-ellipsis-v fa-sm fa-fw text-gray-400\"></i></a><div class=\"dropdown-menu dropdown-menu-right shadow animated--fade-in\"><div class=\"dropdown-header\">Actions:</div><a class=\"dropdown-item\" href=\"")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		var templ_7745c5c3_Var32 templ.SafeURL
-		templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/cluster"))
+		templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/filer"))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 393, Col: 90}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 420, Col: 88}
 		}
 		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "\">Topology View</a></div></div></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-hover\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>Address</th><th>Data Center</th><th>Rack</th><th>Last Updated</th></tr></thead> <tbody>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, "\">File Browser</a> <a class=\"dropdown-item\" href=\"")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		var templ_7745c5c3_Var33 templ.SafeURL
+		templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/cluster"))
+		if templ_7745c5c3_Err != nil {
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 421, Col: 90}
+		}
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "\">Topology View</a></div></div></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-hover\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>Address</th><th>Data Center</th><th>Rack</th><th>Last Updated</th></tr></thead> <tbody>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		for _, filer := range data.FilerNodes {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "<tr><td><a href=\"")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "<tr><td><a href=\"")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			var templ_7745c5c3_Var33 templ.SafeURL
-			templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(fmt.Sprintf("//%s", filer.Address)))
+			var templ_7745c5c3_Var34 templ.SafeURL
+			templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL(fmt.Sprintf("//%s", filer.Address)))
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 412, Col: 106}
-			}
-			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "\" target=\"_blank\">")
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			var templ_7745c5c3_Var34 string
-			templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(filer.Address)
-			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 413, Col: 66}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 440, Col: 106}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, " <i class=\"fas fa-external-link-alt ms-1 text-muted\"></i></a></td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "\" target=\"_blank\">")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var35 string
-			templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinStringErrs(filer.DataCenter)
+			templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinStringErrs(filer.Address)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 417, Col: 65}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 441, Col: 66}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "</td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, " <i class=\"fas fa-external-link-alt ms-1 text-muted\"></i></a></td><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var36 string
-			templ_7745c5c3_Var36, templ_7745c5c3_Err = templ.JoinStringErrs(filer.Rack)
+			templ_7745c5c3_Var36, templ_7745c5c3_Err = templ.JoinStringErrs(filer.DataCenter)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 418, Col: 59}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 445, Col: 65}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var36))
 			if templ_7745c5c3_Err != nil {
@@ -609,64 +609,64 @@ func Admin(data dash.AdminData) templ.Component {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var37 string
-			templ_7745c5c3_Var37, templ_7745c5c3_Err = templ.JoinStringErrs(filer.LastUpdated.Format("2006-01-02 15:04:05"))
+			templ_7745c5c3_Var37, templ_7745c5c3_Err = templ.JoinStringErrs(filer.Rack)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 419, Col: 96}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 446, Col: 59}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var37))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "</td><td>")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			var templ_7745c5c3_Var38 string
+			templ_7745c5c3_Var38, templ_7745c5c3_Err = templ.JoinStringErrs(filer.LastUpdated.Format("2006-01-02 15:04:05"))
+			if templ_7745c5c3_Err != nil {
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 447, Col: 96}
+			}
+			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var38))
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
 		if len(data.FilerNodes) == 0 {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "<tr><td colspan=\"4\" class=\"text-center text-muted py-4\"><i class=\"fas fa-info-circle me-2\"></i> No filer nodes found</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "<tr><td colspan=\"4\" class=\"text-center text-muted py-4\"><i class=\"fas fa-info-circle me-2\"></i> No filer nodes found</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "</tbody></table></div></div></div></div></div><!-- S3 Servers --><div class=\"row mb-4\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex flex-row align-items-center justify-content-between\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-cloud me-2\"></i>S3 Servers</h6><div class=\"dropdown no-arrow\"><a class=\"dropdown-toggle\" href=\"#\" role=\"button\" data-bs-toggle=\"dropdown\"><i class=\"fas fa-ellipsis-v fa-sm fa-fw text-gray-400\"></i></a><div class=\"dropdown-menu dropdown-menu-right shadow animated--fade-in\"><div class=\"dropdown-header\">Actions:</div><a class=\"dropdown-item\" href=\"")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "</tbody></table></div></div></div></div></div><!-- S3 Servers --><div class=\"row mb-4\"><div class=\"col-12\"><div class=\"card shadow mb-4\"><div class=\"card-header py-3 d-flex flex-row align-items-center justify-content-between\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-cloud me-2\"></i>S3 Servers</h6><div class=\"dropdown no-arrow\"><a class=\"dropdown-toggle\" href=\"#\" role=\"button\" data-bs-toggle=\"dropdown\"><i class=\"fas fa-ellipsis-v fa-sm fa-fw text-gray-400\"></i></a><div class=\"dropdown-menu dropdown-menu-right shadow animated--fade-in\"><div class=\"dropdown-header\">Actions:</div><a class=\"dropdown-item\" href=\"")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		var templ_7745c5c3_Var38 templ.SafeURL
-		templ_7745c5c3_Var38, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/cluster/s3"))
+		var templ_7745c5c3_Var39 templ.SafeURL
+		templ_7745c5c3_Var39, templ_7745c5c3_Err = templ.JoinURLErrs(dash.PUrl(ctx, "/cluster/s3"))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 452, Col: 93}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 480, Col: 93}
 		}
-		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var38))
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var39))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "\">View Details</a></div></div></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-hover\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>Address</th><th>Data Center</th><th>Last Updated</th></tr></thead> <tbody>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "\">View Details</a></div></div></div><div class=\"card-body\"><div class=\"table-responsive\"><table class=\"table table-hover\" width=\"100%\" cellspacing=\"0\"><thead><tr><th>Address</th><th>Data Center</th><th>Last Updated</th></tr></thead> <tbody>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
 		for _, s3Node := range data.S3Nodes {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "<tr><td>")
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			var templ_7745c5c3_Var39 string
-			templ_7745c5c3_Var39, templ_7745c5c3_Err = templ.JoinStringErrs(s3Node.Address)
-			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 469, Col: 63}
-			}
-			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var39))
-			if templ_7745c5c3_Err != nil {
-				return templ_7745c5c3_Err
-			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "</td><td>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "<tr><td>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var40 string
-			templ_7745c5c3_Var40, templ_7745c5c3_Err = templ.JoinStringErrs(s3Node.DataCenter)
+			templ_7745c5c3_Var40, templ_7745c5c3_Err = templ.JoinStringErrs(s3Node.Address)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 470, Col: 66}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 497, Col: 63}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var40))
 			if templ_7745c5c3_Err != nil {
@@ -677,39 +677,52 @@ func Admin(data dash.AdminData) templ.Component {
 				return templ_7745c5c3_Err
 			}
 			var templ_7745c5c3_Var41 string
-			templ_7745c5c3_Var41, templ_7745c5c3_Err = templ.JoinStringErrs(s3Node.LastUpdated.Format("2006-01-02 15:04:05"))
+			templ_7745c5c3_Var41, templ_7745c5c3_Err = templ.JoinStringErrs(s3Node.DataCenter)
 			if templ_7745c5c3_Err != nil {
-				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 471, Col: 97}
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 498, Col: 66}
 			}
 			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var41))
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "</td><td>")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			var templ_7745c5c3_Var42 string
+			templ_7745c5c3_Var42, templ_7745c5c3_Err = templ.JoinStringErrs(s3Node.LastUpdated.Format("2006-01-02 15:04:05"))
+			if templ_7745c5c3_Err != nil {
+				return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 499, Col: 97}
+			}
+			_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var42))
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
 		if len(data.S3Nodes) == 0 {
-			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "<tr><td colspan=\"3\" class=\"text-center text-muted py-4\"><i class=\"fas fa-info-circle me-2\"></i> No S3 servers found</td></tr>")
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "<tr><td colspan=\"3\" class=\"text-center text-muted py-4\"><i class=\"fas fa-info-circle me-2\"></i> No S3 servers found</td></tr>")
 			if templ_7745c5c3_Err != nil {
 				return templ_7745c5c3_Err
 			}
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "</tbody></table></div></div></div></div></div><!-- Last Updated --><div class=\"row\"><div class=\"col-12\"><small class=\"text-muted\"><i class=\"fas fa-clock me-1\"></i> Last updated: ")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "</tbody></table></div></div></div></div></div><!-- Last Updated --><div class=\"row\"><div class=\"col-12\"><small class=\"text-muted\"><i class=\"fas fa-clock me-1\"></i> Last updated: ")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		var templ_7745c5c3_Var42 string
-		templ_7745c5c3_Var42, templ_7745c5c3_Err = templ.JoinStringErrs(data.LastUpdated.Format("2006-01-02 15:04:05"))
+		var templ_7745c5c3_Var43 string
+		templ_7745c5c3_Var43, templ_7745c5c3_Err = templ.JoinStringErrs(data.LastUpdated.Format("2006-01-02 15:04:05"))
 		if templ_7745c5c3_Err != nil {
-			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 495, Col: 81}
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/admin.templ`, Line: 523, Col: 81}
 		}
-		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var42))
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var43))
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
-		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "</small></div></div></div>")
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, "</small></div></div></div>")
 		if templ_7745c5c3_Err != nil {
 			return templ_7745c5c3_Err
 		}
@@ -0,0 +1,142 @@
+package app
+
+import (
+	"fmt"
+	"github.com/seaweedfs/seaweedfs/weed/admin/dash"
+)
+
+templ MountClients(data dash.MountClientsData) {
+	<div class="d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom">
+		<h1 class="h2">
+			<i class="fas fa-plug me-2"></i>Mount Clients
+		</h1>
+		<div class="btn-toolbar mb-2 mb-md-0">
+			<div class="btn-group me-2">
+				<button type="button" class="btn btn-sm btn-outline-primary" onclick="exportMountClients()">
+					<i class="fas fa-download me-1"></i>Export
+				</button>
+			</div>
+		</div>
+	</div>
+
+	<div id="mount-clients-content">
+		<!-- Summary Cards -->
+		<div class="row mb-4">
+			<div class="col-xl-12 col-md-12 mb-4">
+				<div class="card border-left-info shadow h-100 py-2">
+					<div class="card-body">
+						<div class="row no-gutters align-items-center">
+							<div class="col mr-2">
+								<div class="text-xs font-weight-bold text-info text-uppercase mb-1">
+									Connected Mount Clients
+								</div>
+								<div class="h5 mb-0 font-weight-bold text-gray-800">
+									{ fmt.Sprintf("%d", data.TotalMountClients) }
+								</div>
+							</div>
+							<div class="col-auto">
+								<i class="fas fa-plug fa-2x text-gray-300"></i>
+							</div>
+						</div>
+					</div>
+				</div>
+			</div>
+		</div>
+
+		<!-- Mount Clients Table -->
+		<div class="card shadow mb-4">
+			<div class="card-header py-3">
+				<h6 class="m-0 font-weight-bold text-primary">
+					<i class="fas fa-plug me-2"></i>Mount Clients
+				</h6>
+			</div>
+			<div class="card-body">
+				if len(data.MountClients) > 0 {
+					<div class="table-responsive">
+						<table class="table table-hover" id="mountClientsTable">
+							<thead>
+								<tr>
+									<th>Address</th>
+									<th>Type</th>
+									<th>Path Prefix</th>
+									<th>Client ID</th>
+									<th>Connected Since</th>
+									<th>Filer</th>
+								</tr>
+							</thead>
+							<tbody>
+								for _, c := range data.MountClients {
+									<tr>
+										<td><code>{ c.Address }</code></td>
+										<td>
+											<span class="badge bg-info text-dark">{ c.ClientType }</span>
+										</td>
+										<td><code>{ c.PathPrefix }</code></td>
+										<td>{ fmt.Sprintf("%d", c.ClientId) }</td>
+										<td>
+											if !c.ConnectedAt.IsZero() {
+												{ c.ConnectedAt.Format("2006-01-02 15:04:05") }
+											} else {
+												<span class="text-muted">N/A</span>
+											}
+										</td>
+										<td>
+											<span class="badge bg-light text-dark">{ c.FilerAddress }</span>
+										</td>
+									</tr>
+								}
+							</tbody>
+						</table>
+					</div>
+				} else {
+					<div class="text-center py-5">
+						<i class="fas fa-plug fa-3x text-muted mb-3"></i>
+						<h5 class="text-muted">No Mount Clients Found</h5>
+						<p class="text-muted">No FUSE or VFS mounts are currently connected to the cluster's filers.</p>
+					</div>
+				}
+			</div>
+		</div>
+
+		<!-- Last Updated -->
+		<div class="row">
+			<div class="col-12">
+				<small class="text-muted">
+					<i class="fas fa-clock me-1"></i>
+					Last updated: { data.LastUpdated.Format("2006-01-02 15:04:05") }
+				</small>
+			</div>
+		</div>
+	</div>
+
+	<script>
+	function exportMountClients() {
+		const rows = Array.from(document.querySelectorAll('#mountClientsTable tbody tr')).map(row => {
+			const cells = row.querySelectorAll('td');
+			if (cells.length > 1) {
+				return {
+					address: cells[0].textContent.trim(),
+					type: cells[1].textContent.trim(),
+					path: cells[2].textContent.trim(),
+					clientId: cells[3].textContent.trim(),
+					connected: cells[4].textContent.trim(),
+					filer: cells[5].textContent.trim()
+				};
+			}
+			return null;
+		}).filter(row => row !== null);
+
+		const csvContent = "data:text/csv;charset=utf-8," +
+			"Address,Type,Path Prefix,Client ID,Connected Since,Filer\n" +
+			rows.map(r => '"' + r.address + '","' + r.type + '","' + r.path + '","' + r.clientId + '","' + r.connected + '","' + r.filer + '"').join("\n");
+
+		const encodedUri = encodeURI(csvContent);
+		const link = document.createElement("a");
+		link.setAttribute("href", encodedUri);
+		link.setAttribute("download", "mount-clients.csv");
+		document.body.appendChild(link);
+		link.click();
+		document.body.removeChild(link);
+	}
+	</script>
+}
@@ -0,0 +1,181 @@
+// Code generated by templ - DO NOT EDIT.
+
+// templ: version: v0.3.1001
+package app
+
+//lint:file-ignore SA4006 This context is only used if a nested component is present.
+
+import "github.com/a-h/templ"
+import templruntime "github.com/a-h/templ/runtime"
+
+import (
+	"fmt"
+	"github.com/seaweedfs/seaweedfs/weed/admin/dash"
+)
+
+func MountClients(data dash.MountClientsData) templ.Component {
+	return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
+		templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
+		if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
+			return templ_7745c5c3_CtxErr
+		}
+		templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
+		if !templ_7745c5c3_IsBuffer {
+			defer func() {
+				templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
+				if templ_7745c5c3_Err == nil {
+					templ_7745c5c3_Err = templ_7745c5c3_BufErr
+				}
+			}()
+		}
+		ctx = templ.InitializeContext(ctx)
+		templ_7745c5c3_Var1 := templ.GetChildren(ctx)
+		if templ_7745c5c3_Var1 == nil {
+			templ_7745c5c3_Var1 = templ.NopComponent
+		}
+		ctx = templ.ClearChildren(ctx)
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 1, "<div class=\"d-flex justify-content-between flex-wrap flex-md-nowrap align-items-center pt-3 pb-2 mb-3 border-bottom\"><h1 class=\"h2\"><i class=\"fas fa-plug me-2\"></i>Mount Clients</h1><div class=\"btn-toolbar mb-2 mb-md-0\"><div class=\"btn-group me-2\"><button type=\"button\" class=\"btn btn-sm btn-outline-primary\" onclick=\"exportMountClients()\"><i class=\"fas fa-download me-1\"></i>Export</button></div></div></div><div id=\"mount-clients-content\"><!-- Summary Cards --><div class=\"row mb-4\"><div class=\"col-xl-12 col-md-12 mb-4\"><div class=\"card border-left-info shadow h-100 py-2\"><div class=\"card-body\"><div class=\"row no-gutters align-items-center\"><div class=\"col mr-2\"><div class=\"text-xs font-weight-bold text-info text-uppercase mb-1\">Connected Mount Clients</div><div class=\"h5 mb-0 font-weight-bold text-gray-800\">")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		var templ_7745c5c3_Var2 string
+		templ_7745c5c3_Var2, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", data.TotalMountClients))
+		if templ_7745c5c3_Err != nil {
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 34, Col: 52}
+		}
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var2))
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 2, "</div></div><div class=\"col-auto\"><i class=\"fas fa-plug fa-2x text-gray-300\"></i></div></div></div></div></div></div><!-- Mount Clients Table --><div class=\"card shadow mb-4\"><div class=\"card-header py-3\"><h6 class=\"m-0 font-weight-bold text-primary\"><i class=\"fas fa-plug me-2\"></i>Mount Clients</h6></div><div class=\"card-body\">")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		if len(data.MountClients) > 0 {
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 3, "<div class=\"table-responsive\"><table class=\"table table-hover\" id=\"mountClientsTable\"><thead><tr><th>Address</th><th>Type</th><th>Path Prefix</th><th>Client ID</th><th>Connected Since</th><th>Filer</th></tr></thead> <tbody>")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+			for _, c := range data.MountClients {
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 4, "<tr><td><code>")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				var templ_7745c5c3_Var3 string
+				templ_7745c5c3_Var3, templ_7745c5c3_Err = templ.JoinStringErrs(c.Address)
+				if templ_7745c5c3_Err != nil {
+					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 70, Col: 31}
+				}
+				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var3))
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 5, "</code></td><td><span class=\"badge bg-info text-dark\">")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				var templ_7745c5c3_Var4 string
+				templ_7745c5c3_Var4, templ_7745c5c3_Err = templ.JoinStringErrs(c.ClientType)
+				if templ_7745c5c3_Err != nil {
+					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 72, Col: 63}
+				}
+				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var4))
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 6, "</span></td><td><code>")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				var templ_7745c5c3_Var5 string
+				templ_7745c5c3_Var5, templ_7745c5c3_Err = templ.JoinStringErrs(c.PathPrefix)
+				if templ_7745c5c3_Err != nil {
+					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 74, Col: 34}
+				}
+				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var5))
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 7, "</code></td><td>")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				var templ_7745c5c3_Var6 string
+				templ_7745c5c3_Var6, templ_7745c5c3_Err = templ.JoinStringErrs(fmt.Sprintf("%d", c.ClientId))
+				if templ_7745c5c3_Err != nil {
+					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 75, Col: 45}
+				}
+				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var6))
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 8, "</td><td>")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				if !c.ConnectedAt.IsZero() {
+					var templ_7745c5c3_Var7 string
+					templ_7745c5c3_Var7, templ_7745c5c3_Err = templ.JoinStringErrs(c.ConnectedAt.Format("2006-01-02 15:04:05"))
+					if templ_7745c5c3_Err != nil {
+						return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 78, Col: 57}
+					}
+					_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var7))
+					if templ_7745c5c3_Err != nil {
+						return templ_7745c5c3_Err
+					}
+				} else {
+					templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 9, "<span class=\"text-muted\">N/A</span>")
+					if templ_7745c5c3_Err != nil {
+						return templ_7745c5c3_Err
+					}
+				}
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 10, "</td><td><span class=\"badge bg-light text-dark\">")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				var templ_7745c5c3_Var8 string
+				templ_7745c5c3_Var8, templ_7745c5c3_Err = templ.JoinStringErrs(c.FilerAddress)
+				if templ_7745c5c3_Err != nil {
+					return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 84, Col: 66}
+				}
+				_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var8))
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+				templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 11, "</span></td></tr>")
+				if templ_7745c5c3_Err != nil {
+					return templ_7745c5c3_Err
+				}
+			}
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 12, "</tbody></table></div>")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+		} else {
+			templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 13, "<div class=\"text-center py-5\"><i class=\"fas fa-plug fa-3x text-muted mb-3\"></i><h5 class=\"text-muted\">No Mount Clients Found</h5><p class=\"text-muted\">No FUSE or VFS mounts are currently connected to the cluster's filers.</p></div>")
+			if templ_7745c5c3_Err != nil {
+				return templ_7745c5c3_Err
+			}
+		}
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 14, "</div></div><!-- Last Updated --><div class=\"row\"><div class=\"col-12\"><small class=\"text-muted\"><i class=\"fas fa-clock me-1\"></i> Last updated: ")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		var templ_7745c5c3_Var9 string
+		templ_7745c5c3_Var9, templ_7745c5c3_Err = templ.JoinStringErrs(data.LastUpdated.Format("2006-01-02 15:04:05"))
+		if templ_7745c5c3_Err != nil {
+			return templ.Error{Err: templ_7745c5c3_Err, FileName: `view/app/mount_clients.templ`, Line: 106, Col: 67}
+		}
+		_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var9))
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 15, "</small></div></div></div><script>\n\tfunction exportMountClients() {\n\t\tconst rows = Array.from(document.querySelectorAll('#mountClientsTable tbody tr')).map(row => {\n\t\t\tconst cells = row.querySelectorAll('td');\n\t\t\tif (cells.length > 1) {\n\t\t\t\treturn {\n\t\t\t\t\taddress: cells[0].textContent.trim(),\n\t\t\t\t\ttype: cells[1].textContent.trim(),\n\t\t\t\t\tpath: cells[2].textContent.trim(),\n\t\t\t\t\tclientId: cells[3].textContent.trim(),\n\t\t\t\t\tconnected: cells[4].textContent.trim(),\n\t\t\t\t\tfiler: cells[5].textContent.trim()\n\t\t\t\t};\n\t\t\t}\n\t\t\treturn null;\n\t\t}).filter(row => row !== null);\n\n\t\tconst csvContent = \"data:text/csv;charset=utf-8,\" +\n\t\t\t\"Address,Type,Path Prefix,Client ID,Connected Since,Filer\\n\" +\n\t\t\trows.map(r => '\"' + r.address + '\",\"' + r.type + '\",\"' + r.path + '\",\"' + r.clientId + '\",\"' + r.connected + '\",\"' + r.filer + '\"').join(\"\\n\");\n\n\t\tconst encodedUri = encodeURI(csvContent);\n\t\tconst link = document.createElement(\"a\");\n\t\tlink.setAttribute(\"href\", encodedUri);\n\t\tlink.setAttribute(\"download\", \"mount-clients.csv\");\n\t\tdocument.body.appendChild(link);\n\t\tlink.click();\n\t\tdocument.body.removeChild(link);\n\t}\n\t</script>")
+		if templ_7745c5c3_Err != nil {
+			return templ_7745c5c3_Err
+		}
+		return nil
+	})
+}
+
+var _ = templruntime.GeneratedTemplate
@@ -32,7 +32,7 @@ templ Layout(view ViewContext, content templ.Component) {
  isStoragePage := strings.HasPrefix(currentPath, "/storage/volumes") || strings.HasPrefix(currentPath, "/storage/ec-shards") || strings.HasPrefix(currentPath, "/storage/collections")

  // Detect if we're on a cluster page (but not storage page) to keep submenu expanded
-  isClusterPage := (strings.HasPrefix(currentPath, "/cluster/masters") || strings.HasPrefix(currentPath, "/cluster/volume-servers") || strings.HasPrefix(currentPath, "/cluster/filers") || strings.HasPrefix(currentPath, "/cluster/s3"))
+  isClusterPage := (strings.HasPrefix(currentPath, "/cluster/masters") || strings.HasPrefix(currentPath, "/cluster/volume-servers") || strings.HasPrefix(currentPath, "/cluster/filers") || strings.HasPrefix(currentPath, "/cluster/s3") || strings.HasPrefix(currentPath, "/cluster/mount-clients"))

  isDefaultWorkerPage := currentPath == "/plugin" ||
    strings.HasPrefix(currentPath, "/plugin/configuration") ||
@@ -139,6 +139,11 @@ templ Layout(view ViewContext, content templ.Component) {
                                            <i class="fas fa-cloud me-2"></i>S3 Servers
                                        </a>
                                    </li>
+                                    <li class="nav-item">
+                                        <a class="nav-link py-2" href={ view.P("/cluster/mount-clients") }>
+                                            <i class="fas fa-plug me-2"></i>Mount Clients
+                                        </a>
+                                    </li>
                                </ul>
                            </div>
                        </li>
@@ -78,6 +78,11 @@ service SeaweedFiler {
    rpc SubscribeLocalMetadata (SubscribeMetadataRequest) returns (stream SubscribeMetadataResponse) {
    }

+    // List the metadata subscribers currently connected to this filer
+    // (FUSE mounts, S3, filer.sync, peer filers, ...).
+    rpc ListMetadataSubscribers (ListMetadataSubscribersRequest) returns (ListMetadataSubscribersResponse) {
+    }
+
    rpc KvGet (KvGetRequest) returns (KvGetResponse) {
    }

@@ -621,6 +626,22 @@ message SubscribeMetadataResponse {
    repeated SubscribeMetadataResponse events = 4; // batch of additional events (backlog catch-up)
    repeated LogFileChunkRef log_file_refs = 5;    // log file chunk refs for client direct-read
 }
+message ListMetadataSubscribersRequest {
+    repeated string client_types = 1; // optional filter by client type, e.g. "mount"; empty = all
+}
+message ListMetadataSubscribersResponse {
+    repeated MetadataSubscriber subscribers = 1;
+}
+message MetadataSubscriber {
+    string client_name = 1;     // "<type>@<address>"
+    string client_type = 2;     // e.g. "mount", "sw-vfs", "s3", "filer:<addr>"
+    string address = 3;         // client peer address
+    string path_prefix = 4;     // subscribed path prefix
+    int32 client_id = 5;
+    int32 client_epoch = 6;
+    int64 connected_at_ns = 7;
+    string filer_address = 8;   // the filer this subscriber is connected to
+}
 // A persisted log file that the client can read directly from volume servers.
 // The file format is: [4-byte size | protobuf LogEntry] repeated.
 // Each LogEntry.Data contains a marshaled SubscribeMetadataResponse.
@@ -42,6 +42,7 @@ const (
 	SeaweedFiler_TraverseBfsMetadata_FullMethodName             = "/filer_pb.SeaweedFiler/TraverseBfsMetadata"
 	SeaweedFiler_SubscribeMetadata_FullMethodName               = "/filer_pb.SeaweedFiler/SubscribeMetadata"
 	SeaweedFiler_SubscribeLocalMetadata_FullMethodName          = "/filer_pb.SeaweedFiler/SubscribeLocalMetadata"
+	SeaweedFiler_ListMetadataSubscribers_FullMethodName         = "/filer_pb.SeaweedFiler/ListMetadataSubscribers"
 	SeaweedFiler_KvGet_FullMethodName                           = "/filer_pb.SeaweedFiler/KvGet"
 	SeaweedFiler_KvPut_FullMethodName                           = "/filer_pb.SeaweedFiler/KvPut"
 	SeaweedFiler_CacheRemoteObjectToLocalCluster_FullMethodName = "/filer_pb.SeaweedFiler/CacheRemoteObjectToLocalCluster"
@@ -81,6 +82,9 @@ type SeaweedFilerClient interface {
 	TraverseBfsMetadata(ctx context.Context, in *TraverseBfsMetadataRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[TraverseBfsMetadataResponse], error)
 	SubscribeMetadata(ctx context.Context, in *SubscribeMetadataRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[SubscribeMetadataResponse], error)
 	SubscribeLocalMetadata(ctx context.Context, in *SubscribeMetadataRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[SubscribeMetadataResponse], error)
+	// List the metadata subscribers currently connected to this filer
+	// (FUSE mounts, S3, filer.sync, peer filers, ...).
+	ListMetadataSubscribers(ctx context.Context, in *ListMetadataSubscribersRequest, opts ...grpc.CallOption) (*ListMetadataSubscribersResponse, error)
 	KvGet(ctx context.Context, in *KvGetRequest, opts ...grpc.CallOption) (*KvGetResponse, error)
 	KvPut(ctx context.Context, in *KvPutRequest, opts ...grpc.CallOption) (*KvPutResponse, error)
 	CacheRemoteObjectToLocalCluster(ctx context.Context, in *CacheRemoteObjectToLocalClusterRequest, opts ...grpc.CallOption) (*CacheRemoteObjectToLocalClusterResponse, error)
@@ -382,6 +386,16 @@ func (c *seaweedFilerClient) SubscribeLocalMetadata(ctx context.Context, in *Sub
 // This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
 type SeaweedFiler_SubscribeLocalMetadataClient = grpc.ServerStreamingClient[SubscribeMetadataResponse]

+func (c *seaweedFilerClient) ListMetadataSubscribers(ctx context.Context, in *ListMetadataSubscribersRequest, opts ...grpc.CallOption) (*ListMetadataSubscribersResponse, error) {
+	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
+	out := new(ListMetadataSubscribersResponse)
+	err := c.cc.Invoke(ctx, SeaweedFiler_ListMetadataSubscribers_FullMethodName, in, out, cOpts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
 func (c *seaweedFilerClient) KvGet(ctx context.Context, in *KvGetRequest, opts ...grpc.CallOption) (*KvGetResponse, error) {
 	cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
 	out := new(KvGetResponse)
@@ -509,6 +523,9 @@ type SeaweedFilerServer interface {
 	TraverseBfsMetadata(*TraverseBfsMetadataRequest, grpc.ServerStreamingServer[TraverseBfsMetadataResponse]) error
 	SubscribeMetadata(*SubscribeMetadataRequest, grpc.ServerStreamingServer[SubscribeMetadataResponse]) error
 	SubscribeLocalMetadata(*SubscribeMetadataRequest, grpc.ServerStreamingServer[SubscribeMetadataResponse]) error
+	// List the metadata subscribers currently connected to this filer
+	// (FUSE mounts, S3, filer.sync, peer filers, ...).
+	ListMetadataSubscribers(context.Context, *ListMetadataSubscribersRequest) (*ListMetadataSubscribersResponse, error)
 	KvGet(context.Context, *KvGetRequest) (*KvGetResponse, error)
 	KvPut(context.Context, *KvPutRequest) (*KvPutResponse, error)
 	CacheRemoteObjectToLocalCluster(context.Context, *CacheRemoteObjectToLocalClusterRequest) (*CacheRemoteObjectToLocalClusterResponse, error)
@@ -601,6 +618,9 @@ func (UnimplementedSeaweedFilerServer) SubscribeMetadata(*SubscribeMetadataReque
 func (UnimplementedSeaweedFilerServer) SubscribeLocalMetadata(*SubscribeMetadataRequest, grpc.ServerStreamingServer[SubscribeMetadataResponse]) error {
 	return status.Error(codes.Unimplemented, "method SubscribeLocalMetadata not implemented")
 }
+func (UnimplementedSeaweedFilerServer) ListMetadataSubscribers(context.Context, *ListMetadataSubscribersRequest) (*ListMetadataSubscribersResponse, error) {
+	return nil, status.Error(codes.Unimplemented, "method ListMetadataSubscribers not implemented")
+}
 func (UnimplementedSeaweedFilerServer) KvGet(context.Context, *KvGetRequest) (*KvGetResponse, error) {
 	return nil, status.Error(codes.Unimplemented, "method KvGet not implemented")
 }
@@ -1020,6 +1040,24 @@ func _SeaweedFiler_SubscribeLocalMetadata_Handler(srv interface{}, stream grpc.S
 // This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
 type SeaweedFiler_SubscribeLocalMetadataServer = grpc.ServerStreamingServer[SubscribeMetadataResponse]

+func _SeaweedFiler_ListMetadataSubscribers_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+	in := new(ListMetadataSubscribersRequest)
+	if err := dec(in); err != nil {
+		return nil, err
+	}
+	if interceptor == nil {
+		return srv.(SeaweedFilerServer).ListMetadataSubscribers(ctx, in)
+	}
+	info := &grpc.UnaryServerInfo{
+		Server:     srv,
+		FullMethod: SeaweedFiler_ListMetadataSubscribers_FullMethodName,
+	}
+	handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+		return srv.(SeaweedFilerServer).ListMetadataSubscribers(ctx, req.(*ListMetadataSubscribersRequest))
+	}
+	return interceptor(ctx, in, info, handler)
+}
+
 func _SeaweedFiler_KvGet_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
 	in := new(KvGetRequest)
 	if err := dec(in); err != nil {
@@ -1275,6 +1313,10 @@ var SeaweedFiler_ServiceDesc = grpc.ServiceDesc{
 			MethodName: "GetFilerConfiguration",
 			Handler:    _SeaweedFiler_GetFilerConfiguration_Handler,
 		},
+		{
+			MethodName: "ListMetadataSubscribers",
+			Handler:    _SeaweedFiler_ListMetadataSubscribers_Handler,
+		},
 		{
 			MethodName: "KvGet",
 			Handler:    _SeaweedFiler_KvGet_Handler,
@@ -165,7 +165,7 @@ func (fs *FilerServer) SubscribeMetadata(req *filer_pb.SubscribeMetadataRequest,
 	ctx := stream.Context()
 	peerAddress := findClientAddress(ctx, 0)

-	isReplacing, alreadyKnown, clientName := fs.addClient("", req.ClientName, peerAddress, req.ClientId, req.ClientEpoch)
+	isReplacing, alreadyKnown, clientName := fs.addClient("", req.ClientName, peerAddress, req.PathPrefix, req.ClientId, req.ClientEpoch)
 	if isReplacing {
 		fs.filer.MetaAggregator.ListenersCond.Broadcast() // nudges the subscribers that are waiting
 	} else if alreadyKnown {
@@ -314,7 +314,7 @@ func (fs *FilerServer) SubscribeLocalMetadata(req *filer_pb.SubscribeMetadataReq
 	// use negative client id to differentiate from addClient()/deleteClient() used in SubscribeMetadata()
 	req.ClientId = -req.ClientId

-	isReplacing, alreadyKnown, clientName := fs.addClient("local", req.ClientName, peerAddress, req.ClientId, req.ClientEpoch)
+	isReplacing, alreadyKnown, clientName := fs.addClient("local", req.ClientName, peerAddress, req.PathPrefix, req.ClientId, req.ClientEpoch)
 	if isReplacing {
 		fs.listenersCond.Broadcast() // nudges the subscribers that are waiting
 	} else if alreadyKnown {
@@ -661,9 +661,9 @@ func (fs *FilerServer) eachEventNotificationFn(req *filer_pb.SubscribeMetadataRe
 	}
 }

-func (fs *FilerServer) addClient(prefix string, clientType string, clientAddress string, clientId int32, clientEpoch int32) (isReplacing, alreadyKnown bool, clientName string) {
+func (fs *FilerServer) addClient(scope string, clientType string, clientAddress string, pathPrefix string, clientId int32, clientEpoch int32) (isReplacing, alreadyKnown bool, clientName string) {
 	clientName = clientType + "@" + clientAddress
-	glog.V(0).Infof("+ %v listener %v clientId %v clientEpoch %v", prefix, clientName, clientId, clientEpoch)
+	glog.V(0).Infof("+ %v listener %v clientId %v clientEpoch %v", scope, clientName, clientId, clientEpoch)
 	if clientId != 0 {
 		fs.knownListenersLock.Lock()
 		defer fs.knownListenersLock.Unlock()
@@ -671,6 +671,15 @@ func (fs *FilerServer) addClient(prefix string, clientType string, clientAddress
 		if !found || epoch < clientEpoch {
 			fs.knownListeners[clientId] = clientEpoch
 			isReplacing = true
+			fs.subscribers[clientId] = &metadataSubscriber{
+				clientName:    clientName,
+				clientType:    clientType,
+				address:       clientAddress,
+				pathPrefix:    pathPrefix,
+				clientId:      clientId,
+				clientEpoch:   clientEpoch,
+				connectedAtNs: time.Now().UnixNano(),
+			}
 		} else {
 			alreadyKnown = true
 		}
@@ -678,14 +687,15 @@ func (fs *FilerServer) addClient(prefix string, clientType string, clientAddress
 	return
 }

-func (fs *FilerServer) deleteClient(prefix string, clientName string, clientId int32, clientEpoch int32) {
-	glog.V(0).Infof("- %v listener %v clientId %v clientEpoch %v", prefix, clientName, clientId, clientEpoch)
+func (fs *FilerServer) deleteClient(scope string, clientName string, clientId int32, clientEpoch int32) {
+	glog.V(0).Infof("- %v listener %v clientId %v clientEpoch %v", scope, clientName, clientId, clientEpoch)
 	if clientId != 0 {
 		fs.knownListenersLock.Lock()
 		defer fs.knownListenersLock.Unlock()
 		epoch, found := fs.knownListeners[clientId]
 		if found && epoch <= clientEpoch {
 			delete(fs.knownListeners, clientId)
+			delete(fs.subscribers, clientId)
 		}
 	}
 }
@@ -0,0 +1,63 @@
+package weed_server
+
+import (
+	"context"
+	"sort"
+
+	"github.com/seaweedfs/seaweedfs/weed/pb/filer_pb"
+)
+
+// metadataSubscriber is the live record of one connected metadata subscriber
+// (a FUSE mount, S3, filer.sync, peer filer, ...). It is created in addClient
+// and removed in deleteClient, so the registry reflects currently-connected
+// clients only. Guarded by FilerServer.knownListenersLock.
+type metadataSubscriber struct {
+	clientName    string // "<type>@<address>"
+	clientType    string // e.g. "mount", "sw-vfs", "s3", "filer:<addr>"
+	address       string
+	pathPrefix    string
+	clientId      int32
+	clientEpoch   int32
+	connectedAtNs int64
+}
+
+// ListMetadataSubscribers returns the metadata subscribers currently connected
+// to this filer, optionally filtered by client type. Each filer only knows its
+// own subscribers, so cluster-wide callers query every filer and aggregate.
+func (fs *FilerServer) ListMetadataSubscribers(ctx context.Context, req *filer_pb.ListMetadataSubscribersRequest) (*filer_pb.ListMetadataSubscribersResponse, error) {
+	typeFilter := make(map[string]bool, len(req.ClientTypes))
+	for _, t := range req.ClientTypes {
+		typeFilter[t] = true
+	}
+
+	filerAddress := string(fs.option.Host)
+
+	fs.knownListenersLock.Lock()
+	subs := make([]*filer_pb.MetadataSubscriber, 0, len(fs.subscribers))
+	for _, s := range fs.subscribers {
+		if len(typeFilter) > 0 && !typeFilter[s.clientType] {
+			continue
+		}
+		subs = append(subs, &filer_pb.MetadataSubscriber{
+			ClientName:    s.clientName,
+			ClientType:    s.clientType,
+			Address:       s.address,
+			PathPrefix:    s.pathPrefix,
+			ClientId:      s.clientId,
+			ClientEpoch:   s.clientEpoch,
+			ConnectedAtNs: s.connectedAtNs,
+			FilerAddress:  filerAddress,
+		})
+	}
+	fs.knownListenersLock.Unlock()
+
+	// Stable ordering so the admin page doesn't shuffle rows between refreshes.
+	sort.Slice(subs, func(i, j int) bool {
+		if subs[i].ClientName != subs[j].ClientName {
+			return subs[i].ClientName < subs[j].ClientName
+		}
+		return subs[i].PathPrefix < subs[j].PathPrefix
+	})
+
+	return &filer_pb.ListMetadataSubscribersResponse{Subscribers: subs}, nil
+}
@@ -113,6 +113,9 @@ type FilerServer struct {
 	// track known metadata listeners
 	knownListenersLock sync.Mutex
 	knownListeners     map[int32]int32
+	// live metadata subscribers (FUSE mounts, S3, peer filers, ...) keyed by
+	// clientId, guarded by knownListenersLock. Exposed via ListMetadataSubscribers.
+	subscribers map[int32]*metadataSubscriber

 	// deduplicates concurrent remote object caching operations
 	remoteCacheGroup singleflight.Group
@@ -183,6 +186,7 @@ func NewFilerServer(defaultMux, readonlyMux *http.ServeMux, option *FilerOption)
 		option:                option,
 		grpcDialOption:        security.LoadClientTLS(util.GetViper(), "grpc.filer"),
 		knownListeners:        make(map[int32]int32),
+		subscribers:           make(map[int32]*metadataSubscriber),
 		inFlightDataLimitCond: sync.NewCond(new(sync.Mutex)),
 		recentCopyRequests:    make(map[string]recentCopyRequest),
 		CredentialManager:     option.CredentialManager,
Author	SHA1	Message	Date
Chris Lu	81daa87f8b	rdma-engine: ranged RDMA READ (offset+size) in rdma-client Match the real read API shape (file_id + offset + size): rdma-client takes optional [offset] [size] and RDMA-READs only that sub-range from the remote MR (remote_addr + offset). Verified over rxe0: reading [3 MiB, 4 MiB) of an 8 MiB target file yields bytes whose md5 matches the source slice.	2026-06-14 18:52:54 -07:00
Chris Lu	49c8d123b5	rdma-engine: standalone cross-process RDMA-READ target + client Split the in-process loopback proof into separate target/client binaries so the {remote_addr, rkey} addressing genuinely crosses a process boundary (the precursor to an engine<->volume-server split): - rdma-target <bind_ip> <port> <file>: loads a file, registers it for REMOTE_READ, serves one client over rxe. - rdma-client <ip> <port> <out>: connects, RDMA-READs the whole buffer into a fresh local buffer, writes it out. Verified over SoftRoCE (rxe0): an 8 MiB real file served by the target process is RDMA-read by a separate client process (status=Success, 8388608 bytes) and md5 matches the source. Real cross-process RDMA of real data; serving from a needle-backed buffer in the volume server is the next (multi-week) step.	2026-06-14 18:50:06 -07:00
Chris Lu	96722ebf4b	rdma-engine: real libibverbs/rdma_cm RDMA READ datapath (proof of plumbing) The engine backend was a mock: post_read fabricated byte=i%256 and never touched a wire (the Go sidecar even discarded the engine reply and re-faked the same pattern). Add a real datapath behind a new 'real-rdma' cargo feature: - src/rdma_real.rs: RealRdmaContext (connect via rdma_cm, register_memory, a genuine IBV_WR_RDMA_READ via rdma_post_read, poll_completion, device_info) + RealResponder (rdma_reg_read REMOTE_READ source that hands the client {addr,rkey,len}). - src/bin/rdma_loopback.rs: proof binary. A responder registers a buffer with a verifiable pattern (REALRDMA magic header, not the mock's i%256); the client RDMA-READs it and verifies the bytes match. - REAL_RDMA.md: build/run + design notes. Verified on a SoftRoCE (rxe) loopback (Linux 6.12.90, rdma_rxe): 1 MiB IBV_WR_RDMA_READ completes status=Success, dest head 'REALRDMA', 1048576 bytes match -> REAL-RDMA-READ-SUCCESS. (rxe on a 2-vCPU VM is a correctness proof, not a benchmark.) vendor/rdma-sys: rdma-sys 0.3.0 with bindgen bumped to 0.69.4 (the crates.io 0.59.2 pin cannot parse rdma-core 56 headers); redirected via [patch.crates-io]. Temporary workaround; see REAL_RDMA.md. Phase-1 proof of plumbing only. A demonstrable zero-copy read of real volume data still needs a needle-backed RDMA target in the volume server, a control-plane handshake, an IPC-contract change, and mount wiring (multi-week; deferred).	2026-06-14 17:54:42 -07:00
Chris Lu	b883fedc23	admin: add connected mount clients page and dashboard section The filer is the authority on who is subscribed to its metadata stream (FUSE/VFS mounts, S3, peer filers, ...), but its in-memory listener registry only tracked clientId->epoch and was not exposed. - Enrich the filer subscriber registry with name/type/address/path/ connected-time, populated in addClient and cleared in deleteClient so it reflects currently-connected clients only. - Add a ListMetadataSubscribers filer gRPC (optional client-type filter). - Admin server fans out to every filer, filters to mount types ("mount" Go weed mount, "sw-vfs" Rust VFS), and renders a new Cluster > Mount Clients page plus a Mount Clients dashboard section. Read-only; no behavior change to the subscribe hot path.	2026-06-14 17:22:05 -07:00