mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-22 07:54:17 +00:00
master
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chris LuandGitHub
|
0ca1c19821 |
s3api: unify auth error handling across s3tables, iceberg and lance (#11381)
* s3api: fail closed when S3 Tables signature verification fails * s3api: avoid nil Account dereference in S3 Tables auth log * iceberg: return auth error instead of falling back to DefaultAllow * lance: return auth error instead of falling back to DefaultAllow * s3api: stop trusting client-supplied s3-account-id The header is set by the server after successful authentication; scrub inbound values alongside the other internal headers, and apply the same admin guard to the header fallback branch of getAccountID that the identity branch already has. * test: cover table-catalog auth wrappers and principal resolution * test: configure anonymous identity where catalog clients do not sign * s3api: scrub s3-account-id after signature verification |
||
 Nguyễn Đăng Minh LựcandGitHub
|
c968084b34 |
iceberg: fix OAuth token expiry handling (401 + token-exchange + configurable TTL) (#11242)
* iceberg: return 401 for invalid or expired Bearer tokens BUG-0001: when the OAuth JWT expired, Server.Auth fell through to the S3 SigV4 authenticator, which rejects the "Authorization: Bearer" scheme with NotImplemented — a 501. Iceberg clients (Java OAuth2Manager, pyiceberg) only refresh tokens on 401, so they retried the dead token forever: RisingWave sinks stalled and Doris catalog queries failed every token TTL (1h) until the client process was restarted. A request carrying a Bearer header is an Iceberg REST client: answer 401 (+ WWW-Authenticate: Bearer, RFC 6750) when the token fails, and only fall through to the S3 authenticator when no Bearer header is present. * iceberg: make OAuth token TTL configurable via ICEBERG_OAUTH_TOKEN_EXPIRY BUG-0001 follow-up: production evidence shows Iceberg Java 1.10.x clients (RisingWave connector node, Doris FE) never re-fetch tokens on 401 — the sink stalled again on token expiry even with the 501→401 fix, and no POST /v1/oauth/tokens appeared in server logs across dozens of retries. 401 is necessary but not sufficient for these clients. The TTL was hardcoded to 3600 with no knob. Read the expiry (seconds) from ICEBERG_OAUTH_TOKEN_EXPIRY, defaulting to 3600, so deployments can issue longer-lived tokens (e.g. 86400) to survive client restart cycles. * iceberg: support OAuth token exchange (RFC 8693) for client refresh Decompiling the Iceberg Java 1.10.1 client bundled with Doris FE showed the missing half of BUG-0001: OAuth2Manager refreshes via token-exchange (AuthConfig.exchangeEnabled defaults to true — the client_credentials re-fetch branch only runs with exchange disabled), so a server that only accepts client_credentials leaves Iceberg clients unable to ever refresh their token, regardless of 401 correctness. Accept grant_type=urn:ietf:params:oauth:grant-type:token-exchange on POST /v1/oauth/tokens: verify the subject_token signature against the issuing credential, allow exchange within a recovery grace window (max(2*TTL, 1h), capped 24h) so clients holding tokens that expired while the grant was unsupported recover without a restart, and mint a fresh access token with the configured TTL. * iceberg: harden OAuth token exchange and Bearer matching per review - match the Bearer scheme case-insensitively (RFC 7235), like authenticateBearer already does - accept optional client authentication on the token-exchange grant (Basic or form credentials, bound to the subject token's client); expired subject tokens now require it. Iceberg Java's proactive refresh sends Bearer-only headers, so the grant cannot require it - reject subject tokens without an exp claim, and re-check the issuer on the verified claims - unauthenticated exchange cannot extend the lifetime past the subject token's own expiry (no chain-refresh from a leaked token) - return 400 invalid_grant per RFC 6749 §5.2 (was 401) - include issued_token_type on exchange responses (RFC 8693) - clamp ICEBERG_OAUTH_TOKEN_EXPIRY to 365d so Duration math cannot overflow into already-expired tokens * iceberg: give authenticated token exchanges a fresh full TTL The remaining-lifetime cap only guards unauthenticated (Bearer-only) exchanges; an authenticated client renewing a live token must get the full configured TTL, matching client_credentials. * iceberg: reject token exchange when no lifetime remains A Bearer-only exchange with under a second of subject lifetime would mint a token with expires_in: 0. Reject with invalid_grant instead. * iceberg: pin near-expiry test token to the next second boundary jwt/v5 serializes exp at one-second precision, so a 300 ms offset can round into the current second and route the test through the expired branch instead of the ttlSeconds<=0 guard. Mint the subject with the next whole-second expiry: live at exchange time, deterministically under a second of remaining lifetime. * iceberg: drop internal ticket reference from comments * iceberg: clamp oversized OAuth TTLs on 32-bit platforms strconv.Atoi on an int-sized value fails with ErrRange on 386, so an oversized ICEBERG_OAUTH_TOKEN_EXPIRY silently fell back to the default instead of clamping. Parse in 64-bit space and clamp, then narrow. * iceberg: make OAuth TTL narrowing explicit * iceberg: disable legacy OAuth in PyIceberg integration tests |
||
|
Chris LuandGitHub
|
0dfaa103d0 |
test: take a table through its whole life, for Iceberg and Lance (#10862)
* lance worker: share the integration tests' scaffolding The recorder that keeps what a handler sent, the config builder and the storage-option fallback all lived inside compaction.rs, so a second test binary would have had to copy them. They move to tests/common. The fallback now reads AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_ENDPOINT_URL from the environment, defaulting to what it used before. A harness can then point these tests at a gateway that checks what it is given rather than one that accepts anything. * lance worker: maintain one named table, for a harness to drive Compacts and cleans up whatever WEED_LANCE_TABLE names, through the handlers' own detect-then-execute path: a proposal the worker would not have made is not one worth running. The existing tests seed the tables they check. This one deliberately does not, so a harness that has already written a table and knows what is in it can have the real handlers maintain it and then read it back. * test: take a table through its whole life, for Iceberg and Lance Created in the catalog, filled by a real client, maintained by the worker, read again, dropped. The step nothing was checking is the read after maintenance: compaction once rewrote every dictionary-encoded column onto a single value and shipped, because the maintenance tests were thorough about sequence numbers, manifest entries and metadata versions and none of them opened the parquet file the worker had just written. So the assertion is a tally - row count, the cardinality of each dictionary-encoded column, and an md5 over whole rows - taken before maintenance and again after, required to be equal. The cardinalities name the failure that happened; the digest catches a rewrite that keeps every column's cardinality and hands the values to the wrong rows. A compaction that merged nothing fails rather than passes, or the read afterwards is checking a file the worker never wrote. The Iceberg half runs two clients. DuckDB is the one the corruption was reported against and the only one here that writes the deprecated PLAIN_DICTIONARY encoding, which parquet-go normalizes away on write, so a Go writer cannot produce it. PyIceberg writes the modern spelling. Pinning parquet-go back to v0.30.1 fails the DuckDB half and passes the PyIceberg one, which is why both are here. Lance maintenance lives in the Rust worker, so it runs there where cargo is installed and through the two lance calls those handlers wrap where it is not. WEED_LANCE_MAINTENANCE picks one instead of letting the test guess. * ci: run the table lifecycle tests CI maintains the Lance table through the lance library rather than the worker: a cold build of the lance crate costs more than the glue it would be checking, and the worker's own tests cover its handlers. The suite drives the Iceberg maintenance worker, so a change to it now triggers this workflow too. * test: let the lifecycle harness fail instead of skipping Setup failures all exited zero, so a cluster that would not come up, or a port allocation that lost, reported a green run for code nothing had executed. That is the failure mode this whole directory exists to close, and it was in the harness itself. Only a checkout without a weed binary skips now, and it runs the tests so each one says so rather than the package quietly passing. Everything else fails. The filer existence probe gets a deadline while I am here: it ran without one, so an unresponsive filer would hang the suite past every timeout the clients have. * test: make the lifecycle checks check what they claim to Three of them could pass without having looked. The DuckDB skip matched "syntax error", "not implemented" and "Failed to load" anywhere in the output, in any phase. A parse error in the SQL this test generates, or a refusal from our own catalog, would have taken the only coverage of the PLAIN_DICTIONARY encoding out of CI and left it green. It now matches the extension failing to install, and only in the phase that installs it. Everything past LOAD is ours and fails. The digests covered id, category and value. Compaction rewrites the whole row, so a defect confined to ts, or to a Lance vector, changed nothing either side of maintenance. Every persisted column goes in now, ts as microseconds so no timezone sits between the two runs. The Lance drop check caught every exception as proof the dataset was gone. pylance turns credential and transport failures into the same ValueError, so it only accepts the message that means not found. * docs: say up front which maintenance path the Lance half takes The opening summary said the worker maintains both tables. It maintains the Iceberg one always and the Lance one only where cargo is installed, which is not what CI does. |