mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-30 19:55:48 +00:00
Khris RichardsonandGitHub
95e0b74fb6
* s3/iam: retry a failed OIDC provider refresh until the store answers RefreshOIDCProvidersFromStore reports a failure and nothing retries it. Its callers can't: a metadata-subscription event reports each change once, so a refresh that found the filer unreachable on it (the filer restarting, say) left a peer's new provider untrusted, or a deleted one trusted, until some unrelated later change. The refresh after a local IAM API mutation has the same shape. Only the startup load retried. A failed refresh now retries in the background with the startup load's backoff until the store answers. At most one retry runs, however many refreshes fail meanwhile, and installing another store cancels it. The startup load uses the same path instead of its own. Seen on a SeaweedFS operator cluster whose filer restarted while an S3OIDCProvider was created: the gateway logged "OIDC provider refresh after /etc/iam/oidc-providers change failed: ... fail to dial". The operator's periodic re-apply happened to recover it; an IAM API client would not. * s3/iam: never retry or apply a superseded OIDC provider store, and never drop a failure during a retry Review of the retry (#11521) found two ways to lose the state it protects. A refresh of store A that failed as store B was installed could start a retry for A after B's install had cancelled retries. Nothing cancelled it, and when A answered it replaced B's providers in STS. The installed store now changes under the retry lock, a store that is no longer current gets no retry, and a snapshot of a replaced store is never handed to STS, even when the refresh listed it just before the swap. A refresh that failed while a retry ran was dropped by the at-most-one guard, though the retry might already have listed an older snapshot, so the change the failed refresh would have loaded stayed unloaded. The retry now runs once more after its success when a failure arrived meanwhile. Each has a test that fails without its guard.