appview: stop serving service tokens past their expiry, and challenge the client when the hold rejects one

Seen in production on 2026-09-11: three cold pulls of a 22-layer image failed
with BLOB_UNKNOWN for layers that exist. The hold had answered 403 "service
token authentication failed: token has expired", and the same blobs served
fine a minute later.

Three things lined up. The registry middleware's validation cache kept a
fetched service token for a flat 45 seconds regardless of its real remaining
life, so a token fetched with 12 seconds left was still handed to the hold
half a minute after it died. The registry JWT is stamped from the auth cache's
expiry, which trailed the real exp by only 10 seconds, while distribution
accepts a JWT for 60 seconds past its exp, so a client could hold an accepted
JWT for most of a minute after the credential behind it was gone. And the
hold's 403 was flattened to BLOB_UNKNOWN, so the client failed instead of
re-authenticating.

Now the validation cache bounds an entry by the token's exp minus a shared
ServiceTokenSafetyMargin of 60 seconds, the same margin the auth cache and the
JWT stamp use, chosen to equal distribution's leeway so the last instant a JWT
is accepted is the service token's real exp. A PDS that grants less than the
margin gets half its remaining life instead of an already-past deadline. When
the hold rejects the service token as expired or missing, the appview drops
both cached copies and returns a 401 challenge so Docker and crane re-run the
token dance and retry; a genuine permission denial stays a 403, and a hold
that is down still maps to blob unknown.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EvFJr4Dwz8p2NDAeXmgmBt

pkg/auth/token/handler.go

@@ -710,11 +710,17 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 			return
 		}
 		if !res.expiresAt.IsZero() {
-			// Cap JWT lifetime at the service-auth's expiry. The cache's
-			// expiresAt already includes a 10s safety margin
-			// (pkg/auth/cache.go:71), so this guarantees the service-auth
-			// is still cache-valid for any /v2/* request the JWT can
-			// authorize. We never extend beyond the configured default.
+			// Cap JWT lifetime at the service-auth's expiry, stamping exactly
+			// the value the fetcher returned. That value already has
+			// auth.ServiceTokenSafetyMargin subtracted from the PDS-granted
+			// exp, and the margin is deliberately >= distribution's
+			// token.Leeway (60s): the registry auth package accepts this JWT
+			// for Leeway past its exp, so the last instant a client can use it
+			// is still inside the service token's real life. Do not shrink the
+			// margin below that leeway or the gap reopens, and Docker never
+			// gets the 401 that would make it re-authenticate.
+			//
+			// We never extend beyond the configured default.
 			until := time.Until(res.expiresAt)
 			if until < issueExp {
 				issueExp = until