The existing tests covered updateCrewTierWithRetry and UpdateCrewTierOnHold.
UpdateCrewTierOnAllHolds -- the function the Stripe webhook actually calls, and
whose error decides whether a paid upgrade is retried or dropped -- had none.
Three cases: the joined error names every failing hold and not the one that
succeeded; a hold that accepts and never answers does not starve the holds
after it (mutation-verified by making the fan-out serial, which leaves the
healthy hold contacted zero times); and a context deadline aborts the retry
loop rather than running to tierUpdateMaxAttempts.
That last one records a real mismatch rather than an intent. Three attempts at
a 5s client timeout need ~15s, and the webhook allows the whole fan-out 10s, so
under a hang the budget funds two attempts and never three -- confirmed against
a blackholed hold on the dev stack, which failed at exactly 10.0s with a bare
context error rather than the "after N attempts" wrapper. If either constant or
the deadline moves, that test is where the arithmetic gets re-checked.
Also covers the other half in pkg/billing: a fan-out failure has to reach
Stripe as a 5xx and leave stripe_processed_events empty. A hold that is briefly
down otherwise costs the customer their tier permanently -- the same shape of
loss as the customer-lookup hole, one layer further out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VwxF2N3HuZ8xSkx6nkirgB
Webhook delivery was neither idempotent nor order-safe, and every failure
returned 400, which Stripe does not retry. A transient DB or hold error
therefore dropped a subscription change silently and permanently.
- New stripe_processed_events table: event_id as primary key dedups
redelivery, and event_created per customer drops stale out-of-order
deliveries.
- HandleWebhook distinguishes ErrWebhookSignature (400, no retry) from
ErrWebhookProcessing (500, Stripe redelivers). The event handlers
return errors instead of swallowing them. ErrBillingDisabled maps to
400: the route is mounted but billing is off, so redelivery can never
succeed and Stripe should stop rather than retry to exhaustion.
- Refuse to boot when billing is enabled with an empty
STRIPE_WEBHOOK_SECRET. Stripe HMACs with the empty key, so an
attacker can reproduce the signature and the endpoint is forgeable.
- UpdateCrewTierOnAllHolds retries each hold (3 attempts, linear
backoff, 5s per request) and returns a joined error so the webhook
can fail and let Stripe redeliver.
The fan-out contacts holds concurrently rather than in sequence. Serially,
one unreachable hold burns the caller's entire 10s budget on its own
retries (3 x 5s plus backoff) and the holds after it are never contacted;
because Stripe redelivers in the same order, a persistently-down first
hold means the rest are never updated at all.
On the hold, the signature-validated sub claim is now the source of truth
for updateCrewTier: a mismatched body userDid is rejected with 403 rather
than retargeting the grant to another DID. "Not crew on this hold" is a
200 no-op, since the appview fans updates out to every managed hold and a
subscriber is not crew everywhere.
That no-op has to be told apart from a storage failure. GetCrewMember
collapsed both into one generic error, so a CAR-store failure read as
"not a member", answered 200, and let the appview record the event as
processed — losing the tier grant permanently, which is exactly the
failure mode this commit exists to prevent. Missing records now carry an
ErrCrewMemberNotFound sentinel, and anything else returns 500 so Stripe
redelivers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Evan JarrettandClaude Opus 5