BadgeprintingEvent registration · badge automation

Rotating HMAC Webhook Signing Secrets Without Dropping Events

Symptom Statement

You rotate the webhook signing secret — a Stripe whsec_, a GitHub secret, or your own HMAC key — flip the environment variable, redeploy, and within seconds the endpoint starts returning a wall of HTTP 401 signature rejects. Every in-flight delivery signed with the old secret now fails verification, the sender’s retry queue swells, and any event whose retry budget expires before you finish the cutover is gone for good. The registration row never confirms, the badge never queues, and the only trace left is a burst of signature mismatch log lines timestamped to the exact minute you rotated. This is the classic hard-cutover failure at the ingress boundary: verification is bound to a single live secret, so the instant that secret changes there is a window where legitimate, correctly-signed traffic is indistinguishable from a forgery. This page is the rotation-specific implementation of the Security Boundary Configuration stage, whose HMAC verification and deny-by-default posture this fix has to preserve while making the secret itself a moving target. The provider-side reconciliation of the underlying charges lives in payment webhook handling; here the concern is narrower — never reject a genuine signature just because a key is mid-rotation.

Hard cutover versus dual-secret overlap window during signing-secret rotation Two timelines compared. The top row, hard cutover, shows a single active secret that switches instantly from key A to key B at time T; deliveries still signed with key A that arrive after T are rejected with HTTP 401 and lost, shown as a red gap. The bottom row, overlap window, keeps both key A and key B active for a grace period spanning T; the verifier tries the incoming signature against every secret in the active set, so a delivery signed with either key verifies and is accepted, shown as a continuous green band with no gap. After the grace period, key A is retired and only key B remains. Rotation: hard cutover drops in-flight events; an overlap window does not T · rotate Hard cutover secret A active secret B active A-signed → 401 lost Overlap window secret A active (grace) secret B active either key verifies · no gap

Root Cause Analysis

The 401 burst at cutover is almost always one of four distinct mistakes. They stack, but each is independently fixable, so name the one you have before you touch the verifier.

  • Single-secret validation with no overlap window. The endpoint verifies against exactly one secret pulled from config. The moment you swap it, every delivery already signed with the previous key fails. The sender is behaving correctly; your verifier simply has no memory of the key it was using thirty seconds ago.
  • No dual-secret grace period. Even teams that know rotation is coming often flip the value in a single deploy. Without a bounded window where both the outgoing and incoming secrets are accepted, there is no safe moment to update the sender’s copy of the key — either the sender is ahead of the verifier or behind it, and one direction always rejects.
  • Cached or pinned old key. The new secret is in the vault, but a per-process cache, a warmed introspection layer, or a stale worker still holds the old value. Some replicas verify with A, some with B, and deliveries fail non-deterministically depending on which pod answers — a rotation that looks half-applied for minutes.
  • Verifying only one secret when the sender supports several. Stripe can sign a single delivery so it validates against more than one endpoint secret, and most HMAC schemes let you publish a key set. If your handler passes only the first secret to construct_event — or compares against one digest — you throw away the exact mechanism the sender gives you to rotate without downtime.

Symptom-to-Resolution Matrix

Single-secret cutover with no overlap

Symptoms

  • A sharp 401 spike that begins the instant the rotation deploy goes live and decays as the sender exhausts its retries.
  • Logs show signature mismatch for deliveries whose IDs the sender’s dashboard still lists as pending retry.

Root cause. Verification is bound to one live secret, so previous-key traffic in flight during the swap is rejected.

Fix

  1. Change the verifier to accept a set of secrets, not a scalar: load WEBHOOK_SIGNING_SECRETS as a comma-separated list and try each in turn.
  2. Verify by attempting stripe.Webhook.construct_event(payload, sig_header, secret) against each secret and accepting on the first that succeeds.
  3. Treat a delivery as forged only after every secret in the active set fails — then, and only then, return 401.

No dual-secret grace period

Symptoms

  • Rotation succeeds in staging with one sender but drops events in production where multiple integrations post to the same endpoint.
  • You cannot find a deploy order that avoids a gap: updating the sender first rejects, updating the verifier first also rejects.

Root cause. Rotation was modeled as an atomic swap instead of an overlap. There is no interval where both keys are valid, so the sender and verifier can never be updated independently.

Fix

  1. Add the new secret to the active set alongside the old one and deploy the verifier first — now it accepts both.
  2. Update every sender to sign with the new secret at your own pace, inside the grace window.
  3. After the window closes and traffic on the old key reaches zero, remove the old secret from the set and deploy again.

Cached or pinned old key

Symptoms

  • The 401 rate is partial and flapping, not total — a fraction of replicas reject while others accept.
  • Restarting a pod “fixes” it for that pod only.

Root cause. A stale per-process or in-memory cache is still serving the old secret after the vault was updated.

Fix

  1. Read the secret set from a single source of truth on each verification, or give any cache a short TTL bounded well under the sender’s retry window.
  2. On rotation, force cache invalidation across all replicas rather than waiting for organic expiry.
  3. Add the resolved secret fingerprint (a short hash prefix, never the secret) to logs so you can see which key each pod is actually using.

Verifying only one secret

Symptoms

  • The sender’s dashboard shows the delivery signed against multiple endpoint secrets, yet your handler rejects it.
  • Adding a second endpoint secret upstream has no effect on your accept rate.

Root cause. The handler passes a single secret to the verifier and discards the sender’s built-in multi-secret rotation support.

Fix

  1. Enumerate the full active secret set on every request and pass each to the verifier until one validates.
  2. Use a constant-time comparison for any hand-rolled HMAC path so the multi-secret loop does not leak timing.
  3. Cap the set size (typically two: current plus previous) so verification cost stays bounded.

Minimal Working Implementation

A single self-contained verifier. It loads an ordered secret set, tries Stripe’s construct_event against each secret, and — for non-Stripe HMAC senders — falls back to a constant-time digest compare against the same set. The verify_signature function returns the parsed event on the first secret that validates and raises only when all of them fail, which is exactly the property that makes an overlap-window rotation lossless. The verification block proves that an old-key signature and a new-key signature both pass while both keys sit in the active set. In stripe-python v5+, stripe.SignatureVerificationError is the canonical exception path.

PYTHON
import os
import hmac
import hashlib
import time
from typing import Optional

import stripe

# --- Configuration --------------------------------------------------------
# Ordered active set: current first, previous second. During a rotation both
# are present; after the grace window the previous entry is removed.
_RAW_SECRETS = os.getenv("WEBHOOK_SIGNING_SECRETS", "whsec_current,whsec_previous")
ACTIVE_SECRETS = [s.strip() for s in _RAW_SECRETS.split(",") if s.strip()]
TOLERANCE_SECONDS = 300  # reject stale timestamps, independent of key rotation


class SignatureRejected(Exception):
    """Raised only after every secret in the active set fails."""


def verify_stripe(payload: bytes, sig_header: str) -> dict:
    """Try each active secret; accept on the first that validates."""
    last_error: Optional[Exception] = None
    for secret in ACTIVE_SECRETS:
        try:
            event = stripe.Webhook.construct_event(payload, sig_header, secret)
            return dict(event)
        except stripe.SignatureVerificationError as exc:
            last_error = exc  # this key failed; try the next one
    raise SignatureRejected(f"no active secret verified the signature: {last_error}")


def _expected_hmac(payload: bytes, secret: str, timestamp: str) -> str:
    signed = f"{timestamp}.".encode() + payload
    return hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()


def verify_hmac(payload: bytes, timestamp: str, signature: str) -> bool:
    """Generic HMAC path for non-Stripe senders, constant-time over the set."""
    if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
        raise SignatureRejected("timestamp outside tolerance window")
    for secret in ACTIVE_SECRETS:
        expected = _expected_hmac(payload, secret, timestamp)
        if hmac.compare_digest(expected, signature):  # constant-time compare
            return True
    raise SignatureRejected("no active secret matched the HMAC digest")


# --- Verification: prove both keys validate while both are active ----------
if __name__ == "__main__":
    body = b'{"id":"evt_test","type":"checkout.session.completed"}'
    ts = str(int(time.time()))
    current, previous = ACTIVE_SECRETS[0], ACTIVE_SECRETS[1]

    # A sender still on the previous key and one on the current key both pass.
    sig_prev = _expected_hmac(body, previous, ts)
    sig_curr = _expected_hmac(body, current, ts)
    assert verify_hmac(body, ts, sig_prev) is True   # old key still accepted
    assert verify_hmac(body, ts, sig_curr) is True   # new key accepted

    # A signature from a retired/forged key is rejected by the whole set.
    forged = _expected_hmac(body, "whsec_retired", ts)
    try:
        verify_hmac(body, ts, forged)
        raise SystemExit("FAIL: forged signature was accepted")
    except SignatureRejected:
        pass
    print("OK: overlap window accepts both active keys, rejects retired key")

The assert block is the regression test for the rotation itself: as long as the previous secret remains in ACTIVE_SECRETS, a delivery signed with either key verifies, so you can update senders on their own schedule and still reject anything signed with a key that has actually been retired.

Memory & Performance Constraints

Verification stays cheap because the active set is tiny by design; the cost you must bound is the per-secret HMAC loop, not memory.

Component Constraint Mitigation
Active secret set Verification cost scales linearly with set size Keep the set to current + previous (2 entries); never let old keys accumulate
Secret cache TTL A long cache TTL delays rotation propagation across replicas Bound the TTL well under the sender’s retry window; force invalidation on rotation
HMAC compute per request SHA-256 over the raw body per secret holds the GIL briefly Order the set current-first so the common case validates on the first try
Timestamp tolerance Too wide a window admits replay; too tight rejects on clock skew Keep TOLERANCE_SECONDS at 300 and enforce NTP sync independently of rotation
Log fingerprints Logging full secrets to trace rotation leaks the key Log only a short hash prefix of the matched secret, never the value

Incident Triage & Rollback

Fast path when the post-rotation 401 alarm fires — target under fifteen minutes, and note that rollback here means re-adding a key, not reverting code.

  1. Confirm it is rotation, not a forgery wave. Correlate the 401 spike start time against your rotation deploy: kubectl rollout history deploy/webhook-ingress. A spike that begins exactly at the deploy is a cutover gap, not an attack.
  2. Check which keys the fleet is using. Grep the resolved fingerprint you log per request: kubectl logs -l app=webhook-ingress --since=5m | grep 'secret_fp' | sort | uniq -c. Mixed fingerprints confirm a stale-cache split-brain.
  3. Re-widen the active set immediately. Put the previous secret back alongside the new one and redeploy so both keys verify again:
BASH
kubectl set env deploy/webhook-ingress \
  WEBHOOK_SIGNING_SECRETS="whsec_new,whsec_old" && \
  kubectl rollout status deploy/webhook-ingress
  1. Ask the sender to redeliver the lost window. In the Stripe dashboard, resend failed deliveries in the affected minutes; for generic senders, replay from their outbound log. Because verification now accepts both keys, every redelivery validates.

Rollback. The safe rollback is never “remove the new key” — it is “ensure both keys are present,” which can only add accept coverage. Retire the old key only after grep 'secret_fp' | grep old | wc -l reports zero old-key traffic across a full retry window.

Post-rollback validation. Confirm the reject rate is back to baseline and no key is split-brained:

BASH
# expect near-zero 401s attributable to signatures:
kubectl logs -l app=webhook-ingress --since=10m | grep -c 'no active secret'

Frequently Asked Questions

How long should the overlap window be? At least as long as the sender’s full retry budget for a single delivery, plus your slowest deploy-and-drain time. For Stripe, whose retries span up to three days, a multi-day window is normal — keep both secrets active until old-key traffic reaches zero, then retire the previous key in a second deploy. The window costs almost nothing because the active set is only two entries.

Does trying multiple secrets weaken security or leak timing? No, provided the set stays small and the comparison is constant-time. stripe.Webhook.construct_event already uses a constant-time digest check, and the generic path uses hmac.compare_digest, so iterating two secrets adds one bounded HMAC computation, not a timing oracle. A forged signature still fails against every key and returns 401.

Can I rotate without any overlap if I control both ends? Only if you can atomically update sender and verifier in the same instant with zero in-flight deliveries, which is never true under real traffic. Even a self-signed internal sender has requests on the wire during the swap. The overlap window is what makes the two ends independently deployable, so use it even when you own both sides.

Related pages

Related Configuring mTLS Between Registration and Print Services Related Hardening the Print Segment With IP Allowlists and VLANs Related Implementing Role-Based Access for Registration APIs: Fixing 403s and PII Leaks from Over-Scoped Service Accounts