BadgeprintingEvent registration · badge automation

Handling Stripe Payment Webhooks for Ticket Purchases: Fixing the Payment Sync Gap & Duplicate Badges

Symptom Statement

Stripe’s dashboard shows a payment_intent in the succeeded state, the buyer’s card was charged, and the webhook endpoint returned 200 OK — yet the registration row stays pending, the badge never prints, and access-control provisioning never fires. Or the opposite failure: a single paid ticket produces two confirmation emails and two identical badge jobs on the print floor. This page addresses that exact pair of symptoms at the Stripe ingress boundary — the payment sync gap (Stripe says paid, the database disagrees) and duplicate provisioning from reprocessed retries. It is the provider-specific implementation of the payment webhook handling stage, which owns signature verification and idempotent acknowledgment for the broader Registration Ingestion & Payment Reconciliation pipeline. The observable tells are consistent: 400/401 spikes on the endpoint during a ticket drop, a climbing count of succeeded intents with no matching confirmed registration, a worker queue that stalls on a repeating KeyError, and a daily manual reconciliation ritual to patch the drift by hand.

Stripe webhook handler — four serial gates, acknowledge before dispatch A raw Stripe POST passes through four gates in series: construct_event HMAC and 300s timestamp tolerance (400/401 reject branch), Redis SET NX EX on event.id (200 duplicate-ACK branch), Pydantic model_validate with extra=ignore (422 branch, dashed arrow to a dead-letter queue), and ACK-then-dispatch (HTTP 200 returned before work, solid arrow to async batch processing and badge generation via Celery .delay()). Stripe webhook handler — four serial gates, acknowledge before dispatch POST /webhooks/stripe raw body (bytes) Stripe-Signature await request.body() GATE 1 construct_event HMAC over raw body + 300s tolerance GATE 2 Redis idempotency SET NX EX (72h) key: stripe:evt:{id} GATE 3 model_validate Pydantic v2 extra = ignore GATE 4 ACK, then dispatch 200 before work Celery .delay() async batch processing → badge generation acks_late=True 400 / 401 reject bad signature or clock skew — inline 200 duplicate ACK retry / sibling event storm suppressed 422 → dead-letter schema drift, fails locally not in worker

Root Cause Analysis

Four distinct, concrete failures produce this pair of symptoms. They are independent — an incident is usually one of them, occasionally two stacked — so triage means identifying which before touching code.

  • Raw-body mutation and clock skew. Framework middleware (request.json(), a body parser, or a WAF rule) re-encodes or mutates the payload before HMAC validation, so the recomputed signature never matches. A secondary cause is server clock skew beyond Stripe’s 300-second tolerance window, which rejects otherwise valid deliveries.
  • Idempotency collisions. Stripe retries any delivery it does not see a 2xx for, with exponential backoff, and it emits overlapping events (checkout.session.completed and payment_intent.succeeded) for the same purchase. Without a deduplication guard, each retry and each sibling event spawns a parallel downstream job — one paid ticket, two badges.
  • Synchronous-write ACK drift. The handler holds a database transaction open while generating its HTTP response. A long-running write exceeds the connection-pool or gateway timeout, the handler crashes before it can ACK, and Stripe still considers the event delivered — the classic sync gap where succeeded never becomes a confirmed registration.
  • Schema drift and queue poisoning. Stripe adds optional fields, renames nested keys, or ships a new API version, and an unpinned handler that unpacks dicts raises an unhandled KeyError/ValidationError. The exception poisons the worker, and every subsequent event backs up behind the stuck payload.

Symptom-to-Resolution Matrix

Signature Verification Failures (HTTP 400/401)

Symptoms

  • The endpoint rejects valid Stripe deliveries; the dashboard shows 400/401 spikes during peak ticket sales.
  • The failure is intermittent in staging but constant behind a new proxy or WAF rule.

Root cause. Middleware mutates or re-encodes the raw payload before HMAC validation, or the server clock has drifted past the 300-second tolerance. Stripe signs the exact bytes it sent; any re-serialization breaks the digest.

Fix

  1. Read raw bytes immediately, before any parser touches the request: raw_body = await request.body().
  2. Pass the unmodified bytes to stripe.Webhook.construct_event(raw_body, sig_header, secret) — never a parsed dict.
  3. Disable automatic JSON parsing on the webhook route so no middleware can re-encode the body.
  4. Enforce NTP sync (chrony or systemd-timesyncd) and verify drift with ntpstat stays well inside the tolerance window.

Duplicate Badge Provisioning & Idempotency Collisions

Symptoms

  • Attendees receive duplicate confirmation emails and the printer queues identical jobs.
  • The database shows multiple registration rows keyed to one payment_intent.

Root cause. Missing idempotency guards. Stripe retries un-ACKed deliveries and fires overlapping event types, so concurrent handlers each dispatch downstream work for the same purchase.

Fix

  1. Claim an idempotency key atomically with Redis SET ... NX EX: stripe:evt:{event_id} with a TTL that covers Stripe’s full retry window.
  2. Deduplicate at the payment_intent level as well as the event level, so checkout.session.completed and payment_intent.succeeded for one purchase collapse to a single job.
  3. Return HTTP 200 the instant a duplicate is detected, so a retry storm is suppressed rather than amplified.

Payment Sync Gap Drift

Symptoms

  • Stripe shows succeeded, the registration DB shows pending, printers sit idle.
  • Reconciliation requires a manual backfill every morning.

Root cause. The handler performs a synchronous database write while generating its response. The transaction outlives the pool or gateway timeout, so the handler dies before ACK while Stripe records the delivery as complete.

Fix

  1. ACK immediately with HTTP 200 right after cryptographic verification and the idempotency claim — before any heavy work.
  2. Dispatch processing to an async worker with acks_late=True, so a crashed worker re-runs the task instead of losing it. The durable-queue machinery lives in async batch processing.
  3. Never hold a DB transaction open across the HTTP response; use exponential backoff with jitter for the downstream retries.

Schema Validation Crashes & Queue Poisoning

Symptoms

  • The worker queue stalls; KeyError or ValidationError floods the logs and subsequent events back up.
  • The failure appears after a Stripe API-version change with no code deploy.

Root cause. Stripe evolves its payloads; an unpinned handler that unpacks dicts raises on a renamed or missing key, and the unhandled exception poisons the worker process.

Fix

  1. Validate with a strict Pydantic v2 model using ConfigDict(extra="ignore") and model_validate() instead of dict unpacking, so a new field is tolerated and a missing required field fails loudly but locally.
  2. Route malformed payloads to a dead-letter queue with max_retries=0 rather than letting them cycle. Deep contract-version handling is owned by schema validation pipelines.
  3. Tag the observed api_version in telemetry so schema drift is visible before it reaches production.

Minimal Working Implementation

A single self-contained handler: raw-body capture, construct_event verification against stripe.SignatureVerificationError, an atomic Redis idempotency claim, strict Pydantic v2 validation, an ACK-before-work async dispatch, and a runnable verification block that proves the schema gate coerces a real Stripe event shape. In stripe-python v5+, stripe.SignatureVerificationError is the canonical path (the stripe.error submodule remains only for backward compatibility).

PYTHON
import os
import logging
from typing import Optional

import stripe
import redis
from fastapi import FastAPI, Request, Response, status
from pydantic import BaseModel, ConfigDict, ValidationError
from celery import Celery

# --- Configuration --------------------------------------------------------
STRIPE_SECRET = os.getenv("STRIPE_WEBHOOK_SECRET", "whsec_test_secret")
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
CELERY_BROKER = os.getenv("CELERY_BROKER_URL", "redis://localhost:6379/1")
IDEMPOTENCY_TTL = 259_200  # 72h — must cover Stripe's full retry window

# --- Clients --------------------------------------------------------------
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True, socket_timeout=2.0)
celery_app = Celery("webhooks", broker=CELERY_BROKER)
celery_app.conf.update(
    task_acks_late=True,          # re-run on worker crash instead of losing the task
    worker_prefetch_multiplier=1,
    task_default_retry_delay=60,
    task_max_retries=5,
)

logger = logging.getLogger("stripe_webhooks")


class StripeEventSchema(BaseModel):
    # extra="ignore" tolerates new Stripe fields; missing required keys still fail.
    model_config = ConfigDict(extra="ignore")
    id: str
    type: str
    api_version: Optional[str] = None
    data: dict


@celery_app.task(bind=True, name="process_ticket_payment")
def process_ticket_payment(self, event_data: dict):
    """Async worker: DB writes, badge-generation trigger, and email dispatch.

    Runs AFTER the endpoint has already ACKed Stripe, so nothing here can
    widen the sync gap. Keep transactions short and hand off heavy work.
    """
    try:
        logger.info("processing payment", extra={"event_id": event_data.get("id")})
        # pool_size=20, max_overflow=10, pool_recycle=300; never hold locks > 2s
    except Exception as exc:  # noqa: BLE001 — retry is the containment path
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)


app = FastAPI()


@app.post("/webhooks/stripe")
async def handle_stripe_webhook(request: Request):
    # Gate 1 — capture raw bytes BEFORE any middleware parsing.
    raw_body = await request.body()
    sig_header = request.headers.get("stripe-signature")
    if not sig_header:
        return Response(status_code=status.HTTP_400_BAD_REQUEST, content="Missing signature")

    # Gate 1 (cont.) — cryptographic verification over the unmodified bytes.
    try:
        event = stripe.Webhook.construct_event(raw_body, sig_header, STRIPE_SECRET)
    except ValueError:
        logger.error("invalid payload")
        return Response(status_code=status.HTTP_400_BAD_REQUEST, content="Invalid payload")
    except stripe.SignatureVerificationError:
        logger.error("signature mismatch")
        return Response(status_code=status.HTTP_401_UNAUTHORIZED, content="Invalid signature")

    # Gate 2 — idempotency. Atomic SET NX EX collapses the check-then-set race,
    # so two concurrent retries can never both dispatch.
    idempotency_key = f"stripe:evt:{event['id']}"
    if not redis_client.set(idempotency_key, "1", nx=True, ex=IDEMPOTENCY_TTL):
        return Response(status_code=status.HTTP_200_OK, content="Duplicate event")

    # Gate 3 — strict schema validation; drift fails locally, never poisons a worker.
    try:
        validated = StripeEventSchema.model_validate(dict(event))
    except ValidationError as exc:
        logger.warning("schema drift", extra={"errors": exc.errors()})
        return Response(
            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, content="Schema validation failed"
        )

    # Gate 4 — ACK to Stripe BEFORE any heavy processing; then dispatch async.
    if validated.type == "checkout.session.completed":
        process_ticket_payment.delay(validated.model_dump())

    return Response(status_code=status.HTTP_200_OK, content="ACK")


# --- Verification: prove the schema gate accepts a real Stripe event shape ---
if __name__ == "__main__":
    sample_event = {
        "id": "evt_1PxYz2AbCdEfGhIj",
        "type": "checkout.session.completed",
        "api_version": "2024-06-20",
        "data": {"object": {"id": "cs_test_123", "payment_status": "paid"}},
        "livemode": False,  # extra key — tolerated by extra="ignore"
    }
    validated = StripeEventSchema.model_validate(sample_event)
    assert validated.id == "evt_1PxYz2AbCdEfGhIj"
    assert validated.type == "checkout.session.completed"
    assert "livemode" not in validated.model_dump()  # dropped, not carried downstream
    print("OK:", validated.model_dump())

The verification block is the fix’s own regression test: it proves an unexpected livemode key is tolerated (schema drift never poisons the worker) while the identity and type fields survive intact for downstream dispatch.

Memory & Performance Constraints

The endpoint is I/O-bound on the idempotency store and the broker, not CPU-bound, so the failure modes are pool exhaustion and unbounded buffering rather than compute.

Component Constraint Mitigation
HTTP payload buffer await request.body() holds the full raw body in memory per in-flight request Reject Content-Length > 1_048_576 at the reverse proxy (Nginx/Cloudflare) before it reaches the worker
Redis idempotency store High write throughput plus memory fragmentation at event scale maxmemory-policy noeviction, monitor used_memory_peak, let the 72h EX TTL auto-evict keys
DB connection pool Pool exhaustion during a traffic spike stalls the async workers pool_size=20, max_overflow=10, pool_recycle=300; never sync-block during the webhook ACK
Celery workers OOM on large event batches or a poisoned task worker_concurrency=4, task_time_limit=30, acks_late=True, worker_prefetch_multiplier=1
Python GIL CPU-bound validation would block the event loop’s I/O Keep the route I/O-bound; offload heavy reconciliation to a separate process pool, scale with Uvicorn workers not threads

Incident Triage & Rollback

Fast path when the sync-gap or duplicate-badge alarm fires — target under 15 minutes to containment. Every step is non-destructive until the final rollback.

  1. Scope the delivery state. Stripe Dashboard → Webhooks → filter Failed Deliveries by 400/401/500. A 401 wave is signature/clock; a 422 wave is schema drift; a timeout wave is the sync gap.
  2. Check idempotency pressure. redis-cli --scan --pattern "stripe:evt:*" | wc -l against expected event volume; a runaway count signals a redelivery storm rather than genuine traffic.
  3. Inspect the worker backlog. celery -A webhooks inspect active and celery -A webhooks inspect reserved. A queue depth over 500 means scale workers or trip the circuit breaker before it poisons further.
  4. Fix upstream, then backfill the gap. Patch the real cause — deploy the rotated signing secret, ship the schema change, or restore Redis — then reconcile the missed succeeded intents:
PYTHON
import stripe

# Backfill: reconcile succeeded intents Stripe delivered but the DB never confirmed.
for pi in stripe.PaymentIntent.list(limit=100).auto_paging_iter():
    if pi.status == "succeeded" and not db.exists(pi.id):
        db.upsert_registration(pi.id, "succeeded")

Rollback. Toggle WEBHOOK_PROCESSING_ENABLED=false to route new deliveries to the polling fallback for a bounded window (the deterministic backstop lives in form API polling strategies), pause badge-print cron jobs, then git revert HEAD~1 --no-edit && docker compose up -d --build. Because dispatch is idempotent on event.id, reverting and re-driving is always safe — a replay can only re-ACK an already-processed event, never double-print.

Post-rollback validation. Replay historical events through the restored handler and confirm state consistency:

BASH
stripe events resend evt_1PxYz2AbCdEfGhIj   # expect HTTP 200
# then confirm no succeeded intent is still pending:
psql -c "SELECT COUNT(*) FROM registrations WHERE status='pending' \
         AND created_at > NOW() - INTERVAL '24 hours';"   # expect 0

Frequently Asked Questions

Why does Stripe show succeeded while my registration stays pending? Because the handler acknowledged the delivery but never completed the write — usually a synchronous DB transaction that outlived a timeout and crashed the handler after Stripe recorded the event as delivered. ACK immediately after verification, then do the write in an async worker with acks_late=True so a crash re-runs the task instead of losing it.

Do I need to verify against the raw body, or is the parsed JSON fine? Always the raw bytes. Stripe signs the exact payload it sent, so any re-serialization by a body parser or WAF changes the bytes and breaks the HMAC. Read await request.body() before any middleware and pass those bytes straight into construct_event.

How do I stop checkout.session.completed and payment_intent.succeeded from double-provisioning? Deduplicate on both the event ID and the payment_intent ID. An atomic Redis SET ... NX EX on stripe:evt:{event_id} collapses retries of the same event, and a second guard keyed on the intent collapses the two sibling event types for one purchase into a single downstream job.

  • Payment Webhook Handling — the parent stage that defines the verify → deduplicate → ACK → delegate contract this Stripe handler implements.
  • Async Batch Processing — the downstream stage that consumes the acknowledged payment and owns the durable queue, retry budget, and dead-letter routing.
  • Form API Polling Strategies — the polling fallback that recovers succeeded intents when webhook delivery degrades, closing sync gaps this handler alone cannot.
  • Schema Validation Pipelines — the sibling stage that owns deep contract versioning beyond the structural gate enforced here.