BadgeprintingEvent registration · badge automation

Handling Stripe Payment Webhooks for Ticket Purchases: Production Debugging & Implementation Guide

When event registration pipelines stall at the payment boundary, badge printing queues freeze, access control provisioning halts, and financial reconciliation drifts. The dominant operational failure is the payment sync gap: Stripe reports succeeded, but the registration database remains pending. Legacy polling strategies introduce latency, exhaust API rate limits, and mask transient network partitions. Modern stacks must treat webhooks as the authoritative source of truth, routing them through a hardened ingestion layer that guarantees exactly-once processing. This architecture directly supports Registration Ingestion & Payment Reconciliation workflows and eliminates the race conditions inherent in synchronous checkout flows.

Symptom-to-Resolution Matrix

1. Signature Verification Failures (HTTP 400/401)

  • Symptom: Webhook endpoint rejects valid Stripe deliveries. Dashboard shows 400/401 spikes during peak ticket drops.
  • Root Cause: Framework middleware (e.g., request.json(), body parsers, or WAF rules) mutates or re-encodes the raw payload before HMAC validation. Secondary cause: server clock skew > 300 seconds from Stripe’s tolerance window.
  • Fix:
  1. Read raw bytes immediately: raw_body = await request.body()
  2. Pass unmodified bytes to stripe.Webhook.construct_event()
  3. Enforce NTP synchronization (chrony or systemd-timesyncd) and verify ntpstat drift < 50ms.
  4. Disable automatic JSON parsing on the webhook route.

2. Duplicate Badge Provisioning & Idempotency Collisions

  • Symptom: Attendees receive duplicate confirmation emails, badge printers queue identical jobs, DB shows multiple registration rows per payment_intent.
  • Root Cause: Missing idempotency guards. Stripe retries failed deliveries with exponential backoff. Concurrent checkout.session.completed and payment_intent.succeeded events trigger parallel downstream jobs.
  • Fix:
  1. Implement Redis SETNX using stripe:evt:{event_id} with a 24-hour TTL.
  2. Deduplicate at the payment_intent level, not just the event level.
  3. Return HTTP 200 immediately after idempotency check to prevent Stripe retry storms.

3. Payment Sync Gap Drift

  • Symptom: Stripe shows succeeded, registration DB shows pending, badge printers idle, manual reconciliation required daily.
  • Root Cause: Synchronous database writes during webhook processing. Long-running transactions exceed connection pool timeouts, causing the webhook to timeout or crash before ACK, while Stripe considers it delivered.
  • Fix:
  1. ACK the webhook immediately (HTTP 200) after cryptographic verification and idempotency check.
  2. Dispatch processing to an async worker queue with acks_late=True.
  3. Implement exponential backoff retries with jitter. Never hold DB transactions during HTTP response generation.

4. Schema Validation Crashes & Queue Poisoning

  • Symptom: Worker queue stalls, KeyError or ValidationError exceptions flood logs, subsequent events back up.
  • Root Cause: Stripe introduces optional fields, renames nested keys, or deprecates legacy payloads without version pinning. Unhandled exceptions poison the worker process.
  • Fix:
  1. Use strict Pydantic v2 models with extra="ignore" and model_validate() instead of dict unpacking.
  2. Route malformed payloads to a Dead Letter Queue (DLQ) with max_retries=0.
  3. Implement schema version tagging in telemetry to track drift before it impacts production.

Deterministic Ingestion Pipeline (Python)

The following implementation enforces raw body capture, cryptographic verification, Redis-backed idempotency, strict schema validation, and async dispatch. It is designed for high-concurrency event tech stacks and aligns with production Payment Webhook Handling standards.

PYTHON
import os
import logging
import stripe
import redis
from fastapi import FastAPI, Request, Response, status
from pydantic import BaseModel, ConfigDict, ValidationError
from celery import Celery
from typing import Optional

# Configuration
STRIPE_SECRET = os.getenv("STRIPE_WEBHOOK_SECRET")
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
CELERY_BROKER = os.getenv("CELERY_BROKER_URL", "redis://localhost:6379/1")

# Clients
redis_client = redis.Redis.from_url(REDIS_URL, decode_responses=True, socket_timeout=2.0)
celery = Celery("webhooks", broker=CELERY_BROKER)
celery.conf.update(
    task_acks_late=True,
    worker_prefetch_multiplier=1,
    task_default_retry_delay=60,
    task_max_retries=5
)

logger = logging.getLogger("stripe_webhooks")

class StripeEventSchema(BaseModel):
    model_config = ConfigDict(extra="ignore")
    id: str
    type: str
    api_version: Optional[str] = None
    data: dict

@celery.task(bind=True, name="process_ticket_payment")
def process_ticket_payment(self, event_data: dict):
    """Async worker: handles DB writes, badge generation triggers, and email dispatch."""
    try:
        # Simulate DB transaction + badge queue push
        # Use connection pooling: pool_size=20, max_overflow=10
        # Never hold locks longer than 2s
        logger.info(f"Processing payment for intent: {event_data.get('id')}")
    except Exception as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

app = FastAPI()

@app.post("/webhooks/stripe")
async def handle_stripe_webhook(request: Request):
    # 1. Capture raw bytes BEFORE any middleware parsing
    raw_body = await request.body()
    sig_header = request.headers.get("stripe-signature")

    if not sig_header:
        return Response(status_code=status.HTTP_400_BAD_REQUEST, content="Missing signature")

    # 2. Cryptographic verification
    try:
        event = stripe.Webhook.construct_event(raw_body, sig_header, STRIPE_SECRET)
    except ValueError as e:
        logger.error(f"Invalid payload: {e}")
        return Response(status_code=status.HTTP_400_BAD_REQUEST, content="Invalid payload")
    except stripe.error.SignatureVerificationError as e:
        logger.error(f"Signature mismatch: {e}")
        return Response(status_code=status.HTTP_401_UNAUTHORIZED, content="Invalid signature")

    # 3. Idempotency guard (SETNX with 24h expiry)
    idempotency_key = f"stripe:evt:{event['id']}"
    if not redis_client.set(idempotency_key, "1", nx=True, ex=86400):
        return Response(status_code=status.HTTP_200_OK, content="Duplicate event")

    # 4. Schema validation (Pydantic v2)
    try:
        validated = StripeEventSchema.model_validate(event)
    except ValidationError as e:
        logger.warning(f"Schema drift detected: {e}")
        # Push to DLQ for manual inspection
        return Response(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, content="Schema validation failed")

    # 5. Async dispatch
    if validated.type == "checkout.session.completed":
        process_ticket_payment.delay(validated.model_dump())

    # Immediate ACK to Stripe
    return Response(status_code=status.HTTP_200_OK, content="ACK")

Memory & Performance Constraints

Component Constraint Mitigation
HTTP Payload Buffer Max 1MB raw body Reject Content-Length > 1_048_576 at reverse proxy (Nginx/Cloudflare)
Redis Idempotency Store High write throughput, memory fragmentation Use maxmemory-policy noeviction, monitor used_memory_peak, set TTL to 86400s
DB Connection Pool Pool exhaustion during traffic spikes pool_size=20, pool_recycle=300, max_overflow=10. Never sync-block during webhook ACK
Celery Workers OOM on large event batches worker_concurrency=4, task_time_limit=30, acks_late=True, prefetch=1
Python GIL CPU-bound validation blocks I/O Offload heavy reconciliation to separate process pool; keep webhook route I/O-bound

Incident Triage & Rollback Procedures

Fast Incident Resolution (< 15 mins)

  1. Verify Delivery State: Check Stripe Dashboard → Webhooks → Failed Deliveries. Filter by 400/401/500.
  2. Check Idempotency Collisions: redis-cli KEYS "stripe:evt:*" | wc -l. If count > expected event volume, verify TTL expiry.
  3. Inspect Worker Backlog: celery -A webhooks inspect active and celery -A webhooks inspect reserved. If queue depth > 500, scale workers or enable circuit breaker.
  4. Force Reconciliation: Run targeted backfill script:
PYTHON
  import stripe
  # Fetch last 24h succeeded payments
  intents = stripe.PaymentIntent.list(status="succeeded", limit=100)
  for pi in intents.auto_paging_iter():
      if not db.exists(pi.id):
          db.upsert_registration(pi.id, "succeeded")

Rollback Strategy

  • Immediate Mitigation: Toggle feature flag WEBHOOK_PROCESSING_ENABLED=false in config. Route traffic to synchronous polling fallback for 15-minute window.
  • Database State Freeze: Pause badge printing cron jobs. Run SELECT COUNT(*) FROM registrations WHERE status='pending' AND created_at > NOW() - INTERVAL '24 hours';
  • Code Revert: git revert HEAD~1 --no-edit && docker compose up -d --build. Verify stripe.Webhook.construct_event signature matches previous stable release.
  • Post-Rollback Validation: Replay 100 historical events via Stripe CLI: stripe events resend evt_xxx. Confirm HTTP 200 and DB state consistency.