Payment Webhook Handling: Signature Verification & Idempotent Ingestion

Payment webhook handling is the strict ingress boundary between an external financial processor and the Registration Ingestion & Payment Reconciliation pipeline. It exists to solve one narrow, unforgiving problem: a gateway will deliver the same paid-ticket event more than once, out of order, and sometimes with a spoofed body, yet exactly one confirmed attendee record must result. The failure mode this stage prevents is the double-charged double-badge — an attendee who pays once but, because a retry was reprocessed, appears twice on the print floor and once again in the financial ledger. Getting this boundary wrong does not surface as a clean error; it surfaces as a reconciliation drift discovered days later, or as a duplicate lanyard handed out at a check-in desk.

To hold that line the handler must stay stateless, idempotent, and ruthlessly bounded. It verifies the cryptographic signature against the raw request body, enforces a rigid schema contract, deduplicates against a durable idempotency store, acknowledges the gateway fast enough to suppress a redelivery, and hands the normalized payload to async batch processing for everything expensive. It performs no seat allocation, no receipt generation, and no badge rendering inline. By isolating verification and normalization from state mutation, the boundary keeps accepting and acknowledging payments even while downstream renderers degrade — preserving ledger integrity while the queue absorbs the load.

Scope Boundary: What Webhook Handling Owns Link to this section#

The webhook handler is a stateless gatekeeper. Its only job is to make an inbound event trustworthy, well-formed, and non-duplicate, then acknowledge it and get out of the way. Any logic that mutates attendee, seat, or ledger state belongs to a downstream stage. Keeping this boundary explicit is what lets the endpoint stay horizontally scalable, restart mid-flight without data loss, and remain available even when the badge renderer is down.

The provider-specific mechanics of this boundary — verifying with the raw body and catching the SDK signature error — are worked end to end in handling Stripe payment webhooks for ticket purchases, which applies the same discipline to Stripe’s concrete stripe.SignatureVerificationError path.

Data Contract: The Normalized Payment Event Link to this section#

Provider payloads vary wildly in shape, so the boundary coerces every one into a single canonical model before it is allowed to advance. Validation runs synchronously at the endpoint using Pydantic v2; anything that fails a type check, is missing a required key, or falls out of range is rejected with a 400 and never reaches the queue. The contract below is the authoritative input/output shape for this stage.

from pydantic import BaseModel, EmailStr, Field, ConfigDict, field_validator


class Metadata(BaseModel):
    model_config = ConfigDict(extra="forbid")

    ticket_tier: str = Field(..., min_length=1)
    registration_session_id: str = Field(..., min_length=1)


class PaymentWebhookPayload(BaseModel):
    # Reject unknown top-level keys so a silently-changed provider schema
    # trips a 400 here instead of leaking an unmodeled field downstream.
    model_config = ConfigDict(extra="forbid")

    provider_event_id: str = Field(..., min_length=1)
    event_type: str
    payment_status: str
    customer_email: EmailStr
    transaction_amount: int = Field(..., gt=0)  # minor units (cents)
    currency: str = Field(..., min_length=3, max_length=3)
    metadata: Metadata

    @field_validator("event_type", mode="before")
    @classmethod
    def normalize_event_type(cls, v: str) -> str:
        # Providers send charge.succeeded / payment_intent.succeeded / etc.
        # Coerce to lower-case so downstream matching is case-stable.
        return str(v).strip().lower()

    @field_validator("currency", mode="before")
    @classmethod
    def upper_currency(cls, v: str) -> str:
        return str(v).strip().upper()

Each field earns its place. provider_event_id is the deduplication key — the single value the idempotency store keys on, so it must be present and non-empty. event_type and payment_status are normalized eagerly with mode="before" validators so that a provider swapping Charge.Succeeded for charge.succeeded never causes a downstream branch to silently miss. transaction_amount is constrained gt=0 and documented as minor units, which removes an entire class of floating-point rounding bugs at the ledger. currency is length-pinned to an ISO 4217 code and upper-cased. The nested metadata object carries the two fields the rest of the pipeline needs to correlate a payment back to a registration — ticket_tier and registration_session_id — and extra="forbid" on both models turns an unexpected new key from a silent data leak into a loud, containable 400.

Deterministic Implementation Link to this section#

The handler executes four gates in a fixed order — signature, schema, idempotency, enqueue — and acknowledges before any expensive work runs. The order is not incidental: signature verification must precede parsing so a spoofed body is never even deserialized, and idempotency must precede enqueue so a retry can never double-dispatch. The FastAPI implementation below is production-shaped, with a minted correlation ID threaded through every log line and a dead-letter fallback when the broker is unreachable.

import hashlib
import hmac
import json
import logging
import time
import uuid
from pathlib import Path

from fastapi import BackgroundTasks, FastAPI, HTTPException, Request, Response
from pydantic import ValidationError
from redis import Redis
from redis.exceptions import RedisError

logger = logging.getLogger("payment_webhook")

redis_client = Redis(host="localhost", port=6379, db=0, decode_responses=True)
SIGNING_SECRET = "whsec_production_secret_here"  # load from a secrets manager
IDEMPOTENCY_TTL = 259_200  # 72h — must cover the provider's full retry window
MAX_SKEW_SECONDS = 300     # reject events older than 5 minutes (replay defense)
DLQ_DIR = Path("/var/spool/webhook-dlq")

app = FastAPI()


def verify_signature(raw_body: bytes, signature_header: str, timestamp_header: str) -> bool:
    """Verify HMAC-SHA256 over `{timestamp}.{raw_body}`.

    Returns False for a stale timestamp (replay) or a mismatched digest.
    Uses hmac.compare_digest to stay constant-time against timing attacks.
    """
    try:
        ts = int(timestamp_header)
    except (ValueError, TypeError):
        return False
    if abs(time.time() - ts) > MAX_SKEW_SECONDS:
        return False
    signed_payload = f"{ts}.".encode() + raw_body
    expected = hmac.new(SIGNING_SECRET.encode(), signed_payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(signature_header, f"v1={expected}")


@app.post("/webhooks/payment")
async def handle_payment_webhook(request: Request, background_tasks: BackgroundTasks):
    correlation_id = str(uuid.uuid4())
    raw_body = await request.body()  # raw bytes — never the parsed body
    sig = request.headers.get("X-Webhook-Signature", "")
    ts = request.headers.get("X-Webhook-Timestamp", "")

    # Gate 1 — cryptographic verification (before any parsing).
    if not verify_signature(raw_body, sig, ts):
        logger.warning("webhook.rejected", extra={
            "correlation_id": correlation_id, "reason": "ERR_INVALID_SIGNATURE",
            "signature_valid": False})
        raise HTTPException(status_code=401, detail="ERR_INVALID_SIGNATURE")

    # Gate 2 — schema enforcement.
    try:
        payload = PaymentWebhookPayload.model_validate_json(raw_body)
    except ValidationError as exc:
        logger.warning("webhook.rejected", extra={
            "correlation_id": correlation_id, "reason": "ERR_SCHEMA_VIOLATION",
            "signature_valid": True, "schema_errors": exc.errors()})
        raise HTTPException(status_code=400, detail="ERR_SCHEMA_VIOLATION")

    cache_key = f"idempotency:{payload.provider_event_id}"

    # Gate 3 — idempotency. A store outage returns 500 so the provider retries;
    # a local fallback cache here would risk duplicate processing on split-brain.
    try:
        already_seen = not redis_client.set(cache_key, correlation_id, nx=True, ex=IDEMPOTENCY_TTL)
    except RedisError:
        logger.error("webhook.idempotency_store_down", extra={
            "correlation_id": correlation_id,
            "provider_event_id": payload.provider_event_id})
        raise HTTPException(status_code=500, detail="ERR_IDEMPOTENCY_STORE")

    if already_seen:
        logger.info("webhook.duplicate_ack", extra={
            "correlation_id": correlation_id,
            "provider_event_id": payload.provider_event_id, "idempotency_hit": True})
        return Response(status_code=200)

    # Gate 4 — acknowledge first, then delegate downstream work.
    background_tasks.add_task(
        dispatch_to_reconciliation_queue, payload.model_dump(mode="json"), correlation_id)
    logger.info("webhook.accepted", extra={
        "correlation_id": correlation_id,
        "provider_event_id": payload.provider_event_id,
        "event_type": payload.event_type, "idempotency_hit": False})
    return Response(status_code=200)


def dispatch_to_reconciliation_queue(normalized_payload: dict, correlation_id: str) -> None:
    """Hand off to the durable broker that feeds async batch processing.

    On broker failure, dead-letter the payload to disk so a scheduled drain
    can replay it — the ACK was already sent, so losing it here is silent.
    """
    try:
        # broker.publish("reconciliation", json.dumps({...}))  # RabbitMQ / SQS / Celery
        logger.info("webhook.enqueued", extra={
            "correlation_id": correlation_id,
            "provider_event_id": normalized_payload["provider_event_id"]})
    except Exception:  # noqa: BLE001 — DLQ is the containment path, not a swallow
        DLQ_DIR.mkdir(parents=True, exist_ok=True)
        (DLQ_DIR / f"{correlation_id}.json").write_text(
            json.dumps({"correlation_id": correlation_id, "payload": normalized_payload}))
        logger.error("webhook.dlq_write", extra={
            "correlation_id": correlation_id,
            "provider_event_id": normalized_payload["provider_event_id"]})

Two subtleties carry most of the correctness. The idempotency check and the write are a single atomic SET ... NX EX, which collapses the classic check-then-set race — two concurrent retries can never both see “not present” and both dispatch. And the correlation ID is minted at ingress and stored as the idempotency value, so when a duplicate arrives you can look up which original request first claimed the key.

Production Debugging and Observability Link to this section#

Every request emits one structured log record at ingress with a fixed field set, so an incident is a query, not a grep. The mandatory fields are correlation_id (UUID minted at ingress), provider_event_id (the gateway’s event identifier), signature_valid (bool), schema_errors (array, present only on a 400), and idempotency_hit (bool). In Datadog these map to @correlation_id, @provider_event_id, and @idempotency_hit facets; in an ELK stack index them as webhook.correlation_id and webhook.provider_event_id keyword fields so duplicate storms are groupable. A single log line at the webhook.duplicate_ack event, faceted by provider_event_id, tells you instantly whether a spike is one event being hammered or a broad redelivery wave.

DLQ routing is deliberately split by failure class. A signature or schema failure is rejected inline with a 4xx and never dead-lettered — the payload is untrustworthy or malformed, and replaying it would only fail again. Only a post-acknowledgment enqueue failure writes to the dead-letter directory keyed on correlation_id, because there the gateway already believes it succeeded and will not retry. That disk DLQ is drained by a scheduled job that republishes to the broker; the durable-queue side of that machinery lives in async batch processing. When a gateway gives up retrying entirely after persistent 5xx responses, the deterministic backstop is form API polling strategies, which queries the provider’s transaction API for unconfirmed session IDs and patches the ledger gaps that webhooks missed.

Track three metrics with alert thresholds: webhook_ack_latency_p95 (page above 800ms — a slow handler invites gateway-side retries that amplify load), idempotency_hit_rate (a sudden climb signals a redelivery storm or a stuck provider), and schema_rejection_count (page when rejections exceed 2% over a 5-minute window — usually a provider payload change). Correlation IDs propagate downstream in the enqueued message so a badge that never printed can be traced back through the queue to the exact inbound webhook that produced it.

Performance and Memory Constraints Link to this section#

The endpoint is I/O-bound on the idempotency store and the broker, not CPU-bound, so the failure modes here are pool exhaustion and unbounded buffering rather than compute.

Incident Triage Checklist Link to this section#

When webhook alarms fire, the target is under 15 minutes to containment. Work the list in order; every step is non-destructive until the final rollback.

1. Scope the blast radius. Query your log platform for webhook.rejected grouped by reason over the last 15 minutes. A 401 spike is a signature problem; a 400 spike is a schema problem; a 500 spike is the idempotency store. This one query routes the rest of the triage.
2. On a 401 wave, check the secret and the clock. Confirm the signing secret was not just rotated on the provider side without deploying the new value, then check server-to-provider clock skew (timedatectl status) against the 5-minute window. The most common self-inflicted cause is body-parser middleware mutating the payload before verification — confirm the handler reads the raw bytes.
3. On a 400 wave, diff the payload. Pull one failing record’s schema_errors array by correlation_id from the logs. A newly-appeared key rejected by extra="forbid" means the provider changed its schema; a missing metadata field usually points at a legacy checkout flow.
4. On a 500 wave, check the idempotency store. redis-cli PING and redis-cli INFO clients to confirm the pool is not exhausted. Inspect a live key: redis-cli GET idempotency:<provider_event_id> returns the correlation ID that first claimed it, or nil if the store dropped it.
5. Confirm nothing is silently lost. List the disk DLQ: ls -1 /var/spool/webhook-dlq | wc -l. A non-zero, climbing count means the broker was unreachable after acknowledgment — those events are parked, not gone.
6. Fix upstream, then drain-replay. Patch the real cause — rotate and deploy the secret, ship the schema change, or restore Redis — then replay the DLQ by republishing each file through dispatch_to_reconciliation_queue. Rollback path: because dispatch is idempotent on provider_event_id, reverting to the previous build and re-driving the DLQ is always safe — a replay can only re-ACK an already-processed event, never double-process it. Validate with ls -1 /var/spool/webhook-dlq | wc -l returning 0.

For the provider-side contract, align the timestamp-skew and signing scheme with the official Stripe webhook signature specification and the Python hmac documentation.

Frequently Asked Questions Link to this section#

Should signature and schema failures be dead-lettered like enqueue failures? No. A 401 or 400 is rejected inline and never dead-lettered — the payload is either untrustworthy or malformed, so replaying it can only fail again. Only a failure that happens after the 2xx acknowledgment, such as an unreachable broker, is dead-lettered, because at that point the gateway believes it succeeded and will not retry on its own.

Why return 500 on an idempotency-store outage instead of processing optimistically? Because optimistic processing during a store outage is exactly how duplicates get created. Returning a 500 lets the gateway’s own retry machinery redeliver once the store recovers. A local fallback cache would risk two workers on a split-brain both believing an event is new and dispatching it twice.

What idempotency TTL should the store use? The TTL must cover the provider’s full retry window — typically 72 hours for major gateways. If keys expire before the provider stops retrying, a late redelivery is treated as new and reprocessed. Size Redis for the peak-window key count and let the EX TTL evict keys automatically rather than sweeping them manually.

Can the handler render a badge or allocate a seat directly if the queue is empty? Never inline. Doing so re-couples fulfillment to ingestion and re-introduces the cascade this boundary exists to prevent — a slow renderer would blow the gateway’s delivery timeout and trigger a retry storm. All expensive work happens downstream of the acknowledgment, in async batch processing.

Related Link to this section#

• Registration Ingestion & Payment Reconciliation — the parent pipeline this ingress boundary feeds; its overview shows every stage from webhook to reconciled record.
• Handling Stripe Payment Webhooks for Ticket Purchases — the concrete Stripe implementation of this contract, including the stripe.SignatureVerificationError path and the payment-sync-gap fix.
• Async Batch Processing — the downstream stage that consumes the acknowledged payload and owns the durable queue, retry budget, and reconciliation gate.
• Form API Polling Strategies — the deterministic backstop that recovers events a gateway stops retrying, patching ledger gaps this boundary alone cannot close.
• Schema Validation Pipelines — the sibling stage that owns deep contract versioning and business-rule validation beyond the structural gate enforced here.