Event Taxonomy Schema Design: The Canonical Ingestion Contract

Event taxonomy schema design is the stage that defines the single canonical shape every attendee record must take before any downstream stage is allowed to touch it. It is one stage of the Core Architecture & Event Taxonomy pipeline, sitting at the ingestion boundary — the first hard checkpoint where raw payloads from ticketing platforms, CRM exports, and API webhooks converge into a unified validation layer. The taxonomy schema enforces strict data contracts at ingress: type safety, required-field presence, hierarchical categorization of access tiers and session tracks, and explicit routing metadata. Every record that clears this gate carries predictable structure that the rest of the system can trust without re-checking.

The failure mode this stage exists to prevent is the schema drift that turns a single vendor payload mutation into a cascade of downstream misprints. Without a hardened schema at ingress, transformations become brittle, badge generation failures propagate unpredictably across print queues and digital credential services, and a malformed CRM export can stall a check-in line. Unlike the attendee field mapping rules that resolve vendor-specific paths into internal fields, this stage owns the definition of those internal fields — the authoritative types, the invariant-versus-mutable split, and the versioning rules that let the taxonomy evolve without breaking previously ingested records. Mapping executes against the contract; schema design authors it.

Scope Boundary: What Schema Design Owns Link to this section#

To stay a stateless, deterministic gate, this stage enforces a hard boundary. It defines and validates the canonical contract and nothing more — it does not resolve vendor paths, render layout, or orchestrate retries. Anything outside that scope is delegated to an adjacent stage so a rendering fault or a broker outage can never propagate backward into schema state.

The rate at which records arrive and the way they are pulled from vendors is not decided here either — that durability is co-owned with the upstream form API polling strategies. Keeping the gate a pure function is what makes it trustworthy: the same raw payload plus the same schema_version always produces the same verdict, which is the property that makes replays and reconciliation safe.

Invariant vs. Mutable Data Contracts Link to this section#

The single most important design decision at this boundary is separating invariant identifiers from mutable attributes. Invariant fields — the unique external identifier, the ticket tier classification, and the immutable registration timestamp — drive routing decisions, entitlement checks, and audit reconciliation. They can never be guessed or silently defaulted, because a fabricated value produces a wrong or unscannable credential. Mutable attributes — dietary preferences, session selections, accessibility requirements, emergency contacts — are subject to post-registration updates and require idempotent merge logic rather than strict rejection.

Encoding that split directly in the schema is what lets validation fail correctly. A missing invariant is a hard error that halts the record; a missing mutable field resolves to a safe baseline and the record proceeds. When designing for real registration flows you will hit nested ticket hierarchies, conditional field requirements, and polymorphic attendee objects — the how to build JSON schemas for multi-ticket event types guide covers the discriminated-union structure that handles those variations without introducing schema drift or collapsing distinct ticket shapes into one permissive object.

The Data Contract Link to this section#

Production validation needs more than basic type checking. It requires a stateless, deterministic gate that aggregates errors, applies safe coercion only where explicitly permitted, and surfaces actionable diagnostics. The contract below uses Pydantic v2 with strict mode enabled so that implicit conversions — a string silently becoming an int, "false" becoming True — cannot mask upstream corruption. Every field is either an enforced invariant or a mutable attribute with an explicit default.

from pydantic import BaseModel, ConfigDict, Field, EmailStr, StrictStr, field_validator
from typing import Optional
from datetime import datetime

class AttendeeIngestSchema(BaseModel):
    # strict: no silent coercion; extra=forbid: reject unknown keys at the gate
    model_config = ConfigDict(strict=True, extra="forbid", str_strip_whitespace=True)

    # --- invariant fields (hard-rejected on failure) ---
    external_id: StrictStr = Field(..., min_length=1, max_length=64)
    ticket_tier: StrictStr = Field(..., pattern=r"^(general|vip|speaker|staff|press)$")
    registered_at: datetime
    email: EmailStr
    schema_version: StrictStr = Field(default="v2.0.0", pattern=r"^v\d+\.\d+\.\d+$")

    # --- mutable attributes (safe defaults; merge post-registration) ---
    display_name: Optional[StrictStr] = Field(default=None, max_length=120)
    dietary_flags: list[StrictStr] = Field(default_factory=list)
    session_ids: list[StrictStr] = Field(default_factory=list)
    fallback_eligible: bool = False

    @field_validator("ticket_tier", mode="before")
    @classmethod
    def canonicalize_tier(cls, v: object) -> object:
        # Only casing is normalized here; unknown tiers still fail the pattern,
        # so a typo becomes a hard error rather than a mis-tiered badge.
        return v.strip().lower() if isinstance(v, str) else v

Each field earns its place. external_id is the identity anchor every downstream stage joins on, so it is length-bounded and non-empty. ticket_tier is constrained to a closed vocabulary by pattern, which means an unrecognized tier is a validation error rather than a badge printed at the wrong access level; its before validator normalizes only casing, never inventing a value. registered_at and email are invariants that gate entitlement and delivery. schema_version travels with every record so that when the taxonomy evolves, historical payloads can be reprocessed against the exact version they were built for. The mutable fields all carry default_factory or None defaults, so their absence never blocks a record. extra="forbid" rejects unknown keys outright, which is what surfaces a vendor adding an undocumented field before it silently leaks downstream.

Deterministic Validation & Quarantine Routing Link to this section#

The gate itself is a pure function: it accepts a raw dictionary and returns either a normalized model instance or a structured verdict describing exactly why the payload was rejected and where it should be routed. Nothing is ever silently dropped. A stable content hash is computed first so the verdict carries a correlation ID even when validation fails before any field is readable.

import hashlib
import json
import logging
from dataclasses import dataclass, field
from typing import Any, Optional
from pydantic import ValidationError

logger = logging.getLogger("ingestion.schema")

CRITICAL_FIELDS = {"external_id", "ticket_tier", "registered_at", "email"}

@dataclass
class ValidationReport:
    payload_hash: str
    is_valid: bool
    errors: list[dict[str, Any]] = field(default_factory=list)
    quarantine_reason: Optional[str] = None
    fallback_chain: Optional[str] = None

def validate_ingest_payload(
    raw: dict[str, Any],
) -> tuple[Optional[AttendeeIngestSchema], ValidationReport]:
    # Stable hash (sorted keys) — reproducible correlation ID, unlike built-in hash()
    payload_hash = hashlib.sha256(
        json.dumps(raw, sort_keys=True, default=str).encode()
    ).hexdigest()[:12]

    try:
        normalized = AttendeeIngestSchema.model_validate(raw)
        logger.info("schema.valid", extra={
            "external_id": normalized.external_id,
            "payload_hash": payload_hash,
            "schema_version": normalized.schema_version,
        })
        return normalized, ValidationReport(payload_hash=payload_hash, is_valid=True)

    except ValidationError as exc:
        errors = [{
            "field_path": ".".join(str(loc) for loc in e["loc"]),
            "constraint": e.get("type", "unknown"),
            "message": e["msg"],
            "input_value": str(e.get("input", ""))[:50],  # truncated: no PII bloat
        } for e in exc.errors()]

        # Route by severity: an invariant breach cannot be salvaged with a default
        failed_critical = any(
            e["field_path"].split(".")[0] in CRITICAL_FIELDS for e in errors
        )
        if failed_critical:
            reason, chain = "CRITICAL_FIELD_VIOLATION", "manual_review_queue"
        else:
            reason, chain = "NON_CRITICAL_MUTABLE_FAILURE", "default_tier_assignment"

        report = ValidationReport(
            payload_hash=payload_hash, is_valid=False, errors=errors,
            quarantine_reason=reason, fallback_chain=chain,
        )
        logger.warning("schema.quarantined", extra={
            "payload_hash": payload_hash, "reason": reason, "chain": chain,
        })
        return None, report

The routing logic is the heart of the stage. A failed invariant — a missing external_id, a malformed email, an out-of-vocabulary ticket_tier — has no safe default, so the record is halted and routed to the manual-review queue with its full error envelope intact. A failed mutable field — an unsupported dietary_flags value, a missing display_name — triggers a fallback chain that applies organizational defaults while preserving the original payload for audit reconciliation. This severity split is what keeps print queues moving without ever trading away data integrity: the gate never stalls the whole batch over a cosmetic field, and it never ships a fabricated identity to satisfy throughput.

Production Debugging & Observability Link to this section#

Fast incident resolution rests on diagnostics that map directly to field-level constraints. The ValidationReport captures the exact field_path, the constraint type that fired, and a truncated input value safe for log retention. The payload_hash is emitted as a trace correlation ID so ops can reconstruct which upstream payload broke without ever storing PII in a log aggregator — the hash points to the record, the record stays in the encrypted store. Failed payloads are serialized with their error envelope and pushed to the dead-letter queue that owns retries, never dropped inline. A representative quarantine event looks like this:

{
  "event": "schema.quarantined",
  "payload_hash": "9f2c1a7be4d0",
  "reason": "CRITICAL_FIELD_VIOLATION",
  "chain": "manual_review_queue",
  "errors": [
    {"field_path": "ticket_tier", "constraint": "string_pattern_mismatch", "input_value": "VVIP"},
    {"field_path": "email", "constraint": "value_error", "input_value": "not-an-address"}
  ],
  "level": "warning"
}

That single line tells the on-call engineer what breached (ticket_tier received an undefined VVIP value), which payload to pull by hash, and that the failure was critical enough to hold for review rather than auto-default. Wire the structured fields into your platform of choice: a Datadog facet on @reason shows the quarantine-rate split at a glance, and an ELK filter on errors.field_path isolates a drift class in seconds. When a single field_path spikes, cross-reference the source system’s API changelog — most ingestion failures originate from vendor payload mutations, not internal schema defects. Because the same envelope shape is shared with the upstream schema validation pipelines, triage tooling stays uniform across the whole intake tier. For mapping the constraint vocabulary back to its source rule, the JSON Schema standard defines the predictable keywords (type, pattern, minLength, required) that Pydantic’s error types mirror.

Performance & Memory Constraints Link to this section#

Validation is CPU-bound and runs once per attendee, so at conference scale the per-record cost and its interaction with the worker pool dominate. The constraints that actually bite during an opening-morning registration burst are below, with the mitigation for each.

The recurring theme is that the gate should stay a pure, allocation-light function and push every stateful or blocking concern — connections, retries, queue depth — outward to the batch tier built to hold it.

Incident Triage Checklist Link to this section#

When quarantine-rate alerts fire, the target is under 15 minutes to containment. Work the list in order; every step is non-destructive until the final rollback.

1. Confirm the blast radius. Query your log platform for schema.quarantined grouped by errors.field_path over the last 15 minutes. A single dominant path (e.g. ticket_tier) means a vendor added or renamed an enum value; a spread across paths means a wholesale payload-shape change.
2. Inspect the DLQ depth. redis-cli LLEN dlq:schema_ingest — a fast-climbing depth confirms an active drift rather than a stale backlog. Peek without draining: redis-cli LRANGE dlq:schema_ingest 0 4.
3. Pull one failing payload by hash: redis-cli HGETALL dlq:schema_ingest:<payload_hash>. Compare its keys against AttendeeIngestSchema to see exactly which constraint fired and whether the value is a new-but-valid tier or genuine corruption.
4. Fix the contract, not the data. If the vendor introduced a legitimate new tier, extend the ticket_tier pattern and bump schema_version; if the field moved, that belongs in the mapping layer, not here. Never hand-edit records in the queue — the fix must reproduce for every replayed payload.
5. Deploy and drain-replay. Ship the version-bumped schema, then requeue the DLQ back through validate_ingest_payload. Watch schema.valid rise and LLEN dlq:schema_ingest fall to zero. Because the gate is stateless, a rolling redeploy completes in under two minutes.
6. Rollback path. If the new pattern wrongly admits a bad value, revert to the previous schema_version — every record carries the version it was validated under, so reprocessing against the prior contract is never destructive.

Frequently Asked Questions Link to this section#

Why enable Pydantic strict mode instead of letting it coerce types? Because silent coercion is exactly how upstream corruption reaches a badge. In lax mode a ticket_tier of 1 or an email that is really an integer would coerce and validate; in strict mode they fail at the gate, where the diagnostic is cheap and the record can be quarantined. The one place coercion is allowed — casing on ticket_tier — is explicit and bounded by a before validator that still rejects unknown values.

Should a payload with an unsupported dietary flag be rejected or defaulted? Defaulted. Dietary flags are mutable attributes, so a failure routes to the default_tier_assignment fallback chain and the record proceeds with the original payload preserved for reconciliation. Hard rejection is reserved for invariant fields — external_id, email, ticket_tier, registered_at — where no safe default exists.

How do I evolve the taxonomy without breaking records already ingested? Bump schema_version, add new fields with explicit defaults so older payloads still validate, and widen enum patterns only additively. Because every record carries the version it was validated under, historical records replay against their original contract while new records use the new one — which is what makes reprints and forensic replays trustworthy.

Related Link to this section#

• Core Architecture & Event Taxonomy — the parent pipeline this ingestion gate anchors; its overview traces every stage from intake to print.
• How to Build JSON Schemas for Multi-Ticket Event Types — the discriminated-union detail for modeling polymorphic ticket shapes within this canonical contract.
• Attendee Field Mapping Rules — the next stage, which resolves vendor-specific payloads into the internal field names this schema defines.
• Badge Layout Architecture — the downstream consumer that renders each validated record, relying on the type safety this gate guarantees.
• Schema Validation Pipelines — the high-throughput intake tier that runs this contract at scale and shares its dead-letter envelope.