Attendee Identity Resolution: One Human, One Canonical Record

The same human rarely arrives once. A speaker is bulk-imported from the CRM, buys a companion ticket through Eventbrite, pays a workshop add-on via Stripe checkout, then walks up on the morning of the event and re-registers at a kiosk because the line for “already registered” looked longer. That is four inbound records for one person, each partial, each keyed differently, and none of them agreeing on the spelling of the name. Attendee identity resolution is the stage that reconciles them into a single canonical record before anything downstream tries to print a badge, and it is one of the four stages owned by the Core Architecture & Event Taxonomy section. Its single job is to decide, deterministically and repeatably, which inbound records refer to the same human, assign them a stable canonical identity, and merge their fields under an explicit precedence order — so the check-in desk sees one attendee, not four, and the printer produces one badge, not a duplicate for every source that touched the record.

This page defines that stage precisely: the match keys that decide sameness, the two-model data contract that separates a raw inbound candidate from the settled canonical record, the deterministic blocking-plus-scoring pipeline that keeps matching tractable at event scale, and the observability surface an on-call engineer needs when a resolution incident surfaces during a registration surge. The two hardest failures it must survive are covered in depth by dedicated guides: deduplicating attendees across multiple registration sources when the same person slips through as two records, and merging attendee records without losing session history when a merge silently drops the agenda or scan history the resolved record was supposed to keep. Everything about the shape of the fields being matched — the canonical types, the tier taxonomy — belongs to the event taxonomy schema design; everything about how a resolved record is then enriched and projected toward the renderer belongs to the attendee field mapping rules. Keeping those seams sharp is what lets this stage own identity, and only identity.

Scope Boundary Link to this section#

Identity resolution stays deterministic and auditable only because its remit is narrow. Anything not in the left column is delegated to an adjacent stage and must not leak into the matching hot path.

The stage holds no long-lived session state of its own beyond the canonical index, makes no rendering decisions, and never invents a field a source did not supply. It receives a stream of IdentityCandidate records and emits either a settled CanonicalAttendee or a diagnostic envelope bound for review — which is what lets a resolution decision be replayed months later during a duplicate-badge dispute and produce the exact same verdict.

Data Contract & Schema Link to this section#

Two models draw the boundary between “what a source sent” and “what we decided”. IdentityCandidate is the normalized inbound projection — one per source touch — carrying the raw match material and its provenance. CanonicalAttendee is the settled record: the survivor identity plus the union of everything the merge kept. Both use Pydantic v2 with coercion disabled, aligned to the canonical types in the event taxonomy schema design.

from __future__ import annotations

import unicodedata
from datetime import datetime, timezone
from enum import Enum
from pydantic import BaseModel, Field, ConfigDict, field_validator


class Source(str, Enum):
    CRM = "crm"
    EVENTBRITE = "eventbrite"
    WALKUP = "walkup"
    STRIPE = "stripe"


# Higher weight wins a field conflict during merge. CRM is the system of record;
# a walk-up record typed under time pressure at a kiosk is trusted least.
SOURCE_PRECEDENCE: dict[Source, int] = {
    Source.CRM: 40,
    Source.EVENTBRITE: 30,
    Source.STRIPE: 20,
    Source.WALKUP: 10,
}


def _fold(value: str) -> str:
    """Normalize for matching: strip diacritics, collapse whitespace, casefold."""
    decomposed = unicodedata.normalize("NFKD", value)
    ascii_only = "".join(ch for ch in decomposed if not unicodedata.combining(ch))
    return " ".join(ascii_only.split()).casefold()


class IdentityCandidate(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")

    source: Source
    source_record_id: str = Field(..., min_length=1, max_length=128)
    email: str = Field(..., pattern=r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
    full_name: str = Field(..., min_length=2, max_length=120)
    org: str | None = Field(default=None, max_length=160)
    external_ids: dict[str, str] = Field(default_factory=dict)
    registered_at: datetime

    @field_validator("email", mode="before")
    @classmethod
    def _normalize_email(cls, v: str) -> str:
        # Lowercase + trim BEFORE pattern validation so casing never forks a match key.
        return v.strip().casefold()

    @property
    def email_key(self) -> str:
        return self.email

    @property
    def name_org_key(self) -> str:
        return f"{_fold(self.full_name)}|{_fold(self.org or '')}"

    @property
    def precedence(self) -> int:
        return SOURCE_PRECEDENCE[self.source]


class CanonicalAttendee(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")

    canonical_id: str = Field(..., min_length=8, max_length=36)
    email: str
    full_name: str
    org: str | None = None
    merged_sources: list[Source] = Field(default_factory=list)
    external_ids: dict[str, str] = Field(default_factory=dict)
    resolved_at: datetime = Field(
        default_factory=lambda: datetime.now(timezone.utc)
    )

Each field earns its place. source is not decoration — it drives SOURCE_PRECEDENCE, the closed weighting that decides which value survives when two records disagree on a name or org, so it is a closed Enum rather than a free string. The email validator runs mode="before" so a [email protected] and a [email protected] collapse to one match key before any comparison, closing the single most common source of near-duplicates. external_ids is a namespaced bag ({"eventbrite_order": "...", "stripe_pi": "..."}) so a shared identifier across two records is a decisive, deterministic match signal that fuzzy scoring never has to guess at. On the canonical side, merged_sources is the audit trail that proves which touches folded into this record, and canonical_id is the stable key every downstream stage — mapping, rendering, and check-in kiosk sync — threads through its own tables.

Deterministic Implementation Link to this section#

Resolution runs in three fixed phases — block, score, merge — and the order is not negotiable. Blocking runs first because comparing every candidate against every other is O(n²); at 40,000 attendees that is 1.6 billion pairs, which no registration window survives. Blocking groups candidates by cheap deterministic keys (normalized email, and a folded name signature) so only plausible pairs are ever scored. Scoring then combines three independent signals into one confidence number, and merging applies precedence only to pairs above the auto-merge threshold. Everything in the ambiguous band is diverted to human review rather than guessed.

from collections import defaultdict
from difflib import SequenceMatcher

AUTO_MERGE = 0.90     # >= auto-merges
REVIEW_FLOOR = 0.62   # [floor, AUTO_MERGE) -> manual review; below -> distinct


def block(candidates: list[IdentityCandidate]) -> list[list[IdentityCandidate]]:
    """Group candidates that share any cheap key, so scoring stays near-linear."""
    by_key: dict[str, list[IdentityCandidate]] = defaultdict(list)
    for c in candidates:
        by_key[f"email:{c.email_key}"].append(c)
        # First token of the folded name is a coarse but cheap blocking signal.
        first_token = c.name_org_key.split(" ", 1)[0][:12]
        by_key[f"name:{first_token}"].append(c)

    seen: set[int] = set()
    blocks: list[list[IdentityCandidate]] = []
    for group in by_key.values():
        ids = frozenset(id(c) for c in group)
        if len(group) > 1 and ids not in seen:
            seen.add(ids)
            blocks.append(group)
    return blocks


def score(a: IdentityCandidate, b: IdentityCandidate) -> float:
    """Blend exact-email, fuzzy name+org, and shared external ids into [0, 1]."""
    if a.email_key == b.email_key:
        email_signal = 1.0
    else:
        email_signal = SequenceMatcher(None, a.email_key, b.email_key).ratio()

    name_org_signal = SequenceMatcher(None, a.name_org_key, b.name_org_key).ratio()

    shared = set(a.external_ids.items()) & set(b.external_ids.items())
    external_signal = 1.0 if shared else 0.0

    # A shared external id is decisive; otherwise lean on email, then name+org.
    return max(
        external_signal,
        0.55 * email_signal + 0.45 * name_org_signal,
    )


def merge(survivor: CanonicalAttendee, cand: IdentityCandidate,
          incoming_precedence: int, survivor_precedence: int) -> CanonicalAttendee:
    """Fold a candidate into the survivor under source precedence; never destructive."""
    win = incoming_precedence > survivor_precedence
    data = survivor.model_dump()
    data["full_name"] = cand.full_name if win else survivor.full_name
    data["org"] = (cand.org or survivor.org) if win else (survivor.org or cand.org)
    data["email"] = cand.email if win else survivor.email
    # Union, never overwrite: external ids and the source audit trail accumulate.
    data["external_ids"] = {**cand.external_ids, **survivor.external_ids} if not win \
        else {**survivor.external_ids, **cand.external_ids}
    if cand.source not in data["merged_sources"]:
        data["merged_sources"] = [*data["merged_sources"], cand.source]
    return CanonicalAttendee.model_validate(data)


def resolve(candidates: list[IdentityCandidate]) -> tuple[list[CanonicalAttendee], list[tuple]]:
    """Return settled canonical records plus (pair, score) rows needing human review."""
    survivors: dict[str, tuple[CanonicalAttendee, int]] = {}
    review: list[tuple] = []

    for group in block(candidates):
        ordered = sorted(group, key=lambda c: (-c.precedence, c.registered_at))
        base = ordered[0]
        canonical_id = base.email_key.split("@", 1)[0][:8].ljust(8, "0")
        current = CanonicalAttendee(
            canonical_id=canonical_id, email=base.email, full_name=base.full_name,
            org=base.org, merged_sources=[base.source], external_ids=base.external_ids,
        )
        current_prec = base.precedence

        for cand in ordered[1:]:
            s = score(base, cand)
            if s >= AUTO_MERGE:
                current = merge(current, cand, cand.precedence, current_prec)
                current_prec = max(current_prec, cand.precedence)
            elif s >= REVIEW_FLOOR:
                review.append(((base.source_record_id, cand.source_record_id), round(s, 3)))
        survivors[canonical_id] = (current, current_prec)

    return [rec for rec, _ in survivors.values()], review


if __name__ == "__main__":
    now = datetime.now(timezone.utc)
    rows = [
        IdentityCandidate(source=Source.CRM, source_record_id="crm-1",
                          email="[email protected]", full_name="Jane Doe",
                          org="Corp", external_ids={"crm": "9001"}, registered_at=now),
        IdentityCandidate(source=Source.STRIPE, source_record_id="pi-1",
                          email="[email protected]", full_name="Jane R Doe",
                          org="Corp Inc", external_ids={"stripe_pi": "pi_1"}, registered_at=now),
        IdentityCandidate(source=Source.WALKUP, source_record_id="kiosk-1",
                          email="[email protected]", full_name="Jané Doe",
                          org=None, external_ids={}, registered_at=now),
    ]
    resolved, needs_review = resolve(rows)
    assert len(resolved) == 1, "three touches of one human must collapse to one record"
    only = resolved[0]
    # CRM has highest precedence, so its name and org survive the conflict.
    assert only.full_name == "Jane Doe"
    assert only.org == "Corp"
    # Every source that touched the human is recorded; ids are unioned, not lost.
    assert set(only.merged_sources) == {Source.CRM, Source.STRIPE, Source.WALKUP}
    assert only.external_ids["stripe_pi"] == "pi_1" and only.external_ids["crm"] == "9001"
    print("OK:", only.model_dump())

The verification block is the stage’s own regression test: three touches of one human — differing in email casing, a diacritic, and a missing org — collapse to a single CanonicalAttendee, the highest-precedence CRM values win the field conflicts, and no external identifier is dropped. That last assert is the boundary between this overview and the two deep guides beneath it: proving a merge loses nothing is exactly the concern owned by merging records without losing session history, while proving two near-duplicates find each other in the first place is owned by deduplicating attendees across multiple registration sources.

Production Debugging & Observability Link to this section#

Every resolution decision must be reconstructable from the log line, because the canonical index is the only durable state and it records outcomes, not the reasoning that produced them. Each decision emits a structured event keyed by correlation_id — carried forward from the originating ingestion event — plus the canonical_id it resolved to and the score that drove the verdict. An auto-merge logs identity_merged at info; an ambiguous pair logs identity_review_flagged at warning with both source record ids and the score; a candidate that matches nothing new logs identity_created at info.

{
  "level": "warning",
  "event": "identity_review_flagged",
  "correlation_id": "reg-7f3a9c2e",
  "canonical_id": "janedoe0",
  "left": {"source": "crm", "source_record_id": "crm-1"},
  "right": {"source": "walkup", "source_record_id": "kiosk-1"},
  "score": 0.71,
  "signals": {"email": 1.0, "name_org": 0.44, "external": 0.0},
  "band": "review"
}

In a Datadog or ELK pipeline, index these fields by name: correlation_id as the cross-stage facet that ties a duplicate-badge complaint back to the exact merge that produced it, event as the monitored dimension (a sustained rise in identity_review_flagged means a source changed its payload shape and its match keys drifted), score and the signals map to distinguish a genuine ambiguous human from a scoring regression, and band to alert when the review queue is filling faster than adjudicators can drain it. Candidates that land in the ambiguous band are wrapped in a diagnostic envelope carrying both raw records and the score breakdown, and routed to the dead-letter queue for human adjudication rather than force-merged. A saturation alert on count(event:identity_review_flagged) / count(event:identity_merged) crossing a few percent is the earliest signal that an upstream integration changed how it emits names or emails.

Performance & Memory Constraints Link to this section#

Resolution is CPU-bound on string comparison and memory-bound on the blocking index, not I/O-bound. The constraints below are the ones that bite during a registration surge.

The GIL constraint is the one most often gotten wrong: because scoring is a tight CPU loop, adding threads adds contention rather than throughput. Horizontal scaling is process-per-core, with each worker owning a disjoint set of blocks so two workers never contend for the same canonical record.

Incident Triage Checklist Link to this section#

When duplicate badges surface at the check-in desk or the review queue starts filling, work these steps in order. Target MTTR is under fifteen minutes.

1. Confirm the symptom class. Check whether records are failing to match (duplicates escaping) or matching too eagerly (wrong humans merged): curl -s localhost:9090/metrics | grep -E 'identity_(created|merged|review)_total'. A spike in identity_created with flat identity_merged means match keys stopped colliding — a normalization or casing regression; a spike in identity_merged with complaints means the threshold is too loose.
2. Inspect the review queue depth. redis-cli -n 5 LLEN dlq:identity:review. A climbing depth confirms ambiguous pairs are being diverted rather than force-merged; peek the newest envelope with redis-cli -n 5 LINDEX dlq:identity:review 0 and read its signals breakdown.
3. Attribute the drift. Group flagged envelopes by source. One integration suddenly dominating the review queue is the usual cause — a renamed field, a stripped diacritic, or an email casing change that forked its match keys.
4. Contain the source. Pause the offending integration at the gate rather than the whole stage: redis-cli SET ingress:pause:<source> 1 EX 900. Clean sources keep resolving while the bad one is held.
5. Roll back or re-tune. If a threshold change caused over-eager merges, restore the prior AUTO_MERGE/REVIEW_FLOOR tags and redeploy; if a normalization change forked keys, roll the _fold/email-validator version back and let the canonical index rebuild the affected blocks.
6. Validate recovery and drain. Confirm the flag/merge ratio returns to baseline, then hand the quarantined pairs to the async batch processing drain job to replay them through the now-healthy scorer.

Frequently Asked Questions Link to this section#

Why block before scoring instead of just comparing every pair? Because all-pairs comparison is quadratic and a mid-size event has tens of thousands of attendees, which is hundreds of millions of comparisons the registration window cannot absorb. Blocking groups candidates by cheap deterministic keys — normalized email, a folded name signature — so only genuinely plausible pairs are ever scored. It trades a small, controllable recall risk (two records that share no blocking key are never compared) for a decisive throughput win, and that risk is bounded by choosing multiple independent blocking keys rather than one.

When should a match auto-merge versus go to human review? Auto-merge only above a high confidence threshold where a shared external identifier or an exact normalized email plus a strong name score leaves little doubt. Everything in the middle band — a name match with a diverging email, or a shared org with different names — goes to a review queue rather than being guessed, because a wrong merge is far more expensive than a duplicate: it fuses two humans’ session history and access credentials, and unwinding it is manual. When in doubt, flag, do not merge.

What stops two concurrent source imports from creating two canonical records for one person? An atomic claim on the canonical_id. The first import to reach a resolved identity claims the id with a Redis SET NX; a second import that resolves to the same human loses the claim, re-reads the existing survivor, and merges into it rather than creating a rival record. That race — and the near-duplicates it produces when the guard is missing — is the exact failure covered by deduplicating attendees across multiple registration sources.

Related Link to this section#

• Core Architecture & Event Taxonomy — the parent section that frames how identity resolution connects to ingestion, mapping, rendering, and dispatch.
• Deduplicating Attendees Across Multiple Registration Sources — the deep guide for when the same human escapes as two records because their match keys never collided.
• Merging Attendee Records Without Losing Session History — the deep guide for when a merge succeeds but silently drops the session or scan history it was meant to keep.
• Event Taxonomy Schema Design — defines the canonical field types and tier taxonomy the match keys are built from.
• Attendee Field Mapping Rules — the downstream stage that enriches and projects the resolved canonical record toward the renderer.
• Check-In Kiosk Sync — the on-site stage that reconciles each kiosk’s local attendee cache against the canonical id this stage publishes.

Up: Core Architecture & Event Taxonomy — the section these four pipeline stages sit inside.