BadgeprintingEvent registration · badge automation

Merging Attendee Records Without Losing Session History

Symptom Statement

The duplicate got resolved — and the attendee’s schedule disappeared with it. Two records for one human are merged into a survivor, and the workshop they signed up for through the CRM, the two sessions they picked on the ticketing form, and the check-in scan from yesterday’s badge collection are all gone, still parented to the record that was deleted. Or worse, the child rows survive but collide: two session:keynote selections that should have collapsed into one now show the attendee registered twice for the same slot, and a re-run of the merge job double-counts every scan. This page addresses that exact symptom — session, agenda, and check-in history lost or corrupted by the act of merging — and it is the merge-side failure within the attendee identity resolution stage, which owns how partial records collapse into one canonical identity. Detecting the duplicate in the first place is the separate concern of deduplicating attendees across multiple registration sources; this page assumes the pair is confirmed and asks the harder question — how do you fold two records into one and provably lose nothing?

Destructive overwrite loses child history; a union merge reparents and dedupes it onto the survivor The left panel shows the destructive path: a loser record carrying a keynote session and a scan is deleted outright, its scalar fields overwrite the survivor, and its child rows are orphaned and lost. The right panel shows the union merge: scalar fields are chosen by source precedence, the loser's session and scan child rows are reparented onto the survivor's canonical id, duplicate sessions collapse to one by a natural key, and the whole operation is guarded so replaying it changes nothing. DESTRUCTIVE · overwrite + delete Survivor sessions: [A] Loser (deleted) sessions: [keynote], scan Survivor sessions: [A] keynote + scan orphaned → lost UNION · reparent + dedupe + idempotent Survivor sessions: [A] Loser sessions: [keynote], scan Union onto survivor reparent child FKs dedupe by natural key precedence for scalars Survivor sessions: [A, keynote] scan kept · replay-safe nothing lost · re-running the merge changes nothing

Root Cause Analysis

Four concrete, independent failures destroy history during a merge. An incident is usually one of them, occasionally two stacked, so triage means naming which before re-running any merge job.

  • Destructive overwrite instead of union-merge. The merge copies the winner’s scalar fields over the survivor and deletes the loser, treating child collections as scalars too. The loser’s sessions and scans are overwritten or dropped rather than unioned, so any history unique to the loser is gone.
  • Missed foreign-key reparenting. The merge updates the attendee row but never re-points the loser’s session_registration.attendee_id and checkin_scan.attendee_id at the survivor’s canonical id. The child rows still exist but are orphaned to a deleted parent — invisible to every query that starts from the survivor.
  • Conflicting field precedence. Two records disagree on tier, name, or contact fields, and the merge picks a value by row order or last-write-wins rather than an explicit source ranking, so a trusted CRM value loses to a walk-up typo — or two child rows for the same session both survive because no natural key decides which is canonical.
  • Merge not idempotent on replay. The job is re-run after a partial failure or a queue redelivery, and because it appends rather than upserts, it re-parents already-moved children a second time and double-counts scans. A merge that is not safe to replay corrupts history the moment it retries.

Symptom-to-Resolution Matrix

Destructive Overwrite of Child Collections

Symptoms

  • After a merge the survivor has fewer sessions than the two inputs combined.
  • Check-in scans from the loser record are simply gone.

Root cause. Child collections are treated as scalar fields and overwritten, not unioned.

Fix

  1. Union child sets explicitly: survivor.sessions = dedupe(survivor.sessions + loser.sessions).
  2. Never DELETE the loser until its children are confirmed reparented in the same transaction.
  3. Treat sessions and scans as sets keyed by a natural identity, never as a replaceable blob.

Orphaned Child Rows

Symptoms

  • The rows exist in the table but no query from the survivor returns them.
  • A count of session_registration is unchanged, yet the attendee shows none.

Root cause. The foreign keys were never re-pointed at the survivor.

Fix

  1. Reparent every child table in one transaction: UPDATE session_registration SET attendee_id = :survivor WHERE attendee_id = :loser.
  2. Enumerate all child tables — sessions, scans, payments, badge jobs — from the schema, not from memory.
  3. Only after every child is reparented, retire the loser (soft-delete with a merged_into pointer, never a hard delete).

Wrong Field Precedence

Symptoms

  • A CRM-sourced tier is overwritten by a walk-up value.
  • Two rows for the same session both survive the merge.

Root cause. Precedence is implicit (row order / last write) instead of an explicit source ranking plus a natural key.

Fix

  1. Rank sources explicitly and let the higher rank win each scalar conflict.
  2. Give every child a natural key (session_code, scan_id) and collapse duplicates on it.
  3. Keep the losing scalar in an audit trail rather than discarding it.

Non-Idempotent Replay

Symptoms

  • Re-running the merge doubles scan counts or re-lists sessions.
  • A retried task corrupts a record that was already correct.

Root cause. The merge appends instead of upserting and carries no replay guard.

Fix

  1. Make every write an upsert keyed on a natural identity so a second run is a no-op.
  2. Record a completed merge_id and short-circuit if it is already applied.
  3. Assert merge(merge(x)) == merge(x) before shipping.

Minimal Working Implementation

A self-contained survivor merge: scalar fields resolved by source precedence, child sessions and scans unioned and deduped by natural key, the loser retired with a merged_into pointer, and an idempotency guard so replaying the merge is a no-op. The final asserts prove nothing is lost and a second application changes nothing.

PYTHON
from __future__ import annotations

from dataclasses import dataclass, field, replace

SOURCE_RANK = {"crm": 40, "eventbrite": 30, "stripe": 20, "walkup": 10}


@dataclass(frozen=True)
class Session:
    session_code: str      # natural key — dedupe on this
    title: str


@dataclass(frozen=True)
class Scan:
    scan_id: str           # natural key — dedupe on this
    gate: str


@dataclass
class Attendee:
    canonical_id: str
    source: str
    full_name: str
    ticket_tier: str
    sessions: list[Session] = field(default_factory=list)
    scans: list[Scan] = field(default_factory=list)
    merged_into: str | None = None
    applied_merges: set[str] = field(default_factory=set)


def _union_by_key(a: list, b: list, key) -> list:
    """Union two child lists, collapsing duplicates on their natural key, order-stable."""
    seen: dict = {}
    for item in (*a, *b):
        seen.setdefault(key(item), item)
    return list(seen.values())


def merge_attendees(survivor: Attendee, loser: Attendee, merge_id: str) -> Attendee:
    """Fold loser into survivor: union children, precedence for scalars, idempotent."""
    # Idempotency guard: replay of an already-applied merge is a no-op.
    if merge_id in survivor.applied_merges:
        return survivor

    win = SOURCE_RANK[loser.source] > SOURCE_RANK[survivor.source]
    merged = replace(
        survivor,
        full_name=loser.full_name if win else survivor.full_name,
        ticket_tier=loser.ticket_tier if win else survivor.ticket_tier,
        # Reparent + union child rows onto the survivor's canonical id, deduped.
        sessions=_union_by_key(survivor.sessions, loser.sessions, lambda s: s.session_code),
        scans=_union_by_key(survivor.scans, loser.scans, lambda s: s.scan_id),
        applied_merges=survivor.applied_merges | {merge_id},
    )
    # Retire the loser with a pointer — never a hard delete.
    loser.merged_into = survivor.canonical_id
    return merged


if __name__ == "__main__":
    survivor = Attendee(
        canonical_id="janedoe0", source="crm", full_name="Jane Doe",
        ticket_tier="SPEAKER",
        sessions=[Session("KEYNOTE", "Opening Keynote")],
        scans=[Scan("scan-1", "north")],
    )
    loser = Attendee(
        canonical_id="janedoe9", source="walkup", full_name="jane doe",
        ticket_tier="GENERAL",
        sessions=[Session("KEYNOTE", "Opening Keynote"),   # duplicate — must collapse
                  Session("WORKSHOP-3", "Async Python")],  # unique — must survive
        scans=[Scan("scan-2", "south")],                   # unique — must survive
    )

    out = merge_attendees(survivor, loser, merge_id="m-001")
    # Nothing lost: the loser's unique workshop and scan are on the survivor now.
    assert {s.session_code for s in out.sessions} == {"KEYNOTE", "WORKSHOP-3"}
    assert {s.scan_id for s in out.scans} == {"scan-1", "scan-2"}
    # Duplicate keynote collapsed, not double-listed.
    assert len(out.sessions) == 2
    # Precedence: CRM outranks walk-up, so the trusted tier survives.
    assert out.ticket_tier == "SPEAKER"
    assert loser.merged_into == "janedoe0"

    # Idempotency: replaying the same merge changes nothing.
    replay = merge_attendees(out, loser, merge_id="m-001")
    assert len(replay.sessions) == 2 and len(replay.scans) == 2
    assert replay is out or replay == out
    print("OK:", {"sessions": len(out.sessions), "scans": len(out.scans), "tier": out.ticket_tier})

The assert block is the fix’s own regression test: the loser’s unique workshop and scan land on the survivor, the duplicate keynote collapses to one instead of double-listing, the higher-ranked CRM tier wins the conflict, and replaying the merge with the same merge_id leaves counts untouched. That idempotency guarantee is what makes the merge safe to retry after a partial failure — the exact property a queue redelivery would otherwise violate. Which records get fed into this merge in the first place is decided upstream by deduplicating attendees across multiple registration sources.

Memory & Performance Constraints

The merge is transaction-bound and lock-sensitive, not compute-heavy — the risks are lock contention and partial writes, not CPU.

Component Constraint Mitigation
Reparent transaction Updating every child table for a high-history attendee holds row locks Batch the reparent updates and keep the transaction short; never hold locks across a network call
Child-set union _union_by_key materializes both child lists in memory Stream child rows by table and dedupe incrementally for attendees with thousands of scans
Idempotency ledger The applied_merges set grows with every merge touching a record Persist applied merge_ids in an indexed side table, not inline on the row, and prune post-event
Concurrent merges Two merges touching an overlapping survivor can interleave writes Serialize per-canonical-id with an advisory lock so only one merge mutates a survivor at a time

Incident Triage & Rollback

Fast path when session history goes missing after a merge — target under fifteen minutes to containment. Every step is non-destructive until the final rollback.

  1. Find the orphans. Count child rows pointing at a retired record: psql -c "SELECT COUNT(*) FROM session_registration sr JOIN attendees a ON a.canonical_id = sr.attendee_id WHERE a.merged_into IS NOT NULL;". A non-zero count means reparenting was skipped for at least one child table.
  2. Confirm loser rows were soft-deleted, not purged. psql -c "SELECT canonical_id, merged_into FROM attendees WHERE merged_into IS NOT NULL ORDER BY resolved_at DESC LIMIT 20;". If the losers are gone entirely, recovery must come from a backup snapshot, not from the live table.
  3. Check for double-counting from replay. redis-cli --scan --pattern "merge:applied:*" | wc -l against the merge-job run count; a mismatch signals a non-idempotent replay inflated the history.
  4. Contain. Pause the merge worker: celery -A identity control cancel_consumer merges, so no further merges corrupt records while you reconcile.

Rollback. Because losers are soft-deleted with a merged_into pointer, most incidents recover forward — re-run the reparent for the skipped child tables — rather than reverting code. If a bad merge build shipped, git revert HEAD~1 --no-edit && docker compose up -d --build, then replay affected merges through the restored idempotent path; the merge_id guard makes re-driving safe.

Post-rollback validation. Confirm no child row is orphaned and the survivor holds the full union:

BASH
psql -c "SELECT COUNT(*) FROM session_registration sr \
         JOIN attendees a ON a.canonical_id = sr.attendee_id \
         WHERE a.merged_into IS NOT NULL;"   # expect 0
# then confirm no session slot is double-listed for a survivor:
psql -c "SELECT attendee_id, session_code, COUNT(*) FROM session_registration \
         GROUP BY 1,2 HAVING COUNT(*) > 1;"  # expect no rows

Frequently Asked Questions

Should I hard-delete the loser record once the merge succeeds? No. Soft-delete it with a merged_into pointer at the survivor’s canonical id. A hard delete makes an incorrect merge unrecoverable without a backup restore, and it destroys the audit trail a duplicate-badge dispute needs. Keeping the retired row with a forwarding pointer means a wrong merge can be reversed and any late-arriving child row can still find its way to the survivor.

How do I make a merge safe to re-run after a worker crash? Make every write an upsert keyed on a natural identity — session_code for sessions, scan_id for scans — and record the completed merge_id so a replay short-circuits. Because a crashed worker with acks_late re-runs its task, a non-idempotent merge would double the history on retry; an idempotent one treats the second run as a no-op. Assert merge(merge(x)) == merge(x) in tests so the property cannot regress silently.

Which value wins when the two records disagree on ticket tier? The higher-ranked source, decided by an explicit precedence table, not by row order or last-write-wins. The CRM is the system of record and outranks a walk-up kiosk record typed under time pressure, so its tier survives the conflict while the losing value is kept in an audit trail rather than discarded. Never let arrival order decide a field that selects a template family or an access scope downstream.

Related pages

Related Deduplicating Attendees Across Multiple Registration Sources