Check-In Kiosk Sync: Keeping Self-Service Terminals Consistent with the Registration Store

A bank of self-service check-in kiosks at the venue door is a distributed system that nobody planned to build, and it is one of the stages owned by the On-Site Check-In & Print Routing section. Each terminal accepts a badge scan or a name search, records that an attendee has arrived, and triggers a print — but it does all of that against a local copy of the registration roster, because the one guarantee an event floor cannot make is uninterrupted network. A dropped uplink at 8:55am, thirty seconds before the keynote doors open, must not stop the line. So every kiosk caches the roster it needs, writes check-ins locally, and syncs them back to the central registration store when it can. This component exists to make that eventual sync correct: to guarantee that an attendee who checked in at kiosk 3 while the wifi was down is not shown as un-arrived when the venue router recovers, and that the same person walking up to kiosk 7 two minutes later is recognized as already checked in rather than issued a second badge.

The failure this stage prevents is silent divergence — two kiosks holding conflicting truth about one attendee, and the central store picking the wrong one when they reconnect. Everything about who an attendee actually is across multiple registration sources is delegated upstream to attendee identity resolution; this stage assumes a resolved attendee_id and concerns itself only with the arrival fact and its ordering. The detailed recovery drill for a terminal that has been offline for an extended window is documented in reconciling offline kiosk check-ins after network loss. This page defines the contract, the deterministic reconciliation rule, and the observability an on-call engineer needs when the sync lag alarm fires mid-event.

Scope Boundary Link to this section#

The kiosk sync layer stays reasoned-about only because it refuses to answer questions that belong to adjacent stages. It moves arrival facts, orders them, and resolves collisions on one axis — time of arrival for a known identity — and delegates everything else.

The reconciler holds no attendee PII beyond the identifier and the arrival record. It never repairs identity, never issues a credential, and never touches a printer. That narrowness is what lets a check-in from a kiosk that was offline for forty minutes be replayed hours later and land in exactly the state it would have reached had the network never dropped.

Data Contract & Schema Link to this section#

Every arrival crosses the sync boundary as a KioskCheckinEvent. The contract is deliberately minimal — four load-bearing fields plus a client-generated idempotency key — because the more a kiosk is allowed to assert, the more ways two kiosks can disagree. Ordering is carried by an explicit per-kiosk sequence number rather than inferred from the timestamp, because kiosk clocks drift and a venue’s cheap tablet hardware is the last place to trust datetime.now() as a total order.

from datetime import datetime, timezone
from pydantic import BaseModel, Field, field_validator, ConfigDict

class KioskCheckinEvent(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid", frozen=True)

    attendee_id: str = Field(..., min_length=8, max_length=36, pattern=r"^[a-zA-Z0-9\-]+$")
    kiosk_id: str = Field(..., pattern=r"^kiosk-[0-9]{1,3}$")
    checked_in_at: datetime
    seq: int = Field(..., ge=1)
    method: str = Field(..., pattern=r"^(scan|name_search|manual)$")

    @field_validator("checked_in_at", mode="before")
    @classmethod
    def require_utc(cls, v: object) -> object:
        # Kiosks MUST stamp UTC; a naive datetime is a clock-config bug, reject it.
        if isinstance(v, datetime) and v.tzinfo is None:
            raise ValueError("checked_in_at must be timezone-aware UTC")
        return v

    @property
    def dedupe_key(self) -> str:
        # Stable identity of THIS write, independent of arrival order.
        return f"{self.kiosk_id}:{self.seq}"

attendee_id is the correlation key and the reconciliation key — all conflict resolution is scoped to one identifier, so it is bounded and never defaulted. kiosk_id names the origin terminal and, paired with seq, forms the idempotency token dedupe_key: a Stripe-style “have I already applied this exact write” guard that makes a redelivered backlog harmless. seq is a monotonic counter local to each kiosk, incremented once per accepted check-in and persisted alongside the buffer, so a kiosk that reboots mid-event resumes counting rather than restarting at 1 and colliding with its own history. checked_in_at is the business time used to pick a winner between kiosks; the mode="before" validator rejects a naive datetime outright because a timezone-less timestamp from a mis-configured tablet is the single most common cause of a phantom “arrived in the future” record. frozen=True makes an event immutable once constructed — a buffered arrival is a fact, not a mutable row — which is what lets the reconciler treat replay as safe.

Deterministic Implementation Link to this section#

Two reconciliation disciplines are in play at once, and conflating them is the classic mistake. The arrival status an attendee sees — checked in or not, which door, at what time — is resolved first-arrival-wins: the earliest checked_in_at for an attendee_id is canonical, and a later event from another kiosk does not overwrite it. But the record of what happened is append-only: every event, including the losing duplicates, is retained as an audit row. Last-writer-wins is wrong for arrival status here because the “last writer” is usually a redelivered backlog draining out of a kiosk that was offline — chronologically old data arriving late — and letting it clobber the canonical arrival would rewrite history backwards. So the rule is: canonical status is decided by business time, never by arrival time; the audit log keeps everything.

from dataclasses import dataclass, field

@dataclass
class ReconcilerState:
    # canonical[attendee_id] -> the winning arrival event
    canonical: dict[str, KioskCheckinEvent] = field(default_factory=dict)
    # audit is append-only: every accepted event, winners and duplicates alike
    audit: list[KioskCheckinEvent] = field(default_factory=list)
    # applied dedupe_keys — the idempotency guard against replayed backlogs
    applied: set[str] = field(default_factory=set)


@dataclass(frozen=True)
class ReconcileResult:
    accepted: bool          # was this a new write (not a replay)?
    is_duplicate: bool      # did the attendee already have a canonical arrival?
    canonical: KioskCheckinEvent


def reconcile(state: ReconcilerState, event: KioskCheckinEvent) -> ReconcileResult:
    """Merge one kiosk event. Idempotent on (kiosk_id, seq); first-arrival-wins."""
    # 1. Idempotency: a replayed backlog event is a no-op against the winner.
    if event.dedupe_key in state.applied:
        current = state.canonical[event.attendee_id]
        return ReconcileResult(accepted=False, is_duplicate=True, canonical=current)

    state.applied.add(event.dedupe_key)
    state.audit.append(event)  # append-only: nothing is ever discarded

    # 2. First-arrival-wins on business time, NOT on arrival-at-reconciler time.
    current = state.canonical.get(event.attendee_id)
    if current is None:
        state.canonical[event.attendee_id] = event
        return ReconcileResult(accepted=True, is_duplicate=False, canonical=event)

    if event.checked_in_at < current.checked_in_at:
        # A late-draining backlog carried an EARLIER real arrival — it wins status,
        # but we already logged the (now-superseded) later one in the audit trail.
        state.canonical[event.attendee_id] = event
        return ReconcileResult(accepted=True, is_duplicate=True, canonical=event)

    # Later or equal arrival for an already-arrived attendee: a duplicate walk-up.
    return ReconcileResult(accepted=True, is_duplicate=True, canonical=current)


if __name__ == "__main__":
    def evt(aid, kid, iso, seq):
        return KioskCheckinEvent(
            attendee_id=aid, kiosk_id=kid,
            checked_in_at=datetime.fromisoformat(iso).replace(tzinfo=timezone.utc),
            seq=seq, method="scan",
        )

    st = ReconcilerState()
    # Kiosk 1 records the real arrival at 08:55 but is offline; kiosk 7 sees a
    # walk-up at 08:57 and syncs FIRST because its uplink is healthy.
    r_late = reconcile(st, evt("att-10000001", "kiosk-7", "2026-07-16T08:57:00", 4))
    assert r_late.is_duplicate is False           # first to reconcile, provisional winner
    r_early = reconcile(st, evt("att-10000001", "kiosk-1", "2026-07-16T08:55:00", 2))
    assert r_early.canonical.kiosk_id == "kiosk-1"  # earlier BUSINESS time wins status
    assert r_early.is_duplicate is True             # attendee was already arrived
    # Replay of kiosk-1's backlog is a harmless no-op.
    r_replay = reconcile(st, evt("att-10000001", "kiosk-1", "2026-07-16T08:55:00", 2))
    assert r_replay.accepted is False
    assert len(st.audit) == 2                       # both real writes retained; replay dropped
    print("OK canonical:", st.canonical["att-10000001"].kiosk_id, "audit rows:", len(st.audit))

Three properties make this safe under a reconnect storm. First, reconcile is idempotent on dedupe_key, so when a kiosk that buffered nine events reconnects and its client retries the drain, replays collapse to no-ops instead of doubling the audit trail. Second, the winner is chosen by checked_in_at — business time — so the order events arrive at the reconciler is irrelevant; a healthy kiosk that syncs a 08:57 walk-up before an offline kiosk drains its real 08:55 arrival still ends with the 08:55 record as canonical. Third, nothing is destructive: a “loss” only demotes an event from canonical to audit, so the daily who-actually-arrived reconciliation and any forensic dispute have the complete record. The one thing this block does not decide is whether the duplicate walk-up should have reprinted a badge — that suppression is the already_checked_in signal fanned back to the kiosks, and reprint policy proper belongs to badge reprint handling.

Production Debugging & Observability Link to this section#

Because a kiosk’s local buffer is invisible from the center until it drains, the only durable diagnostic is the structured event the reconciler emits per merge. Each carries attendee_id, kiosk_id, seq, the sync_lag_ms between checked_in_at and reconciliation time, and the verdict. A healthy online kiosk shows single-digit-second lag; a spike into minutes is the earliest sign a terminal went offline and is now draining a backlog.

{
  "level": "info",
  "event": "kiosk_event_reconciled",
  "trace_id": "checkin-att-10000001",
  "attendee_id": "att-10000001",
  "kiosk_id": "kiosk-3",
  "seq": 7,
  "verdict": "duplicate_appended",
  "sync_lag_ms": 2417000,
  "canonical_kiosk": "kiosk-1"
}

In a Datadog or ELK pipeline, index kiosk_id as the primary facet so a single misbehaving terminal is isolatable, sync_lag_ms as a distribution to alert on the p99 crossing a few seconds (the reconnect-backlog signal), and verdict as a monitored dimension — a rising rate of duplicate_appended means attendees are being recognized at more than one terminal, which is either innocent (someone scanned twice) or a symptom that the already_checked_in fan-out is not reaching kiosks. Correlate trace_id forward to the print receipt and backward to the resolved identity from attendee identity resolution. A per-kiosk gap detector on seq is the highest-value alert of all: if the reconciler has applied seq 1,2,3,6 for kiosk-3, events 4 and 5 are stranded in a buffer that has not fully drained, and that gap — not the raw lag — is what tells you an offline terminal still has unsynced arrivals.

Performance & Memory Constraints Link to this section#

The reconciler is memory- and lock-bound during the door-open surge, not CPU-bound: the work per event is a dict lookup and a set insert, but thousands of kiosks draining backlogs at once contend on the same per-attendee state.

The lock-granularity choice is the one most often gotten wrong: a single mutex around reconcile turns a 20-kiosk reconnect into a serial queue and blows the MTTR. Because the canonical map is keyed by attendee_id, contention only occurs for the rare attendee scanned at two terminals within the same window, so a striped lock keyed on the identifier keeps throughput flat as kiosks are added.

Incident Triage Checklist Link to this section#

When the sync-lag alarm fires or duplicate badges start appearing on the floor, work these in order. Target MTTR is under fifteen minutes.

1. Identify the lagging terminals. Query the reconciler metrics for kiosks whose p99 sync_lag_ms exceeds the threshold: curl -s localhost:9090/metrics | grep kiosk_sync_lag_ms | sort -t'"' -k2. One kiosk high and the rest flat means a single dropped uplink; all high means the central store or its broker is the bottleneck.
2. Find the sequence gaps. For a suspect terminal, list applied sequences and look for holes: redis-cli -n 5 SMEMBERS applied:kiosk-3 | sort -n | awk 'NR>1 && $1!=p+1{print "gap after "p} {p=$1}'. A gap confirms stranded events still buffered on the tablet.
3. Confirm the buffer is draining, not stuck. On the kiosk (or via its health endpoint) check the outbound queue depth trend: curl -s http://kiosk-3.local:8080/health | jq '.buffer_depth' twice, thirty seconds apart. Falling means it is recovering on its own; flat-and-high means the drain client is wedged and needs a restart.
4. Contain duplicate prints. If the already_checked_in fan-out is lagging and duplicates are printing, pause reprints for the affected door rather than the whole floor: redis-cli SET checkin:suppress_reprint:door-north 1 EX 600, which the print stage honors while sync catches up.
5. Drain the backlog deterministically. Trigger the offline-replay job — which merges the buffer through the same idempotent reconcile path — following reconciling offline kiosk check-ins after network loss; because merges are idempotent on (kiosk_id, seq), re-running the drain is always safe.
6. Validate recovery. Confirm the sequence gaps have closed and sync_lag_ms returned to baseline, then spot-check one previously-stranded attendee: redis-cli -n 5 HGET canonical att-10000001 kiosk_id should name the earliest-arrival kiosk, not the last to reconnect.

Frequently Asked Questions Link to this section#

Why order by a sequence number instead of the check-in timestamp? Because the timestamp and the ordering answer two different questions. checked_in_at decides which arrival is canonical between kiosks and is business-meaningful, but kiosk hardware clocks drift and cannot provide a reliable total order within a single terminal’s stream. The per-kiosk seq gives an exact, gap-detectable order of what one kiosk recorded, which is what lets you prove events 4 and 5 are still stranded in a buffer when the reconciler has already applied 1, 2, 3, and 6.

Why first-arrival-wins rather than last-writer-wins? Because on this floor the “last writer” is almost always a redelivered backlog draining out of a kiosk that was offline — chronologically old data that happens to arrive late. Last-writer-wins would let that late delivery overwrite a correct, earlier arrival and rewrite history backwards. Deciding the canonical arrival by the earliest business time makes the reconciler indifferent to the order events reach it, which is the only property that survives a reconnect storm.

If a duplicate never overwrites the winner, why store it at all? Because arrival status and the audit record are separate concerns. The append-only log of every event — winners and duplicates alike — is what backs the daily attendance reconciliation, dispute resolution (“I definitely checked in at the west door”), and the sequence-gap detector. Demoting a duplicate to an audit row costs almost nothing and keeps the system’s memory complete, which is what makes a forensic replay months later reproduce the exact same canonical state.

Related Link to this section#

• On-Site Check-In & Print Routing — the parent section that frames how kiosk sync connects to queueing, reprints, and printer failover on the event floor.
• Attendee Identity Resolution — resolves the attendee_id this stage treats as given, deduplicating people across registration sources before they ever reach a kiosk.
• Reconciling Offline Kiosk Check-Ins After Network Loss — the step-by-step recovery drill for draining a terminal’s buffer through the idempotent reconciler after an extended outage.
• Print Queue Orchestration — consumes the confirmed arrival and owns the badge job it turns into, including the already_checked_in reprint suppression.
• Badge Reprint Handling — decides whether a recognized-again attendee should be reprinted at all, and how to do so without minting a second credential.

Up: On-Site Check-In & Print Routing — the section these on-floor stages sit inside.