BadgeprintingEvent registration · badge automation

Reconciling Offline Kiosk Check-Ins After Network Loss

Symptom Statement

The venue Wi-Fi hiccups for ninety seconds during the morning rush, and the self-service kiosks keep working — they check attendees in against a local cache, open the gate, and hand out badges as if nothing happened. Then the network comes back, every kiosk flushes its backlog at once, and the server state tears itself apart: the same attendee appears checked in twice from two kiosks, a later local write overwrites an earlier server-confirmed one, badge jobs replay and print a second copy, and the head-count on the ops dashboard no longer matches the turnstiles. This page addresses that exact failure — the divergence between what each kiosk recorded while partitioned and what the server believes after everyone reconnects. It is the reconciliation half of the check-in kiosk sync stage, which owns how a fleet of intermittently-connected kiosks converges back to a single source of truth. The tells are consistent: a spike of 409 conflicts in the sync endpoint the instant connectivity returns, duplicate checked_in rows keyed to one attendee, badge counts that exceed unique check-ins, and a manual de-dupe spreadsheet passed around after the event.

Offline kiosk check-ins reconciled through a durable append-only log with idempotency keys and a monotonic sequence A kiosk partitioned from the network writes each check-in to a local append-only event log, stamping every entry with a stable idempotency key derived from kiosk id and a monotonic per-kiosk sequence number, plus a captured wall-clock time. When the network is restored the kiosk flushes the log in sequence order to the server reconciler. The reconciler claims each idempotency key exactly once, so a replayed entry is a no-op rather than a second row; it resolves conflicts by preferring the entry with the highest monotonic sequence for that attendee rather than last-writer-wins on wall clock. A confirmed unique check-in flows onward to attendee identity resolution and the print queue, while a duplicate replay is diverted to a suppressed branch that prints nothing. Partition-tolerant check-in: append locally, flush in order, claim each key once Kiosk (partitioned) append-only local log key = kiosk:seq monotonic seq++ fsync durable Flush on reconnect replay in seq order carries idempotency key + seq + captured_at at-least-once delivery Server reconciler claim key once (SET NX) highest seq wins not last wall-clock skew-immune Confirmed check-in → identity resolution → print queue one badge, once head-count exact duplicate replay → suppressed key already claimed — no row, no second badge job

Root Cause Analysis

Four independent failures produce the reconnect divergence. An incident is usually one of them, occasionally two stacked, so triage means naming which before you touch the sync path.

  • No local durable queue. The kiosk holds unsynced check-ins in process memory or a best-effort in-RAM cache. A reload, a battery swap, or a browser tab crash during the outage loses them outright, or — worse — a partial flush leaves some check-ins on the server and some still local with no record of the boundary, so a retry re-sends the wrong subset.
  • Last-writer-wins clobbering. The sync endpoint resolves a collision by keeping whichever write arrives last, or whichever carries the newest updated_at. An earlier, server-confirmed truth (staff manually corrected a badge tier at the help desk) is silently overwritten by a stale offline write that flushed later, because arrival order and wall-clock order both lie under partition.
  • Clock skew across kiosks. Kiosks are laptops and tablets whose clocks drift, sit in different time zones, or never synced NTP on the venue network. Any conflict resolution keyed on captured_at timestamps ranks writes by a clock you do not control, so the “newest” write is an artifact of which device was fast, not which event happened last.
  • Replayed events double-printing badges. At-least-once flush delivery means the server can receive the same check-in twice — once before an ACK was lost, once on retry. Without an idempotency guard, each arrival dispatches a fresh badge job, so a single attendee walks away with two printed badges and the print floor burns stock it cannot get back.

Symptom-to-Resolution Matrix

Lost or Partially-Flushed Check-Ins

Symptoms

  • After a reload or crash during the outage, attendees who used a kiosk are missing from the server entirely.
  • A retry re-checks-in people who were already synced, because the kiosk cannot tell where its last successful flush stopped.

Root cause. Unsynced events live in volatile memory with no durable boundary marker, so a crash loses them and a resume cannot compute the un-acked suffix.

Fix

  1. Persist every check-in to an append-only log on durable local storage the moment it happens — IndexedDB for a browser kiosk, SQLite with PRAGMA synchronous=FULL for a native one — before the gate opens.
  2. Assign each entry a per-kiosk monotonic_seq and never mutate a written entry; the log is append-only, so a crash can only lose the in-flight tail, never rewrite history.
  3. Track a persisted last_acked_seq watermark and, on reconnect, flush only entries with seq > last_acked_seq, advancing the watermark solely on server ACK.

Clobbered Server Truth (Last-Writer-Wins)

Symptoms

  • A help-desk correction or an earlier kiosk check-in is silently replaced by a stale offline write that flushed later.
  • The final server record disagrees with what staff know happened first.

Root cause. Conflict resolution keeps the last arrival or the newest updated_at, which under partition reflects flush order and clock drift, not causal order.

Fix

  1. Resolve conflicts on the (attendee_id) key by keeping the entry with the highest monotonic_seq from an authoritative sequence, never the last to arrive.
  2. Treat a server-side manual correction as its own high-sequence event so it cannot be undone by a replayed offline write.
  3. Make the write a conditional upsert — apply only if the incoming sequence is greater than the stored one — so a late, lower-sequence flush is a no-op rather than a clobber.

Clock-Skew-Ranked Conflicts

Symptoms

  • Which duplicate “wins” changes depending on which kiosk’s clock was ahead.
  • Reordering appears random and is not reproducible.

Root cause. Ordering keyed on device captured_at ranks by an uncontrolled wall clock instead of a causal counter.

Fix

  1. Keep captured_at for display and audit only; never use it to break a conflict.
  2. Order and de-conflict on the monotonic sequence, which is a logical clock you own end to end.
  3. If cross-kiosk ordering matters, pair the per-kiosk sequence with the kiosk_id as a tiebreaker so the ordering is total and deterministic.

Replayed Events Double-Printing

Symptoms

  • Badge count exceeds unique check-ins; attendees hold two identical badges.
  • The print queue shows paired jobs seconds apart for one attendee_id.

Root cause. At-least-once flush plus no dedup guard lets a retried event dispatch a second badge job.

Fix

  1. Stamp each entry with a stable idempotency_key ({kiosk_id}:{seq}) computed at capture time, not at flush time, so a retry carries the identical key.
  2. Claim the key atomically at the reconciler with Redis SET key value NX; a failed claim means “already processed” and returns a no-op ACK.
  3. Gate the badge dispatch on a first-claim result only, so a replay never reaches the print queue orchestration stage.

Minimal Working Implementation

A self-contained reconciler: a Pydantic v2 model for the flushed entry, an idempotency claim, a sequence-guarded conditional upsert, and a verification block that proves a replayed event and a stale lower-sequence write are both no-ops while the highest-sequence truth survives. The stores are simple in-memory stand-ins so the module runs as-is; in production the claim set is Redis and the state table is Postgres.

PYTHON
from __future__ import annotations
from datetime import datetime, timezone
from pydantic import BaseModel, ConfigDict, Field, field_validator


class KioskCheckIn(BaseModel):
    """One flushed check-in entry from a kiosk's append-only local log."""
    model_config = ConfigDict(strict=True, extra="forbid")

    kiosk_id: str = Field(..., pattern=r"^kiosk-[0-9]{2,4}$")
    monotonic_seq: int = Field(..., ge=0)          # per-kiosk logical clock
    attendee_id: str = Field(..., min_length=6, max_length=36)
    captured_at: datetime                          # display/audit ONLY, never for ordering
    status: str = Field(..., pattern=r"^(checked_in|corrected)$")

    @field_validator("captured_at", mode="before")
    @classmethod
    def _aware(cls, v):
        # Normalise to UTC so a skewed local clock can't even be compared as truth.
        dt = v if isinstance(v, datetime) else datetime.fromisoformat(v)
        return dt.astimezone(timezone.utc) if dt.tzinfo else dt.replace(tzinfo=timezone.utc)

    @property
    def idempotency_key(self) -> str:
        # Stable across retries: derived from capture-time identity, not flush time.
        return f"{self.kiosk_id}:{self.monotonic_seq}"


class Reconciler:
    def __init__(self) -> None:
        self._claimed: set[str] = set()                 # stands in for Redis SET NX
        self._state: dict[str, KioskCheckIn] = {}       # attendee_id -> winning entry
        self.printed: list[str] = []                    # badge jobs actually dispatched

    def _claim(self, key: str) -> bool:
        """Atomic first-claim. Real impl: redis.set(key, '1', nx=True, ex=...)."""
        if key in self._claimed:
            return False
        self._claimed.add(key)
        return True

    def reconcile(self, entry: KioskCheckIn) -> str:
        # Gate 1 — idempotency: a replayed flush is a no-op, no second badge.
        if not self._claim(entry.idempotency_key):
            return "duplicate"

        # Gate 2 — conflict resolution by monotonic sequence, NOT wall clock.
        current = self._state.get(entry.attendee_id)
        if current is not None and entry.monotonic_seq <= current.monotonic_seq:
            return "stale"        # late, lower-sequence flush cannot clobber truth

        self._state[entry.attendee_id] = entry
        # Only a genuine first confirmation dispatches a badge job downstream.
        if current is None:
            self.printed.append(entry.attendee_id)
        return "applied"


if __name__ == "__main__":
    r = Reconciler()
    base = {"kiosk_id": "kiosk-07", "attendee_id": "att_9f3ac210",
            "captured_at": "2026-07-16T09:01:00", "status": "checked_in"}

    # First arrival: applied, one badge dispatched.
    assert r.reconcile(KioskCheckIn.model_validate({**base, "monotonic_seq": 1})) == "applied"
    # Exact replay of the same entry (at-least-once retry): no-op, no second badge.
    assert r.reconcile(KioskCheckIn.model_validate({**base, "monotonic_seq": 1})) == "duplicate"
    # A help-desk correction with a higher sequence wins.
    assert r.reconcile(KioskCheckIn.model_validate(
        {**base, "monotonic_seq": 5, "status": "corrected"})) == "applied"
    # A stale offline write that flushed late (lower seq) cannot clobber the correction.
    assert r.reconcile(KioskCheckIn.model_validate(
        {**base, "monotonic_seq": 2})) == "stale"

    assert r._state["att_9f3ac210"].status == "corrected"
    assert r.printed == ["att_9f3ac210"]   # exactly one badge for one attendee
    print("OK:", {"winning_seq": r._state["att_9f3ac210"].monotonic_seq,
                  "badges": len(r.printed)})

The assertions are the fix’s own regression test: a replay claims nothing new, a stale lower-sequence write is refused, the highest-sequence correction is the surviving truth, and exactly one badge job leaves the reconciler for one attendee.

Memory & Performance Constraints

The reconciler is I/O-bound on the claim store and the state table during the reconnect thundering herd, when an entire fleet flushes at once.

Component Constraint Mitigation
Reconnect flush burst Every kiosk flushes its full backlog simultaneously the instant Wi-Fi returns Flush in bounded batches with jittered backoff per kiosk so the herd spreads over seconds, not one spike
Idempotency claim set One key per check-in accumulates across the whole event Set an EX TTL that outlives the longest plausible offline window (a few hours), let Redis auto-evict
Local append-only log Unbounded growth on a long-lived kiosk with a large event Truncate entries at or below last_acked_seq once the server ACK is durable
State-table upsert Sequence-guarded conditional writes contend on hot attendee_id rows Make the upsert a single atomic WHERE incoming_seq > stored_seq statement; no read-modify-write race
Badge dispatch fan-out A first-claim flood can overrun the print floor at peak Hand dispatch to print queue orchestration, which owns rate limiting and back-pressure

Incident Triage & Rollback

Fast path when the reconnect-divergence alarm fires — target under fifteen minutes to containment. Every step is non-destructive until the final rollback.

  1. Scope the conflict wave. Check the sync endpoint for the 409/stale/duplicate ratio the moment connectivity returned: redis-cli --scan --pattern "ckin:claim:*" | wc -l against expected unique check-ins. A count near the unique total is healthy; a large excess is a replay storm.
  2. Confirm no data loss. Compare each kiosk’s last_acked_seq watermark against its highest local monotonic_seq; a gap means un-flushed entries still sit on-device and must be drained before any rollback.
  3. Freeze new dispatch. Pause the badge dispatch downstream so triage does not add prints: redis-cli SET ckin:dispatch:paused 1 EX 900. Reconciliation keeps running; only printing halts.
  4. Attribute the divergence. Group stale results by kiosk_id; a single offending kiosk usually points at a clock or a stuck flush watermark, a fleet-wide pattern points at a last-writer-wins regression in the endpoint.
  5. Rollback. If a bad deploy reintroduced wall-clock conflict resolution, git revert HEAD~1 --no-edit && kubectl rollout restart deploy/checkin-reconciler. Because every apply is idempotent on idempotency_key and guarded on monotonic_seq, re-driving the full flush after rollback can only converge — a replay re-claims nothing and a stale write is refused.

Post-rollback validation. Re-drive the flush and confirm convergence:

BASH
# No attendee should hold more than one confirmed check-in row:
psql -c "SELECT attendee_id, COUNT(*) FROM check_ins GROUP BY attendee_id \
         HAVING COUNT(*) > 1;"    # expect 0 rows
# Badge jobs must not exceed unique confirmed check-ins:
psql -c "SELECT (SELECT COUNT(*) FROM badge_jobs) <= \
         (SELECT COUNT(DISTINCT attendee_id) FROM check_ins) AS ok;"  # expect t

Frequently Asked Questions

Why not just resolve conflicts by timestamp — isn’t the newest check-in the right one? No, because the “newest” timestamp comes from an uncontrolled device clock. Kiosks drift, sit in different time zones, and often never sync NTP on a locked-down venue network, so ranking by captured_at ranks by which device was fast, not which event happened last. Keep the timestamp for display and audit, and resolve conflicts on a monotonic sequence you own, which is a logical clock that survives skew.

Do I need a full CRDT or vector clocks for this? Usually not. A per-kiosk monotonic sequence plus a stable idempotency key handles the two real requirements — exactly-once application and skew-immune conflict resolution — for check-in, which is close to a last-write-wins register keyed per attendee. Vector clocks earn their complexity when many nodes concurrently mutate the same field with genuine causal branching; a kiosk check-in is a single-writer append, so the lighter scheme converges correctly with far less to get wrong.

What stops a replayed flush from printing a second badge? The idempotency claim. Each entry carries a stable idempotency_key computed at capture time, so a retry carries the identical key. The reconciler claims it atomically with SET NX; a failed claim means “already processed” and returns a no-op without dispatching a badge job. Only a genuine first claim reaches the print queue.

  • Check-In Kiosk Sync — the parent stage that defines how a fleet of intermittently-connected kiosks converges back to one source of truth.
  • Attendee Identity Resolution — resolves the flushed attendee_id to a single canonical person, so a duplicate check-in and a duplicate identity never compound.
  • Print Queue Orchestration — the downstream stage that receives a confirmed, deduplicated check-in and owns rate-limited, back-pressured badge dispatch.