BadgeprintingEvent registration · badge automation

Failing Over to a Backup Badge Printer Mid-Event

Symptom Statement

The rush is at full tilt and the primary badge printer dies — a fatal jam, a thermal shutdown, a yanked power cable. It had one badge half-printed and forty more queued behind it. Now those forty-one attendees are stuck: the badge that was mid-print may or may not have come out, and the queued jobs are stranded on a device that will not wake up. Whatever you do next must satisfy two non-negotiable constraints at once — no attendee leaves without a badge (nothing lost), and no attendee gets two (nothing duplicated). This page is the failover runbook for exactly that moment, and it belongs to the On-Site Print Failover stage, which owns detecting a dead printer and moving its work to a backup safely. The tells are abrupt: one printer’s jobs stop acknowledging entirely, its health probe flips red, a lane goes cold mid-transaction, and the reserved-but-unacked count for that device is frozen. Naive recovery here is dangerous — a blind re-dispatch reprints the badge that already came out, and a blind failover to a printer that is only briefly unreachable double-prints the whole backlog. The fix is a disciplined sequence: probe, drain, then re-dispatch idempotently.

Probe, drain, and idempotently re-dispatch a dead printer's jobs to a backup A primary printer has failed with one job in flight and several queued. Failover runs three ordered steps. First, a health probe confirms the printer is genuinely down, not briefly unreachable, before any reroute. Second, a drain step reclaims the in-flight job and the queued jobs from the dead printer, marking the in-flight job's outcome unknown. Third, each reclaimed job is re-dispatched to the backup printer through an idempotency gate: a Redis SET NX on the print-completion key admits a job only if it was never confirmed printed, so an already-printed badge is skipped and every unprinted badge is produced exactly once on the backup. Probe → drain → idempotent re-dispatch: nothing lost, nothing duplicated primary — DEAD 1 in-flight (unknown) 40 queued 1 · probe down for N checks? not a blip else: wait, don't reroute 2 · drain reclaim in-flight + queued jobs 3 · dedupe gate SET NX done:job_id skip if already printed admit if never confirmed backup printer each unprinted badge ×1 already-printed skipped — no reprint The dedupe gate is what makes re-dispatch safe: a confirmed badge can never be reprinted.

Root Cause Analysis

The danger in mid-event failover is not the failure itself — printers jam — but the recovery doing damage. Four concrete gaps turn a recoverable jam into lost or duplicated badges.

  • No health probe (or a twitchy one). Without a probe that distinguishes a genuinely dead printer from a two-second network blip, failover either fires too late (attendees wait on a corpse) or too early (it reroutes the backlog to a backup, then the “dead” printer wakes and prints its copies too — every badge twice).
  • Non-idempotent re-dispatch. Re-queuing a dead printer’s jobs onto the backup with fresh job identities, or with no completion check, reprints anything that already came out. The half-printed badge and any job the primary finished but did not get to acknowledge are the ones that duplicate.
  • In-flight job lost on crash. The job the printer was actively rendering when it died sits in a reserved/unacked limbo. If the broker’s visibility timeout is long or the worker never nacks, that one attendee’s badge is silently dropped — nothing reprints it because nothing knows it was in flight.
  • No drain-before-reroute. Rerouting new traffic to the backup while the dead printer’s own queue is never reclaimed leaves the backlog stranded. The lane recovers for new arrivals but the forty already-queued attendees never get a badge — you fixed the future and abandoned the present.

Symptom-to-Resolution Matrix

Missing or Flapping Health Probe

Symptoms

  • Failover fires on a transient blip and the “dead” printer later wakes and reprints its jobs.
  • Or the opposite: a genuinely dead printer holds its jobs for minutes because nothing declares it down.

Root cause. No stable liveness signal, so the system cannot tell a real death from a momentary unreachability.

Fix

  1. Probe each printer on a short interval (reachable + no error state + media present) and require N consecutive failures before declaring it down, so a single blip cannot trigger failover.
  2. Once declared down, mark it unhealthy so the load-balancing dispatcher stops sending it new work immediately.
  3. Require a successful probe and a manual probe-print before a recovered printer is re-added, so it never resurrects with a stale queue.

Non-Idempotent Re-Dispatch

Symptoms

  • After failover some attendees hold two identical badges.
  • The backup printed a badge that the primary had already produced before it died.

Root cause. Re-dispatch does not check whether a job was already printed, so already-completed work is reproduced on the backup.

Fix

  1. Key every job by a stable job_id that survives re-dispatch — never mint a new ID on reroute.
  2. Gate each print behind an atomic completion claim: SET print:done:{job_id} 1 NX. A job that was already confirmed printed fails the claim and is skipped.
  3. Mark completion only after the printer confirms the badge, so a job left ambiguous by the crash is re-attempted rather than assumed done.

In-Flight Job Lost on Crash

Symptoms

  • Exactly one attendee per failure is missing a badge — the one being printed at the moment of death.
  • That job appears in neither the backup’s output nor the dead printer’s recovered queue.

Root cause. The actively-rendering job is reserved-but-unacked and falls into a gap between “not queued” and “not confirmed done.”

Fix

  1. Use acks_late=True so a job is acknowledged only after the badge is confirmed, and a worker/printer crash returns it to the queue instead of dropping it.
  2. On failover, explicitly reclaim the reserved in-flight job for the dead printer and re-dispatch it through the same completion gate — if it did finish, the gate skips it; if it did not, the backup prints it.
  3. Keep the broker visibility timeout short enough that a lost in-flight job is redeliverable within the failover window, not minutes later.

No Drain-Before-Reroute

Symptoms

  • New arrivals print fine on the backup, but the jobs already queued on the dead printer never come out.
  • Queue depth on the dead device stays frozen at its failure-time value.

Root cause. Failover rerouted new traffic but never reclaimed the dead printer’s existing backlog.

Fix

  1. Drain the dead printer’s queue in full — move every queued job to the backup before or alongside rerouting new traffic.
  2. Run the drain through the completion gate so a job the primary managed to print before dying is not reprinted.
  3. Only consider failover complete when the dead printer’s queue depth reaches zero, not when new traffic starts flowing. This drain-and-reclaim discipline is the same one used when clearing a stuck badge print queue at peak check-in.

Minimal Working Implementation

A self-contained failover routine over a Redis-backed pair of printers. health_ok requires N consecutive good probes; failover reclaims the dead printer’s in-flight job and its whole queue, then re-dispatches each through an atomic completion gate (SET NX on print:done:{job_id}) so a badge already confirmed printed is skipped and every unprinted badge is produced exactly once on the backup. The verification block proves both halves: the already-printed job does not reprint, and the queued and in-flight jobs do.

PYTHON
import json
import os
import redis

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
QUEUE_PREFIX = "print:q:"          # per-printer job list
INFLIGHT_PREFIX = "print:inflight:"  # the single reserved job a printer is rendering
DONE_PREFIX = "print:done:"        # completion marker per job_id — the dedupe gate
FAIL_PREFIX = "print:probefail:"   # consecutive probe-failure counter per printer
DOWN_AFTER = 3                     # consecutive probe failures before declaring down

r = redis.Redis.from_url(REDIS_URL, decode_responses=True)


def record_probe(printer: str, reachable: bool) -> bool:
    """Update the probe counter; return True while the printer is still considered up."""
    key = FAIL_PREFIX + printer
    if reachable:
        r.delete(key)
        return True
    fails = r.incr(key)
    return fails < DOWN_AFTER


def dispatch_if_unprinted(job: dict, target: str) -> str:
    """Place a job on target only if it was never confirmed printed. Idempotent."""
    job_id = job["job_id"]
    # SET NX succeeds only if no completion marker exists → this job is unprinted.
    if r.set(DONE_PREFIX + job_id, "claimed", nx=True):
        r.rpush(QUEUE_PREFIX + target, json.dumps(job))
        return "redispatched"
    return "skipped_already_printed"


def confirm_printed(job_id: str) -> None:
    """Called by the printer worker AFTER a badge physically prints."""
    r.set(DONE_PREFIX + job_id, "printed")


def failover(dead: str, backup: str) -> dict:
    """Move the dead printer's in-flight and queued jobs to the backup, safely."""
    moved, skipped = [], []

    # 1. Reclaim the in-flight job first — it is the one most at risk of being lost.
    raw_inflight = r.getdel(INFLIGHT_PREFIX + dead)
    reclaimed = [raw_inflight] if raw_inflight else []

    # 2. Drain the whole queue of the dead printer.
    while (raw := r.lpop(QUEUE_PREFIX + dead)) is not None:
        reclaimed.append(raw)

    # 3. Re-dispatch each reclaimed job through the completion gate.
    for raw in reclaimed:
        job = json.loads(raw)
        outcome = dispatch_if_unprinted(job, backup)
        (moved if outcome == "redispatched" else skipped).append(job["job_id"])

    return {"moved": moved, "skipped": skipped, "dead_depth": r.llen(QUEUE_PREFIX + dead)}


if __name__ == "__main__":
    r.flushdb()

    # Arrange: primary has one in-flight job and two queued; one queued job (job-x)
    # was already confirmed printed just before the crash.
    r.set(INFLIGHT_PREFIX + "primary", json.dumps({"job_id": "job-inflight", "attendee_id": "a-0"}))
    r.rpush(QUEUE_PREFIX + "primary", json.dumps({"job_id": "job-x", "attendee_id": "a-x"}))
    r.rpush(QUEUE_PREFIX + "primary", json.dumps({"job_id": "job-y", "attendee_id": "a-y"}))
    confirm_printed("job-x")  # primary finished this badge before dying

    # The probe must see DOWN_AFTER consecutive failures before we fail over.
    assert record_probe("primary", reachable=False) is True   # 1st miss — still up
    assert record_probe("primary", reachable=False) is True   # 2nd miss — still up
    assert record_probe("primary", reachable=False) is False  # 3rd miss — declared down

    # Act: fail over to the backup.
    result = failover("primary", "backup")

    # job-x was already printed → skipped, never reprinted.
    assert "job-x" in result["skipped"], result
    # in-flight and the unprinted queued job move to the backup exactly once.
    assert set(result["moved"]) == {"job-inflight", "job-y"}, result
    assert result["dead_depth"] == 0                    # fully drained
    backup_ids = {json.loads(x)["job_id"] for x in r.lrange(QUEUE_PREFIX + "backup", 0, -1)}
    assert backup_ids == {"job-inflight", "job-y"}, backup_ids

    # Re-running failover is safe — nothing left to move, nothing reprints.
    again = failover("primary", "backup")
    assert again["moved"] == [] and again["dead_depth"] == 0
    print("OK: moved", sorted(result["moved"]), "· skipped printed", result["skipped"])

The completion gate is the whole safety argument. dispatch_if_unprinted uses SET NX on print:done:{job_id}, so a badge that was confirmed printed before the crash fails the claim and is skipped, while the in-flight job — whose outcome the crash left unknown — is re-attempted. Re-running failover is a no-op, which means a nervous operator running it twice cannot double-print.

Memory & Performance Constraints

Failover is a burst operation under time pressure; the constraints are about doing it atomically and not stranding state.

Component Constraint Mitigation
Probe counter A flapping printer churns the probefail counter and can oscillate up/down Require N consecutive failures to go down and a clean probe plus probe-print to come back up
In-flight reclaim getdel on the in-flight key must be atomic or the job is lost or doubled Use GETDEL (single round-trip) so reclaim and clear cannot interleave with a late ack
Completion markers print:done:* grows one key per job for the whole event TTL markers past the event window; they are bounded by attendee count, not retries
Drain loop A long LPOP loop on a deep dead queue blocks the failover path Drain in a pipeline batch; failover latency scales with backlog depth, so keep per-printer queues shallow via load-balancing
Broker visibility timeout Too long strands the in-flight job; too short redelivers prematurely Tune to just above one badge’s print time with acks_late=True so a crash redelivers promptly

Incident Triage & Rollback

Fast path when a printer dies mid-rush. Target under fifteen minutes; every step before rollback is non-destructive.

  1. Confirm death, not a blip. Check the probe-failure counter: redis-cli GET print:probefail:primary. Fewer than the threshold means wait — do not fail over on a single miss. At or past it, proceed.
  2. Capture the at-risk state. Read the in-flight job and queue depth before touching anything: redis-cli GET print:inflight:primary and redis-cli LLEN print:q:primary. This is the exact set of attendees you must not lose.
  3. Fail over to the backup. Run the failover routine (failover("primary", "backup")). It reclaims the in-flight job, drains the queue, and re-dispatches each job through the completion gate.
  4. Confirm the drain completed. redis-cli LLEN print:q:primary must read 0, and the backup’s depth should rise by the number of unprinted jobs — the already-printed one is correctly skipped.

Rollback. If you failed over prematurely and the primary recovers, do not simply re-enable it with its old queue — that queue may re-drive already-moved jobs. Instead, keep the completion gate authoritative: re-add the primary only after a clean probe-print, and let dispatch_if_unprinted reject anything already printed. To revert a bad failover deploy, git revert HEAD~1 --no-edit && docker compose up -d --build; because every dispatch is gated on print:done:{job_id}, re-running failover after the revert cannot duplicate a badge. Reprints that genuinely need a fresh credential belong to badge reprint handling, not to failover.

Post-rollback validation. Confirm nothing was lost and nothing doubled:

BASH
redis-cli LLEN print:q:primary   # expect 0 — fully drained
redis-cli LLEN print:q:backup    # expect += count of unprinted jobs only
# every reclaimed job must have exactly one completion marker:
redis-cli --scan --pattern "print:done:*" | wc -l   # expect == distinct jobs handled

Frequently Asked Questions

How do I avoid failing over on a printer that was only briefly unreachable? Require N consecutive probe failures before declaring the printer down, not a single miss. A short probe interval with a small consecutive-failure threshold (e.g. three misses over a few seconds) rides out a network blip while still reacting within the failover window. Never reroute on one failed probe — that is exactly what causes the “dead” printer to wake and double-print its backlog.

What happens to the badge that was half-printed when the printer died? Its outcome is unknown, so failover re-attempts it through the completion gate. If the primary had actually confirmed it printed, the SET NX on print:done:{job_id} fails and it is skipped; if it never confirmed, the gate admits it and the backup prints it. That is the point of confirming completion only after a physical print — an ambiguous job is safely re-attempted, never assumed done.

Is failover the same thing as reprinting a lost badge? No. Failover moves a dead printer’s existing jobs to a backup under the original job_id, so it is idempotent and never issues a new credential — a re-dispatched job either prints once or is skipped. Reprinting a badge an attendee physically lost or damaged is a separate flow with its own credential-revocation concerns, owned by badge reprint handling.