BadgeprintingEvent registration · badge automation

On-Site Print Failover: Rerouting Around a Dead Printer Without Losing or Duplicating Badges

A badge printer jams, runs out of ribbon, or simply dies mid-job while a hundred attendees stand in line, and the question is not whether to reroute their badges to the healthy printer two tables over — it is how to do that without printing some badges twice and losing others entirely. This is the failure this stage owns, and it is one of the stages inside the On-Site Check-In & Print Routing section. The hard part is the in-flight job: the one the dead printer had already accepted but may or may not have physically printed before it failed. Reroute it blindly and you risk a duplicate; skip it and you risk an attendee who checked in but never got a badge. Failover has to make a decision about that ambiguous job under time pressure, drain the rest of the dead printer’s queue to a healthy device, and guarantee that re-dispatching a job is idempotent — that the same badge, keyed on its job id, prints exactly once across the whole printer fleet no matter how many times it is rerouted.

This stage sits directly downstream of queue management: the assignment of jobs to printers, retry budgets, and queue depth are owned by print queue orchestration, and failover consumes that queue rather than replacing it. What failover adds is the health-probe loop that declares a printer dead, the drain that moves its outstanding jobs, and the idempotency guard that makes re-dispatch safe. The full mid-event runbook — swapping to a backup device while the line keeps moving — is documented in failing over to a backup printer mid-event. This page defines the health and decision contract, the drain-and-re-dispatch implementation, and the observability an operator needs when a printer drops mid-rush.

A health probe declares printer A dead, drains its queued jobs, and re-dispatches them idempotently to printer B A health probe polls printer A on a fixed interval, checking that it is reachable, not jammed, and has ribbon and stock. Three consecutive failed probes trip printer A from healthy to dead, producing a FailoverDecision. On that decision the drain step reads printer A's outstanding jobs: the queued jobs that were never started, and the single in-flight job the printer had accepted. Queued jobs are re-dispatched to healthy printer B keyed on job id, so each prints exactly once. The in-flight job is held for an acknowledgement check: if printer A confirmed the physical print before dying, the job is marked done and not reprinted; if unconfirmed, it is re-dispatched to printer B under the same idempotency key, so a badge that A actually printed cannot come out of B a second time. Printer B, still healthy, prints the drained jobs. health probe drain · dead printer idempotent re-dispatch Health probe reach · jam · ribbon interval = 5s 3 fails → dead Printer A · dead queued: j2, j3, j4 in-flight: j1 (?) ack confirmed? → done | re-dispatch FailoverDecision drain queued jobs resolve in-flight j1 dedupe on job_id prints exactly once re-dispatch j2,j3,j4 (+j1?) Printer B · healthy prints drained jobs each job_id once probe keeps watching the probe watches every printer — B can fail over next

Scope Boundary

Failover is a reactive controller with one trigger and one guarantee: a printer is declared dead, and its work moves exactly once. It does not decide the initial job routing, does not render badges, and does not manage the queue’s retry economics — it reroutes around a fault.

In-Scope (this component owns) Out-of-Scope (delegated to adjacent stages)
Probing printer health and declaring a device dead after N consecutive failures Assigning jobs to printers under normal load — owned by print queue orchestration
Draining a dead printer’s queued and in-flight jobs to a healthy target Rendering the badge PDF and its layout — owned by upstream badge generation
Resolving the ambiguous in-flight job via an acknowledgement check The queue’s retry budget and backpressure policy — owned by print queue orchestration
Guaranteeing idempotent re-dispatch keyed on job_id — print exactly once Deciding whether a job should exist at all — an upstream check-in concern
Emitting the FailoverDecision record for audit and reconciliation The full mid-event swap runbook — owned by failing over to a backup printer

The failover controller holds no badge content — it moves job references, keyed on job_id, and never re-renders. That reference-only discipline is what makes re-dispatch cheap and idempotent: moving a job is a claim on a key, not a copy of a PDF, so replaying a drain a dozen times converges on the same single print.

Data Contract & Schema

Failover reasons over two records. PrinterHealth is the rolling verdict the probe maintains per device; FailoverDecision is the immutable record produced when a device is declared dead and its jobs are moved. The health model deliberately separates reachable from ready: a printer can answer the network and still be unable to print because it is jammed or out of ribbon, and treating those as the same signal is how you either fail over too eagerly (network blip) or too late (silent jam).

PYTHON
from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field, ConfigDict

class PrinterState(str, Enum):
    HEALTHY = "healthy"
    DEGRADED = "degraded"   # reachable but low ribbon/stock — drain proactively
    DEAD = "dead"           # unreachable or jammed past the failure threshold

class PrinterHealth(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")

    printer_id: str = Field(..., pattern=r"^printer-[a-z0-9]{1,8}$")
    reachable: bool
    jammed: bool
    ribbon_pct: int = Field(..., ge=0, le=100)
    consecutive_failures: int = Field(..., ge=0)
    probed_at: datetime

    def classify(self, fail_threshold: int) -> PrinterState:
        # Unreachable or jammed past the threshold is dead; low ribbon is degraded.
        if self.consecutive_failures >= fail_threshold or self.jammed:
            return PrinterState.DEAD
        if not self.reachable or self.ribbon_pct < 10:
            return PrinterState.DEGRADED
        return PrinterState.HEALTHY

class FailoverDecision(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid", frozen=True)

    decision_id: str = Field(..., min_length=8, max_length=64)
    dead_printer_id: str = Field(..., pattern=r"^printer-[a-z0-9]{1,8}$")
    target_printer_id: str = Field(..., pattern=r"^printer-[a-z0-9]{1,8}$")
    drained_job_ids: list[str] = Field(..., min_length=1)
    inflight_job_id: str | None = None   # the ambiguous job, if any
    inflight_acked: bool = False         # did the dead printer confirm the physical print?
    decided_at: datetime

PrinterHealth.classify is the single place the healthy/dead line is drawn, so the policy — how many failed probes, what ribbon floor — lives in one auditable method rather than scattered across the probe loop. consecutive_failures (not a single failed probe) is what trips DEAD, because a lone timeout on a congested venue network is noise; three in a row is a signal. jammed short-circuits straight to DEAD regardless of failure count, because a jam is a confirmed physical fault, not a transient. On the FailoverDecision, drained_job_ids is the manifest of what moved and is the key to reconciliation, while inflight_job_id and inflight_acked together encode the whole ambiguous-job problem: a job the dead printer had accepted, plus whether it confirmed printing before dying. frozen=True makes the decision an immutable audit fact — you can prove, months later, exactly which jobs were rerouted where and why.

Deterministic Implementation

The controller runs two loops. The probe loop maintains each printer’s PrinterHealth and flips a device to DEAD on the classification. The failover loop, triggered by that flip, drains the dead printer’s jobs and re-dispatches them — and the whole design turns on one guarantee: re-dispatch is idempotent on job_id, so a job prints exactly once across the fleet regardless of how many failovers touch it. The ambiguous in-flight job is resolved by an acknowledgement check against the dead printer’s last confirmed print: if it acked, the badge is already out and the job is closed; if it did not, the job re-dispatches under its original job_id, and the idempotency guard on the target printer ensures that even if the dead printer secretly did print it, the target refuses to print the same job_id twice.

PYTHON
from dataclasses import dataclass, field

@dataclass
class PrintFleet:
    # printed[job_id] -> printer_id that has committed the physical print
    printed: dict[str, str] = field(default_factory=dict)
    # queues[printer_id] -> list of job_ids waiting on that device
    queues: dict[str, list[str]] = field(default_factory=dict)
    applied_decisions: set[str] = field(default_factory=set)


class NoHealthyTarget(Exception):
    """Every alternate printer is also dead — escalate, do not silently drop."""


def dispatch(fleet: PrintFleet, printer_id: str, job_id: str) -> bool:
    """Idempotent print. Returns True if THIS call committed the print, else False.

    The guard is fleet-wide on job_id: if ANY printer already printed this job,
    no other printer will print it again — this is what makes re-dispatch safe.
    """
    if job_id in fleet.printed:
        return False  # already printed somewhere — a no-op, never a duplicate
    fleet.printed[job_id] = printer_id
    return True


def execute_failover(fleet: PrintFleet, decision: FailoverDecision) -> dict[str, int]:
    """Drain the dead printer and re-dispatch its jobs idempotently to the target."""
    # Idempotency at the DECISION level: replaying a whole failover is a no-op.
    if decision.decision_id in fleet.applied_decisions:
        return {"printed": 0, "skipped": 0, "replayed": 1}
    fleet.applied_decisions.add(decision.decision_id)

    printed = skipped = 0
    jobs = list(decision.drained_job_ids)

    # Resolve the ambiguous in-flight job FIRST.
    if decision.inflight_job_id is not None:
        if decision.inflight_acked:
            # The dead printer confirmed the physical print — record it, do not reprint.
            fleet.printed.setdefault(decision.inflight_job_id, decision.dead_printer_id)
            skipped += 1
        else:
            jobs.append(decision.inflight_job_id)  # unconfirmed — must be re-dispatched

    # Re-dispatch every drained job to the healthy target, exactly once each.
    for job_id in jobs:
        if dispatch(fleet, decision.target_printer_id, job_id):
            printed += 1
        else:
            skipped += 1

    fleet.queues[decision.dead_printer_id] = []  # dead printer's queue is now empty
    return {"printed": printed, "skipped": skipped, "replayed": 0}


if __name__ == "__main__":
    fleet = PrintFleet(queues={"printer-a": ["j2", "j3", "j4"], "printer-b": []})

    # Printer A dies. j1 was in-flight and A did NOT confirm it printed.
    health = PrinterHealth(
        printer_id="printer-a", reachable=False, jammed=True, ribbon_pct=40,
        consecutive_failures=3, probed_at=datetime.fromisoformat("2026-07-16T10:03:00"),
    )
    assert health.classify(fail_threshold=3) is PrinterState.DEAD

    decision = FailoverDecision(
        decision_id="fo-a-1003", dead_printer_id="printer-a", target_printer_id="printer-b",
        drained_job_ids=["j2", "j3", "j4"], inflight_job_id="j1", inflight_acked=False,
        decided_at=datetime.fromisoformat("2026-07-16T10:03:05"),
    )
    result = execute_failover(fleet, decision)
    assert result["printed"] == 4                      # j1..j4 each printed once on B
    assert fleet.printed["j1"] == "printer-b"           # unconfirmed in-flight reprinted
    # Replaying the SAME failover prints nothing — decision-level idempotency.
    assert execute_failover(fleet, decision)["replayed"] == 1
    # Even a fresh decision cannot double-print an already-printed job.
    dup = FailoverDecision(
        decision_id="fo-a-1004", dead_printer_id="printer-a", target_printer_id="printer-b",
        drained_job_ids=["j2"], decided_at=datetime.fromisoformat("2026-07-16T10:04:00"),
    )
    assert execute_failover(fleet, dup)["printed"] == 0  # j2 already out — no duplicate
    print("OK printed:", fleet.printed)

Three properties make this safe under a cascading failure. First, dispatch is idempotent fleet-wide on job_id: the guard checks whether any printer has committed the job, so re-dispatching after a failover — or after a second failover when the backup itself dies — can never produce a duplicate badge. Second, the ambiguous in-flight job is resolved conservatively: an acked job is recorded as printed and not reissued, while an unacked job is re-dispatched, and the fleet-wide guard absorbs the rare case where the dead printer actually printed it but never acked. Third, execute_failover is idempotent at the decision level too, so a retried or replayed drain — the norm when an operator re-triggers failover manually because they weren’t sure it took — is a clean no-op rather than a second reroute. The one thing this deliberately does not do is invent a new job for a badge that was never queued; a missing check-in is upstream, not a failover concern.

Production Debugging & Observability

Failover is invisible until it fires, so the FailoverDecision and each re-dispatch must be fully logged — the reconciliation question after an incident is always “did every checked-in attendee get exactly one badge,” and only the structured trail answers it. Each decision emits an event carrying the dead and target printers, the drained job manifest, and the in-flight resolution.

JSON
{
  "level": "warning",
  "event": "printer_failover_executed",
  "trace_id": "failover-printer-a-1003",
  "decision_id": "fo-a-1003",
  "dead_printer_id": "printer-a",
  "target_printer_id": "printer-b",
  "drained_count": 3,
  "inflight_job_id": "j1",
  "inflight_acked": false,
  "reprinted_inflight": true
}

In a Datadog or ELK pipeline, index dead_printer_id as a facet so a device that fails repeatedly is spotted and pulled from service, drained_count as a distribution to size how much load just shifted onto the target (a large drain onto an already-busy printer is the next bottleneck), and reprinted_inflight because a sustained rate of unacked in-flight jobs points at printers whose firmware acks late or not at all — the root cause of most duplicate-versus-lost ambiguity. Correlate trace_id back to the queue owned by print queue orchestration and forward to the per-job_id print-commit log. The highest-value alarm is a reconciliation over job_id: count(printed) GROUP BY job_id HAVING count > 1 must always be empty — a value above one is a duplicate badge and a hard bug in the idempotency guard, and it should page immediately.

Performance & Memory Constraints

The controller is probe-latency- and target-capacity-bound, not compute-bound: the decision itself is cheap, but the probe interval sets how fast a dead printer is noticed and the target printer’s throughput caps how fast a drain clears.

Component Constraint Mitigation
Probe interval vs detection lag A long interval leaves a dead printer accepting jobs into a black hole Probe every few seconds; on a failed probe, quarantine new dispatch to that device immediately, before the death is confirmed
Target printer capacity Draining a full queue onto one healthy printer overloads it and grows the line Spread the drain across all healthy devices via print queue orchestration, not a single failover target
printed idempotency map Grows with total jobs; must be consistent across controller replicas Back it with Redis SET NX keyed on job_id, not process memory, so every replica sees the same commit set
In-flight ack window Firmware that acks slowly widens the ambiguous window per job Keep a short, bounded ack timeout; treat an ack-timeout as unacked and lean on the fleet-wide guard against duplicates
Probe false positives A congested network trips a healthy printer to dead Require N consecutive failures and separate reachable from jammed so a network blip degrades rather than kills

The idempotency store is the constraint that must not be gotten wrong: if the printed set lives in a single controller’s memory and a second replica runs a concurrent failover, both can believe a job_id is unprinted and each dispatch it. Backing the guard with a shared atomic SET NX on job_id is what makes “exactly once” hold across replicas, which is the entire point of the stage.

Incident Triage Checklist

When a printer drops mid-rush or the duplicate/lost-badge alarm fires, work these in order. Target MTTR is under fifteen minutes.

  1. Confirm the printer is actually dead. Read its latest health verdict: curl -s http://printer-a.local:9100/health | jq '{reachable,jammed,ribbon_pct}'. A jammed:true is a real fault to clear physically; reachable:false with everything else fine may be a network blip — check the switch before failing over.
  2. Verify the drain executed. Confirm a FailoverDecision fired and the dead queue is empty: redis-cli LLEN queue:printer-a should read 0, and redis-cli GET failover:last:printer-a should show a recent decision_id.
  3. Check for duplicates or gaps. Run the exactly-once reconciliation: redis-cli --scan --pattern 'printed:*' | wc -l against the checked-in count, and scan for any job_id printed on two devices: psql -c "SELECT job_id,count(*) FROM print_commits GROUP BY 1 HAVING count(*)>1;" (must be empty).
  4. Contain the target’s load. If the drain overloaded printer B, stop routing new jobs to it and spread across the rest of the fleet: redis-cli SET printer:pause_new:printer-b 1 EX 300 so it clears its backlog without the line stalling further.
  5. Re-drive stragglers idempotently. For any checked-in attendee still without a badge, re-dispatch through the idempotent execute_failover path; because both dispatch and the decision are keyed for exactly-once, re-driving is always safe and can only fill a gap, never duplicate. The full swap procedure is in failing over to a backup badge printer mid-event.
  6. Validate recovery. Confirm the duplicate query is empty, every checked-in attendee has exactly one commit, and the recovered or replaced printer passes three consecutive probes before it is returned to the routing pool.

Frequently Asked Questions

How do you avoid printing the in-flight job twice when you cannot tell if the dead printer printed it? With two layers. First, an acknowledgement check: if the dead printer confirmed the physical print before dying, the job is recorded as done and never re-dispatched. Second, and more importantly, the fleet-wide idempotency guard keyed on job_id: even if the printer actually printed the badge but died before acking, re-dispatching it to the backup is safe because the target refuses to print a job_id that any printer has already committed. The unacked job is re-dispatched optimistically, and the guard absorbs the rare double.

Why require several failed probes instead of failing over on the first timeout? Because a single failed probe on a crowded venue network is almost always noise, not a dead printer, and failing over on it would thrash jobs between devices for no reason. Requiring N consecutive failures — while separately treating a confirmed jammed signal as immediate death — distinguishes a transient network blip (which should at most mark the printer degraded and quarantine new dispatch) from a genuine fault that warrants moving the whole queue.

What happens if the failover target also dies? The same machinery handles it: the probe declares the backup dead, a new FailoverDecision drains its queue to the next healthy device, and because every re-dispatch is keyed on job_id fleet-wide, jobs that already printed on the first target are not reprinted on the second. If no healthy printer remains, execute_failover cannot silently drop work — it raises so the incident escalates to a human rather than jobs vanishing. Spreading the original drain across multiple devices, via print queue orchestration, is what keeps one backup from becoming the next single point of failure.

Up: On-Site Check-In & Print Routing — the section these on-floor stages sit inside.

Continue in this section

Onsite Checkin Print Routing Failing Over to a Backup Badge Printer Mid-Event