BadgeprintingEvent registration · badge automation

Load-Balancing Badge Jobs Across Multiple On-Site Printers

Symptom Statement

You brought four badge printers to spread the check-in load, but only one of them is doing real work. Its queue is thirty deep and the lane it feeds has a line out the door, while the other three sit warm and idle with two or three jobs between them. Same event, same hardware, wildly uneven wait times — one lane clears in twenty seconds and the next takes four minutes. This is the printer-imbalance symptom: jobs pile onto a subset of printers while capacity you paid for goes unused, and the badge print queue as a whole drains far slower than the fleet could sustain. This page addresses that exact imbalance for a Redis-backed multi-printer setup, and it sits under the Print Queue Orchestration stage, which owns how badge jobs are assigned, acknowledged, and retried across on-site hardware. The tells are consistent: one printer’s sub-queue depth dwarfs the others, per-lane wait time has a wide spread rather than a tight band, adding a printer to the fleet changes nothing, and a printer that goes offline still has jobs routed to it. The goal is to make each job land on the printer that can produce it soonest — without ever assigning the same job to two printers.

Least-depth health-weighted dispatch versus static hashing across four printers On the left, static hashing sends every job for one lane to a single printer, whose queue is deep while two others are shallow and a fourth is offline but still receiving jobs. On the right, a dispatcher reads the live depth and health of every printer, filters out the offline one, and assigns each incoming job to the shortest healthy queue, keyed idempotently on job ID so no job lands on two printers. The result is four balanced queues of similar depth feeding four lanes with even wait times. Route each job to the shortest healthy queue, idempotent on job ID badge job job_id dispatcher read depth + health pick shortest healthy SADD job_id → dedupe printer A · depth 6 printer B · depth 5 printer C · depth 6 printer D · OFFLINE excluded — probe failing skipped lane 1 · even wait lane 2 · even wait lane 3 · even wait balanced depth → tight wait-time band

Root Cause Analysis

Uneven printer utilization has four common causes. They all produce the same visible symptom — one deep queue and idle capacity — so the fix depends entirely on which routing decision is wrong.

  • Static hashing to a printer. Jobs are assigned by hash(attendee_id) % printer_count or a fixed lane-to-printer map. Hashing is oblivious to how busy each printer already is, so a run of large templates or a hot lane piles onto whichever printer the hash favors, regardless of its depth. Worse, % printer_count reshuffles every assignment the moment a printer joins or leaves.
  • No health-weighted routing. The dispatcher treats all printers as equal even when one is slow, low on media, or recovering from a jam. A printer that accepts jobs but produces them at half speed keeps getting its full share, so its queue grows while healthier printers idle.
  • Sticky lane-to-printer mapping. Each check-in lane is hard-wired to one printer for the whole event. If one lane is busier — the VIP entrance, the press desk, the door nearest the parking lot — its dedicated printer is swamped while another lane’s printer barely runs. The mapping is fair per lane but not per printer.
  • No work-stealing. Once a job is committed to a printer’s queue it stays there, even if that printer later jams and a neighbor goes idle. Without a rebalancing step, a transient slowdown becomes permanent imbalance: the deep queue never sheds work to the shallow one.

Symptom-to-Resolution Matrix

Static Hashing Ignores Live Depth

Symptoms

  • One printer’s sub-queue is many times deeper than the others despite even overall traffic.
  • Adding a printer briefly reshuffles assignments and imbalance returns.

Root cause. Assignment is a pure function of the job’s key and the printer count, with no input from current queue depth, so it cannot react to a hot streak.

Fix

  1. Replace the hash with a least-depth choice: read every printer’s current depth and assign to the shallowest.
  2. Break ties deterministically (lowest printer ID) so the choice is reproducible and testable.
  3. Read depth and commit the assignment atomically so two concurrent dispatchers cannot both pick the same “shortest” queue.

No Health Weighting

Symptoms

  • A printer keeps receiving its full share while visibly slow, jammed, or low on stock.
  • Its queue climbs even though the dispatcher believes the fleet is balanced.

Root cause. The dispatcher has no health signal, so a degraded printer is indistinguishable from a healthy one until its queue is already deep.

Fix

  1. Attach a health probe per printer (reachable, media present, not in an error state) and refresh it on a short interval.
  2. Exclude unhealthy printers from candidate selection entirely — an offline or jammed printer should receive zero new jobs, which is the same guarantee that keeps on-site print failover from routing into a dead device.
  3. Weight by effective capacity, not just up/down, so a half-speed printer gets a smaller share rather than an equal one.

Sticky Lane-to-Printer Mapping

Symptoms

  • Wait time correlates with which lane an attendee joined, not overall load.
  • One printer is idle while the lane next to it has a line.

Root cause. A fixed lane-to-printer binding means a busy lane cannot borrow an idle neighbor’s printer.

Fix

  1. Decouple lanes from printers: publish all jobs to one shared pool and let the dispatcher place each on the best printer regardless of origin lane.
  2. Keep only affinity hints (prefer the nearest printer when depths are equal) rather than hard bindings, so proximity is honored without starving a printer.

No Work-Stealing After Commitment

Symptoms

  • A printer jams, recovers, and stays under-loaded while a neighbor never sheds its backlog.
  • Imbalance persists long after the event that caused it.

Root cause. Jobs are committed to a printer’s local queue and never move, so a transient slowdown leaves a permanent skew.

Fix

  1. Assign late: keep jobs in the shared pool and let each printer pull its next job only when it is ready, so idle printers naturally steal work.
  2. If you must pre-assign, add a rebalancer that migrates queued (not in-flight) jobs from the deepest queue to the shallowest, re-keyed idempotently so a migrated job is never printed twice.

Minimal Working Implementation

A self-contained least-depth, health-weighted dispatcher over a Redis-backed fleet. Each printer has its own list; a health map marks printers up or down. dispatch picks the shallowest healthy printer and pushes the job there, but only after an idempotency claim on the job_id, so re-running dispatch for the same job is a no-op rather than a second badge. The verification block proves both properties: jobs spread to the shallowest queues, and a duplicate dispatch lands nowhere.

PYTHON
import json
import os
import redis

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
PRINTERS = ["printer-a", "printer-b", "printer-c"]
QUEUE_PREFIX = "print:q:"          # per-printer list, e.g. print:q:printer-a
HEALTH_PREFIX = "print:health:"    # "up" / "down" per printer
ASSIGNED_KEY = "print:assigned"    # SET of job_ids already dispatched — dedupe guard

r = redis.Redis.from_url(REDIS_URL, decode_responses=True)


def is_healthy(printer: str) -> bool:
    # Absent key means never probed → treat as unhealthy (deny-by-default).
    return r.get(HEALTH_PREFIX + printer) == "up"


def shortest_healthy_printer() -> str | None:
    """Return the healthy printer with the smallest queue depth.

    Ties break on printer name so the choice is deterministic and testable.
    """
    candidates = [
        (r.llen(QUEUE_PREFIX + p), p)
        for p in PRINTERS
        if is_healthy(p)
    ]
    if not candidates:
        return None
    return min(candidates)[1]  # (depth, name) — min by depth, then name


def dispatch(job: dict) -> dict:
    """Place a job on the shortest healthy printer, idempotent on job_id."""
    job_id = job["job_id"]
    # Claim the job first. SADD returns 0 if it was already dispatched.
    if r.sadd(ASSIGNED_KEY, job_id) == 0:
        return {"action": "duplicate", "job_id": job_id, "printer": None}

    target = shortest_healthy_printer()
    if target is None:
        # No healthy printer: release the claim so a later retry can place it.
        r.srem(ASSIGNED_KEY, job_id)
        return {"action": "no_capacity", "job_id": job_id, "printer": None}

    r.rpush(QUEUE_PREFIX + target, json.dumps(job))
    return {"action": "assigned", "job_id": job_id, "printer": target}


if __name__ == "__main__":
    # Arrange: three printers, one of them down, two shallow queues.
    keys = [QUEUE_PREFIX + p for p in PRINTERS] + [ASSIGNED_KEY]
    r.delete(*keys)
    r.set(HEALTH_PREFIX + "printer-a", "up")
    r.set(HEALTH_PREFIX + "printer-b", "up")
    r.set(HEALTH_PREFIX + "printer-c", "down")   # excluded from selection

    # Act: dispatch four jobs; they must spread across the two healthy printers.
    results = [dispatch({"job_id": f"job-{i}", "attendee_id": f"a-{i}"}) for i in range(4)]
    placed = {res["job_id"]: res["printer"] for res in results}

    # The down printer never receives work.
    assert r.llen(QUEUE_PREFIX + "printer-c") == 0, "routed to a down printer"
    # Load is balanced: two on each healthy printer, none more than one apart.
    depth_a = r.llen(QUEUE_PREFIX + "printer-a")
    depth_b = r.llen(QUEUE_PREFIX + "printer-b")
    assert abs(depth_a - depth_b) <= 1, (depth_a, depth_b)
    assert depth_a + depth_b == 4

    # Re-dispatching an already-placed job is a no-op — no duplicate badge.
    again = dispatch({"job_id": "job-0", "attendee_id": "a-0"})
    assert again["action"] == "duplicate", again
    assert r.llen(QUEUE_PREFIX + "printer-a") + r.llen(QUEUE_PREFIX + "printer-b") == 4
    print("OK: balanced", {"printer-a": depth_a, "printer-b": depth_b}, "· dupe blocked")

The verification block asserts the two properties that matter on the floor: the down printer receives zero jobs (health gating works), and re-dispatching job-0 changes no queue depth (idempotency works). The idempotency claim is taken before placement and released only when no printer can accept the job, so a job is either placed exactly once or safely left for retry — never placed twice.

Memory & Performance Constraints

The dispatcher is called once per check-in, so its cost is per-job latency and Redis round-trips, not throughput of a batch.

Component Constraint Mitigation
Depth read fan-out shortest_healthy_printer issues one LLEN per printer per job Read depths in a single pipeline/MGET-style batch; cache health for a short TTL so only depth is hot
Assignment race Two dispatchers can both read the same shallowest depth and pile on Take the idempotency claim first, then place; or move selection into a Lua script so read-and-push is atomic
Health probe interval Probing too often adds load; too rarely routes into a just-failed printer Probe on a 2–5s interval and fail closed — an unprobed printer is treated as down
Dedupe set growth print:assigned grows one entry per job for the whole event TTL the set past the event window or clear it at teardown; it is bounded by attendee count, not traffic bursts
Skew under bursty arrivals A thundering herd at doors-open can still transiently favor one printer Prefer late binding (printers pull when ready) over eager push so idle printers self-balance

Incident Triage & Rollback

Fast path when the lanes diverge. Target under fifteen minutes; every step before rollback is read-only.

  1. Quantify the imbalance. for p in printer-a printer-b printer-c; do redis-cli LLEN print:q:$p; done. A single deep queue against shallow ones confirms routing skew rather than a global stall (that case is clearing a stuck badge print queue at peak check-in).
  2. Check health truthfulness. redis-cli MGET print:health:printer-a print:health:printer-b print:health:printer-c. A printer marked up that is physically jammed is the health-signal failure; a down printer still holding jobs is the sticky-mapping failure.
  3. Confirm the selection is depth-aware. Dispatch a synthetic job and watch which queue it lands on — it must be the shallowest healthy one. If it lands on the deep queue, the deployed dispatcher is still hashing.
  4. Shed the hot printer. Mark the swamped printer down (redis-cli SET print:health:printer-a down) so new jobs route elsewhere, then migrate its queued backlog to the shared pool for rebalancing.

Rollback. If a bad rebalance stranded jobs, replay each migrated job through dispatch; the idempotency claim means already-placed jobs are skipped and only genuinely unplaced ones land. To revert to the previous router, git revert HEAD~1 --no-edit && docker compose up -d --build. Because placement is idempotent on job_id, switching dispatchers mid-event cannot double-print — a job already on a printer’s queue is never re-placed.

Post-rollback validation. Confirm depths reconverge and no job is double-claimed:

BASH
for p in printer-a printer-b printer-c; do redis-cli LLEN print:q:$p; done  # expect a tight band
redis-cli SCARD print:assigned   # expect == number of jobs dispatched, no duplicates

Frequently Asked Questions

Should I balance by queue depth or by wait time? Depth is the right signal at check-in speed. Wait time is a lagging estimate that needs per-printer throughput modeling, and at badge speeds a shallower queue almost always prints sooner. Use depth as the primary key and health as a gate; only reach for time-weighting when printers have genuinely different speeds, and even then weight depth by effective capacity rather than switching signals entirely.

Why take the idempotency claim before choosing a printer instead of after placing the job? Because claiming first closes the window where two concurrent dispatchers both decide to place the same job_id. If no healthy printer is available, the claim is released so a retry can place the job later. Claiming after placement would let a crash between “placed on printer A” and “recorded as assigned” produce a second placement on the next attempt.

Do I need work-stealing if I already route to the shortest queue? Only if you pre-assign jobs to printer queues eagerly. If printers pull their next job from a shared pool when ready, idle printers steal work for free and no rebalancer is needed. Work-stealing matters when a job is committed to one printer that then jams — late binding avoids the problem, and on-site print failover handles the harder case where the printer dies with jobs in flight.

Related pages

Related Clearing a Stuck Badge Print Queue at Peak Check-In