BadgeprintingEvent registration · badge automation

Clearing a Stuck Badge Print Queue at Peak Check-In

Symptom Statement

It is 08:00, the doors just opened, and the badge print queue is filling faster than it drains. The queue depth on your dashboard climbs past 200, then 400, and the printers have gone quiet — no paper is moving even though jobs are clearly waiting. The check-in lanes back up, staff start hand-writing name tags, and the on-call phone lights up. This is the peak check-in stall: a queue that accepts work but stops producing badges, so wait time grows without bound until someone intervenes. This page addresses that exact symptom for the Redis-backed job queue that feeds your printers, and it is the recovery runbook for the Print Queue Orchestration stage, which owns the ordering, acknowledgment, and retry contract for on-site badge jobs. The tells are consistent: the head of the queue never advances, worker active counts sit pinned or drop to zero, one job ID keeps reappearing in the error log, and the depth gauge is monotonic — it only goes up. The goal here is narrow and time-boxed: drain the stall in under fifteen minutes without losing a single attendee’s badge and without printing anyone’s badge twice.

Unblocking a stuck badge print queue by parking the poison job at the head A Redis list holds badge jobs in FIFO order. A poison job sits at the head and cannot be acknowledged, so every job behind it — job 2, job 3, job 4 — is blocked and the queue depth climbs while attendees wait. The recovery pops the head atomically, inspects it, and routes it down a dashed branch to a dead-letter queue keyed by its job ID. The remaining jobs then drain to the printer pool along a solid branch. A verification step confirms the poison job appears exactly once in the dead-letter queue and never on paper. Head-of-line block: park the poison job, then the rest drains head (blocked) poison job unacked, retrying forever job 2 job 3 job 4 … blocked behind head — depth climbs inspect + park RPOPLPUSH → DLQ key: job_id dead-letter queue one entry · replay later now drains printer pool healthy, draining verify: poison in DLQ exactly once depth falling · no duplicate badge on paper LLEN print:queue trends down

Root Cause Analysis

A queue that accepts jobs but stops draining is almost always one of four failures. They present identically on the depth gauge, so triage means finding which before you touch anything — the wrong containment for the wrong cause loses badges.

  • A poison job at the head. One malformed job — a record with a missing attendee_id, an unresolvable template reference, or a corrupt glyph payload — throws on every dequeue attempt. With ordered delivery and automatic retry, the worker picks it up, crashes or nacks, and it returns to the head. Nothing behind it ever runs. This is head-of-line blocking and it is the single most common peak-time stall.
  • Jobs routed to an offline printer. A printer jams, loses power, or drops off the network, but the dispatcher keeps assigning its share of jobs. Those jobs sit reserved, never acknowledged, and the lane they feed goes silent while the queue behind them grows. The queue is not stuck globally — it is stuck for one destination that no longer exists.
  • Worker prefetch hoarding. Workers with a high worker_prefetch_multiplier reserve a large batch of jobs into local memory at connection time. If those workers then stall or die, their reserved jobs are invisible and undrained until the broker’s visibility timeout expires — the queue looks deep, but the jobs are hoarded off-broker, not waiting to be dispatched.
  • Spooler backpressure not propagated. The CUPS/IPP spooler ahead of the physical printer fills its own buffer and starts rejecting or slow-accepting new jobs. If the worker treats a slow spooler accept as success, it keeps pulling from Redis and pushing into a spooler that cannot absorb it; if it treats it as failure, it nacks and re-heads the job. Either way the Redis depth stops falling because the real bottleneck is downstream and the signal never reached the queue.

Symptom-to-Resolution Matrix

Poison Job at the Head of the Queue

Symptoms

  • Queue depth (LLEN print:queue) climbs and never falls; the head element never changes between polls.
  • The same job_id reappears in the worker error log every few seconds with the same stack trace.

Root cause. A single un-processable job throws on every dequeue and, because delivery is ordered with retry, returns to the head — blocking every job behind it indefinitely.

Fix

  1. Identify the head without consuming it: redis-cli LINDEX print:queue -1 (or 0, depending on which end you push to) and read the offending job_id.
  2. Pop the head atomically and move it aside with RPOPLPUSH print:queue print:dlq so it is parked, not lost — the parked copy lives in the dead-letter queue for later replay.
  3. Confirm the queue resumes draining (depth falls) and the parked job_id no longer reappears in the error log.
  4. Reprocess the parked job out-of-band once the underlying data is fixed, so the attendee still gets a badge.

Jobs Routed to an Offline Printer

Symptoms

  • One check-in lane is silent while others print; depth is concentrated on one destination’s sub-queue.
  • Jobs for that printer sit reserved/unacked and never complete.

Root cause. The dispatcher is still assigning work to a printer that is jammed, powered off, or off-network, and those jobs cannot be acknowledged.

Fix

  1. Mark the printer unhealthy so no new jobs route to it, then requeue its stranded jobs onto the shared pool.
  2. Let a healthy printer pick them up — this is exactly what load-balancing badge jobs across multiple on-site printers automates, and what on-site print failover does when a printer dies mid-rush.
  3. Physically clear the printer, and only re-add it to the healthy set once a probe print succeeds.

Worker Prefetch Hoarding

Symptoms

  • celery inspect reserved shows a large batch held by a worker that is doing no useful work.
  • Redis depth looks high but active throughput is near zero.

Root cause. A high prefetch multiplier let workers reserve jobs into memory; a stalled worker then hoards them off-broker until the visibility timeout expires.

Fix

  1. Set worker_prefetch_multiplier=1 and task_acks_late=True so a job is reserved only while actually being printed and is redelivered on worker death.
  2. Restart the stalled worker to force its reserved batch back onto the broker immediately rather than waiting out the visibility timeout.
  3. Cap concurrency to the number of physical printers a worker feeds — reserving more jobs than you can print only hides them.

Unpropagated Spooler Backpressure

Symptoms

  • Redis depth plateaus (stops falling) while the CUPS/IPP spooler queue is full.
  • Workers log slow or timing-out submit calls to the print server.

Root cause. The spooler downstream of the worker is saturated, but its backpressure never reaches the Redis dequeue loop, so the worker keeps pulling into a buffer that cannot absorb more.

Fix

  1. Gate the dequeue on real spooler capacity: check the destination’s pending count before popping the next job, and pause when it exceeds a ceiling.
  2. Treat a spooler timeout as a retry-in-place with backoff, not an immediate nack that re-heads the job and starves everything behind it.
  3. Alert on spooler depth as a first-class signal so the stall is visible upstream before Redis depth flatlines.

Minimal Working Implementation

A self-contained recovery routine for the most common cause: a poison job wedged at the head of a Redis list. It inspects the head non-destructively, parks a genuinely poisoned job into a dead-letter list keyed so it can be replayed exactly once, requeues transient failures, and includes a verification block that proves the poison job leaves the live queue and lands in the DLQ exactly once — never on paper twice.

PYTHON
import json
import os
import redis

REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
QUEUE_KEY = "print:queue"          # RPUSH to enqueue, workers BRPOPLPUSH from the head
DLQ_KEY = "print:dlq"              # parked poison jobs, replayed out-of-band
DLQ_SEEN_KEY = "print:dlq:seen"    # SET of job_ids already parked — dedupe guard
MAX_ATTEMPTS = 3

r = redis.Redis.from_url(REDIS_URL, decode_responses=True)


def is_poison(job: dict) -> bool:
    """A job we can never print: missing identity or unresolvable template."""
    if not job.get("attendee_id"):
        return True
    if not job.get("template_id"):
        return True
    if job.get("attempts", 0) >= MAX_ATTEMPTS:
        return True
    return False


def park_job(raw: str, job: dict) -> bool:
    """Move a poison job to the DLQ exactly once. Returns True if newly parked."""
    job_id = job["job_id"]
    # SADD returns 1 only the first time this job_id is added — idempotent park.
    if r.sadd(DLQ_SEEN_KEY, job_id) == 1:
        r.rpush(DLQ_KEY, raw)
        return True
    return False


def unblock_head() -> dict:
    """Pop the current head, park it if poisoned, otherwise requeue it.

    Returns a summary of the action taken. Safe to run repeatedly: a poison
    job is parked once, a transient job is bumped back to the tail so the
    rest of the queue can drain past it.
    """
    raw = r.lpop(QUEUE_KEY)  # workers consume the head; LPOP mirrors that end
    if raw is None:
        return {"action": "empty", "job_id": None}

    job = json.loads(raw)
    job_id = job["job_id"]

    if is_poison(job):
        parked = park_job(raw, job)
        return {"action": "parked" if parked else "already_parked", "job_id": job_id}

    # Transient failure: increment attempts and requeue at the tail, not the head,
    # so it can retry later without blocking the jobs behind it.
    job["attempts"] = job.get("attempts", 0) + 1
    r.rpush(QUEUE_KEY, json.dumps(job))
    return {"action": "requeued", "job_id": job_id, "attempts": job["attempts"]}


if __name__ == "__main__":
    # Arrange: a poison job (no attendee_id) wedged ahead of two good jobs.
    r.delete(QUEUE_KEY, DLQ_KEY, DLQ_SEEN_KEY)
    poison = {"job_id": "job-poison", "attendee_id": None, "template_id": "vip"}
    good1 = {"job_id": "job-1", "attendee_id": "a-1", "template_id": "vip"}
    good2 = {"job_id": "job-2", "attendee_id": "a-2", "template_id": "vip"}
    for j in (poison, good1, good2):
        r.rpush(QUEUE_KEY, json.dumps(j))

    # Act: clear the head, then confirm the good jobs are still queued.
    first = unblock_head()
    assert first["action"] == "parked", first
    assert first["job_id"] == "job-poison"

    # The poison job is parked exactly once even if recovery runs again.
    r.lpush(QUEUE_KEY, json.dumps(poison))          # simulate a re-head
    second = unblock_head()
    assert second["action"] == "already_parked", second
    assert r.llen(DLQ_KEY) == 1                      # never duplicated
    assert r.scard(DLQ_SEEN_KEY) == 1

    # The two good jobs remain and can now drain.
    remaining = {json.loads(x)["job_id"] for x in r.lrange(QUEUE_KEY, 0, -1)}
    assert remaining == {"job-1", "job-2"}, remaining
    print("OK: poison parked once, queue drains,", r.llen(QUEUE_KEY), "jobs left")

The verification block is the fix’s own regression test: parking is keyed on job_id through a Redis SET, so re-running recovery on a job that re-heads can only re-observe the existing DLQ entry — it can never park a second copy, which is what guarantees the parked attendee is reprinted once, not twice.

Memory & Performance Constraints

At doors-open the queue is write-heavy and latency-sensitive; the failure modes are memory growth and blocking pops, not compute.

Component Constraint Mitigation
Redis queue depth An un-draining list grows unbounded in RAM during a surge Alert on LLEN print:queue slope, not just absolute depth; cap enqueue and shed to a holding list past a ceiling
Poison retry loop A re-heading job spins the worker at full CPU with no output Cap attempts and park on the cap; never retry a structurally invalid job in place
Worker prefetch High worker_prefetch_multiplier hoards jobs into worker memory off-broker Set multiplier to 1 with acks_late=True so depth reflects real, dispatchable backlog
DLQ dedupe set print:dlq:seen grows one entry per parked job across the event Give it a TTL past the event window, or clear it during teardown; it is bounded by unique failures, not traffic
Blocking pop BRPOPLPUSH with no timeout can pin a worker if the queue empties Use a bounded block timeout so idle workers stay responsive to health checks

Incident Triage & Rollback

Fast path when the depth gauge alarms at peak. Target under fifteen minutes to containment; every step before rollback is non-destructive.

  1. Confirm the stall is real, not lag. redis-cli LLEN print:queue twice, ten seconds apart. Rising with no fall is a stall; a steady high number that ticks down is just surge lag — do not intervene.
  2. Read the head. redis-cli LINDEX print:queue -1 and note the job_id. Cross-check it against the worker error log: a repeating job_id with a repeating stack trace is the poison signature.
  3. Check for hoarding and offline destinations. celery -A printing inspect reserved (large idle batch = prefetch hoarding) and celery -A printing inspect active. If one destination is silent, it is the offline-printer case — hand it to on-site print failover.
  4. Park the poison job. Run the recovery routine, or by hand: redis-cli RPOPLPUSH print:queue print:dlq. Watch depth begin to fall.

Rollback. If parking the wrong job, replay it immediately from the DLQ back onto the queue tail: redis-cli RPOPLPUSH print:dlq print:queue and delete its dedupe marker with redis-cli SREM print:dlq:seen <job_id>. Because every downstream print is idempotent on job_id, replaying a parked job can only re-attempt an unprinted badge, never duplicate a printed one. If the stall is worker-side, celery -A printing control shutdown and restart with worker_prefetch_multiplier=1 to force reserved jobs back onto the broker.

Post-rollback validation. Confirm the queue is draining and nothing is stranded in the DLQ that should have printed:

BASH
redis-cli LLEN print:queue     # expect a falling number
redis-cli LLEN print:dlq       # expect only genuinely un-processable jobs
redis-cli SCARD print:dlq:seen # expect == LLEN print:dlq (no orphaned markers)

Frequently Asked Questions

How do I tell a poison job apart from a queue that is just backed up under load? Watch the head, not the depth. Under honest load the head element changes every poll and depth ticks down between surges; with a poison job the head never advances and the same job_id repeats in the error log. Check LINDEX print:queue -1 twice — if the head is identical and depth only rises, you have head-of-line blocking, not load.

Is it safe to just delete the stuck job to get things moving? No — deleting loses that attendee’s badge with no record. Park it instead with RPOPLPUSH into the dead-letter queue so it is preserved and can be reprinted once the underlying data is fixed. Deletion also leaves you blind to why it was poison, so the next identical job stalls you again.

Will parking and replaying a job print someone’s badge twice? Not if every downstream print is idempotent on job_id, which it must be. Recovery parks on a Redis SET keyed by job_id, and the printer worker dedupes on the same key, so a replay can only complete an unprinted job — the same guarantee that lets on-site print failover re-dispatch in-flight jobs safely.

Related pages

Related Load-Balancing Badge Jobs Across Multiple On-Site Printers