Redis as a queue: when it works and when it loses jobs

A Redis list is a workable job queue until a worker crashes, Redis restarts, or memory fills. Each of those loses jobs under the default settings, and each has a specific fix.

Originally written on 12 February 2018. Migrated from a WordPress blog and reformatted.

Redis lists provide a queue with two commands: LPUSH to enqueue and BRPOP to dequeue with blocking. Laravel, Sidekiq, and RQ all build on this. It is fast, simple, and already running. The question is what happens to a job when something fails, and the answer depends on configuration that defaults to the wrong values for a queue.

The basic pattern

producer:   LPUSH jobs '{"id": 41, "type": "email", ...}'
worker:     BRPOP jobs 0        # blocks until a job is available

BRPOP removes the job from the list and returns it. From that moment the job exists only in the worker's memory. If the worker crashes before finishing, the job is gone.

Failure mode 1: the worker dies mid-job

Use BRPOPLPUSH instead. It atomically pops from the queue and pushes onto a second list, conventionally one per worker:

worker:     BRPOPLPUSH jobs processing:worker-3 0
            ... do the work ...
            LREM processing:worker-3 1 '<the job>'

The job is now in processing:worker-3 until the worker explicitly removes it on completion. A supervisor process can scan processing:* lists for jobs older than a threshold and push them back onto jobs. This is the reliable queue pattern from the Redis documentation, and most frameworks implement it. Check that yours does; some default to plain BRPOP.

Failure mode 2: Redis restarts

By default, Redis persists with RDB snapshots at intervals of minutes. A restart loses every job enqueued since the last snapshot. For a queue, enable AOF:

appendonly yes
appendfsync everysec

Now a restart loses at most one second of enqueues. appendfsync always loses nothing and costs a disk sync per write, which is usually too slow; everysec is the standard trade.

Failure mode 3: memory pressure

The default maxmemory-policy on many installations is allkeys-lru, or noeviction without a maxmemory set. Under allkeys-lru, Redis evicts the least recently used keys when memory is full, and a queue list that has not been read recently is a candidate. Jobs disappear with no error.

If the instance is shared with a cache, set:

maxmemory-policy volatile-lru

Only keys with a TTL are then eligible for eviction. Queue keys must never carry a TTL. Better still, run the queue on a separate instance with noeviction, so that a full queue fails loudly at LPUSH time instead of silently later.

Failure mode 4: the producer cannot replay

Even with all of the above, a queue can lose a job during a failover to a replica that had not yet received the write. Whether that matters depends on the producer. A job that will be regenerated by tomorrow's batch is recoverable. A job representing a webhook that was already acknowledged to a third party is not. For the second kind, record the intent in the primary database before enqueueing, and reconcile from that table periodically.

When to use something else

Redis is a reasonable queue for background jobs that are cheap to lose or easy to replay. It is not a durable log: there is no consumer offset, no replay from an earlier point, and no ordering guarantee across multiple lists. When any of those are requirements, the answer is Kafka or a managed queue with acknowledgements, not a more careful Redis configuration.

Checklist

• BRPOPLPUSH with a processing list and a reaper, not bare BRPOP
• appendonly yes, appendfsync everysec
• volatile-lru on shared instances, noeviction on dedicated ones, no TTL on queue keys
• For jobs that cannot be regenerated, an intent record outside Redis