Every application of any size ends up with work that happens outside a request: sending email, generating reports, processing uploads, calling a slow third party. It is the least-observed part of most systems, because nobody is waiting for it and nothing obviously breaks when it fails. That combination is exactly why it is where quiet, expensive bugs accumulate.
The first decision is what to put in the job payload, and the common mistake is to serialise the whole object. Pass an identifier and let the worker fetch current state. A job that carries a stale copy of a record will happily act on data that changed while it sat in the queue - sending a notification about a cancelled order, or overwriting a newer value with an older one.
Retries need designing rather than enabling. Exponential backoff with jitter, so a failing dependency is not hammered by every worker simultaneously the moment it starts recovering. A maximum attempt count. And a dead-letter queue for jobs that will never succeed - without one, a permanently failing job either retries forever, consuming capacity, or vanishes silently. Someone must be alerted when the dead-letter queue is non-empty, or it becomes a place where failures go to be forgotten.
Idempotency is not optional, because at-least-once delivery means a job can and will run twice - after a worker crashes mid-execution, after a network partition, after a manual retry during an incident. Every job that has an effect on the world needs to be safe to run again: check whether the email was already sent, whether the payment was already captured, before doing it. Teams that treat double execution as unlikely find out through duplicate charges.
Separate your queues by urgency and by risk. A single queue means a batch of ten thousand report generations delays the password reset email behind it, and one poisonous job type can starve everything else. Separate queues with separate workers give you isolation and let you scale the expensive work independently of the urgent work.
Then monitor the things that only jobs have: queue depth, oldest message age, and failure rate per job type. Oldest message age is the most useful single number - a queue that is deep but draining is healthy, while a queue with a message from four hours ago is broken regardless of its depth. And alert on jobs that stopped running altogether, because a scheduled job that silently fails to fire produces no errors at all, and that is the failure people discover weeks later.
