Background jobs
BullMQ on the Redis you already run, drained by a separate process.
Anything that can be slow, can fail, or should retry belongs off the request
path. packages/jobs is the queue layer and apps/worker is the process that
drains it.
Three queues ship:
| Queue | Registered in | What it does |
|---|---|---|
welcome | packages/jobs/src/jobs/welcome.ts | Sends the welcome email after sign-up |
webhook-delivery | packages/jobs/src/jobs/webhook-delivery.ts | Delivers one outbound webhook, with retries |
process-project | packages/jobs/src/jobs/process-project.ts | The worked example to copy |
The worker is a separate process
bun run --filter=@workspace/worker dev # localIt registers every worker, and it listens for exhausted retries:
worker.on("failed", (job, error) => {
if (job && job.attemptsMade < (job.opts.attempts ?? 1)) return // still retrying
sentryCapture(error, { queue: worker.name, jobId: job?.id })
})The guard matters. Without it every intermediate retry pages you, and the alert that means "this is never going to succeed" is buried among the ones that mean "it will work in nine seconds".
Deploy the worker as its own supervised process. Nothing user-facing breaks when it dies: pages render, the API answers, and every welcome email, webhook delivery and export silently stops. Monitor the process, not the website.
Redis, configured twice on purpose
The cache and the queues share one Redis server and disagree about one option:
// packages/jobs/src/queue.ts
function connection(): ConnectionOptions {
const cacheClient = getRedis()
return {
...cacheClient.options,
maxRetriesPerRequest: null,
// The cache client sets enableOfflineQueue:false to fail fast on request
// paths; BullMQ's blocking commands need it back on.
enableOfflineQueue: true,
}
}The cache wants to fail immediately, because a page waiting on a dead Redis is worse than a page rendering without a cache. A queue wants the opposite: a job enqueued during a blip should wait rather than vanish.
Know what that combination does to a producer. With maxRetriesPerRequest: null
and the offline queue on, an await queue.add(...) issued while Redis is
unreachable is neither sent nor rejected. It waits for a reconnection that may
never come, and the catch block around it never runs.
If you enqueue from a request handler, put a deadline on it yourself, or pass
enableOfflineQueue: false on that specific connection so the add rejects while
the socket is unwritable. A Promise.race against a timeout is enough.
BullMQ does not require maxRetriesPerRequest: null on every connection; it
forces the value only on blocking ones. Read out of the installed bullmq
5.76.4: queue-base.js defaults hasBlockingConnection to false and passes
it through as blocking, redis-connection.js gates the override on that flag,
and both Queue and Worker call super() without setting it. The Worker's
blocking client is a separate connection built with blocking: true, so the
split is per connection rather than per class.
The null on a producer is therefore our choice, and the warning above is its consequence.
Adding a queue
Write the job module in packages/jobs/src/jobs/, exporting an enqueue*
function and a register*Worker function. Copy process-project.ts, which
exists for this.
Register the worker in apps/worker/src/index.ts. The array at the top is the
only wiring; the failure handler applies to everything in it.
Enqueue from wherever the work is triggered. Handlers return once the job is queued, not once it has run.
Outbound webhooks ride the same queue
dispatchEvent(userId, event, payload) writes one webhook_deliveries row per
enabled endpoint subscribed to that event, then enqueues a delivery job per
row. The job carries only the row id, not the payload, so a manual replay
mutates the same row instead of creating a second history.
Deliveries are signed to the Standard Webhooks spec, give up after 5 attempts, and time out after 10 seconds per request so one hung receiver cannot hold a worker slot. Endpoints, their event subscriptions and a replay button live in the admin app.