BullMQ Tutorial: Queues, Workers, Retries and Scheduling

Some work doesn't belong inside a web request: sending emails, generating PDFs, calling slow AI models, processing uploads, syncing with other APIs. Put it on a queue, return to the user straight away, and let a worker process jobs in the background. (Your app needs background jobs)

BullMQ is the most widely used job queue for Node.js. It stores jobs in Redis. (Redis: when a small app needs it)

Setup

npm install bullmq ioredis

// queue.ts
import { Queue } from 'bullmq'
import IORedis from 'ioredis'

export const connection = new IORedis(process.env.REDIS_URL!, { maxRetriesPerRequest: null })

export const emailQueue = new Queue('emails', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: { count: 1000 },
    removeOnFail: { count: 5000 },
  },
})

maxRetriesPerRequest: null is required for workers' blocking connections.

Adding jobs

// in your route handler
await emailQueue.add('welcome', { userId: user.id }, { jobId: `welcome:${user.id}` })
res.status(201).json(user)

The name ('welcome') lets one queue hold several job types.
Data should be small and serialisable — pass IDs, not whole objects. The worker loads fresh data.
A custom jobId makes adding idempotent: adding the same ID twice doesn't create a duplicate while the first still exists. (Idempotency keys)

The worker

Run workers as a separate process from your web server:

// worker.ts
import { Worker } from 'bullmq'
import { connection } from './queue'

const worker = new Worker(
  'emails',
  async job => {
    if (job.name === 'welcome') {
      const user = await db.user.findUnique({ where: { id: job.data.userId } })
      if (!user) return                       // nothing to do
      await sendWelcomeEmail(user)
    }
  },
  { connection, concurrency: 5 },
)

worker.on('failed', (job, err) => console.error(`Job ${job?.id} failed:`, err.message))

process.on('SIGTERM', async () => {
  await worker.close()                        // finish current jobs, then exit
  process.exit(0)
})

Keep it running with systemd or PM2. (systemd service file, Graceful shutdown)

Retries and failures

With attempts: 5 and exponential backoff, a failing job is retried after roughly 2s, 4s, 8s, 16s. Throwing an error marks an attempt as failed. (Exponential backoff)

For errors that will never succeed (invalid data), throw an UnrecoverableError to skip remaining retries.

Make jobs safe to run twice. A worker can crash after doing the work but before marking it complete, so the job runs again. Check "already sent?" before sending, or use idempotency keys with external APIs.

Delayed and scheduled jobs

// run in 24 hours
await emailQueue.add('trial-ending', { userId }, { delay: 24 * 60 * 60 * 1000 })

// repeat: every day at 08:00
await emailQueue.upsertJobScheduler('daily-digest', { pattern: '0 8 * * *' }, { name: 'digest' })

(Cron expressions explained)

Concurrency and rate limits

concurrency: 5 — up to five jobs at once per worker process.
Run more worker processes to scale horizontally.
A queue-level limiter caps throughput, useful for APIs with rate limits:

new Worker('ai-tasks', processor, { connection, limiter: { max: 50, duration: 60_000 } })

Progress and results

await job.updateProgress(40)
return { pdfUrl }      // stored as the job's return value

Your app can poll a job's state or listen to QueueEvents to update the UI.

Monitoring

Use a dashboard such as Bull Board (open source) to see waiting, active, failed and completed jobs, retry failures, and inspect errors. Alert when the failed count or queue length grows. (Know when your app is down)

Redis setup that matters

Set Redis maxmemory-policy to noeviction — evicting keys would silently lose jobs.
Enable persistence (AOF) if losing queued jobs on a Redis restart is unacceptable.
Keep Redis private, never exposed to the internet. (Docker Compose networking)

Do you even need Redis?

If you already use Postgres and your volume is modest, a Postgres-backed queue (pg-boss, Graphile Worker, or your own SKIP LOCKED table) avoids running Redis, and jobs can be created in the same transaction as your data. (Postgres SKIP LOCKED queue, Message queues explained)

EasySpawn runs your web process and BullMQ workers side by side on one server, with Redis or Postgres next to them and restarts handled — so background jobs keep going when you're not watching. See how it works or join the waitlist.

BullMQ Tutorial: Background Jobs in Node.js With Redis