Background jobs that survive a second attempt

Give the operation a durable identity

Separate the identity of the business operation from the identity of a worker attempt. A report for account A and reporting period B should keep the same operation key across retries. Store the request and its state in your database, and put the record ID in the queue payload.

Use explicit states such as requested, generating, ready, and failed. Define who can move between them. A retry that discovers a completed artifact can return that artifact instead of generating a second one. Keep payloads small and exclude credentials or unnecessary personal data from job logs.

A useful review exercise is to draw a vertical line after every side effect in the worker. Imagine the process stopping at each line. If the next attempt cannot determine what happened, the state model needs another checkpoint or a reconciliation step.

Protect the effect, not just the queue entry

An idempotent operation reaches the same intended final state when it is repeated. For database-only work, a unique operation key and a transaction can keep the result and its completion marker consistent. An early already processed check on its own is insufficient: two workers can both pass that check before either writes.

External APIs create a different boundary. When a provider supports idempotency keys, use a stable key for the business operation. If it does not, plan how to look up the provider's result and reconcile an uncertain outcome. A local completed flag cannot make an external request atomic with your database update.

Keep jobs small enough that their retry behavior is understandable. Generating a file and delivering a notification can be separate steps with separate completion records. Splitting work helps diagnosis, but each step still needs its own duplicate-handling strategy.

Reference: BullMQ: idempotent jobs ↗

Make retries bounded and deliberate

Retry a temporary outage differently from invalid input. In BullMQ, attempts controls the total number of attempts, and a backoff policy introduces time between failed attempts. Exponential backoff with jitter can spread retry traffic instead of having every worker retry at the same instant.

This illustrative queue configuration allows four attempts in total. It assumes an existing queue with its Redis connection configured. Retaining failed jobs makes investigation possible; choose retention values for your workload rather than copying them into production unchanged.

await reportQueue.add(
  "generate-report",
  { reportRequestId: request.id },
  {
    attempts: 4,
    backoff: {
      type: "exponential",
      delay: 2_000,
      jitter: 0.5,
    },
    removeOnComplete: { count: 500 },
    removeOnFail: false,
  },
);

Reference: BullMQ: retrying failing jobs ↗

Close the gap between saving and enqueueing

Saving a report request and then publishing a queue message are two writes. If the database commit succeeds and publishing fails, the request exists but no worker knows about it. Reversing the order creates a different failure: a worker can receive a message for data that never committed.

A transactional outbox records the request and an event in the same database transaction. A separate dispatcher reads committed outbox records and publishes jobs. The dispatcher can still publish twice if it crashes before marking an event as delivered, so consumers must remain idempotent. The outbox addresses the handoff gap; it does not remove the need to handle duplicates.

Reference: AWS: transactional outbox pattern ↗

Test the recovery path before adding capacity

A happy-path test proves that one worker can finish one job. A recovery test proves that the product remains correct when execution is interrupted. Make these cases part of the workflow's acceptance criteria:

• Run the same operation twice and verify the same durable result.
• Run two attempts concurrently and verify that only one effect wins.
• Stop a worker after an external call but before the completion write.
• Temporarily interrupt queue publishing and verify outbox recovery.
• Exhaust retries, inspect the failure, then replay the operation safely.