The Lifeline service uses producer records to periodically rescue orphaned jobs, i.e. jobs that are stuck in the executing state because the node was shut down before the job could finish. In addition, it performs the following maintenance tasks:

• Discard jobs left available with exhausted attempts due to rare edge cases
• Repair stuck workflows with deleted dependencies or missed scheduling events
• Repair stuck chains with deleted dependencies or missed scheduling events
• Repair jobs in partitioned queues that are missing a partition key
• Repair chunk jobs that are missing a computed chunk_id

Without Lifeline you'll need to manually rescue stuck jobs or perform maintenance.

Using the Plugin

To use Lifeline, configure it through the top-level :lifeline option in config.exs:

config :my_app, Oban,
  engine: Oban.Pro.Engine,
  lifeline: Oban.Pro.Lifeline
  ...

No configuration is necessary because the defaults are tuned for most systems. Each rescue query runs with a generous 45 second timeout. Both the rescue interval and query timeout accept millisecond values or an Oban.Period tuple:

lifeline: {Lifeline, rescue_interval: {1, :minute}, timeout: {45, :seconds}}

Automatic Repairs

The plugin automatically performs several repair operations during each rescue cycle to keep workflows, chains, partitions, and chunks healthy.

• Chain repair — Chain jobs waiting on deleted or stuck predecessors are released to continue processing.

• Chunk repair — Chunk jobs that are missing a chunk_id in their metadata have their chunk ID computed from the job's chunk_by configuration and set automatically. This can happen when chunk settings are changed after jobs are already enqueued.

• Partition repair — Jobs in partitioned queues that are missing a partition_key in their metadata (e.g., jobs scheduled before a queue was partitioned) have their partition key computed and set automatically.

• Workflow repair — Jobs held waiting for dependencies that were deleted or missed a scheduling event are released. This handles edge cases where workflow jobs get stuck due to incomplete dependency resolution.

Each repair operation processes up to repair_limit jobs per cycle, 1000 by default. Workflow repairs aren't subject to this limit, since they're naturally bounded by the set of stuck workflows.

Identifying Rescued Jobs

Rescued jobs can be identified by a rescued value in meta. Each rescue increments the rescued count by one.

Rescuing Exhausted Jobs

When a job's attempt matches its max_attempts its retries are considered "exhausted". Normally, the Lifeline plugin transitions exhausted jobs to the discarded state and they won't be retried again. It does this for a couple of reasons:

1. To ensure at-most-once semantics. Suppose a long-running job interacted with a non-idempotent service and was shut down while waiting for a reply; you may not want that job to retry.

2. To prevent infinitely crashing BEAM nodes. Poorly behaving jobs may crash the node (through NIFs, memory exhaustion, etc.) We don't want to repeatedly rescue and rerun a job that repeatedly crashes the entire node.

When exhausted jobs are discarded, the Oban.Pro.Worker.on_discarded/2 callback is called with an :exhausted reason, which is useful for error reporting or notifications.

Discarding exhausted jobs may not always be desired. Use the retry_exhausted option if you'd prefer to retry exhausted jobs when they are rescued, rather than discarding them:

lifeline: {Lifeline, retry_exhausted: true}

During rescues, with retry_exhausted: true, a job's max_attempts is incremented and it is moved back to the available state.

Retrying every exhausted job is often too broad. To retry selectively, pass a match spec with workers, queues, or both. Only exhausted jobs matching a listed worker or queue are retried, while the rest are still discarded:

lifeline: {Lifeline, retry_exhausted: [workers: [MyApp.SafeWorker], queues: [:safe]]}

Instrumenting with Telemetry

The Lifeline plugin adds the following metadata to the [:oban, :plugin, :stop] event:

• :rescued_jobs — a list of jobs transitioned back to available

• :discarded_jobs — a list of jobs transitioned to discarded

Note: jobs only include id, queue, and state fields.