Pruner

Pruner#

Oban Pro replaces the standard pruner with a rules-based pruner that gives fine-grained control over how jobs are retained. Where the standard pruner treats every job the same, the Pro pruner lets you define rules that prune specific queues, workers, or states differently, run on a cron schedule, and archive jobs rather than deleting them.

The Pro pruner is active as soon as oban_pro is imported. Without any additional configuration, it keeps a single implicit rule, the "default", that retains completed, cancelled, and discarded jobs for one day, the same as the standard pruner.

Pruning Rules#

A rule describes which jobs to prune and how long to retain them. Each rule has:

  • A name, which acts as a unique identifier

  • A retention limit, either max_len or max_age

  • An optional match on queue, worker, or state

Rules are provided as a list through the rules option:

[[pruner.rules]]
name = "events"
queue = "events"
max_age = "10 minutes"

[[pruner.rules]]
name = "discarded"
state = "discarded"
max_age = "7 days"

Or programmatically when running embedded:

oban = Oban(
    pool=pool,
    pruner={
        "rules": [
            {"name": "events", "queue": "events", "max_age": "10 minutes"},
            {"name": "discarded", "state": "discarded", "max_age": "7 days"},
        ]
    },
)

Retention Limits#

A rule’s retention limit determines how jobs are retained, set with either max_len (retain by count) or max_age (retain by age). Exactly one is required.

Setting max_len keeps the most recent matching jobs. For example, to keep only the most recent 10,000 jobs from the events queue:

{"name": "events", "queue": "events", "max_len": 10_000}

Setting max_age keeps matching jobs younger than the given age. For example, to retain jobs from a less active media queue for two days:

{"name": "media", "queue": "media", "max_age": "2 days"}

Ages are periods like "30 seconds", "1 hour", or "2 weeks", using second, minute, hour, day, week, or month (always 30 days), in singular or plural form. A plain number of seconds or a timedelta works as well.

With either limit, use "infinity" to retain jobs indefinitely rather than setting an arbitrarily high value:

{"name": "audit", "worker": "myapp.workers.AuditWorker", "max_len": "infinity"}

Worker matches use the fully qualified worker name, and you can pass the worker class itself, e.g. "worker": AuditWorker.

Precedence#

Rules are evaluated in order, and each rule excludes the jobs already matched by the rules before it. In other words, a job is pruned or retained based on the first rule that matches it, so order rules from most to least specific.

Rules with identical matches are the exception. They don’t exclude each other, so a job is pruned when any of them would prune it. For example, to retain at most 1,000 jobs from the last day:

({"name": "events-len", "queue": "events", "max_len": 1_000},)
({"name": "events-age", "queue": "events", "max_age": "1 day"},)

The implicit "default" rule always runs last, catching anything no earlier rule matched. To change the default, either set a top-level max_age, as with the standard pruner, or include a rule named "default":

[pruner]
max_age = "1 hour"  # Keep unmatched jobs for an hour

Indexing Worker Matches#

Rules that match on worker can’t use Oban’s standard indexes. If you prune a high volume of jobs with worker matches, add a compound index so pruning doesn’t fall back to sequential scans:

CREATE INDEX CONCURRENTLY IF NOT EXISTS oban_jobs_worker_state_id_index
ON oban_jobs (worker, state, id);

Managing Rules at Runtime#

Rules are persisted in the database, so you can add, update, or remove them while the system is running rather than only through a config change. Rules can also be managed from Oban Web.

By default, rules that aren’t in the configuration are deleted on the next start. To keep rules added at runtime across restarts, use the "manual" sync mode.

The oban_pro.pruner module provides functions to manage rules:

  • all() — list every rule, ordered by position

  • get(name) — fetch a single rule by name

  • insert(**opts) — add a rule, or replace one with a matching name

  • update(name, **opts) — change an existing rule’s options

  • delete(name) — remove a rule (the "default" can’t be deleted, only updated)

Rules are returned as Rule instances. For example, to start pruning a new queue without a deploy:

from oban_pro import pruner

await pruner.insert(name="media", queue="media", max_age="1 week")

Changes take effect on the next pruning pass.

Rules accept the same options as configured rules. Without a position, an inserted rule is appended after the existing rules. With a position, it’s placed at that spot and the remaining rules are renumbered:

await pruner.insert(name="urgent", queue="urgent", max_len=100, position=0)

Be aware that insert() replaces a rule with a matching name, so prefer update() for targeted changes. Options that update() doesn’t receive keep their current value:

await pruner.update("media", max_age="2 days")

Both update() and delete() accept a lock_version to guard against overwriting a rule that changed since it was read. A mismatch raises StaleRuleError:

rule = await pruner.get("media")

await pruner.update("media", lock_version=rule.lock_version, max_len=500)

get(), update(), and delete() return None when no rule matches the name.

Pausing#

To stop a rule from pruning without deleting it, pause it. A paused rule is skipped entirely and has no effect on later rules, so its matching jobs fall through to the rules behind it:

await pruner.update("media", paused=True)

Resume it the same way:

await pruner.update("media", paused=False)

Sync Mode#

On start, configured rules are reconciled against the rules already in the database. The sync_mode option controls what happens to persisted rules that aren’t in the configuration:

  • "automatic" (the default) — persisted rules absent from the configuration are deleted, making the configuration the source of truth. The "default" rule is always preserved.

  • "manual" — configured rules are inserted or updated, and any other persisted rules are left in place. This preserves rules added at runtime, e.g. from Oban Web.

[pruner]
sync_mode = "manual"

In a cluster shared with Elixir nodes, or where rules are managed from Oban Web, use "manual" so that starting a Python node doesn’t delete rules it didn’t configure.

In manual mode, a configured rule can be removed by marking it for deletion:

[[pruner.rules]]
name = "media"
delete = true

What Configuration Overwrites#

When a configured rule changes, its options (match, retention, archive, limit, and timeout) are reapplied on the next start in either mode, replacing any runtime changes to that rule.

Pausing is never overwritten. A rule paused at runtime stays paused across restarts even when it’s also listed in config, because pausing is an operational decision.

Ordering depends on the mode:

  • "automatic" — positions are rewritten from the configuration on every start, matching the order rules are listed in.

  • "manual" — a newly configured rule is appended after the rules already persisted. After that, ordering belongs to whatever changed it last, so rules rearranged at runtime keep their order across restarts.

In both modes the "default" rule stays last.

Keeping Up With Inserts#

By default each rule deletes at most 10,000 jobs per pruning pass, within a 60 second statement timeout. The limit prevents connection timeouts and excessive table locks, but a busy system can easily insert more than 10,000 jobs per minute. If jobs accumulate despite active pruning, raise the limit.

Set at the top level, limit and timeout act as defaults for any rule that doesn’t set its own. Here we raise the limit to 25,000 and allow 90 seconds per statement:

[pruner]
limit = 25_000
timeout = 90

Setting a Schedule#

By default, pruning happens at the top of every minute based on the cron schedule * * * * *. Set any cron schedule for greater control over when pruning happens. For example, to prune once an hour instead:

[pruner]
schedule = "0 * * * *"

Schedules are evaluated in the timezone, which defaults to "Etc/UTC". To prune once a day at midnight in your local timezone:

[pruner]
limit = 100_000
schedule = "0 0 * * *"
timezone = "America/Chicago"

Pruning less frequently can reduce load, particularly with many rules. Be sure to set a higher limit to compensate for the accumulated jobs.

Note

The Pro pruner runs on a schedule rather than an interval, so the standard pruner’s interval option isn’t supported. Use schedule instead.

Archiving Jobs#

Some jobs are a historic record worth keeping past the point they’d normally be pruned. Rather than deleting matching jobs, a rule can archive them instead. Archived jobs are copied to the oban_jobs_archive table before they’re removed from oban_jobs, keeping the job’s full record (args, meta, errors, timestamps, etc.) for long-term storage and later inspection.

Set archive on any rule whose jobs you’d like to keep:

[[pruner.rules]]
name = "audit"
worker = "myapp.workers.AuditWorker"
max_age = "1 week"
archive = true

After one week, finished AuditWorker jobs are moved to the archive table rather than deleted.

Archiving runs in the same statement as deleting, so a job is never lost between the two tables. The archive is append-only, and managing its retention is up to you, e.g. with table partitioning or a scheduled cleanup job.

Workflow Preservation#

By default, jobs that are part of an active workflow are retained to prevent partial deletion of workflow data while the workflow is still running. A workflow is active while any of its jobs are suspended, scheduled, retryable, available, or executing.

The check includes sub-workflows and parent workflows, so related jobs across a workflow hierarchy are preserved until the entire workflow finishes.

To disable workflow preservation and prune all eligible jobs regardless of workflow status:

[pruner]
preserve_workflows = false

Disabling preservation can improve pruning performance for systems with many workflows, but may leave incomplete workflow histories when workflows are pruned while still running.

Implementation Notes#

  • Pruning is best-effort and performed out-of-band. All limits are soft, so jobs beyond a specified age may not be pruned immediately after they finish.

  • Pruning only applies to completed, cancelled, and discarded jobs. It never deletes a suspended, scheduled, available, executing, or retryable job.

  • Only the leader prunes, which prevents deadlocks between transactions on multiple nodes.

  • The pruner emits an oban.pruner.prune telemetry span with pruned_count and archived_count metadata.