# Pruner

Oban Pro replaces the standard pruner with a rules-based pruner that gives fine-grained control
over how jobs are retained. Where the standard pruner treats every job the same, the Pro pruner
lets you define rules that prune specific queues, workers, or states differently, run on a cron
schedule, and archive jobs rather than deleting them.

The Pro pruner is active as soon as `oban_pro` is imported. Without any additional configuration,
it keeps a single implicit rule, the `"default"`, that retains `completed`, `cancelled`, and
`discarded` jobs for one day, the same as the standard pruner.

## Pruning Rules

A rule describes which jobs to prune and how long to retain them. Each rule has:

* A `name`, which acts as a unique identifier
* A retention limit, either `max_len` or `max_age`
* An optional match on `queue`, `worker`, or `state`

Rules are provided as a list through the `rules` option:

```toml
[[pruner.rules]]
name = "events"
queue = "events"
max_age = "10 minutes"

[[pruner.rules]]
name = "discarded"
state = "discarded"
max_age = "7 days"
```

Or programmatically when running embedded:

```python
oban = Oban(
    pool=pool,
    pruner={
        "rules": [
            {"name": "events", "queue": "events", "max_age": "10 minutes"},
            {"name": "discarded", "state": "discarded", "max_age": "7 days"},
        ]
    },
)
```

### Retention Limits

A rule's retention limit determines how jobs are retained, set with either `max_len` (retain by
count) or `max_age` (retain by age). Exactly one is required.

Setting `max_len` keeps the most recent matching jobs. For example, to keep only the most recent
10,000 jobs from the `events` queue:

```python
{"name": "events", "queue": "events", "max_len": 10_000}
```

Setting `max_age` keeps matching jobs younger than the given age. For example, to retain jobs
from a less active `media` queue for two days:

```python
{"name": "media", "queue": "media", "max_age": "2 days"}
```

Ages are periods like `"30 seconds"`, `"1 hour"`, or `"2 weeks"`, using `second`, `minute`,
`hour`, `day`, `week`, or `month` (always 30 days), in singular or plural form. A plain number of
seconds or a `timedelta` works as well.

With either limit, use `"infinity"` to retain jobs indefinitely rather than setting an
arbitrarily high value:

```python
{"name": "audit", "worker": "myapp.workers.AuditWorker", "max_len": "infinity"}
```

Worker matches use the fully qualified worker name, and you can pass the worker class itself, e.g.
`"worker": AuditWorker`.

### Precedence

Rules are evaluated in order, and each rule excludes the jobs already matched by the rules before
it. In other words, a job is pruned or retained based on _the first rule_ that matches it, so
order rules from most to least specific.

Rules with identical matches are the exception. They don't exclude each other, so a job is pruned
when any of them would prune it. For example, to retain at most 1,000 jobs from the last day:

```python
({"name": "events-len", "queue": "events", "max_len": 1_000},)
({"name": "events-age", "queue": "events", "max_age": "1 day"},)
```

The implicit `"default"` rule always runs last, catching anything no earlier rule matched. To
change the default, either set a top-level `max_age`, as with the standard pruner, or include a
rule named `"default"`:

```toml
[pruner]
max_age = "1 hour"  # Keep unmatched jobs for an hour
```

### Indexing Worker Matches

Rules that match on `worker` can't use Oban's standard indexes. If you prune a high volume of jobs
with worker matches, add a compound index so pruning doesn't fall back to sequential scans:

```sql
CREATE INDEX CONCURRENTLY IF NOT EXISTS oban_jobs_worker_state_id_index
ON oban_jobs (worker, state, id);
```

## Managing Rules at Runtime

Rules are persisted in the database, so you can add, update, or remove them while the system is
running rather than only through a config change. Rules can also be managed from Oban Web.

By default, rules that aren't in the configuration are deleted on the next start. To keep rules
added at runtime across restarts, use the `"manual"` [sync mode](#sync-mode).

The `oban_pro.pruner` module provides functions to manage rules:

* `all()` — list every rule, ordered by position
* `get(name)` — fetch a single rule by name
* `insert(**opts)` — add a rule, or replace one with a matching name
* `update(name, **opts)` — change an existing rule's options
* `delete(name)` — remove a rule (the `"default"` can't be deleted, only updated)

Rules are returned as `Rule` instances. For example, to start pruning a new queue without a
deploy:

```python
from oban_pro import pruner

await pruner.insert(name="media", queue="media", max_age="1 week")
```

Changes take effect on the next pruning pass.

Rules accept the same options as configured rules. Without a `position`, an inserted rule is
appended after the existing rules. With a `position`, it's placed at that spot and the remaining
rules are renumbered:

```python
await pruner.insert(name="urgent", queue="urgent", max_len=100, position=0)
```

Be aware that `insert()` replaces a rule with a matching name, so prefer `update()` for targeted
changes. Options that `update()` doesn't receive keep their current value:

```python
await pruner.update("media", max_age="2 days")
```

Both `update()` and `delete()` accept a `lock_version` to guard against overwriting a rule that
changed since it was read. A mismatch raises `StaleRuleError`:

```python
rule = await pruner.get("media")

await pruner.update("media", lock_version=rule.lock_version, max_len=500)
```

`get()`, `update()`, and `delete()` return `None` when no rule matches the name.

### Pausing

To stop a rule from pruning without deleting it, pause it. A paused rule is skipped entirely and
has no effect on later rules, so its matching jobs fall through to the rules behind it:

```python
await pruner.update("media", paused=True)
```

Resume it the same way:

```python
await pruner.update("media", paused=False)
```

### Sync Mode

On start, configured rules are reconciled against the rules already in the database. The
`sync_mode` option controls what happens to persisted rules _that aren't in_ the configuration:

* `"automatic"` (the default) — persisted rules absent from the configuration are deleted, making
  the configuration the source of truth. The `"default"` rule is always preserved.

* `"manual"` — configured rules are inserted or updated, and any other persisted rules are left in
  place. This preserves rules added at runtime, e.g. from Oban Web.

```toml
[pruner]
sync_mode = "manual"
```

In a cluster shared with Elixir nodes, or where rules are managed from Oban Web, use `"manual"` so
that starting a Python node doesn't delete rules it didn't configure.

In manual mode, a configured rule can be removed by marking it for deletion:

```toml
[[pruner.rules]]
name = "media"
delete = true
```

#### What Configuration Overwrites

When a configured rule changes, its options (match, retention, `archive`, `limit`, and `timeout`)
are reapplied on the next start in either mode, replacing any runtime changes to that rule.

Pausing is never overwritten. A rule paused at runtime stays paused across restarts even when it's
also listed in config, because pausing is an operational decision.

Ordering depends on the mode:

* `"automatic"` — positions are rewritten from the configuration on every start, matching the
  order rules are listed in.

* `"manual"` — a newly configured rule is appended after the rules already persisted. After that,
  ordering belongs to whatever changed it last, so rules rearranged at runtime keep their order
  across restarts.

In both modes the `"default"` rule stays last.

## Keeping Up With Inserts

By default each rule deletes at most 10,000 jobs per pruning pass, within a 60 second statement
timeout. The limit prevents connection timeouts and excessive table locks, but a busy system can
easily insert more than 10,000 jobs per minute. If jobs accumulate despite active pruning, raise
the limit.

Set at the top level, `limit` and `timeout` act as defaults for any rule that doesn't set its
own. Here we raise the limit to 25,000 and allow 90 seconds per statement:

```toml
[pruner]
limit = 25_000
timeout = 90
```

## Setting a Schedule

By default, pruning happens at the top of every minute based on the cron schedule `* * * * *`.
Set any cron schedule for greater control over when pruning happens. For example, to prune once
an hour instead:

```toml
[pruner]
schedule = "0 * * * *"
```

Schedules are evaluated in the `timezone`, which defaults to `"Etc/UTC"`. To prune once a day at
midnight in your local timezone:

```toml
[pruner]
limit = 100_000
schedule = "0 0 * * *"
timezone = "America/Chicago"
```

Pruning less frequently can reduce load, particularly with many rules. Be sure to set a higher
limit to compensate for the accumulated jobs.

```{note}
The Pro pruner runs on a schedule rather than an interval, so the standard pruner's `interval`
option isn't supported. Use `schedule` instead.
```

## Archiving Jobs

Some jobs are a historic record worth keeping past the point they'd normally be pruned. Rather
than deleting matching jobs, a rule can _archive_ them instead. Archived jobs are copied to the
`oban_jobs_archive` table before they're removed from `oban_jobs`, keeping the job's full record
(args, meta, errors, timestamps, etc.) for long-term storage and later inspection.

Set `archive` on any rule whose jobs you'd like to keep:

```toml
[[pruner.rules]]
name = "audit"
worker = "myapp.workers.AuditWorker"
max_age = "1 week"
archive = true
```

After one week, finished `AuditWorker` jobs are moved to the archive table rather than deleted.

Archiving runs in the same statement as deleting, so a job is never lost between the two tables.
The archive is append-only, and managing its retention is up to you, e.g. with table partitioning
or a scheduled cleanup job.

## Workflow Preservation

By default, jobs that are part of an active workflow are retained to prevent partial deletion of
workflow data while the workflow is still running. A workflow is active while any of its jobs are
`suspended`, `scheduled`, `retryable`, `available`, or `executing`.

The check includes sub-workflows and parent workflows, so related jobs across a workflow hierarchy
are preserved until the entire workflow finishes.

To disable workflow preservation and prune all eligible jobs regardless of workflow status:

```toml
[pruner]
preserve_workflows = false
```

Disabling preservation can improve pruning performance for systems with many workflows, but may
leave incomplete workflow histories when workflows are pruned while still running.

## Implementation Notes

* Pruning is best-effort and performed out-of-band. All limits are soft, so jobs beyond a
  specified age may not be pruned immediately after they finish.

* Pruning only applies to `completed`, `cancelled`, and `discarded` jobs. It never deletes a
  suspended, scheduled, available, executing, or retryable job.

* Only the leader prunes, which prevents deadlocks between transactions on multiple nodes.

* The pruner emits an `oban.pruner.prune` telemetry span with `pruned_count` and
  `archived_count` metadata.