The DynamicPartitioner plugin manages a partitioned oban_jobs table for optimized query performance, minimal database bloat, and efficiently pruned historic jobs.

Migrating Off Partitioned Tables

A partitioned table can't be converted back in place, so moving off requires a transition stage. Both strategies are listed below in order of complexity, beginning with the least disruptive.

Strategy 1: Drain Into a Standard Table

The safest path runs two Oban instances side by side while the partitioned table drains.

Start by adding a standard oban_jobs table in a new prefix:

defmodule MyApp.Repo.Migrations.AddStandardObanJobs do
  use Ecto.Migration

  def up do
    Oban.Migration.up(prefix: "standard")
    Oban.Pro.Migration.up(prefix: "standard")
  end

  def down do
    Oban.Pro.Migration.down(prefix: "standard")
    Oban.Migration.down(prefix: "standard")
  end
end

Then point the primary instance at the new prefix and add a second instance that finishes processing jobs in the partitioned table, without any plugins, so it won't insert new jobs:

queues = [
  default: 10,
  other_queue: 10,
  and_another: 10
]

config :my_app, Oban.Partitioned, queues: queues

config :my_app, Oban,
  prefix: "standard",
  queues: queues,
  plugins: [
    ...

Now, start both Oban instances within your application's supervisor:

 children = [
   MyApp.Repo,
   {Oban, Application.fetch_env!(:my_app, Oban)},
+  {Oban, Application.fetch_env!(:my_app, Oban.Partitioned)},
   ...
 ]

New jobs are inserted into the standard table while existing jobs keep processing through the Oban.Partitioned instance. Once all of the partitioned jobs have executed you're free to remove the extra instance, remove the plugin, and drop the partitioned table. Dropping the parent cascades to every sub-partition:

defmodule MyApp.Repo.Migrations.DropPartitionedObanJobs do
  use Ecto.Migration

  def change do
    drop_if_exists table(:oban_jobs)
    drop_if_exists table(:oban_jobs_old)
  end
end

Be very careful to ensure you're dropping the partitioned table and not the standard one. Pass a prefix if the partitioned table doesn't live in public.

Strategy 2: Revert the Migration

If the original table is still around as oban_jobs_old, reverting the partitioning migration restores it in place:

defmodule MyApp.Repo.Migrations.RevertPartitionedObanJobs do
  use Ecto.Migration

  def change do
    Oban.Pro.Migrations.DynamicPartitioner.down()
  end
end

This drops the partitioned table along with every job in it, so it's only appropriate when those jobs are expendable and the original table is still current enough to use.

Configuration

Until you've migrated off, the plugin manages sub-partitions for the partitioned table:

config :my_app, Oban,
  plugins: [Oban.Pro.Plugins.DynamicPartitioner]
  ...

The plugin preemptively creates sub-partitions for finished job states (completed, cancelled, discarded) as well as prunes partitions older than the retention period. By default, older jobs are retained for 3 days.

You can override the retention period for states individually. For example, to retain completed jobs for 2 days, cancelled for 7, and discarded for 30:

plugins: [{
  Oban.Pro.Plugins.DynamicPartitioner,
  retention: [completed: 2, cancelled: 7, discarded: 30]
}]

Pruning sub-partitions is an extremely fast operation akin to dropping a table. As a result, there is zero lingering bloat and a separate pruner isn't advised, unless you're pruning a subset of jobs aggressively after a few minutes, hours, etc. DynamicPartitioner will warn you if the standard Pruner is enabled at the same time.

The partitioner attempts once an hour to pre-create partitions two days in advance. That schedule and buffer should be suitable for most applications. However, you can increase the buffer period and set an alternate schedule if necessary.

For example, to increase the buffer to 3 days and run at 05:00 in the Europe/Paris timezone:

plugins: [{
  Oban.Pro.Plugins.DynamicPartitioner,
  buffer: 3,
  schedule: "0 5 * * *",
  timezone: "Europe/Paris"
}]

Instrumenting with Telemetry

The DynamicPartitioner plugin adds the following metadata to the [:oban, :plugin, :stop] event:

• :created_count — the number of partitions created
• :deleted_count - the number of partitions deleted