Process large data sets gradually with a chain of small, resumable jobs.

Backfills are useful for updating existing records, deleting stale data, or performing external work without a long-running migration or one large transaction. Each job processes a limited window of rows and advances a cursor until no rows remain.

Usage

Define a backfill with use Oban.Pro.Backfill and implement backfill/2. Pass the cursor to an Oban.Pro.Backfill.Cursor helper to process the next window and continue the backfill:

defmodule MyApp.UserBackfill do
  use Oban.Pro.Backfill, queue: :backfills, limit: 1_000

  import Ecto.Query

  @impl Oban.Pro.Backfill
  def backfill(cursor, _extra) do
    MyApp.User
    |> where([u], is_nil(u.migrated_at))
    |> Cursor.update_all(cursor, set: [migrated_at: ^DateTime.utc_now()])
  end
end

Start the backfill from a migration:

defmodule MyApp.Repo.Migrations.BackfillUserMigratedAt do
  use Ecto.Migration

  def up do
    Oban.Pro.Backfill.start(MyApp.UserBackfill)
  end
end

The migration inserts the first job when it runs in dev or prod. In test, start/1 skips insertion so migration tests don't leave backfill jobs behind.

Non-Database Backfills

Not every backfill is a single statement. When each row needs individual work, such as an external API or inference call, use Cursor.each/4:

@impl Oban.Pro.Backfill
def backfill(cursor, _extra) do
  Cursor.each(MyApp.User, cursor, fn user ->
    embedding = MyApp.Inference.embed(user.bio)

    MyApp.Repo.update_all(where(MyApp.User, id: ^user.id), set: [embedding: embedding])
  end)
end

Use Cursor.fetch/3 when you need the whole window for a batch request or to process rows concurrently with Task.async_stream/3. It returns the rows and the advanced cursor. A window may run more than once on retry, so any external work must be safe to repeat.

Options

Set options as defaults with use Oban.Pro.Backfill, or override them with new/2 and start/2:

• :key — The unique, ordered column used to track progress. It may be an integer or UUID. Defaults to :id.

• :limit — The maximum number of rows per window. Defaults to 5_000.

• :order — Process keys in :asc or :desc order. Descending order requires a key with a meaningful order, such as an integer or UUIDv7. Defaults to :asc.

• :throttle — Pause between windows by the given period, e.g. {1, :second}, to limit load. Defaults to no delay.

• :timeout — Limit how long a single window may run, in milliseconds, before Oban cancels it. Defaults to no limit.

• :dry_run — Process every window while rolling back its database writes. Use it to verify that a backfill terminates and estimate how many windows it requires. External side effects aren't rolled back. Defaults to false.

Extra Data

Pass data needed by every window as the first argument to new/2, or with the extra: option to start/2. The data arrives as the second argument to backfill/2 and preserves Elixir terms such as tuples, atoms, and structs.

MyApp.UserBackfill.new(%{tenant_id: 42}, limit: 500)

Idempotency

A window may run more than once after an error or node shutdown. Make the work in backfill/2 safe to repeat. Backfill doesn't wrap the callback in a transaction, though a dry_run does use a transaction to roll back database writes.