Batches#
Batches link the execution of many jobs as a group and run optional callbacks after the jobs are processed. This lets your application coordinate any number of jobs running in parallel, then react once they’ve all completed, or as soon as one of them fails.
Usage#
A batch is made of jobs that share a batch_id. As a simple example, here’s a worker that
delivers daily emails and logs when every email in the batch is sent:
from oban import worker
@worker(queue="mailers")
class EmailWorker:
async def process(self, job):
await mailer.daily_update(job.args["email"])
async def batch_completed(self, job):
logger.info("batch %s complete", job.meta["batch_id"])
Build a batch by adding jobs to it, then enqueue it like any other list of jobs:
from oban_pro import Batch
batch = Batch().add([EmailWorker.new({"email": email}) for email in emails])
await oban.enqueue_many(batch)
After all jobs in the batch are completed, the batch_completed callback runs.
Handler Callbacks#
Workers opt into callbacks by defining one or more of these methods:
callback |
runs after |
|---|---|
|
all jobs were executed at least once |
|
the first job is |
|
all jobs are |
|
the first job is |
|
all jobs are |
|
the first job is |
Each callback runs in a separate, isolated job, so it may be retried or discarded like any other
job. A callback receives the callback job, with the batch_id and the callback’s name in meta,
and returns the same results as process.
Callbacks only run once per batch. They’re checked whenever a batch job finishes, retries, or
fails, so batch_retryable may run while other jobs are still executing.
Callbacks require a @worker class. Functions decorated with @job can belong to a batch, but
can’t define callbacks, so pair them with a callback_worker.
Hybrid Batches#
Batches can be built from a mix of workers. Pass an explicit callback_worker to control which
worker runs the callbacks:
batch = (
Batch(callback_worker=NotifyWorker)
.add([MailWorker.new(args) for args in mail_args])
.add([PushWorker.new(args) for args in push_args])
)
Without a callback_worker, callbacks run on the worker of whichever job triggered them, which
makes callback handling unpredictable for mixed batches.
Customizing Batches#
Batch ID#
By default, a batch_id is generated as a time-ordered UUIDv7, which is unique across workers
and nodes. Pass your own id when you need control over it:
Batch(id="custom-batch-id")
Batch Name#
An optional name describes the purpose of the batch. Unlike the id, the name doesn’t have to
be unique, so it works as a general purpose label:
Batch(name="nightly-etl")
Callback Options#
By default, callback jobs have empty args, run in the same queue as the job that triggered
them, and inherit other options from the callback worker. Use callback_opts to set args,
max_attempts, meta, priority, queue, or tags for callback jobs.
For example, here we pass a webhook URL in args:
Batch(callback_opts={"args": {"webhook": "https://web.hook"}})
And here we change the callback queue and lower the priority:
Batch(callback_opts={"queue": "callbacks", "priority": 9})
Keep callback_opts small, because they’re stored in each batch job’s meta.
Fetching Batch Jobs#
Load every job in a batch, excluding callback jobs, with all_jobs. Here we notify admins after a
batch of account jobs completes:
@worker()
class AccountWorker:
async def process(self, job): ...
async def batch_completed(self, job):
jobs = await Batch.for_job(job).all_jobs()
account_ids = [job.args["account_id"] for job in jobs]
await mailer.notify_admins(account_ids)
Every job in the batch is loaded into memory, so keep that in mind for batches with many thousands of jobs.
Appending to Batches#
Add more jobs to an existing batch from any of its jobs with append. The batch_id and other
options carry over to the new jobs:
batch = Batch.append(job).add(
[EmailWorker.new({"email": email}) for email in late_emails]
)
await oban.enqueue_many(batch)
Callback jobs are only inserted once. Appending to a batch where callbacks already ran, e.g.
batch_completed, won’t run them again.
Batches From Workflows#
Combine workflows and batches by turning a workflow into a batch. Workflow jobs keep their dependencies, and the batch callbacks run once the jobs are processed:
workflow = (
Workflow()
.add("extract", ExtractWorker.new())
.add("transform", TransformWorker.new(), deps=["extract"])
.add("load", LoadWorker.new(), deps=["transform"])
)
await oban.enqueue_many(Batch.from_workflow(workflow, callback_worker=NotifyWorker))
Cancelling Batches#
Cancel every job in a batch that isn’t already finished with cancel_jobs:
await Batch.for_job(job).cancel_jobs()
Jobs are cancelled with a bulk update, without loading the workers, and callbacks such as
batch_cancelled are inserted right away.
Callbacks are also inserted right away when batch jobs are cancelled or deleted with functions
such as oban.cancel_job or oban.delete_many_jobs. Jobs discarded while rescuing orphans, or
cancelled any other way, e.g. directly in the database, are picked up by the lifeline. It inserts any
missing callbacks on its next run, within a minute by default.