# Batches Batches link the execution of many jobs as a group and run optional callbacks after the jobs are processed. This lets your application coordinate any number of jobs running in parallel, then react once they've all completed, or as soon as one of them fails. ## Usage A batch is made of jobs that share a `batch_id`. As a simple example, here's a worker that delivers daily emails and logs when every email in the batch is sent: ```python from oban import worker @worker(queue="mailers") class EmailWorker: async def process(self, job): await mailer.daily_update(job.args["email"]) async def batch_completed(self, job): logger.info("batch %s complete", job.meta["batch_id"]) ``` Build a batch by adding jobs to it, then enqueue it like any other list of jobs: ```python from oban_pro import Batch batch = Batch().add([EmailWorker.new({"email": email}) for email in emails]) await oban.enqueue_many(batch) ``` After all jobs in the batch are `completed`, the `batch_completed` callback runs. ## Handler Callbacks Workers opt into callbacks by defining one or more of these methods: | callback | runs after | | ----------------- | ------------------------------------------------------- | | `batch_attempted` | _all_ jobs were executed at least once | | `batch_cancelled` | the _first_ job is `cancelled` | | `batch_completed` | _all_ jobs are `completed` | | `batch_discarded` | the _first_ job is `discarded` | | `batch_exhausted` | _all_ jobs are `completed`, `cancelled`, or `discarded` | | `batch_retryable` | the _first_ job is `retryable` | Each callback runs in a separate, isolated job, so it may be retried or discarded like any other job. A callback receives the callback job, with the `batch_id` and the callback's name in `meta`, and returns the same results as `process`. Callbacks only run once per batch. They're checked whenever a batch job finishes, retries, or fails, so `batch_retryable` may run while other jobs are still executing. Callbacks require a `@worker` class. Functions decorated with `@job` can belong to a batch, but can't define callbacks, so pair them with a `callback_worker`. ## Hybrid Batches Batches can be built from a mix of workers. Pass an explicit `callback_worker` to control which worker runs the callbacks: ```python batch = ( Batch(callback_worker=NotifyWorker) .add([MailWorker.new(args) for args in mail_args]) .add([PushWorker.new(args) for args in push_args]) ) ``` Without a `callback_worker`, callbacks run on the worker of whichever job triggered them, which makes callback handling unpredictable for mixed batches. ## Customizing Batches ### Batch ID By default, a `batch_id` is generated as a time-ordered UUIDv7, which is unique across workers and nodes. Pass your own `id` when you need control over it: ```python Batch(id="custom-batch-id") ``` ### Batch Name An optional `name` describes the purpose of the batch. Unlike the `id`, the name doesn't have to be unique, so it works as a general purpose label: ```python Batch(name="nightly-etl") ``` ### Callback Options By default, callback jobs have empty `args`, run in the same queue as the job that triggered them, and inherit other options from the callback worker. Use `callback_opts` to set `args`, `max_attempts`, `meta`, `priority`, `queue`, or `tags` for callback jobs. For example, here we pass a webhook URL in `args`: ```python Batch(callback_opts={"args": {"webhook": "https://web.hook"}}) ``` And here we change the callback queue and lower the priority: ```python Batch(callback_opts={"queue": "callbacks", "priority": 9}) ``` Keep `callback_opts` small, because they're stored in each batch job's `meta`. ## Fetching Batch Jobs Load every job in a batch, excluding callback jobs, with `all_jobs`. Here we notify admins after a batch of account jobs completes: ```python @worker() class AccountWorker: async def process(self, job): ... async def batch_completed(self, job): jobs = await Batch.for_job(job).all_jobs() account_ids = [job.args["account_id"] for job in jobs] await mailer.notify_admins(account_ids) ``` Every job in the batch is loaded into memory, so keep that in mind for batches with many thousands of jobs. ## Appending to Batches Add more jobs to an existing batch from any of its jobs with `append`. The `batch_id` and other options carry over to the new jobs: ```python batch = Batch.append(job).add( [EmailWorker.new({"email": email}) for email in late_emails] ) await oban.enqueue_many(batch) ``` Callback jobs are only inserted once. Appending to a batch where callbacks already ran, e.g. `batch_completed`, won't run them again. ## Batches From Workflows Combine workflows and batches by turning a workflow into a batch. Workflow jobs keep their dependencies, and the batch callbacks run once the jobs are processed: ```python workflow = ( Workflow() .add("extract", ExtractWorker.new()) .add("transform", TransformWorker.new(), deps=["extract"]) .add("load", LoadWorker.new(), deps=["transform"]) ) await oban.enqueue_many(Batch.from_workflow(workflow, callback_worker=NotifyWorker)) ``` ## Cancelling Batches Cancel every job in a batch that isn't already finished with `cancel_jobs`: ```python await Batch.for_job(job).cancel_jobs() ``` Jobs are cancelled with a bulk update, without loading the workers, and callbacks such as `batch_cancelled` are inserted right away. Callbacks are also inserted right away when batch jobs are cancelled or deleted with functions such as `oban.cancel_job` or `oban.delete_many_jobs`. Jobs discarded while rescuing orphans, or cancelled any other way, e.g. directly in the database, are picked up by the lifeline. It inserts any missing callbacks on its next run, within a minute by default.