Telemetry#

Oban emits telemetry events as jobs run and as it does its background work, such as staging, pruning, and leadership. This guide covers logging those events, attaching your own handlers for metrics, tracing, or error reporting, the metadata each event carries, and wrapping your own work in the same kind of events.

Logging#

The built-in logger handles Oban’s events and writes each one as JSON through Python’s logging module. The CLI attaches it automatically. When running embedded, attach it when your application starts:

from oban.telemetry import logger as telemetry_logger

telemetry_logger.attach()

By default, it logs every job’s start, stop, and exception at the INFO level, background work at DEBUG when it succeeds and ERROR when it fails, and queue shutdowns that orphan jobs at WARNING. Job events include the job’s id, worker, queue, attempt, args, meta, and tags, along with its final state and timings once it finishes.

Logs are written to the oban logger unless given another one. Choose the level for job events with level:

import logging

telemetry_logger.attach(level=logging.DEBUG, logger=logging.getLogger("myapp.oban"))

Pass events to log only some events, such as just job exceptions:

telemetry_logger.attach(events=["oban.job.exception"])

Use telemetry_logger.detach() to stop logging.

Custom Handlers#

Attach a handler to report events wherever you need them. Each handler gets an id, a list of events, and a function called with the event’s name and metadata:

from oban import telemetry


def record_duration(name, meta):
    job = meta["job"]

    metrics.timing(
        "oban.job.duration", meta["duration"], worker=job.worker, state=meta["state"]
    )


telemetry.attach(
    "job-metrics", ["oban.job.stop", "oban.job.exception"], record_duration
)

Handlers are called synchronously, in the order they were attached, so keep them quick and hand slow work off elsewhere. A handler that raises is logged, and doesn’t affect other handlers or the job that emitted the event.

Every event includes the name of the Oban instance that emitted it, so handlers can find the instance with Oban.get_instance(name), or tell instances apart when running more than one. That includes custom spans emitted while a job runs. Every event also includes a system_time, a timestamp in nanoseconds since the epoch.

Remove a handler by its id with telemetry.detach("job-metrics"). Attaching with an id that’s already in use adds another handler rather than replacing the first, so detach before attaching again.

Checking Event Names#

Attaching to a name that’s never emitted fails silently, so a typo means a handler that never runs. Check your subscriptions against telemetry.EVENTS, the complete list of emitted events:

events = ["oban.job.stop", "oban.job.exception", "oban.pruner.prune.stop"]
unknown = [name for name in events if name not in telemetry.EVENTS]

if unknown:
    raise ValueError(f"Unknown Oban events: {', '.join(unknown)}")

Job Events#

Each job emits oban.job.start when it starts running, followed by either oban.job.stop or oban.job.exception when it finishes:

Event

Metadata

oban.job.start

job

oban.job.stop

job, state, duration, queue_time

oban.job.exception

job, state, duration, queue_time, error fields

The duration is how long the job ran, and queue_time how long it waited between when it was scheduled and when it started, both in nanoseconds.

Only failed attempts emit an exception event, with state as retryable or discarded, and the error_type, error_message, and traceback of the raised exception. Everything else emits a stop event, with state as completed, or as cancelled or scheduled for jobs that returned Cancel or Snooze. That makes exception events the natural place to report errors:

def report_error(_name, meta):
    error_tracker.capture(
        meta["error_message"],
        extra={
            "id": meta["job"].id,
            "worker": meta["job"].worker,
            "traceback": meta["traceback"],
        },
    )


telemetry.attach("job-errors", ["oban.job.exception"], report_error)

Background Events#

Oban’s background work reports itself as spans, emitting .start before each run, then .stop when it succeeds or .exception when it raises:

Prefix

Metadata

Count Of

oban.leader.election

leader

oban.lifeline.rescue

count

Jobs rescued

oban.metrics.broadcast

oban.producer.ack

queue, count

Jobs acknowledged

oban.producer.get

queue, count

Jobs fetched

oban.pruner.prune

count

Jobs pruned

oban.refresher.cleanup

count

Expired producers removed

oban.refresher.refresh

count, restored_count

Producers refreshed

oban.scheduler.evaluate

cron_at, count

Jobs enqueued

oban.stager.stage

count, available_queues

Jobs staged

Stop and exception events add the span’s duration in nanoseconds, and exception events add the error_type, error_message, and traceback. Counts are only known once the work is done, so they’re on the stop event rather than the start. The refresher’s restored_count is how many of this node’s producers were missing and recreated.

For example, to track how many jobs the pruner deletes:

def record_pruned(_name, meta):
    metrics.increment("oban.pruned", meta["count"])


telemetry.attach("pruner", ["oban.pruner.prune.stop"], record_pruned)

Job Operation Events#

Operations that change jobs from outside of execution, such as enqueueing, cancelling, or retrying, report themselves as spans the same way background work does:

Prefix

Called By

Start Metadata

oban.jobs.cancel

cancel_job, cancel_many_jobs

job_ids

oban.jobs.delete

delete_job, delete_many_jobs

job_ids

oban.jobs.enqueue

enqueue, enqueue_many

jobs

oban.jobs.retry

retry_job, retry_many_jobs

job_ids

oban.jobs.update

update_job, update_many_jobs

job_ids, changes

Single job functions emit the same events as their bulk counterparts. The requested job_ids include any ids that didn’t match, while stop events add the jobs that were actually changed along with their count. Jobs that were skipped, such as a completed job that can’t be cancelled, aren’t included. For update, the changes are whatever was passed, either a dict or a callable.

For example, to record which jobs were cancelled:

def record_cancelled(_name, meta):
    for job in meta["jobs"]:
        audit_log.write("cancelled", job_id=job.id, worker=job.worker)


telemetry.attach("job-audit", ["oban.jobs.cancel.stop"], record_cancelled)

Inserting with a conn emits the stop event once the insert runs, before your transaction commits, so jobs from a transaction that later rolls back still appear in the event.

Queue Shutdown#

Each queue emits oban.queue.shutdown as it stops, with the queue, the duration of the shutdown in nanoseconds, and the orphaned ids of jobs still running when the grace period ended. Orphaned jobs are left for the lifeline to rescue, and the logger reports any shutdown that orphans jobs as a warning.

Custom Spans#

Wrap your own work in a span to emit events the same way Oban does, such as within a worker. Metadata added to the collector is merged into the stop or exception event:

with telemetry.span("myapp.report", {"report_id": report_id}) as collector:
    rows = await generate_report(report_id)

    collector.add({"row_count": len(rows)})

A span that raises emits its exception event, then re-raises the exception.