Telemetry#
Oban emits telemetry events as jobs run and as it does its background work, such as staging, pruning, and leadership. This guide covers logging those events, attaching your own handlers for metrics, tracing, or error reporting, the metadata each event carries, and wrapping your own work in the same kind of events.
Logging#
The built-in logger handles Oban’s events and writes each one as JSON through Python’s logging
module. The CLI attaches it automatically. When running embedded, attach it when your application
starts:
from oban.telemetry import logger as telemetry_logger
telemetry_logger.attach()
By default, it logs every job’s start, stop, and exception at the INFO level, background work at
DEBUG when it succeeds and ERROR when it fails, and queue shutdowns that orphan jobs at
WARNING. Job events include the job’s id, worker, queue, attempt, args, meta, and tags, along
with its final state and timings once it finishes.
Logs are written to the oban logger unless given another one. Choose the level for job events
with level:
import logging
telemetry_logger.attach(level=logging.DEBUG, logger=logging.getLogger("myapp.oban"))
Pass events to log only some events, such as just job exceptions:
telemetry_logger.attach(events=["oban.job.exception"])
Use telemetry_logger.detach() to stop logging.
Custom Handlers#
Attach a handler to report events wherever you need them. Each handler gets an id, a list of events, and a function called with the event’s name and metadata:
from oban import telemetry
def record_duration(name, meta):
job = meta["job"]
metrics.timing(
"oban.job.duration", meta["duration"], worker=job.worker, state=meta["state"]
)
telemetry.attach(
"job-metrics", ["oban.job.stop", "oban.job.exception"], record_duration
)
Handlers are called synchronously, in the order they were attached, so keep them quick and hand slow work off elsewhere. A handler that raises is logged, and doesn’t affect other handlers or the job that emitted the event.
Every event includes the name of the Oban instance that emitted it, so handlers can find the
instance with Oban.get_instance(name), or tell instances apart when running more than one. That
includes custom spans emitted while a job runs. Every event also includes a system_time, a
timestamp in nanoseconds since the epoch.
Remove a handler by its id with telemetry.detach("job-metrics"). Attaching with an id that’s
already in use adds another handler rather than replacing the first, so detach before attaching
again.
Checking Event Names#
Attaching to a name that’s never emitted fails silently, so a typo means a handler that never
runs. Check your subscriptions against telemetry.EVENTS, the complete list of emitted events:
events = ["oban.job.stop", "oban.job.exception", "oban.pruner.prune.stop"]
unknown = [name for name in events if name not in telemetry.EVENTS]
if unknown:
raise ValueError(f"Unknown Oban events: {', '.join(unknown)}")
Job Events#
Each job emits oban.job.start when it starts running, followed by either oban.job.stop or
oban.job.exception when it finishes:
Event |
Metadata |
|---|---|
|
|
|
|
|
|
The duration is how long the job ran, and queue_time how long it waited between when it was
scheduled and when it started, both in nanoseconds.
Only failed attempts emit an exception event, with state as retryable or discarded, and the
error_type, error_message, and traceback of the raised exception. Everything else emits a
stop event, with state as completed, or as cancelled or scheduled for jobs that returned
Cancel or Snooze. That makes exception events the natural place to report errors:
def report_error(_name, meta):
error_tracker.capture(
meta["error_message"],
extra={
"id": meta["job"].id,
"worker": meta["job"].worker,
"traceback": meta["traceback"],
},
)
telemetry.attach("job-errors", ["oban.job.exception"], report_error)
Background Events#
Oban’s background work reports itself as spans, emitting .start before each run, then .stop
when it succeeds or .exception when it raises:
Prefix |
Metadata |
Count Of |
|---|---|---|
|
|
|
|
|
Jobs rescued |
|
||
|
|
Jobs acknowledged |
|
|
Jobs fetched |
|
|
Jobs pruned |
|
|
Expired producers removed |
|
|
Producers refreshed |
|
|
Jobs enqueued |
|
|
Jobs staged |
Stop and exception events add the span’s duration in nanoseconds, and exception events add the
error_type, error_message, and traceback. Counts are only known once the work is done, so
they’re on the stop event rather than the start. The refresher’s restored_count is how many of
this node’s producers were missing and recreated.
For example, to track how many jobs the pruner deletes:
def record_pruned(_name, meta):
metrics.increment("oban.pruned", meta["count"])
telemetry.attach("pruner", ["oban.pruner.prune.stop"], record_pruned)
Job Operation Events#
Operations that change jobs from outside of execution, such as enqueueing, cancelling, or retrying, report themselves as spans the same way background work does:
Prefix |
Called By |
Start Metadata |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Single job functions emit the same events as their bulk counterparts. The requested job_ids
include any ids that didn’t match, while stop events add the jobs that were actually changed along
with their count. Jobs that were skipped, such as a completed job that can’t be cancelled, aren’t
included. For update, the changes are whatever was passed, either a dict or a callable.
For example, to record which jobs were cancelled:
def record_cancelled(_name, meta):
for job in meta["jobs"]:
audit_log.write("cancelled", job_id=job.id, worker=job.worker)
telemetry.attach("job-audit", ["oban.jobs.cancel.stop"], record_cancelled)
Inserting with a conn emits the stop event once the insert runs, before your transaction
commits, so jobs from a transaction that later rolls back still appear in the event.
Queue Shutdown#
Each queue emits oban.queue.shutdown as it stops, with the queue, the duration of the
shutdown in nanoseconds, and the orphaned ids of jobs still running when the grace period ended.
Orphaned jobs are left for the lifeline to rescue, and the logger reports any shutdown that orphans
jobs as a warning.
Custom Spans#
Wrap your own work in a span to emit events the same way Oban does, such as within a worker. Metadata added to the collector is merged into the stop or exception event:
with telemetry.span("myapp.report", {"report_id": report_id}) as collector:
rows = await generate_report(report_id)
collector.add({"row_count": len(rows)})
A span that raises emits its exception event, then re-raises the exception.