# Telemetry Oban emits telemetry events as jobs run and as it does its background work, such as staging, pruning, and leadership. This guide covers logging those events, attaching your own handlers for metrics, tracing, or error reporting, the metadata each event carries, and wrapping your own work in the same kind of events. ## Logging The built-in logger handles Oban's events and writes each one as JSON through Python's `logging` module. The CLI attaches it automatically. When running embedded, attach it when your application starts: ```python from oban.telemetry import logger as telemetry_logger telemetry_logger.attach() ``` By default, it logs every job's start, stop, and exception at the `INFO` level, background work at `DEBUG` when it succeeds and `ERROR` when it fails, and queue shutdowns that orphan jobs at `WARNING`. Job events include the job's id, worker, queue, attempt, args, meta, and tags, along with its final state and timings once it finishes. Logs are written to the `oban` logger unless given another one. Choose the level for job events with `level`: ```python import logging telemetry_logger.attach(level=logging.DEBUG, logger=logging.getLogger("myapp.oban")) ``` Pass `events` to log only some events, such as just job exceptions: ```python telemetry_logger.attach(events=["oban.job.exception"]) ``` Use `telemetry_logger.detach()` to stop logging. ## Custom Handlers Attach a handler to report events wherever you need them. Each handler gets an id, a list of events, and a function called with the event's name and metadata: ```python from oban import telemetry def record_duration(name, meta): job = meta["job"] metrics.timing( "oban.job.duration", meta["duration"], worker=job.worker, state=meta["state"] ) telemetry.attach( "job-metrics", ["oban.job.stop", "oban.job.exception"], record_duration ) ``` Handlers are called synchronously, in the order they were attached, so keep them quick and hand slow work off elsewhere. A handler that raises is logged, and doesn't affect other handlers or the job that emitted the event. Every event includes the `name` of the Oban instance that emitted it, so handlers can find the instance with `Oban.get_instance(name)`, or tell instances apart when running more than one. That includes custom spans emitted while a job runs. Every event also includes a `system_time`, a timestamp in nanoseconds since the epoch. Remove a handler by its id with `telemetry.detach("job-metrics")`. Attaching with an id that's already in use adds another handler rather than replacing the first, so detach before attaching again. ### Checking Event Names Attaching to a name that's never emitted fails silently, so a typo means a handler that never runs. Check your subscriptions against `telemetry.EVENTS`, the complete list of emitted events: ```python events = ["oban.job.stop", "oban.job.exception", "oban.pruner.prune.stop"] unknown = [name for name in events if name not in telemetry.EVENTS] if unknown: raise ValueError(f"Unknown Oban events: {', '.join(unknown)}") ``` ## Job Events Each job emits `oban.job.start` when it starts running, followed by either `oban.job.stop` or `oban.job.exception` when it finishes: | Event | Metadata | | -------------------- | ----------------------------------------------------- | | `oban.job.start` | `job` | | `oban.job.stop` | `job`, `state`, `duration`, `queue_time` | | `oban.job.exception` | `job`, `state`, `duration`, `queue_time`, error fields | The `duration` is how long the job ran, and `queue_time` how long it waited between when it was scheduled and when it started, both in nanoseconds. Only failed attempts emit an exception event, with `state` as `retryable` or `discarded`, and the `error_type`, `error_message`, and `traceback` of the raised exception. Everything else emits a stop event, with `state` as `completed`, or as `cancelled` or `scheduled` for jobs that returned `Cancel` or `Snooze`. That makes exception events the natural place to report errors: ```python def report_error(_name, meta): error_tracker.capture( meta["error_message"], extra={ "id": meta["job"].id, "worker": meta["job"].worker, "traceback": meta["traceback"], }, ) telemetry.attach("job-errors", ["oban.job.exception"], report_error) ``` ## Background Events Oban's background work reports itself as spans, emitting `.start` before each run, then `.stop` when it succeeds or `.exception` when it raises: | Prefix | Metadata | Count Of | | ------------------------- | --------------------------- | ------------------------- | | `oban.leader.election` | `leader` | | | `oban.lifeline.rescue` | `count` | Jobs rescued | | `oban.metrics.broadcast` | | | | `oban.producer.ack` | `queue`, `count` | Jobs acknowledged | | `oban.producer.get` | `queue`, `count` | Jobs fetched | | `oban.pruner.prune` | `count` | Jobs pruned | | `oban.refresher.cleanup` | `count` | Expired producers removed | | `oban.refresher.refresh` | `count`, `restored_count` | Producers refreshed | | `oban.scheduler.evaluate` | `cron_at`, `count` | Jobs enqueued | | `oban.stager.stage` | `count`, `available_queues` | Jobs staged | Stop and exception events add the span's `duration` in nanoseconds, and exception events add the `error_type`, `error_message`, and `traceback`. Counts are only known once the work is done, so they're on the stop event rather than the start. The refresher's `restored_count` is how many of this node's producers were missing and recreated. For example, to track how many jobs the pruner deletes: ```python def record_pruned(_name, meta): metrics.increment("oban.pruned", meta["count"]) telemetry.attach("pruner", ["oban.pruner.prune.stop"], record_pruned) ``` ## Job Operation Events Operations that change jobs from outside of execution, such as enqueueing, cancelling, or retrying, report themselves as spans the same way background work does: | Prefix | Called By | Start Metadata | | ------------------- | -------------------------------- | -------------------- | | `oban.jobs.cancel` | `cancel_job`, `cancel_many_jobs` | `job_ids` | | `oban.jobs.delete` | `delete_job`, `delete_many_jobs` | `job_ids` | | `oban.jobs.enqueue` | `enqueue`, `enqueue_many` | `jobs` | | `oban.jobs.retry` | `retry_job`, `retry_many_jobs` | `job_ids` | | `oban.jobs.update` | `update_job`, `update_many_jobs` | `job_ids`, `changes` | Single job functions emit the same events as their bulk counterparts. The requested `job_ids` include any ids that didn't match, while stop events add the `jobs` that were actually changed along with their `count`. Jobs that were skipped, such as a completed job that can't be cancelled, aren't included. For `update`, the `changes` are whatever was passed, either a dict or a callable. For example, to record which jobs were cancelled: ```python def record_cancelled(_name, meta): for job in meta["jobs"]: audit_log.write("cancelled", job_id=job.id, worker=job.worker) telemetry.attach("job-audit", ["oban.jobs.cancel.stop"], record_cancelled) ``` Inserting with a `conn` emits the stop event once the insert runs, before your transaction commits, so jobs from a transaction that later rolls back still appear in the event. ## Queue Shutdown Each queue emits `oban.queue.shutdown` as it stops, with the `queue`, the `duration` of the shutdown in nanoseconds, and the `orphaned` ids of jobs still running when the grace period ended. Orphaned jobs are left for the lifeline to rescue, and the logger reports any shutdown that orphans jobs as a warning. ## Custom Spans Wrap your own work in a span to emit events the same way Oban does, such as within a worker. Metadata added to the collector is merged into the stop or exception event: ```python with telemetry.span("myapp.report", {"report_id": report_id}) as collector: rows = await generate_report(report_id) collector.add({"row_count": len(rows)}) ``` A span that raises emits its exception event, then re-raises the exception.