Monitoring the Health of Oban Applications

Oban tries its damnedest to keep queues chugging along in the face of unexpected issues. But stuff happens. Queues back up, jobs run longer than we'd like, and sometimes bugs (no, never our bugs) jam up the whole system. You want to know these things are going down before a customer tells you, because for every customer that reaches out there's a gaggle that were silently peeved.

It doesn't have to be like that. Between telemetry events, periodic jobs, and some diligent logging, Oban exposes the tools you need to alert on trouble early. No third-party collector services necessary (though possibly helpful).

🙋 Is This Only for Elixir?

It's not! The techniques here apply to Oban for Python as well. The exception is querying metrics, which needs Met on an Elixir node.

Taxonomy of Issues

In our experience, there are three categories of issues that you should monitor to keep your Oban system well-oiled. In order of "devastating" (everything seems broken) to "perplexing" (that doesn't look right), here they are:

Totally Stuck - no queues are running and cron jobs aren't getting scheduled
Backed Up - queues can't keep up with throughput and backlog keeps growing
Left Hanging - queues are clogged with some jobs stuck executing forever

Each of those highly scientific categories has its own distinct flavor of breakage that you can detect proactively and remediate.

Totally Stuck

A totally stuck system is the most catastrophic scenario, but thankfully the least ambiguous to identify. Queues stall and simply stop processing any jobs. Within the confines of Oban, where your database is up and rest of the application is running, this is usually a config issue.

Identification

A simple technique for detecting stuck queues is to watch logs or metrics and alert when no events have occurred within a window of time. Make sure you're ready for production with logs firing and you're halfway there. There's a catch though:

If you have periods of low activity where nothing is happening you'll get a false positive and be paged for no reason. To prevent that inactivity, set up a periodic job in a dedicated queue. The job acts as a sentinel, and its purpose is to force some activity.

Start with a health queue that won't run other jobs (so it can't back up), and schedule a cron job that checks in every minute:

defmodule MyApp.Sentinel do
  # Using the Pro worker for inline cron, you can also do it the old-fashioned way
  use Oban.Pro.Worker, queue: :health, cron: "* * * * *"

  @impl Oban.Pro.Worker
  def process(_job) do
    :my_app
    |> Application.fetch_env!(:some_check_in_url)
    |> Req.get!()

    :ok
  end
end

With a monitoring service like Honeybadger, the check-in reports your Oban is alive and kicking. If the scheduled job doesn't run and misses a check-in, you'll get a pointed alert.

Remediation

The alert tells you when jobs aren't processing, but it doesn't tell you why they aren't. As noted earlier, it's usually down to configuration. The most common issues are:

Broken Leadership - a non-worker node has leadership or a zombie node is still hanging on
Broken Notifications - using PG based PubSub without a functional Distributed Erlang cluster

These used to be issues, but that was prior to Oban v2.23.1, where we added fallbacks to ensure processing continued despite any distribution issues. Read on if you're on a previous Oban version, or want to understand how to fix these issues for other reasons.

Broken Leadership

A leader node that can't run plugins won't schedule cron jobs. Pruning and rescuing orphan jobs stop too, so jobs keep piling up. This typically happens in a hybrid web/worker environment, where web nodes don't process jobs.

Prevent this situation by either disabling plugins and leadership with plugins: false, or disabling leadership with peer: false. Don't use plugins: [], queues: [], because Elixir deep merges config, and an empty list combines with existing options rather than overwriting it.

Broken Notifications

Notifications must reach queues to alert them of available jobs. When notifications can't go through, queues may stop fetching work entirely. Oban tries to detect unhealthy notifiers and log warnings when connectivity is broken. However, in a distributed cluster a node will still receive messages from itself, which makes it impossible to accurately detect a disconnect.

Ensure you have a functional cluster when using the Oban.Notifiers.PG notifier, or that you're not behind a connection pooler for Oban.Notifiers.Postgres.

Backed Up

Queues are designed to hold jobs, but you run into trouble when jobs accumulate faster than your system can process them. Maybe your app imports a batch of 100k records every morning, then spends the next hour or so working through them. That's completely normal. The trouble starts when jobs keep arriving faster than you can finish them.

As designed, the health sentinel from earlier won't detect queue issues. To spot queue backups you need to monitor the backlog size and overall throughput.

Identification

If you're using Oban Web, you already have Oban.Met gathering the counts and metrics you need. If you're not using Oban Web...why not, it's free! Anyhow, Met shares all those values between nodes, and you can use the same data that powers the dashboard without polling for your own metrics.

Python apps can enable metrics to feed the dashboard, but there's no Met in Python to query them from your own code. If an Elixir node runs alongside your Python workers, it can query everything. Otherwise, the dashboard is your ticket to backlog and throughput, and a telemetry handler on oban.job.stop can feed your own collector.

Start by tracking how many jobs are waiting in each queue:

Oban.Met.latest(Oban, :full_count,
  filters: [state: "available"],
  group: "queue",
  lookback: 5
)

That will give you the freshest count, broken down by queue, so you can see where jobs are accumulating. Then, to see how quickly you're working through the accumulated jobs, check the exec_count in one-minute buckets for the past five minutes:

Oban.Met.timeslice(Oban, :exec_count,
  filters: [state: "completed"],
  group: "queue",
  lookback: 300,
  by: 60
)

The counts tell you how much work is available, and how quickly it's being completed. To make them useful beyond an iex session, you'll want to export them to your metrics collector of choice.

Here's a cron function (new in Pro v1.8) that runs every minute, in the health queue, and sends the latest metrics to a StatsD-type collector. From there, you can set up per-queue alerts when throughput is falling behind:

@job cron: "* * * * *", queue: :health
def report_queue_metrics do
  queues = Oban.Met.labels("queue")

  available =
    Oban.Met.latest(Oban, :full_count,
      filters: [state: "available"],
      group: "queue",
      lookback: 5
    )

  completed =
    Oban
    |> Oban.Met.timeslice(:exec_count,
      filters: [state: "completed"],
      group: "queue",
      lookback: 60,
      by: 60
    )
    |> Enum.reduce(%{}, fn {_bucket, count, queue}, acc ->
      Map.update(acc, queue, count, &(&1 + count))
    end)

  Enum.each(queues, fn queue ->
    :ok = Statsd.gauge("#{queue}.available", Map.get(available, queue, 0))
    :ok = Statsd.gauge("#{queue}.completed", Map.get(completed, queue, 0))
  end)
end

This assumes the enclosing module uses Oban.Pro.Decorator and Statsd is your configured StatsD client.

Only you know the proper thresholds for your application, based on how much delay you can tolerate. Nightly imports may have far more leeway than a queue delivering timely password resets.

Remediation

A growing backlog is a mismatch between incoming work and system capacity. Knowing there's an issue doesn't tell you why it happened, and you'll need to do some investigation.

Check whether you're receiving more jobs, whether jobs are taking longer, or whether you've reduced the queue's concurrency. It could be time to increase concurrency, increase available CPUs, (auto?)-scale to more nodes, process jobs together in chunks, or tune some global concurrency.

Just be sure you get to the bottom of the slowdown before turning the dial.

Left Hanging

Without a timeout, an Oban job can run forever. Well, "forever" meaning until the application restarts, but that's long enough to cause trouble. An infinite await, a hanging network call, or an unbounded query can leave a job squatting without accomplishing anything.

One hanging job reduces a queue's capacity, and enough of them will stop processing altogether. Meanwhile, your application is running, the health checks are fine, and everything looks alive.

Identification

Start by deciding how long each worker should reasonably take. A job that transcodes video naturally takes more time than one that sends an email. Again, only you know the right numbers for your system.

Met already tracks execution time through exec_time, and it includes a worker label to help identify which jobs are taking too long. As we did earlier, check the longest executions per worker in one-minute buckets:

Oban.Met.timeslice(Oban, :exec_time,
  group: "worker",
  lookback: 300,
  by: 60,
  operation: :max
)

That gives you five minutes of execution times for jobs that succeeded, failed, or were cancelled.

Export those values alongside the queue metrics from the previous section, and alert when workers exceed their expected duration (with some padding). Note that Met records in native time units by default, so convert the values to something more useful, like milliseconds, before exporting.

There's an important catch here: exec time is only recorded after an attempt ends. Jobs that hang forever will never report their duration, making those outliers effectively invisible. That's why alerting alone isn't enough.

Remediation

A timeout/1 callback defined on the worker will put an upper bound on each execution. For example, to set a static two-minute timeout:

@impl Oban.Worker
def timeout(_job), do: to_timeout(minute: 2)

Choose a timeout that leaves headroom for variation, and set your threshold below it. That will help you spot workers that regularly take longer than expected, even when they finish before the timeout.

When a job times out, execution is terminated with an Oban.TimeoutError exception and potentially retried if there are remaining attempts. As a bonus, the failure contributes to exec_time measurements, which become visible in the timing report.

If the work itself expires—like a notification that would arrive a day late—job deadlines in Pro will cancel it on its next run after the deadline passes. Setting force: true can cancel the job mid-execution too.

use Oban.Pro.Worker, deadline: {1, :day}, force: true

Use timeouts to limit execution, and deadlines to limit how long the job sticks around. Check a job's network calls, database queries, and potential deadlocks and apply inner limits wherever possible. That way you'll get more meaningful timeouts, and a chance for the job to handle the failure before Oban takes care of it.

Knowing is a Fraction of the Battle

The issues we cover here have plagued Oban users for years now. Wherever possible, we've improved Oban (and Pro) itself to make entire categories of issues irrelevant.

Put these tips into practice in your own app, and catch issues before your customers do. Chances are, you lovely humans are only going to glance over this...but your LLMs are going to love it.

As usual, if you have any questions or comments, ask in the Elixir Forum. For future announcements and insight into what we're working on next, subscribe to our newsletter.