← All Articles

Oban v2.11, Pro v0.10, and Web v2.9 Released

Oban v2.11, Pro v0.10, and Web v2.9 Released

Over this bundle of releases we’ve focused on minimizing resource usage and pivoted features from Pro into Oban. We’ve taken Oban from “open-core” to “basic-core”—where all the functionality needed to run a safe, stable system is available from the start. While we expanded the base of Oban v2.11, two considerable features bolstered Pro v0.10.

On to the highlights, starting with Oban. Or, choose your own adventure, and go straight to Pro.

Oban v2.11

Centralized Leadership

Coordination between nodes running Oban is crucial to how many plugins operate. Staging jobs once-a-second from multiple nodes is wasteful; as is pruning, rescuing, or scheduling cron jobs. Prior Oban versions used transactional advisory locks to prevent plugins from running concurrently.

However, there were some issues:

  • Plugins don’t know if they’ll take the advisory lock, so they still need to run a query periodically.

  • Nodes don’t usually start simultaneously, and time drifts between machines. There’s no guarantee that the top of the minute for one node is the same as another’s—chances are, they don’t match.

Here’s a chart that illustrates the difference in database queries per minute for idle queues between v2.10 and v2.11:

Queries per minute by Oban version

A new table-based leadership mechanism guarantees only one node in a cluster, where “cluster” means a bunch of nodes connected to the same Postgres database, will run plugins. That’s why, in the chart above, Oban v2.11’s query count stays so low—only one node is staging jobs.

If you’re wondering why we’re using an unlogged table for coordination rather than PubSub or Distributed Erlang, keep reading.

All will be revealed in the next section 🪄.

See the Upgrade Guide for instructions on how to create the peers table and get started with leadership. If you’re curious about the implementation details or want to use leadership in your application, take a look at docs for the Oban.Peer module.

Alternative PG (Process Groups) Notifier

Oban relies heavily on PubSub, and until now it only provided a Postgres adapter. Postgres is magnificent, and has a highly performant PubSub option, but it doesn’t work in every environment (we’re looking at you, PG Bouncer 🤬).

Fortunately, many Elixir applications run in a cluster connected by distributed Erlang. That means Process Groups, aka PG, is available for many applications.

Side note: there’s a subset of applications that use transaction pooling and don’t have clustering—ahem, Heroku—and that’s why we have table-backed leadership. Advisory lock based leadership would have been so simple…

So, we pulled Oban Pro’s PG notifier into Oban to make it available for everyone! If your app runs in a proper cluster, you can switch over to the PG notifier with one line of configuration:

config :my_app, Oban,
  notifier: Oban.Notifiers.PG,
  ...

Now there are two notifiers to choose from, each with their own strengths and weaknesses:

Oban.Notifiers.Postgres

  • Pros: Doesn’t require distributed Erlang, publishes insert events to trigger queues
  • Cons: Doesn’t work with transaction pooling, doesn’t work in tests because of the sandbox

Oban.Notifiers.PG

  • Pros: Works PG Bouncer in transaction mode, works in tests
  • Cons: Requires distributed Erlang, doesn’t publish insert events

Use whichever notifier suits your needs best. Or, use them both in different environments. G’head, we do 😉.

Basic Lifeline Plugin

When a queue’s producer crashes or a node shuts down before a job finishes executing, the job may be left in an executing state. The worst part is that these jobs—which we call “orphans”—are completely invisible until you go searching through the jobs table.

Oban Pro has always had a “Lifeline” plugin for just this occasion—and now we’ve brought a basic Lifeline plugin to Oban.

To automatically rescue orphaned jobs, include Lifeline in your configuration:

config :my_app, Oban,
  plugins: [Oban.Plugins.Lifeline],
  ...

Presto! Now Lifeline will search and rescue orphans after they’ve lingered for 60 minutes.

A crucial tidbit about the basic lifeline, from the docs:

The basic Lifeline plugin only checks execution time and it may transition jobs that are genuinely still executing; causing duplicate execution. For more accurate rescuing or to rescue jobs that have exhausted retry attempts you want the DynamicLifeline plugin.

Reindexer Plugin

Over time various Oban indexes (indeed, any indexes) may grow without VACUUM cleaning them up properly. When this happens, rebuilding the indexes will release bloat and free up space in your Postgres instance. Sure, you could handle that with a cron job, but we’ve made a plugin to take care of it for you.

config :my_app, Oban,
  plugins: [Oban.Plugins.Reindexer],
  ...

The Reindexer plugin automates index maintenance by periodically rebuilding all of your Oban indexes concurrently, without any locks. By default, reindexing happens once a day at midnight, and it’s configurable with a standard cron expression.

config :my_app, Oban,
  plugins: [{Oban.Plugins.Reindexer, schedule: "@weekly"}],
  ...

Hopefully, Postgres will get its act together and index bloat will be a distant memory.

Oban Pro v0.10

This Pro release brings enhancements over some basic Oban functionality.

Encryption, Recording, and Structure

First off, there’s a new Pro worker. Oban.Pro.Worker is a drop-in replacement for Oban.Worker with expanded capabilities such as:

  • Encrypted—transparent encryption of args at rest for applications that handle sensitive data.

  • Structured—struct backed jobs with key enforcement to prevent typos and codify args structure.

  • Recorded—capture job output and stash it on the job for inspection, or use in downstream jobs for batches or workflows.

Yes, Batch, Chunk, and Workflow workers are based on the Pro worker, so you can use all of the Pro options there as well.

Here’s a sample worker that’s configured for encryption, recording, and enforced structure:

defmodule MyApp.SuperWorker do
  use Oban.Pro.Worker,
    queue: :super,
    encrypted: [key: {Application, :fetch_env!, :secret_key}],
    recorded: true,
    structured: [keys: [:id, :ssn, :pin], required: [:id]]

  @impl Oban.Pro.Worker
  def process(%Job{args: %__MODULE__{} = args}) do
    # Use the decrypted and structured args and record the result!
    MyApp.Business.predict(args.id, args.ssn, args.pin)
  end
end

Explore each new option in the Pro Worker guide.

Dynamic Queues

DynamicQueues is to basic queues as DynamicCron is to basic cron—that is, it adds persistence across restarts, globally, across all connected nodes.

DynamicQueues are ideal for applications that dynamically start, stop, and modify queues at runtime. For example, what if your application runs one queue per customer to isolate work? You wouldn’t re-deploy a new app every time a customer signs up—it’s easy to use DynamicQueues instead:

DynamicQueues.insert([customer_42: [global_limit: 20]])

If the limit turns out to be too high and greedy ol’ customer 42 is hogging all the resources, scale them down (and rest assured that the scale will keep when the app restarts):

DynamicQueues.update(:customer_42, global_limit: 10)

There’s also a declarative syntax for specifying which nodes a queue will run on. Here’s a taste:

DynamicQueues.insert([
  basic: [limit: 10, only: {:node, :=~, "web|worker"}],
  audio: [limit: 20, only: {:node, :=~, "worker"}],
  video: [limit: 30, only: {:node, :=~, "worker"}],
  learn: [limit: 10, only: {:sys_env, "EXLA", "CUDA"}]
])

With that configuration basic will run on any web or worker node, audio and video will only run on workers, and learn will run anywhere that has EXLA configured.

Check out the DynamicQueues guide for installation, configuration, and a whole lot more usage examples.

Oban Web v2.9

This round of releases focused on Oban and Pro, but Oban Web got a little snuggle, too. Along with some minor bug fixes, Web works seamlessly with Pro’s new worker features.

Pro Worker Support

Jobs that use Oban.Pro.Worker features like encryption, recording, and enforced structure now display an indicator on the details page. What’s more, recorded jobs display the job’s return value directly in the details page:

Job detail's recorded output

Handy for spot-checking a job’s output right in the browser, isn’t it?

That’s a Wrap

All of the features in this bundle came directly from customer requests. Thanks for your feedback and issues. You’re helping to refine Oban.

See the full Oban Changelog, Web Changelog, and Pro Changelog for a complete list of enhancements and bug fixes. Or, get started with the Oban v2.11 upgrade guide.

As usual, if you have any questions or comments, ask in the Elixir Forum or the #oban channel on Elixir Slack. For future announcements and insight into what we're working on next, subscribe to our newsletter.