Whether you're enabling disparate teams to collaborate, gradually migrating from one language to another, or leveraging packages that are lacking in one ecosystem, having a mechanism to transparently exchange durable jobs between Elixir and Python opens up new possibilities.

On that tip, let's build a small example to demonstrate how trivial bridging can be. We'll call it "Badge Forge".

Forging Badges

"Badge Forge," like "Fire Saga" before it, is a pair of nouns that barely describes what our demo app does. But, it's balanced and why hold back on the whimsy?

More concretely, we're building a micro app that prints conference badges. The actual PDF generation happens through WeasyPrint, a Python library that turns HTML and CSS into print-ready documents. It's mature and easy to use. For the purpose of this demo, we'll pretend that running ChromaticPDF is unpalatable and Typst isn't available.

There's no web framework involved, just command-line output and job processing. Don't fret, we'll bring in some visualization later.

Sharing a Common Database

Some say you're cra-zay for sharing a database between applications. We say you're already willing to share a message queue, and now the database is your task broker, so why not? It's happening.

Oban for Python was designed for interop with Elixir from the beginning. Both libraries read and write to the same oban_jobs table, with job args stored as JSON, so they're fully language-agnostic. When an Elixir app enqueues a job destined for a Python worker (or vice versa), it simply writes a row. The receiving side picks it up based on the queue name, processes it, and updates the status. That's the whole mechanism:

Each side maintains its own cluster leadership, so an Elixir node and a Python process won't compete for leader responsibilities. They coordinate through the jobs table, but take care of business independently.

Both sides can also exchange PubSub notifications through Postgres for real-time coordination. The importance of that tidbit will become clear soon enough.

Printing in Action

This is more of a demonstration than a tutorial. We don't expect you to build along, but we hope you'll see how little code it takes to form a bridge.

With a wee config in place and both apps pointing at the same database, we can start generating badges.

Enqueueing Jobs

Generation starts on the Elixir side. This function enqueues a batch of (fake) jobs destined for the Python worker:

def enqueue_batch(count \\ 100) do
  generate = fn _ ->
    args = %{
      id: Ecto.UUID.generate(),
      name: fake_name(),
      company: fake_company(),
      type: Enum.random(~w(attendee speaker sponsor organizer))
    }

    Oban.Job.new(args, worker: "badge_forge.generator.GenerateBadge", queue: :badges)
  end

  1..count
  |> Enum.map(generate)
  |> Oban.insert_all()
end

Notice the worker name is a string, "badge_forge.generator.GenerateBadge", matching the Python worker's fully qualified name. The job lands in the badges queue, where a Python worker is listening.

The Python Side

The Python worker receives badge requests and generates PDFs using WeasyPrint:

from oban import Job, Oban, worker
from weasyprint import HTML

@worker(max_attempts=5, queue="badges")
class GenerateBadge:
    async def process(self, job: Job) -> None:
        id = job.args["badge_id"]
        name = job.args["name"]
        html = render_badge_html(name, job.args["company"], job.args["type"])
        path = BADGES_DIR / f"{name}.pdf"

        # Generate the pdf content
        HTML(string=html).write_pdf(path)

        # Construct a job manually
        job = Job(
            args={"id": id, "name": name, "path": str(path)},
            queue="printing",
            worker="BadgeForge.PrintCenter",
        )

        # Use the active Oban instance and enqueue the job
        await Oban.get_instance().enqueue(job)

When a job arrives, it pulls the attendee info from the args, renders an HTML template, and writes the PDF to disk. After completion, it enqueues a confirmation job back to Elixir.

The Elixir Side

The Elixir side listens for confirmations and prints the result:

defmodule BadgeForge.PrintCenter do
  use Oban.Worker, queue: :printing

  require Logger

  @impl Oban.Worker
  def perform(%Job{args: %{"id" => id, "name" => name, "path" => path}}) do
    Logger.info("Printing badge #{id} for #{name}: #{path}...")

    do_actual_printing_here(...)

    :ok
  end
end

With that, there's two-way communication through the jobs table.

Sample Output

To print conference badges you need a conference. You should have a conference. We're printing badges for the fictional "Oban Conf" being held this year in Edinburgh. It will be both hydrating and engaging. Kicking off a batch of ten jobs from Elixir:

iex> BadgeForge.enqueue_batch(10)
:ok

On the Python side, we see automatic logging for each job with output like this (output has been prettified):

[INFO] oban: {
  "id":14,
  "worker":"badge_forge.generator.GenerateBadge",
  "queue":"badges",
  "attempt":1,
  "max_attempts":20,
  "args":{
    "id":"7bfb7c39-c354-4cce-ad5b-f1be2814b17e",
    "name":"Alasdair Fraser",
    "type":"speaker",
    "company":"Wavelength Tech"
  },
  "meta":{},
  "tags":[],
  "event":"oban.job.stop",
  "state":"completed",
  "duration":2.51,
  "queue_time":5.45
}

The job completed successfully, and back in the Elixir app, we see that the print completed:

[info] Printing badge 7bfb7c39 for Alasdair Fraser: /some/path...

The output looks something like this:

Apologies to any "Alasdair Frasers" out there, your name was pulled from the nether and there isn't a real conference. As consolation, if you contact us, you have stickers coming.

Visualizing the Activity

Seeing jobs in terminal logs is fine, but watching them flow through a dashboard is far more satisfying. We recently shipped a standalone Oban Web Docker image for situations like this; where you want monitoring without mounting it in your app. It's also useful when your primary app is actually Python...

With docker running, point the DATABASE_URL at your Oban-ified database and pull the image:

docker run -d \
  -e DATABASE_URL="postgres://user:pass@host.docker.internal:5432/badge_forge_dev" \
  -p 4000:4000 \
  ghcr.io/oban-bg/oban-dash

That starts Oban Web running in the background to monitor jobs from all connected Oban instances. Queue activity and metrics are exchanged via PubSub, so the Web instance can store them for visualization. Trigger a few (hundred) jobs, navigate to the dashboard on localhost:4000, and look at 'em roll:

Bridging Both Ways

Badge Forge is whimsical, some say "useless", but the pattern is practical! When you need tools that are stronger in one ecosystem, you can bridge it. This goes both ways. A Python app can reach for Elixir's strengths just as easily.

Check out the full demo code for the boilerplate and config we rested over here.

Why on Earth?

2025 was supposed to be the year of Elixir interop. We're fashionably late, and what better way to celebrate than bringing Oban to another ecosystem? Python seemed like a natural fit. One of us, (not naming any names), wouldn't even consider Oban for Go...

Python is undeniably ubiquitous across most domains of computing. There are packages and frameworks available in Python that are sorely missing in other ecosystems, including Elixir.

Oban has proven to be a valuable tool for developers looking to build reliable applications in Elixir. So why not Python? As Elixir evangelists we recognize the value in expanding Oban into a new community, championing Elixir, and enticing developers to consider its benefits.

Oban for Python

The OSS oban-py package is available on GitHub, published to PyPI, and the documentation is hosted here. The initial release is v0.5.0; it's not yet v1.0 but full featured enough that we shan't call it a v0.1.

If you're coming from any one of the other Python background job systems, some of the standouts of Oban in Python are below (if you're an Elixirist familiar with Oban, skip beyond the bullet list):

• No message broker — Just PostgreSQL. No Redis, no RabbitMQ, no separate infrastructure to manage. Jobs live in the same database as your application data.

• Historic job retention — Oban keeps jobs after execution rather than immediately deleting them — completed, failed, cancelled, all of it. You receive a full audit trail and can query job history.

• Independent concurrency — Each queue has its own concurrency limit. Configure emails at 5, reports at 2 and they won't compete for a shared worker pool. A slow report can't starve your email queue.

• Runtime queue control — Pause, resume, or scale queue concurrency without restarting workers. Especially useful for maintenance windows or throttling during incidents.

• The CLI — This one's not a surprise, but we're super keen on it. The schmancy CLI shown above handles installation, upgrades, and running worker processes with auto-discovery of workers and cron schedules.

We encourage you to give it a go! Clock the docs, kick the tires, peruse the source, throw your agents at it, and interrogate us (via email only).

Oban Pro for Python

Yes, we're dropping Oban Pro for Python at the same time. The initial Pro release is also v0.5.0, and the docs are here. It ships with some of our favorite Pro features and something BEAM-ish:

• Workflows — Durable job composition with sequential, fan-out, and fan-in patterns. Nest sub-workflows, pass results downstream automatically, or graft new jobs onto running workflows at runtime. State lives in Postgres, not memory.

• Smart concurrency — Rate limit jobs globally across your cluster (e.g., 60 API calls per minute, enforced across all nodes), or partition queues to apply limits per worker, tenant, or any argument you choose.

• Multi-process execution — Call it "BEAM mode" for Python. Bypass the GIL and allow jobs to fan out across all processors, using available CPU cores automatically.

Oban Pro for Python is its own product with its own license. If you've purchased Pro for Elixir, the pricing and flow is similar if not intuitive.

We're launching in beta, and the first 10 subscribers that use the coupon code OBAN4PY will be grandmothered in at 50% off for the lifetime of their Oban Pro for Python subscription.

Elixir Interop

From the outset we were determined to make the Elixir and Python implementations fully compatible. This is about interop and expansion after all! To that end, the underlying table structures are nearly identical (some column types are subtly different for the sake of optimization).

It's entirely possible to enqueue jobs from an Oban instance running in Elixir and run them in Python, or vice-versa!

In fact, recorded job output is stored in erlang term format, so you can retrieve output from jobs run in an entirely different platform. Since the PubSub notification format is identical, it's even conceivable to pause/resume/scale queues running on heterogeneous nodes.

Building Oban for Python encouraged us to revisit legacy decisions baked into the Elixir implementation. The exercise will inform Oban 3.0 and Pro 2.0. Interop goes both ways.

To the Future

This is a v0.5, not a v1.0. There's much more to design and build—full parity with Oban, missing Pro features, complete interop with the Oban Web dashboard. Perhaps alternate database support? We're working on it!

Building Oban for Python was a significant undertaking, and we're genuinely excited to see what people do with it. If you try it out, do let us know how it goes.

The good news? With the release of Oban v2.20 and Oban Pro v1.6, you get it all. Anything they can do, we can do better too. LLM call chaining? Dynamic expansion at runtime? Pausing for human feedback? Yes we can and did.

In fact, you can compose functional workflows, graft sub-workflows on the fly, and pause mid-flow to get a human involved—all without giving up durability and transparency from the comfort of your own app.

Cascading Workflows

Cascading workflows are undoubtedly the most useful tool in the agentic workflow toolbox. But we wrote an entire article touting the power of cascading workflows and why you should use them, so we won't bore you by repeating ourselves.

They're a dead simple way to link functions together with shared context, automatic retries, and distributed across nodes. They look like this:

def invoke(%{user_id: user_id, query: query} = source) do
  Workflow.new()
  |> Workflow.put_context(%{source: source, user_id: user_id, model: "gpt-4o-mini"})
  |> Workflow.add_cascade(:plan, &plan/1)
  |> Workflow.add_cascade(:draft, &draft/1, deps: :plan)
  |> Workflow.add_cascade(:revise, &revise/1, deps: :draft)
  |> Workflow.add_cascade(:persist, &persist/1, deps: :revise)
  |> Oban.insert_all()
end

Each of the steps is a plain elixir function that receives a cumulative context map. Bosh. That's it. On to the other new stuff!

Hold for Human

Not every workflow can, or should, be fully automated. Sometimes you need to pause to get a human in the loop. There are high-stakes, high-risk, or irreversible actions where mistakes become costly in money or trust. Or maybe you're one of those control freaks that like to verify citations before publishing a legal document.

Thanks to the recent addition of Oban.update_job/3, it's trivial to pause an in-progress workflow and await human intervention.

Imagine a marketing campaign pipeline that will draft an email, pass it off to a human for review, and deliver it upon approval. For the initial hold, we can make use of tags on the current job and start by notifying somebody that the job needs attention then enter a snooze loop to wait:

# This injects the `current_job/0` helper used to retrieve the job later
use Oban.Pro.Decorator

def hold_for_human(_context) do
  job = current_job()

  case job.tags do
    ["approved"] ->
      :ok

    ["denied"] ->
      {:cancel, "the human said so"}

    _ ->
      MyApp.Campaign.notify_a_human(job)

      {:snooze, {1, :hour}}
  end
end

Later, after the human has reviewed the details, they will approve or deny the rest of the workflow by tagging the job accordingly, then tell it to stop snoozing:

def resume_by_human(job_id, status) when status in ~w(approved denied) do
  with {:ok, job} <- Oban.update_job(job_id, &%{tags: [status | &1.tags]}) do
    Oban.retry_job(job)
  end
end

At that point, the job will complete and the rest of the workflow can continue on its merry way.

This is a powerful pattern that's useful for agent and non-agentic workflows alike.

Workflow Grafting

Grafting lets you define a placeholder in a workflow, then expand it into a sub-workflow after the workflow has already started. Those placeholders are called grafts, and the expansion is the grafting bit.

The beauty of grafting is that downstream jobs that depend on the graft will automatically depend on the expanded sub-workflow as well. That means downstream jobs won't run until both the graft and the grafted sub-workflow have completed.

Let's expand on the "human in the loop" example above to demonstrate. Imagine that we don't know the recipients of the marketing email until the workflow is already running:

def start(campaign_id) do
  Workflow.new()
  |> Workflow.put_context(%{campaign_id: campaign_id})
  |> Workflow.add_cascade(:draft, &draft_campaign/1)
  |> Workflow.add_graft(:load, &load_accounts/1, deps: :draft)
  |> Workflow.add_cascade(:finish, &finish_campaign/1, deps: [:draft, :load])
  |> Workflow.insert()
end

That builds out a cascade, and sticks a graft into the middle. When that runs, it calls load_accounts/1 to create a sub-workflow of all the notification jobs for each account dynamically:

def load_accounts(%{campaign_id: campaign_id, draft: draft}) do
  account_ids = MyApp.Campaign.recipients_for_campaign(campaign_id, draft)

  {account_ids, &notify_account/2}
  |> Workflow.apply_graft()
  |> Oban.insert_all()
end

The final job will still wait for both the load_accounts/1 job and the grafted notify_account/2 sub-workflow to complete.

So grafting allows dynamic expansion of workflows-you don't have to know up front which (or how many) jobs need to run, only that at some point during the workflow you'll figure it out.

For more involved pipelines, this type of dynamically constructed workflow is virtually impossible (or highly invasive) to manage without grafting!

The More You Know

With cascades, complex workflows become a reliable chain of functions. With human-in-the-loop holds, you can instill human expertise and comingle it with automation. And with grafting, a new type of dynamic workflow is possible sans duct-tape and black magic.

The patterns we've highlighted are necessary for agentic workflows, but they're equally powerful for regular, non-agentic workloads if "determinism" is your thang.

We're excited to see how you build your workflows. How have you worked humans into them? Has grafting changed what you build? Drop us a line and share—your patterns might inspire the next wave of features.

Said pageant doesn't require any domain knowledge or business logic; just some whimsy and a really loose understanding of graphs. On to the Fire Saga!

Say What About a Fire Saga?

It's called "Fire Saga" (🔥📖), because we like Eurovision and thought it sounded catchy. A saga is an epic story, they're both nouns...it's awesome. Don't get too wrapped up in the name.

Anyhow, FireSaga is a workflow that generates a collection of children's stories based on a topic using generative AI. The authors are selected at random, then a short children's story and an illustration are generated for each author before it's all packaged together with cover art in a tidy markdown file.

It's all coordinated using an Oban Workflow.

Why Oban Workflows?

Performing a set of coordinated tasks with dependencies between them is precisely what workflows are meant to do. Think of workflows as a pipe operator for directed, distributed, persistent map-reduce.

Let's break that buzzword soup down:

• Directed-jobs progress in a fixed order, whether running linearly, fanning out, fanning in, or running in parallel.

• Distributed-jobs run seamlessly on all available nodes transparently, without extra coordination from your application.

• Persistent-processes crash, applications restart, and a workflow must be able to pick up where it left off.

A few more essential traits for good measure:

• Retryable-LLMs are notoriously unreliable, whether from timeouts, rate limiting, or unexpected nonsense responses. Retries are crucial to having reliable output.

• Introspection-you need to track progress to see how a workflow is progressing, check timing information, and see errors wreaking havoc on the pipeline.

See? Some say, workflows are a perfect fit for a multi-stage, interdependent, story generating pipeline making unreliable API calls. Well, we do.

Building the Workflow

It would be possible to build FireSaga using standard jobs in older style workflows. In that case, each job would be in a separate module, and you would use Workflow functions to manually fetch recorded output from upstream dependencies and weave it all together.

That'd be fine...but Pro v1.6 introduced context sharing, cascade mode, and sub workflows, which automates all of that.

To make a long saga short, "cascade mode" streamlines defining a workflow by passing accumulated context to each step. A context map, as well as output from each dependency, are passed to every function.

Kick it off with a workflow building function named build/1:

def build(opts) when is_list(opts) do
  topic = Keyword.fetch!(opts, :topic)
  count = Keyword.fetch!(opts, :chapters)
  range = 0..(count - 1)

  Workflow.new()
  |> Workflow.put_context(%{count: count, topic: topic})
  |> Workflow.add_cascade(:authors, &gen_authors/1)
  |> Workflow.add_cascade(:stories, {range, &gen_story/2}, deps: :authors)
  |> Workflow.add_cascade(:images, {range, &gen_image/2}, deps: ~w(authors stories))
  |> Workflow.add_cascade(:title, &gen_title/1, deps: :stories)
  |> Workflow.add_cascade(:cover, &gen_cover/1, deps: :stories)
  |> Workflow.add_cascade(:print, &print/1, deps: ~w(authors cover images stories title))
end

There's a lot going on in there:

• We stash a context map containing the number of chapters and chosen topic with put_context/2 to share data with all workflow steps.

• We're using add_cascade/4 to create steps that receive both the context map and the output from their dependencies.

• The function captures (e.g., &gen_authors/1) define the actual work performed at each step. That way, the compiler can verify functions exist and add_cascade/4 verifies they have the correct arity.

• For the :stories and :images steps, we use the {enum, capture} variant of add_cascade/4 to create sub-workflows that fan out across our range of chapters.

• Dependencies are clearly specified with the deps option, ensuring steps only run after their prerequisites complete, (including deps on an entire sub-workflow).

• The final :print step has dependencies on all previous steps, ensuring it only runs once everything else is complete.

It's easier to understand the flow of data with some visualization. Time for to_mermaid/1:

[chapters: 3, topic: "hamsters and gooey kablooies"]
|> FireSaga.build()
|> Oban.Pro.Workflow.to_mermaid()

Passing the output through to mermaid outputs a spiffy, helpful diagram:

That's all for the workflow. On to the actual functions.

Cascade Functions

Cascade functions are surprisingly simple—they're just regular functions that accept a context map as their single argument. You can define them in any module without special annotations or boilerplate.

Let's check out the gen_authors/1 implementation:

def gen_authors(%{count: count}) do
  prompt = """
  Generate an unordered list of thirty prominent children's authors.
  Use a leading hyphen rather than numbers for each list item.
  """

  prompt
  |> LLM.chat!()
  |> String.split("\n", trim: true)
  |> Enum.map(&String.trim_leading(&1, "- "))
  |> Enum.take_random(count)
end

It's using a simple LLM module to call a chat endpoint (you can check it out on GitHub). The output is a list of authors (Beverly Cleary, Shel Silverstein, etc.), which is recorded and made available in the downstream functions.

Next downstream is gen_story/2, which runs as part of a sub-workflow and receives slightly different arguments. The first argument is an element from the enumerable (in this case, the index), followed by the accumulated context:

def gen_story(index, %{authors: authors, topic: topic}) do
  author = Enum.at(authors, index)

  LLM.chat!("""
  You are #{author}. Write a one paragraph story about "#{topic}" including a title.
  """)
end

The gen_image/2 function is another sub-workflow step that receives two arguments. Since the images step depends on both authors and stories, it has access to their outputs in the context map. This allows it to generate an illustration specifically tailored to each author's story (um, usually, it stinks at Shel Silverstein):

def gen_image(index, %{authors: authors, stories: stories}) do
  author = Enum.at(authors, index)
  story = Map.get(stories, to_string(index))

  LLM.image!("""
  You are #{author}. Create an illustration for the children's story: #{story}
  """)
end

Both gen_title/1 and gen_cover/1 follow a similar structure, using output from dependencies and making LLM calls, so we'll omit them here. Finally, the print/1 function pulls it all together:

def print(context) do
  context.images
  |> Enum.map(fn {idx, url} -> {url, "image_#{idx}.png"} end)
  |> Enum.concat([{context.cover, "cover.png"}])
  |> Enum.each(fn {url, path} -> Req.get!(url, into: File.stream!(path)) end)

  @template
  |> EEx.eval_string(Keyword.new(context))
  |> then(&File.write!("story.md", &1))
end

We love it when a plan comes together.

What About the Output?

How entertaining is a story-generating pipeline without some sample output? (Very little, unless you really like mermaid diagrams. You know who you are 😉)

Here's a sample chapter that FireSaga generated about the topic "bento box lunches" in the style of Dav Pilkey:

Captivating stuff. Gummy candies and veggie heroes you say!?

Workflows in Action

So, that's cascading workflows. They eliminate all the boilerplate of multiple workers and collocate an entire pipeline of functions into a single module. It's a much more convenient - dare we say, "elegant" - way to build workflows.

If you'd like to experiment with FireSaga yourself, check out the repository. (You'll need two things to get started—an Oban Pro license and some OpenAI keys).

⚠️ This release includes a required (and extremely helpful) migration. Follow the upgrade guide closely!

First up, our favorite addition...

🗂️ Sub-Workflows

Workflows gain powerful new capabilities for organizing complex job relationships with two major enhancements: add_workflow/4 and add_many/4.

Use add_workflow/4 to nest entire workflows within others to create hierarchical job dependencies. This makes complex workflows more maintainable by grouping related jobs together:

extr_flow = 
  Workflow.new(workflow_name: "extract")
  |> Workflow.add(:extract, WorkerA.new(%{source: "database"}))
  |> Workflow.add(:transform, WorkerB.new(%{}), deps: :extract)

# Add sub-workflow as a dependency
Workflow.new()
|> Workflow.add(:setup, WorkerC.new(%{mode: "initialize"}))
|> Workflow.add_workflow(:extract, extr_flow, deps: :setup)
|> Workflow.add(:finalize, WorkerC.new(%{}), deps: :extract)

Workflows can depend on other workflows, and downstream deps will wait until the sub-workflow completes before executing.

Need to run similar jobs in parallel? Use add_many/4 to add multiple jobs with a single dependency name:

# Add multiple email jobs that can run in parallel
email_jobs = Enum.map(users, &EmailWorker.new(%{user_id: &1.id}))

workflow =
  Workflow.new()
  |> Workflow.add_many(:emails, email_jobs)
  |> Workflow.add(:report, ReportWorker.new(), deps: :emails)

The add_many/4 step creates a sub workflow from either a list or a map, and the full recorded results can be extracted with a single call:

def process(job) do
  map_of_results = Workflow.all_recorded(job, with_subs: true)
end

🖼️ Context and Workflow Status

Workflows that rely on common data can now share data without duplicating arguments using put_context/3:

workflow = 
  Workflow.new()
  |> Workflow.put_context(%{user_id: 123, app_version: "1.2.3"})
  |> Workflow.add(:job_a, WorkerA.new(%{}))
  |> Workflow.add(:job_b, WorkerB.new(%{}))

# Later in a worker:
def process(job) do
  context = Workflow.get_context(job)
  # Use context map...
end

It's now easier to check workflow progress with status/1, which provides execution stats:

%{
  total: 5,
  state: "executing", 
  counts: %{completed: 3, executing: 2},
  duration: 15_620
} = Workflow.status(workflow_id)

🧰 Queue Partitioning Overhaul

Queue partitioning is completely redesigned for dramatic performance improvements. Jobs are now assigned partition keys on insert rather than at runtime, enabling more efficient querying and eliminating head-of-line blocking when one partition has many jobs.

The new design has none of the issues of the previous solution:

• Job processing is completely fair. Jobs from a single partition can't block processing of other partitions after bulk insert. No priority or scheduling workarounds are necessary.
• Querying in partitioned queues relies on a single, partial index
• Partitioning uses a single, optimized query without any unions or dynamic sections. That allows ecto to prepare and cache a single plan for faster fetching and less memory usage.

In a benchmark of 10k jobs spread across 20 partitions (200k jobs), processing took 17s in v1.6, down from 360s in v1.5 (20x faster) with far less load on the database.

🧨 Global Burst Mode

Global partitioning gained an advanced feature called "burst mode" that allows you to maximize throughput by temporarily exceeding per-partition global limits when there are available resources.

Each global partition is typically restricted to the configured allowed value. However, with burst mode enabled, the system can intelligently allocate more jobs to each active partition, potentially exceeding the per-partition limit while still respecting the overall queue concurrency.

This is particularly useful when:

1. You have many potential partitions but only a few are active at any given time
2. You want to maximize throughput while maintaining some level of fairness between partitions
3. You need to ensure your queues aren't sitting idle when capacity is available

Here's an example of a queue that will 5 jobs from a single partition concurrently under load, but can burst up to 100 for a single partition when there is available capacity:

config :my_app, Oban,
  queues: [
    exports: [
      local_limit: 100,
      global_limit: [allowed: 5, burst: true, partition: [args: :tenant_id]]
    ]
  ]

🍋 Preserve DynamicQueues Updates

DynamicQueues now preserves queue changes made at runtime across application restarts. This brings two key improvements:

1. Runtime changes to queues (via Web or CLI) persist until explicitly changed in configuration
2. A new :automatic sync mode that can manage queue deletions based on configuration

# Automatic mode - Deletes queues missing from configuration
config :my_app, Oban,
  plugins: [{DynamicQueues, sync_mode: :automatic, queues: [...]}]

In automatic mode, any queue that exists in the database but isn't defined in the configuration will be automatically deleted during startup. This is useful when you want to ensure your runtime queue configuration exactly matches what's defined in your application config.

Now when you pause a queue through the dashboard or change its limits via API, those changes will persist across application restarts until you explicitly update those options in your configuration.

🎨 Decorated Job Enhancements

Decorated jobs gain a few new capabilities. You can now use current_job/0 to access the underlying job struct from within decorated functions, making it easier to work with job context or pass job details to other functions. Additionally, you can mark any decorated job as recorded at runtime with the recorded option, enabling workflow composition and return value access without separate modules.

defmodule Business do
  use Oban.Pro.Decorator

  @job queue: :default, recorded: true
  def process_account(account_id) do
    job = current_job()

    IO.puts("Processing account #{account_id} with job #{job.id}")

    {:ok, %{processed: true}}
  end
end

Take a look through the complete changelog, or check out the v1.6 Upgrade Guide for complete upgrade steps and migration caveats.

Open Sourcing Oban Web

📯 Oban Web is now open source and free (as in champagne 🥂)! From now on, Oban Web will be published to Hex and available for use in all your Oban powered applications.

Why? Here's some history.

Our original Oban announcement on the Elixir Forum included a screenshot of Oban Web. That was nearly six years ago now, and boy have we come a long way. Originally launched as a private beta, Web was our first foray into building a viable business model for Oban. Years ago we realized our customers needed a more complex set of tools. Thus, we released Oban Pro. Pro is where the serious business happens now, and we want more people to have Web available from the start.

If you're a current customer, or wonder what this means for our existing subscriptions, bounce to the Simplified Plans section.

Let's cover Oban v2.19 first, then get back to more Web news.

Oban v2.19

The latest Oban release includes a new database engine, an installer for improved UX, better logging, and more. Here are some of the highlights.

🐬 MySQL Support

Oban officially supports MySQL with the new Dolphin engine. Oban works with modern (read "with full JSON support") MySQL versions from 8.4 on, and has been tested on the highly scalable Plantescale database.

Running on MySQL is as simple as specifying the Dolphin engine in your configuration:

config :my_app, Oban,
  engine: Oban.Engines.Dolphin,
  queues: [default: 10],
  repo: MyApp.Repo

With this addition, Oban can run in an estimated 10% more Elixir applications!

⚗️ Automated Installer

Installing Oban into a new application is simplified with a new igniter powered mix task. The new oban.install task handles installing and configuring a standard Oban installation, and it will deduce the correct engine and notifier automatically based on the database adapter.

mix igniter.install oban

This oban.install task is currently the recommended way to install Oban. As a bonus, the task composes together with other igniter installers, making it possible to install phoenix and oban with a single igniter.new command:

mix archive.install hex igniter_new

mix igniter.new oban_demo \
  --with phx.new \
  --with-args="--database postgres" \
  --install oban

📔 Logging Enhancements

Logging in a busy system may be noisy due to job events, but there are other events that are particularly useful for diagnosing issues. A new events option for attach_default_logger/1 allows selective event logging, so it's possible to receive important notices such as notifier connectivity issues, without logging all job activity:

Oban.Telemetry.attach_default_logger(
  events: ~w(notifier peer stager)a
)

Along with filtering, there are new events to make diagnosing operational problems easier.

A peer:election event logs leadership changes to indicate when nodes gain or lose leadership. Leadership issues are rare, but insidious, and make diagnosing production problems especially tricky.

[
  message: "peer became leader",
  source: "oban",
  event: "peer:election",
  node: "worker.1",
  leader: true,
  was_leader: false
]

Activity for all official plugins is now logged via plugin:stop and plugin:exception events. That includes runtime information to help monitor activity and diagnose issues. For example, every time Cron runs successfully it will output details about the execution time and all of the inserted job ids:

[
  source: "oban",
  duration: 103,
  event: "plugin:stop",
  plugin: "Oban.Plugins.Cron",
  jobs: [1, 2, 3]
]

There are other notable changes such as the new Oban.check_all_queues/1 function, starting queues in parallel on init, and official JSON integration. Read more in the full Changelog.

Oban Web v2.11

Beyond preparations for open sourcing, Web v2.11 includes substantial work toward component cleanup and unification. Here are some of our favorite new features:

🐬🪶 MySQL and SQLite

All official engines are fully supported. Listing, filtering, ordering, and searching through jobs works for Postgres, MySQL, and SQLite. That includes the particularly gnarly issue of dynamically generating and manipulating JSON for filter auto-suggestions! Nested args queries, such as args.address.city:Edinburgh work equally well with each engine.

🎛️ Instance Select

The dashboard will now support switching between Oban instances.

A new instance select menu in the header allows switching between running Oban instances at runtime. There's no need to mount additional dashboards in your application's router to handle multiple instances. The most recently selected instance persists between mounts.

- oban_dashboard "/oban_a", oban_name: Oban.A
- oban_dashboard "/oban_b", oban_name: Oban.B
- oban_dashboard "/oban_c", oban_name: Oban.C
+ oban_dashboard "/oban"

This also eliminates the need for additional router configuration. The dashboard will select the first running Oban instance it finds (with a preference for the default Oban).

- oban_dashboard "/oban", oban_name: MyOban
+ oban_dashboard "/oban"

☯️ Rebuilt Queues Page

The queue and jobs tables are fully rebuilt with shared, reusable components and matching functionality.

This enables far more powerful interaction with queues:

• Uniform Navigation - click on any part of the queue row to navigate to details.

• Sidebar - a new queue sidebar shows status counts and enables filtering by statuses such as paused or terminating.

• Filtering - queues are auto-complete filterable just like jobs, making it possible to find queues running on a particular node or narrow down by status.

• Shared Sorting - queue sorting now behaves identically to jobs, through a shared dropdown.

• Condensed Rows - simplify the queue page by removing nested row components. Extra queue details are in the sub-queue page.

🕯️ Operate on Full Selection

Apply bulk actions to all selected jobs, not just those visible on the current page.

This expands the select functionality to extend beyond the current page and include all filtered jobs, up to a configurable limit. The limit defaults to 1000 and may be overridden with a resolver callback.

🪪 Licensing and Installation

Oban Web v2.11 is licensed under Apache 2.0, just like Oban and Elixir itself. Previous versions are commercially licensed, therefore private, and won't be published to Hex.

See the updated, much slimmer, installation guide to get started.

Oban Met v1.0

Oban Met is the secret sauce that powers the charts and runtime details shown in the Oban Web dashboard. It is a distributed, compacting, multidimensional, telemetry-powered time series datastore (Zang!) for Oban that requires no configuring. It gathers data for queues, job counts, execution metrics, active crontabs, historic metrics, and more.

Web is virtually useless without Met, so we've open sourced it as well ✨!

While Met is designed for use with Web, it's still mighty useful on its own for accessing runtime metrics and counts. Take a look at the docs for some useful examples.

Simplified Plans

As of today, Web and Web+Pro subscriptions are no longer available. Only Oban Pro.

Existing web subscriptions and license keys will continue working until you upgrade Web to v2.11+. However, you'll continue paying for your subscription until you're ready to upgrade (those are private packages that we're serving still).

Legacy web customers will receive an email with an exclusive coupon to upgrade to Pro continue receiving support and to beneift from lush features such as: workflows, chunks, decorators, structured args, hooks, global limits, dynamic plugins, and much more.

Special thanks to all of our customers ❤️‍🔥 that supported Oban and its ecosystem for the past five years. You are integral to us making open source Oban, and now Web, possible.

On this our last release of the week, we're giving you the jazz-hands-big-finish.

Hybrid Compositions

It's time to put all of the week's goodness together!

The demo builds a pseudo video processing and object detection workflow. Each job in the workflow is a decorated function which uses FLAME internally for elasticity. Then, the workflow is wrapped in a batch to get lifecycle notifications for the entire workflow.

Let's walk through it, without narration, to keep it consistent. (Sometime soon, we'll re-record the demos from this week with narration):

Oban v2.18.0 and Pro v1.5.0-rc.0

To ensure a smooth transition, due to this being a behemoth of a Pro release, we're issuing a Pro v1.5 release candidate. Please download, give it a try, and report any bugs or rough spots.

There's also an Oban v2.18 companion release that brings queue shutdown telemetry, job observability additions, and changes to aid in the distributed PostgreSQL we highlighted earlier this week.

What Next?

We're home free! Read the complete changelog for all the features big and small that we didn't cover. If you're interested in trying out the RC, check out the upgrade guide. We're crushing the summer and in chasing the dream of bringing you better features and optimizations with Oban.

Hope to see you at Elixir Conf in Orlando August 27-30!

More Launch Week

• Day 1 — Unified Migrations, Preemptive Chaining, and Worker Aliases
• Day 2 — Enhanced Uniqueness and Distributed PostgreSQL
• Day 3 — Job Decorators
• Day 4 — Overhauled Batches and Improved Workflows

On today's menu, Workers gone wild! We're surveying the next evolution of Pro's primary compositioning tools.

Overhauled Batches

First up, we've reimagined Batches. One of Pro's original three features, batches link the execution of many jobs as a group and run optional callback jobs after jobs are processed.

Composing batches used to rely on a dedicated worker, one that couldn't be composed with other worker types. Now, there's a stand alone Oban.Pro.Batch module that's used to dynamically build, append, and manipulate batches from any type of job, and with much more functionality.

Batches gain support for streams (creating and appending with them), clearer callbacks, and allow setting any Oban.Job option on callback jobs.

Check it out:

Improved Workflows

Workflows began the transition from a dedicated worker to a stand-alone module several versions ago. Now that transition is complete, and workflows can be composed from any type of job.

All workflow management functions have moved to a centralized Oban.Pro.Workflow module. An expanded set of functions, including the ability to cancel an entire workflow, conveniently work with either a workflow job or id, so it's possible to maneuver workflows from anywhere.

Perhaps the most exciting addition, because it's visual and we like shiny things, is the addition of mermaid output for visualization. Mermaid has become the graphing standard, and it's an excellent way to visualize workflows in tools like LiveBook.

Take a look:

More Launch Week

• Day 1 — Unified Migrations, Preemptive Chaining, and Worker Aliases
• Day 2 — Enhanced Uniqueness and Distributed PostgreSQL
• Day 3 — Job Decorators
• Day 5 — Hybrid Compositions and Release Day

One more day to our Oban release week spectacular! Please stop by tomorrow, when we'll put it all together.

Today, we're focused on a single, powerful new feature. One that simplifies how applications can define, configure, and compose jobs.

Job Decorator

The new Oban.Pro.Decorator module converts functions into Oban jobs with a teeny-tiny @job true annotation. Decorated functions, such as those in contexts or other non-worker modules, can be executed as fully fledged background jobs with retries, priority, scheduling, uniqueness, and all the other guarantees you have come to expect from Oban jobs.

See the decorator in action:

Decorated jobs are a convenient way to run functions in the background, but there's abundance within.

Advanced Decorators

The @job decorator also supports most standard Job options, validated at compile time. As expected, the options can be overridden at runtime through an additional generated clause. Along with generated insert_ functions, there's also a new_ variant that be used to build up job changesets for bulk insert, and a relay_ variant that operates like a distributed async/await.

Any Elixir term may be passed as an argument, not just JSON encodable values. That enables passing native data-types such as tuples, keywords, or structs that can't easily be used in regular jobs.

Finally, the generated functions also respect patterns and guards, so you can write assertive clauses that defend against bad inputs or break logic into multiple clauses.

Check it out:

Imagine the possibilities when you combine the flexibility of decorated functions with distributed execution with FLAME. Dramatic pause... You can take a standard Elixir function and make it asynchronous, persistent, and scale it elastically with a few lines of code 🪄🧙. More on that soon...

More Launch Week

• Day 1 — Unified Migrations, Preemptive Chaining, and Worker Aliases
• Day 2 — Enhanced Uniqueness and Distributed PostgreSQL
• Day 4 — Overhauled Batches and Improved Workflows
• Day 5 — Hybrid Compositions and Release Day

See you again tomorrow!

Today, we're showcasing a couple of database driven features that expand where Pro can be deployed and drastically improve performance while reducing database load.

Distributed PostgreSQL

There were a smattering of PostgreSQL features used in Oban and Pro that prevented it from running in distributed PostgreSQL clients such as Yugabyte.

A few table creation options prevented running the migrations due to unsupported database features. Then there were advisory locks, which are part of how Oban normally handles unique jobs, and how Pro coordinates queues globally.

We're festive with fist-bumps and delighted to report that we've worked around both of these limitations and it's possible to run Oban and Pro on Yugabyte with most of the same functionality as regular PostgreSQL (global, rate limits, queue partitioning).

Take a look:

Running Yugabyte is also possible with the upcoming Oban release, but without any unique job support.

Enhanced Uniqueness

Oban's standard unique options are robust, but they require multiple queries and centralized locks to function. Not to mention, the period controls cause more confusion than they're worth...most of the time.

Now Pro supports an simplified, opt-in unique mode designed for speed, correctness, scalability, and simplicity. The enhanced hybrid and simple modes allows slightly fewer options while boosting insert performance 1.5x-3.5x, from reducing database load with fewer queries, improving memory usage, and staying correct across multiple processes/nodes.

See the new mode in action:

More Launch Week

• Day 1 — Unified Migrations, Preemptive Chaining, and Worker Aliases
• Day 3 — Job Decorators
• Day 4 — Overhauled Batches and Improved Workflows
• Day 5 — Hybrid Compositions and Release Day

See you tomorrow!

Throughout the week we'll be highlighting and demoing our favorite features of this upcoming release. We got carried away with this Pro release, and there's far too much to pack into a single announcement.

As you'll see over the course of the week, it's a paradigm shift for Oban Pro that opens up the way you write workers, configure queues, and compose jobs. Check back daily as we unroll this behemoth of a release.

Unified Migrations

Oban has had centralized, versioned migrations from the beginning. When there's a new release with database changes, you run the migrations and it figures out what to change on its own. Pro behaved differently for reasons that made sense when there was a single producers table, but it doesn't track with multiple tables and custom indexes.

We've solved that problem.

Now Pro has unified migrations to keep all the necessary tables and indexes updated and fresh, and you'll be warned at runtime if the migrations aren't current.

Here's an example of the migration experience:

Chained Jobs

Switching workflows from reactive to preemptive, where jobs won't execute until they're ready, was a massive speed boost in Pro v1.4. Afterwards we thought, "aren't chains really infinite linear workflows with automatic links between jobs?" Let's do the same thing for chains!

Much easier said than done, but absolutely worth it (we'll spare you the hours of discussion we had about this in the heat). Preemptive chaining doesn't clog queues with waiting jobs, and it chews through a backlog without any polling.

Chains are also a standard Oban.Pro.Worker option now. There's no need to define a chain specific worker, just add the option and you're guaranteed a FIFO chain of jobs.

Take a look at how easy it is to chain jobs:

This is the first of several changes to workers that open up job composition possibilities. We'll have much more on that later 😉.

Worker Aliases

Module names change as applications grow, and workers are no exception. Renaming a worker module when there are jobs queued up causes a flood of failing jobs, and it's a common foot gun.

Worker aliases solve the perennial issue of breaking existing jobs when workers are renamed.

Check it out:

It's a small quality of life feature, but one we're sure will help teams grow their Oban powered applications.

More Launch Week

• Day 2 — Enhanced Uniqueness and Distributed PostgreSQL
• Day 3 — Job Decorators
• Day 4 — Overhauled Batches and Improved Workflows
• Day 5 — Hybrid Compositions and Release Day

Meet us back here tomorrow!

As their summary put it:

Today you get Sorentwo for the price of one! We are joined by Shannon & Parker Selbert, both halves of the mom-and-pop software shop behind Oban, the robust job processing library that’s been delivering our emails & processing our audio for years.

Listen to the full episode of Changelog & Friends 35: The Oban Pros below (we'll add a transcript when one's available).

Most production apps, at least those ran by people that care about their well-being, use an error monitor like Sentry, HoneyBadger, AppSignal, etc. to notify attentive devs when an error occurs.

For Oban, identifying precisely which job generated which error can be difficult without additional context and some custom grouping. Error reporters are tailored toward reporting errors for web requests or more blatant exceptions. Fortunately, with a few careful reporting tweaks, we can make job error reports just as detailed and actionable.

🙋 Why Focus on Sentry?

This article uses Sentry for its examples because it's the most widely used official Elixir client. AppSignal and HoneyBadger are excellent alternatives with comparable mechanisms for all of the tips shown below.

Attaching the Error Handler

The standard Oban playbook outlines how to report exceptions via a standard telemetry event. Oban Pro makes it easier, and more reliable, with a global work hook. In either case, you have all the necessary information normalized and ready to report.

A basic handler looks something like this:

def handle_event([:oban, :job, :exception], _measure, %{job: job}, _conf) do
  %{reason: exception, stacktrace: stacktrace} = job.unsaved_error

  Sentry.capture_exception(exception, stacktrace: stacktrace)
end

Error details including the reason, kind, and stacktrace are stored in a job's virtual unsaved_error field. That's ultimately what is formatted and stashed in the errors field that's saved to the database. The same fields are available in the telemetry event's meta, but pulling it from unsaved_error works with a global hook as well.

Returned tuples like {:error, :boom} or {:cancel, :boop} are standardized as a Oban.PerformError and crashes are converted to an Oban.CrashError.

Normalization makes reporting simpler and removes the need to carefully inspect the error for reporting. It also means we can always use Sentry.capture_exception/2, because all error reasons are converted to exceptions. Unfortunately, normalization can also lead to incorrectly grouping unrelated exceptions together.

Fingerprinting Errors

Exceptions with the same name, no stacktrace, and slightly differing messages look the same to an error reporter. From the Sentry docs on error grouping:

By default, Sentry will run one of our built-in grouping algorithms to generate a fingerprint based on information available within the event such as stacktrace, exception, and message.

Without better hinting you'll find yourself with thousands of unrelated PerformError reports from various jobs all grouped together when they don't have anything in common. That's not helpful. It makes debugging harder and masks real errors since you only receive an email on an error's first occurrence.

That's why Sentry, and other error reporting tools, provide a fingerprint parameter to give hints to the service about how things should be grouped together. A fingerprint built from the job's worker and the exception module is granular enough to separate the same error from different workers:

fingerprint = [inspect(exception.__struct__), inspect(job.worker)]
# ["Oban.PerformError", "MyApp.BusinessWorker"]

However, in jobs that make liberal use of error tuples, the exception will always be Oban.PerformError, and we can use the exception's message to be more specific:

fingerprint = [
  inspect(exception.__struct__),
  inspect(job.worker),
  Exception.message(exception)
]
# ["Oban.PerformError", "MyApp.BusinessWorker", "record not found"]

Now add the fingerprint to the context options:

opts = [fingerprint: fingerprint, stacktrace: stacktrace]

Sentry.capture_exception(exception, opts)

Additional Context

Distinguishing between different error notifications is a start. The next step is injecting more contextual details into those notifications to help diagnose the issue and drill down to the root cause. The primary mechanisms for better context are extra and tags maps.

The extra map is for custom, structured data. We can slice off a portion of the job's fields and pass those along. Any fields that will help identify the job and recognize a pattern are candidates for the extra map:

extra = Map.take(job, ~w(args attempt id max_attempts meta queue tags worker)a)
opts = [extra: extra, fingerprint: fingerprint, stacktrace: stacktrace]

Sentry.capture_exception(exception, opts)

The extra fields are now shown in each report:

Grouping with Tags

Tags are an even better way to identify related events because they're indexed and searchable. Fields like worker and queue, that are shared among many jobs, are perfect for tags.

tags = %{oban_worker: job.worker, oban_queue: job.queue, oban_state: job.state}
opts = [extra: extra, fingerprint: fingerprint, stacktrace: stacktrace, tags: tags]

Sentry.capture_exception(exception, opts)

While there's little chance the names will conflict with existing tags, prefixing them with oban_ keeps them grouped and distinct. Now the tags are displayed with the runtime and server information:

Consistent Stacktraces

Erlang/Elixir stacktraces are a finicky beast due to tail call optimization and automatic truncation. They're elusive outside of a catch block. A useful stacktrace is only provided when an exception or crash is caught. Error tuple returns don't have any associated stacktrace, and we can't retrieve one that's of any use.

Sentry, and other reporters, show the last stacktrace entry in the report title. Fortunately, all we really need is an entry for the worker and process/1 (or perform/1 for standard workers), which is trivial to build manually:

stacktrace =
  case {Oban.Worker.from_string(job.worker), stacktrace} do
    {{:ok, worker}, []} -> [{job.worker, :process, 1, []}]
    _ -> stacktrace
  end

Sentry requires that the first element of a stacktrace is a module, aka an atom. Note the use of from_string/1 to safely convert the worker string to a module name while guarding against a missing module.

Having a relevant stacktrace entry makes it possible to spot the worker while scanning through notices. In the screenshot below, the top entry has a trace and the bottom doesn't:

Finishing Up

Perhaps you're thinking "those are all useful tips, why not make a library out of it and save us some time?" Well, there are many error reporting services out there, and we don't want to play favorites too much.

Besides, there are different takes on the exact components of a fingerprint, various approaches toward tags, and you may have additional context that's important to submit. A one-size-fits-all solution would be limiting.

Appendix: Putting it All Together

Here's one last example that puts all the tips together in a single block you can use as a starting point:

def handle_event([:oban, :job, :exception], _measure, %{job: job}, _conf) do
  %{reason: exception, stacktrace: stacktrace} = job.unsaved_error

  fingerprint = [
    inspect(exception.__struct__),
    inspect(job.worker),
    Exception.message(exception)
  ]

  stacktrace =
    case {Oban.Worker.from_string(job.worker), stacktrace} do
      {{:ok, worker}, []} -> [{job.worker, :process, 1, []}]
      _ -> stacktrace
    end
  
  extra = Map.take(job, ~w(args attempt id max_attempts meta queue tags worker)a)
  tags = %{oban_worker: job.worker, oban_queue: job.queue, oban_state: job.state}
  opts = [extra: extra, fingerprint: fingerprint, stacktrace: stacktrace, tags: tags]
  
  Sentry.capture_exception(exception, opts)
end

Performance matters, particularly for a background job system so closely integrated with your application's database. That's why recent Oban Pro releases have focused extensively on optimization—to ensure Oban scales right along with your system as traffic increases.

Today, we'll present two recent Pro features designed to alleviate database bloat and dramatically reduce overall transactions.

Bundled Acking

Job fetching is optimized to retrieve multiple jobs with a single query. However, "acking", the process of updating a job after it finishes executing, requires a separate query for each job. That puts stress on an application's shared Ecto pool and can cause contention when there aren't enough connections to go around. The pool can only scale so far because database connections aren't free.

One alternative approach is to make acking async. During the one million jobs a minute benchmark we experimented with async acking, but found the loss of consistency guarantees unacceptable. Since then, the issue of excessive transactions hasn't changed, so we revisited the optimization with Pro's Smart engine.

Async Acking in the Smart Engine

Async acking alleviates pool contention and increases throughput by bundling calls together and flushing them with a single database call. Acks are recorded with a lock-free mechanism and flushed securely in a single transaction, along with the next fetch query. A system of retries and careful shutdown routines ensures acks are recorded to the database before shutdown, et voilà, async acking is as consistent as the synchronous version.

Comparing Queries Between Engines

The following chart (populated with this script) compares how many queries are required to process a varying number of jobs using Oban's Basic engine and Pro's async Smart engine. The concurrency limit is set to 20 for both engines:

At 1,000 jobs the Basic engine uses 1,157 queries (1k acks and 157 fetches) across 1,053 transactions, while the Smart engine only uses 214 queries over 55 transactions, and the disparity grows from there. The higher a queue's throughput, and the higher the concurrency limit, the higher the reduction in queries.

Async acking works transparently, without any configuration changes, and it just shipped in Pro v1.3. Fewer queries, fewer transactions, and less pool contention with a seamless upgrade.

Table Partitioning

As is standard for any update-heavy Postgres workload, processing jobs at scale accumulates a lot of dead tuples, e.g. rows that were deleted or updated but not physically removed from the table. It's not until a CPU hungry vacuum process flags them as available for re-use that the rows are cleaned up—even then, they still occupy space on disk.

In contrast, table partitioning, where a table is broken into logical sub-tables, has tremendous maintenance advantages. Tables are smaller, vacuuming is faster, and there's less bloat after vacuuming. The best part—bulk deletion, aka pruning, is virtually instantaneous.

Partitioning the Jobs Table

Pro v1.2 shipped with support for partitioned tables managed by a DynamicPartitioner plugin and its companion migration. The oban_jobs table is strategically partitioned by job state and then sub-partitioned by date for cancelled, completed, and discarded states. The DynamicPartitioner is then responsible for the daily maintenance of creating and deleting new sub-partitions.

Frequent queries such as job fetching are faster for high-volume tables because they're scoped to smaller, active partitions. Also, since dropping a partition immediately reclaims storage space, pruning completely avoids the vacuum overhead from bulk deletes.

Benchmarking the Partitioned Table

A synthetic benchmark of 1m jobs a day for 7 days in Postgres 15 showed outstanding improvements. Partitioning faired better in every category.

The combined tables were 40% smaller (6,281MB to 4,121MB), with 95% less bloat after vacuum (4,625MB to 230MB):

Indexes were 37% smaller before vacuum (3,265MB to 2,068MB), and 18% smaller afterwards (123MB to 101MB):

Finally, vacuuming was 2.5x faster (28,638ms to 11,529ms), reindexing was 2.1x faster (6,248ms to 2,939ms), and astoundingly, job pruning was over 1000x faster (51,170ms to 49ms). Partitioned pruning is so fast that it isn't visible in the chart:

Partitioning has evolved since Oban was released 5 years ago, when we supported Postgres versions that didn't yet offer declarative partitioning. Since then, it has matured with every release and it massively improves Oban's ability to handle high-volume workloads with minimal bloat.

Keep on Scaling

The Elixir community is flourishing and we have impressive applications running millions of Oban jobs a day. Postgres' capabilities keep growing, Elixir gets better with each release, and we strive to keep refining Oban Pro to scale along with you.

Upgrade to Pro v1.3 for a tremendous reduction in database queries, and consider switching to partitioned tables to tackle bloat for high throughput systems.

📊 Realtime Charts

Charts for realtime metrics are enabled out of the box without any external dependencies. Charts are helpful for monitoring health and troubleshooting from within Oban Web because not all apps can, or will, run an extra timeseries or metrics database. And, because they're displayed alongside the original jobs, you can identify outliers in aggregate and then drill into individual jobs.

Highlights

• Select between execution counts, full counts, execution time, and wait time
• Aggregate time series by state, node, worker, or queue label
• Rollup metrics by time from 1s to 2m, spanning 90s to 3h of historic data
• Measure execution and wait times across percentiles, from 100th down to 50th at standard intervals

🔍 Filtering with Auto-complete

Filtering is entirely overhauled with a new auto-complete interface, new qualifier syntax, and vastly more performant queries. Full text searching with unindexable operators such as ilike and tsvector was removed in favor of highly optimized exact-match queries. With the new query syntax all searching is faster, and searching nested args is over 100x faster thanks to index usage.

One additional performance improvement for large oban_jobs tables is threshold querying. In order to minimize load on the application's database, only the most recent 100k jobs (approximately) are filtered. The 100k limit can be disabled or configured for each state, i.e. you could restrict filtering completed jobs but access the full history of cancelled jobs.

Highlights

• Filter by args, meta, node, priority, queue, tags, and worker
• Typeahead with keyboard shortcuts for focusing, selecting, and completing suggestions
• Highly optimized suggestion queries across a configurable number of recent jobs
• Locally cached for immediate feedback and minimal load on your application database
• Auto-completion of nested args and meta keys and values at any depth

⏱️ Metrics

The foundation of charts, filtering, optimized counts, and realtime monitoring is the new Oban.Met package. It introduces a distributed time-series data store and replaces both Oban.Plugins.Gossip and Oban.Web.Plugins.Stats with zero-config implementations that are much more efficient.

Highlights

• Telemetry powered execution tracking for time-series data that is replicated between nodes, filterable by label, arbitrarily mergeable over windows of time, and compacted for longer playback.
• Centralized counting across queues and states with exponential backoff to minimize load and data replication between nodes.
• Ephemeral data storage via data replication with handoff between nodes. All nodes have a shared view of the cluster's data and new nodes are caught up when they come online.

In the future Oban.Met modules will be public, documented, and available for use from your own applications.

❤️‍🩹 Upgrading

Most apps can upgrade by pinning to the latest release and removing unused plugins from their configuration. Oban Met starts supervised collectors automatically in fitting environments and the old plugins aren't necessary.

plugins: [
- Oban.Plugins.Gossip,
- Oban.Web.Plugins.Stats,
...

Point deps to the release candidate once the plugins are removed:

{:oban_web, "2.10.0-rc.2", repo: :oban}

💛 Until Next Time

We've planned some of these features since the first sketch of an Oban dashboard and are delighted to finally share them. Please Play with the demo, try giving the RC a spin, and share your feedback with us!

See the full Web Changelog for a complete list of changes, enhancements, and bug fixes.

First, it contains some mega-features and performance optimizations that shouldn't languish in a changelog (or the snazzy releases section).

Second, this is the final Pro release before the big v1.0! Yes, Pro was pre-1.0, but only in name. We've always treated Pro like a 1.0+ and avoided breaking changes. Now it's time to toss out some lingering deprecations and proudly signal that Pro is mature and stable.

Now, on to the features, starting with one that's been distilling for nearly three years.

⚖️ Horizontal Auto-Scaling Worker Nodes

The new DynamicScaler plugin monitors queue throughput and horizontally scales worker nodes to optimize processing. Horizontal scaling is applied at the node level, not the queue level, so you can distribute processing over more physical hardware. With auto-scaling, you can spin up additional nodes during load spikes, and pare down to a single node during a lull.

The DynamicScaler calculates an optimal scale by predicting the future size of a queue based on throughput per node, not simply available jobs. You provide an acceptable range of node sizes, which queues to track, and a cloud module—auto-scaling takes care of the rest (with observability, of course).

It's designed to integrate with popular cloud infrastructure like AWS, GCP, K8S, and Fly via a simple, flexible behaviour. For example, here we declare auto-scaling rules for two distinct queues and hypothetical AWS autoscaling groups:

{Oban.Pro.Plugins.DynamicScaler,
 scalers: [
   [queues: :reports, range: 0..1, cloud: {MyApp.Cloud, asg: "rep-asg"}],
   [queues: :exports, range: 1..4, cloud: {MyApp.Cloud, asg: "exp-asg"}]
 ]}

With that in place, the reports group can scale down to 0 when the queue is empty and up to 1 when there are jobs available. Meanwhile, exports will never scale below 1 node but can surge up to 4 nodes during a spike.

Beyond filters for scaling by a particular queue, there are scaling steps to optimize responsiveness, and tunable cooldown periods to prevent unnecessary scaling.

💸 Auto-Scaling Can Save Real Money

DynamicScaling based on queue activity is especially exciting because it can peel dollars off your cloud hosting bill. Depending on your scale, auto-scaling can pay for your Pro license by itself (and then some). This table explores the dollars saved by scaling down 16 hours a day, 30 days a month, during off-peak hours across hefty instances on popular clouds:

ℹ️ Based on prorated on-demand prices captured in April, 2023

Imagine the potential savings from only scaling up production workers to meet demand, or spinning staging instances over night!

🏗️ Args Schema for Structured Workers

Structured args are indispensable for enforcing an args schema. However, the legacy keyword list syntax with field: :type, implicit enums, and an asterisk symbol for required fields was simply awkward. We're correcting that awkwardness with a new args_schema/1 macro for defining structured workers. The args_schema macro defines a DSL that's a subset of Ecto's schema, optimized for JSON compatibility and without the need for dedicated changeset functions.

Structured args are validated before they're inserted into the database, and cast into structs with defined fields, atom keys, enums, and nested data before processing.

Here's a schema that demonstrates multiple field types, the required option, enums, and embedded structures:

use Oban.Pro.Worker

alias __MODULE__, as: Args

args_schema do
  field :id, :id, required: true
  field :mode, :enum, values: ~w(enabled disabled paused)a

  embeds_one :data do
    field :office_id, :uuid, required: true
    field :addons, {:array, :string}
  end

  embeds_many :address do
    field :street, :string
    field :city, :string
    field :country, :string
  end
end

@impl Oban.Pro.Worker
def process(%{args: %Args{id: id, mode: mode} = args) do
  %Args.Data{office_id: office_id} = args.data
  [%Args.Address{street: street} | _] = args.addresses

  ...
end

The legacy (and legacy-legacy) syntax is still viable and it generates the appropriate field declarations automatically. However, we strongly recommend updating to the new syntax for your own sanity.

🍪 Chunk Partitioning

Previously, chunk workers executed jobs in groups based on size or timeout, with the grouping always consisting of jobs from the same queue, regardless of worker, args, or other job attributes. However, sometimes there's a need for more flexibility in grouping jobs based on different criteria.

To address this, we have introduced partitioning, which allows grouping chunks by worker and/or a subset of args or meta. This improvement enables you to methodically compose chunks of jobs with the same args or meta, instead of running a separate queue for each chunk.

Here's an example that demonstrates using GPT to summarize groups of messages from a particular author:

defmodule MyApp.MessageSummarizer do
  use Oban.Pro.Workers.Chunk,
      by: [:worker, args: :author_id],
      size: 100,
      timeout: :timer.minutes(5)

  @impl true
  def process([%{"author_id" => author_id} | _] = jobs) do
    messages =
      jobs
      |> Enum.map(& &1.args["message_id"])
      |> MyApp.Messages.all()

    {:ok, summary} = MyApp.GPT.summarize(messages)

    # Push the summary
  end
end

By leveraging the enhanced partitioning capabilities, you can now effectively group and process jobs based on specific criteria, providing a more streamlined approach to managing your workloads.

Chunk queries have been optimized for tables of any size to compensate for their newfound advanced complexity. As a result, even with hundreds of thousands of available jobs, queries run in milliseconds.

🗄️ Batch Performance

In short, batches are lazier, their queries are faster, and they'll put less load on your database. If query tuning and performance optimizations are your things, read on for the details!

Some teams run batches containing upwards of 50k and often run multiple batches simultaneously, chewing through millions of jobs a day. That load level exposed some opportunities for performance tuning that we're excited to provide for batches of all sizes.

• Debounce batch callback queries so that only one database call is made for each batch within a short window of time. By default, batching debounces for 100ms, but it's configurable at the batch level if you'd prefer to debounce for longer.

• Query the exact details needed by a batch's supported callbacks. If a batch only has a cancelled handler, then no other states are checked.

• Optimize state-checking queries to avoid using exact counts and ignore completed jobs entirely, as that's typically the state with most jobs.

• Use index-powered match queries to find existing callback jobs, again, only for the current batch's supported callbacks.

💛 That's a Wrap

All of the features in this bundle came directly from customer requests. Thanks for helping to refine Pro with your feedback and issues.

See the full Pro Changelog for a complete list of enhancements and bug fixes.

Layering Abstractions

Erlang, and therefore Elixir, provides a legendary concurrency story through lightweight processes (spawn) and message passing (send). Those two functions are technically all you need to build actor-model-based concurrency. If you really wanted to, you could build an entire application purely with spawn and send. Presumably, it would be tedious, and you'd slowly reimplement an ad-hoc, informally-specified, bug-ridden version of half of OTP.

While Erlang provides basic concurrency primitives, decades of in-the-field experience has guided the creation of elegant concurrency abstractions such as GenServer for long-lived generic processes, and Supervisor for maintaining trees of processes. So, rather than stitching systems together with spawn and send, applications are composed of standard, well-behaved GenServers and Supervisors.

Elixir provides ergonomic abstractions that simplify advanced patterns based on OTP's abstractions. For instance, Registry adds a process-aware wrapper around ETS tables, and Agent brings a GenServer tailored for state management. Then there is the Task module for one-off OTP-friendly processes.

What Tasks Are and What They Aren't

Tasks are a more powerful concurrency abstraction than bare spawned processes, making it simple to convert sequential code into concurrent code with varying guarantees. They're ideal for operations like fetching several URLs or querying the database in parallel. Depending on how tasks are initialized, they have a spectrum of responsibility between best-effort and loosely supervised:

1. Best-Effort (Task.start/1)—Tasks without process linking, supervision, concurrency controls, and no shutdown guarantees.

2. Supervised (Task.Supervisor.async/2)—Tasks with process linking, simple enumerability, hard concurrency limits, and configurable shutdown periods.

Supervised tasks improve observability, constrain resources, and provide shutdown guarantees. But, tasks lack essential functionality for mission-critical work. Consider the following:

• Enqueueing—How can I wait to execute a task when the supervisor hits a concurrency limit? How do I separate fast and slow tasks to prevent bottlenecks?

• Scheduling—How do I run a task at a specific time in the future? What if too many scheduled jobs all need to start simultaneously? How do I reschedule them?

• Retries—How do I restart tasks with transient failures? How do I delay and stagger retries with some backoff to prevent concurrent access problems?

• Uniqueness—How do I prevent the same task from executing concurrently on the same node? What if I ran a task a few seconds ago, and the result is still usable?

• Distribution—How do I distribute tasks evenly between every node in my cluster? What if I only need to run tasks on some nodes?

• Instrumentation—How can I measure the run time for various tasks and integrate them with my other application metrics?

• Runtime Visibility—What function and arguments is each task currently doing? How long has it been doing it?

• Historical Observability—What tasks are complete? When did they start and how long did they take?

That's a lot of missing functionality, and there's a more significant issue. Once you've implemented a solution for all of those missing pieces (or at least the parts you need right now) there is an essential component missing.

Persistence is Crucial

What happens when your application inevitably restarts, whether intentionally or from cascading failures? To retain tasks between restarts, you need persistent storage.

There are abundant persistence options from the Erlang native RabbitMQ to the inescapable Redis. Any persistent store could work with enough effort. However, the best fit, in our opinion, is PostgreSQL (surely not a surprise, as Oban says, "powered by modern PostgreSQL" right on the tin).

Aside from a well-earned reputation as a flexible, reliable, and highly performant relational database, PostgreSQL's killer feature is that it's probably in your application.

Persisting tasks, or at least a task-like wrapper, in a database neatly solves many of the problems we identified earlier:

• Enqueueing—With atomic operations, a SQL table can behave like a queue.

• Scheduling—With timestamps, we can defer execution until a specific time.

• Uniqueness—With persistence, we can query for duplicate tasks.

• Distribution—With a central database, nodes can pull tasks when ready.

• Historical Observability—With retention, we can look at completed tasks.

Coordinating with a database is certainly slower than spawning a BEAM process, but the upside of persistence is immense. Tasks can be enqueued atomically within the same transaction as your other application code. More importantly, you're assured that critical tasks won't disappear unexpectedly during a routine application restart.

Once foundational persistence is sorted, the other layers can fall into place.

Picking Up Where Tasks Leave Off

You could rebuild GenServers, Supervisors, Agents, Tasks, or a Registry, but they already exist in Elixir as a springboard for you to build on.

As Elixir builds on top of OTP, Oban expands on those primitives (and some phenomenal packages) to formalize how well-behaved, observable, reliable, and persistent tasks should operate. In fact, Oban links processes through a Registry, manages queues with a DynamicSupervisor, and executes every job within a supervised Task!

Even in an environment with the subjectively best concurrency story of any runtime, you still need additional functionality for mission-critical tasks.

That's where Oban starts.

Dive into Oban's docs to learn how Oban layers missing functionality on tasks and leverages persistence.

You know—and we know that you know—you're here to see some schmancy charts that back up the claim of "one million jobs a minute." Good news! We'll jump right into benchmark results and charts. Afterwards, if you're feeling adventurous, stick around to dig into the nitty-gritty of how much load Oban places on a system, how it minimizes PostgreSQL overhead, and our plans for eking out even more jobs per second.

The Benchmark

This is the story of a benchmark that demonstrates the jobs-per-second throughput that Oban is capable of. The benchmark aims to process one million jobs a minute, that's 16,1667 jobs/sec, in a single queue on a single node. Running multiple queues or multiple nodes can achieve dramatically higher throughput, but where's the fun in that?

Not to spoil the story, but it didn't take much to accomplish our goal. Oban is built on PostgreSQL, which has a stellar benchmarking story and widely documented results for write-heavy workloads, which is precisely Oban's profile.

Our benchmark inserts one million jobs and then tracks the time it takes to complete processing them all with a throughput measurement every second. (Reporting and metrics aren't shown, but you can find the original script in this gist).

# Start Oban without any queues
Oban.start_link(repo: Repo)

# Create an atomic counter and encode it to binary for use in jobs
cnt = :counters.new(1, [])
:ok = :counters.put(cnt, 1, 0)

bin_cnt =
  cnt
  |> :erlang.term_to_binary()
  |> Base.encode64()

# Insert 1m jobs in batches of 5k to avoid parameter limitations
Task.async_stream(1..200, fn _ ->
  Oban.insert_all(fn _ ->
    for _ <- 1..5_000, do: CountWorker.new(%{bin_cnt: bin_cnt})
  end)
end)

The CountWorker job itself simply deserializes the counter and atomically increments it:

defmodule CountWorker do
  use Oban.Worker

  @impl Oban.Worker
  def perform(%Job{args: %{"bin_cnt" => bin_cnt}}) do
    bin_cnt
    |> Base.decode64!()
    |> :erlang.binary_to_term()
    |> :counters.add(1, 1)
  end
end

Now, to process all one million jobs in a minute our queue needs to chew through at least 16,667 (⌈1,000,000 / 60⌉) jobs per second . Theoretically, if fetching and dispatching jobs was instantaneous, and there wasn't any throttling, setting concurrency to 34 (⌈16,667 jobs / (1000 ms / 2 cooldown)⌉) would get the job done.

# Start the queue that all jobs are inserted
Oban.start_queue(queue: :default, limit: 34, dispatch_cooldown: 2)

In practice, back in reality, the limit needs to be quite a bit higher to compensate for the actual work of fetching and dispatching jobs. This chart illustrates rerunning the benchmark with increasing concurrency until it tops one million completed jobs:

The magic concurrency number, on this machine 👇, is 230. It peaks at around 17,699 jobs/sec and finishes one million jobs in 57s.

👉 Erlang/OTP 25, Elixir 1.14.0-dev, PostgreSQL 14.2, Apple M1 Pro (10 Cores), 32 GB RAM, 1TB SSD

But what's the system load like, you ask? HTOP screen captures have the answer. First the single BEAM process:

And here's the Postgres load showing all 10 connections and the various daemons:

Cores 1-4 are 29%-46% engaged while the other six sit around like freeloaders with only a smattering of activity.

The Limits

Now that we know we can hit the one million jobs/min goal, why not find out how quickly we can do it?

With a single queue on a single node the only knobs we have to tweak processing speed are concurrency (limit) and cooldown (dispatch_cooldown). Concurrency determines how many jobs will run at once, while cooldown limits how frequently a queue fetches new jobs from the database.

The default cooldown period is 5ms, and the minimum is 1ms. Some measurement (er, trial and error) shows that a 2ms cooldown period is optimal because it strikes a balance between quick fetches and not thrashing the database. With cooldown dialed in, all that's left to explore is concurrency.

Running the benchmark again with increasing concurrency from 230-3,000 paints a telling picture:

The total time to complete 1m jobs decreases steadily until it bottoms out at around 2,000 concurrently. After that point fetching slows down, the BEAM gets overloaded, and overall processing time starts to rise. But, when concurrency is set to 2,000 it completes in 30 seconds—that's two million jobs a minute.

The benchmark also tracks the average jobs per second over time. If we dig out the peak jobs/sec for at each concurrency limit we get this companion chart:

Throughput tops out around 32,000 jobs/sec, over two billion jobs/day, and well beyond the realm of reason for a single queue. Real jobs do real work in an application, and that's where the real load comes from.

The Not-So Secret Sauce

There's a little extra sauce at work to reach such high throughput when concurrency is set at 2,000. Beyond Oban's core historic sauce—index-only scans for fetching, a single compound index, debounced fetching via cooldown—two new changes unlocked substantial performance gains.

Async Acking

While job fetching is optimized into a single query, "acking" when a job is complete requires a separate query for every job. That puts a lot of contention on the shared database pool and the associated PostgreSQL connections. Or at least it did, before the addition of an async acking option. Async acking trades consistency guarantees for throughput by batching calls together and flushing them with a single database call. The reduction in pool contention and database calls increases throughput by 200-210%.

Partitioned Supervisor

Perhaps you noticed earlier that the benchmarks are running with Elixir 1.14.0-dev? That's to make use of the unreleased (as of this post) PartitionSupervisor. Without partitioning, the Task.Supervisor that supervises all of the job processes is a bottleneck. Swapping in the PartitionSupervisor spins up one supervisor for each core and increases throughput 5-10%, virtually for free.

Neither of these changes are released or on main yet, but watch for a variant of them in the future.

The End

Oban's primary goals may be reliability, consistency, and observability, but we strive to keep it as speedy as possible. If Oban is your application's performance bottleneck, it should either be because your business is booming (congratulations 🎉), or there are insidious bugs at play (we have work to do 😓)!

On to the highlights, starting with Oban. Or, choose your own adventure, and go straight to Pro.

Oban v2.11

Centralized Leadership

Coordination between nodes running Oban is crucial to how many plugins operate. Staging jobs once-a-second from multiple nodes is wasteful; as is pruning, rescuing, or scheduling cron jobs. Prior Oban versions used transactional advisory locks to prevent plugins from running concurrently.

However, there were some issues:

• Plugins don't know if they'll take the advisory lock, so they still need to run a query periodically.

• Nodes don't usually start simultaneously, and time drifts between machines. There's no guarantee that the top of the minute for one node is the same as another's—chances are, they don't match.

Here's a chart that illustrates the difference in database queries per minute for idle queues between v2.10 and v2.11:

A new table-based leadership mechanism guarantees only one node in a cluster, where "cluster" means a bunch of nodes connected to the same Postgres database, will run plugins. That's why, in the chart above, Oban v2.11's query count stays so low—only one node is staging jobs.

If you're wondering why we're using an unlogged table for coordination rather than PubSub or Distributed Erlang, keep reading.

All will be revealed in the next section 🪄.

See the Upgrade Guide for instructions on how to create the peers table and get started with leadership. If you're curious about the implementation details or want to use leadership in your application, take a look at docs for the Oban.Peer module.

Alternative PG (Process Groups) Notifier

Oban relies heavily on PubSub, and until now it only provided a Postgres adapter. Postgres is magnificent, and has a highly performant PubSub option, but it doesn't work in every environment (we're looking at you, PG Bouncer 🤬).

Fortunately, many Elixir applications run in a cluster connected by distributed Erlang. That means Process Groups, aka PG, is available for many applications.

Side note: there's a subset of applications that use transaction pooling and don't have clustering—ahem, Heroku—and that's why we have table-backed leadership. Advisory lock based leadership would have been so simple...

So, we pulled Oban Pro's PG notifier into Oban to make it available for everyone! If your app runs in a proper cluster, you can switch over to the PG notifier with one line of configuration:

config :my_app, Oban,
  notifier: Oban.Notifiers.PG,
  ...

Now there are two notifiers to choose from, each with their own strengths and weaknesses:

Oban.Notifiers.Postgres

• Pros: Doesn't require distributed Erlang, publishes insert events to trigger queues
• Cons: Doesn't work with transaction pooling, doesn't work in tests because of the sandbox

Oban.Notifiers.PG

• Pros: Works PG Bouncer in transaction mode, works in tests
• Cons: Requires distributed Erlang, doesn't publish insert events

Use whichever notifier suits your needs best. Or, use them both in different environments. G'head, we do 😉.

Basic Lifeline Plugin

When a queue's producer crashes or a node shuts down before a job finishes executing, the job may be left in an executing state. The worst part is that these jobs—which we call "orphans"—are completely invisible until you go searching through the jobs table.

Oban Pro has always had a "Lifeline" plugin for just this occasion—and now we've brought a basic Lifeline plugin to Oban.

To automatically rescue orphaned jobs, include Lifeline in your configuration:

config :my_app, Oban,
  plugins: [Oban.Plugins.Lifeline],
  ...

Presto! Now Lifeline will search and rescue orphans after they've lingered for 60 minutes.

A crucial tidbit about the basic lifeline, from the docs:

The basic Lifeline plugin only checks execution time and it may transition jobs that are genuinely still executing; causing duplicate execution. For more accurate rescuing or to rescue jobs that have exhausted retry attempts you want the DynamicLifeline plugin.

Reindexer Plugin

Over time various Oban indexes (indeed, any indexes) may grow without VACUUM cleaning them up properly. When this happens, rebuilding the indexes will release bloat and free up space in your Postgres instance. Sure, you could handle that with a cron job, but we've made a plugin to take care of it for you.

config :my_app, Oban,
  plugins: [Oban.Plugins.Reindexer],
  ...

The Reindexer plugin automates index maintenance by periodically rebuilding all of your Oban indexes concurrently, without any locks. By default, reindexing happens once a day at midnight, and it's configurable with a standard cron expression.

config :my_app, Oban,
  plugins: [{Oban.Plugins.Reindexer, schedule: "@weekly"}],
  ...

Hopefully, Postgres will get its act together and index bloat will be a distant memory.

Oban Pro v0.10

This Pro release brings enhancements over some basic Oban functionality.

Encryption, Recording, and Structure

First off, there's a new Pro worker. Oban.Pro.Worker is a drop-in replacement for Oban.Worker with expanded capabilities such as:

• Encrypted—transparent encryption of args at rest for applications that handle sensitive data.

• Structured—struct backed jobs with key enforcement to prevent typos and codify args structure.

• Recorded—capture job output and stash it on the job for inspection, or use in downstream jobs for batches or workflows.

Yes, Batch, Chunk, and Workflow workers are based on the Pro worker, so you can use all of the Pro options there as well.

Here's a sample worker that's configured for encryption, recording, and enforced structure:

defmodule MyApp.SuperWorker do
  use Oban.Pro.Worker,
    queue: :super,
    encrypted: [key: {Application, :fetch_env!, :secret_key}],
    recorded: true,
    structured: [keys: [:id, :ssn, :pin], required: [:id]]

  @impl Oban.Pro.Worker
  def process(%Job{args: %__MODULE__{} = args}) do
    # Use the decrypted and structured args and record the result!
    MyApp.Business.predict(args.id, args.ssn, args.pin)
  end
end

Explore each new option in the Pro Worker guide.

Dynamic Queues

DynamicQueues is to basic queues as DynamicCron is to basic cron—that is, it adds persistence across restarts, globally, across all connected nodes.

DynamicQueues are ideal for applications that dynamically start, stop, and modify queues at runtime. For example, what if your application runs one queue per customer to isolate work? You wouldn't re-deploy a new app every time a customer signs up—it's easy to use DynamicQueues instead:

DynamicQueues.insert([customer_42: [global_limit: 20]])

If the limit turns out to be too high and greedy ol' customer 42 is hogging all the resources, scale them down (and rest assured that the scale will keep when the app restarts):

DynamicQueues.update(:customer_42, global_limit: 10)

There's also a declarative syntax for specifying which nodes a queue will run on. Here's a taste:

DynamicQueues.insert([
  basic: [limit: 10, only: {:node, :=~, "web|worker"}],
  audio: [limit: 20, only: {:node, :=~, "worker"}],
  video: [limit: 30, only: {:node, :=~, "worker"}],
  learn: [limit: 10, only: {:sys_env, "EXLA", "CUDA"}]
])

With that configuration basic will run on any web or worker node, audio and video will only run on workers, and learn will run anywhere that has EXLA configured.

Check out the DynamicQueues guide for installation, configuration, and a whole lot more usage examples.

Oban Web v2.9

This round of releases focused on Oban and Pro, but Oban Web got a little snuggle, too. Along with some minor bug fixes, Web works seamlessly with Pro's new worker features.

Pro Worker Support

Jobs that use Oban.Pro.Worker features like encryption, recording, and enforced structure now display an indicator on the details page. What's more, recorded jobs display the job's return value directly in the details page:

Handy for spot-checking a job's output right in the browser, isn't it?

That's a Wrap

All of the features in this bundle came directly from customer requests. Thanks for your feedback and issues. You're helping to refine Oban.

See the full Oban Changelog, Web Changelog, and Pro Changelog for a complete list of enhancements and bug fixes. Or, get started with the Oban v2.11 upgrade guide.

Today we'd like to give you a tour of how we use Oban in that app to run the dashboard demo, dogfood new features, asynchronously process customer payments, and handle critical webhooks.

As a brief aside, before we get into the fun stuff, the application itself is named "Lysmore." "Lys" is Norwegian for "light" and is an intentional misspelling of "Lismore," the island across the bay from Oban in Scotland. If you notice the Lysmore module in some code samples, that's why.

Running the Web Dashboard Demo

Undoubtedly, our most entertaining use of Oban is for the live Web dashboard demo. A playful combination of randomly generated workers using fake data and random failures makes the demo a chaotic simulation of a production workload. A typical generated worker looks something like this (with hardcoded values and inlined functions for simplicity):

defmodule Oban.Workers.WelcomeMailer do
  use Oban.Worker, queue: :mailers

  alias Faker.Internet

  def gen(opts \\ []) do
    args = %{
      email: Internet.email(),
      homepage: Internet.url(),
      username: Internet.user_name()
    }

    new(args, opts)
  end

  @impl Worker
  def perform(_job) do
    if :rand.uniform(100) < 15, do: raise(RuntimeError, "Something went wrong!")

    100..60_000
    |> Enum.random()
    |> Process.sleep()
  end
end

That worker will randomly error 15% of the time and take anywhere from 100ms to 60s to run, plenty of time for people to track progress, click into the details and possibly cancel the job.

Anybody on the internet can get a taste of the dashboard and possibly cause some harmless chaos of their own in the meantime—some miscreants love to pause queues or scale the concurrency down to one.

The demo is a beautiful canary because it uses the latest OSS, Web, and Pro releases, utilizing all the plugins and most available features. With error monitoring, we receive notifications that help us diagnose and fix issues from a constantly running production instance, often (but not always) before any customers report a problem! It's crowdsourcing and dogfooding rolled into one, leading us to...

Dogfooding Web and Pro Features

During development, we use Lysmore to mount the web dashboard in an actual Phoenix project. That is essential for complete integration tests and getting a sense of how well everything plays together. Through that integration, proven out and refined new features before they're released. Let's look at a few examples.

CSP (Content Security Policy)

Using a nonce for CSP (Content Security Policy) was a customer's security requirement that we verified locally. Fun fact, Phoenix's live reload iframe doesn't like CSP. Now the public demo loads a different nonce for every dashboard request in production, using this config in router.ex:

scope "/" do
  pipe_through :live_browser

  oban_dashboard "/oban",
    resolver: LysmoreWeb.Resolver,
    csp_nonce_assign_key: %{
      img: :img_csp_nonce,
      style: :style_csp_nonce,
      script: :script_csp_nonce
    }
end

Access Controls and Auditing

Access controls for authorization, authentication, and auditing landed in Web a little while ago. Obviously, we don't restrict access to the public demo, and all of the actions (deleting, pausing, scaling, etc.) are allowed. Regardless, we use a simple resolver module that is easily modified to restrict access while testing:

defmodule LysmoreWeb.Resolver do
  defmodule User do
    defstruct [:id, guest?: true, admin?: false]
  end

  @behaviour Oban.Web.Resolver

  @impl Oban.Web.Resolver
  def resolve_user(_conn), do: %User{id: 0, guest?: true}

  @impl Oban.Web.Resolver
  def resolve_access(_user), do: :all

  @impl Oban.Web.Resolver
  def resolve_refresh(_user), do: 1
end

In the future, if the dashboard supports more invasive operations, e.g., editing crontabs, we're prepared to disable some features.

Smart Engine

Pro's SmartEngine introduced oft-requested global concurrency and rate limiting, which are built on lightweight locks and required multiple nodes to exercise queue producer interaction. The public demo uses rate-limiting and global limiting for a couple of queues:

queues: [
  analysis: 20,
  default: 30,
  events: 15,
  exports: 8,
  mailers: [local_limit: 10, rate_limit: [allowed: 20, period: 30]],
  media: [global_limit: 10]
],

If you've ever noticed that the mailers queue has a lot of available jobs, that aggressive rate-limit is the reason.

When we deployed the SmartEngine, we briefly scaled our cluster up to 5 nodes to identify any bottlenecks. That proved fruitful because we identified a global concurrency deadlock and quickly shipped an improved algorithm that used some jitter.

Handling License Payments

In addition to the public Oban instance that powers the demo, we also run a separate private instance using a private prefix in PostgreSQL. That instance is entirely isolated and only runs a few queues where we integrate with Stripe to manage customers, attach payment methods, and create or update subscriptions.

Here is the full config for that instance, living right alongside the public config:

config :lysmore, ObanPrivate,
  engine: Oban.Pro.Queue.SmartEngine,
  repo: Lysmore.Repo,
  name: ObanPrivate,
  prefix: "private",
  queues: [default: 3],
  plugins: [
    Oban.Plugins.Gossip,
    Oban.Pro.Plugins.Lifeline,
    Oban.Web.Plugins.Stats
  ]

Note that there isn't any pruning involved. There are relatively few private jobs that run, and the data is vital for troubleshooting, so we hold on to it.

All of the Stripe interactions are essential for setting up payments. They're also possible failure points—anything can happen when we make an HTTP call to a third-party service, even one as reliable as Stripe. What happens if, as a customer signs up, we send invalid data, make an invalid API call, or we get rate-limited for too much activity (yeah, we wish)? We can't have that affect our customers.

Our solution for ensuring that we are resilient to failures is to wrap each operation in an idempotent job. As an example, let's look at the worker that handles updating customer details, upgrading their payment details, or changing their subscription:

defmodule Lysmore.Accounts.UpdateWorker do
  use Oban.Worker, unique: [period: 120]

  alias Lysmore.Accounts

  @impl Oban.Worker
  def perform(%{args: %{"id" => id, "plan" => plan}}) do
    {:ok, user} = Accounts.fetch_user(id)

    user =
      user
      |> attach_payment()
      |> update_customer()
      |> create_or_update_subscription(plan)

    {:ok, user}
  end

  def perform(%{args: %{"id" => id, "info" => info}}) do
    {:ok, user} = Accounts.fetch_user(id)

    update_customer(user, info)
  end

We handle either a plan change or a generic customer info change through the beauty of pattern matching. Each of the attach_payment/1 type functions is built to be idempotent—if a change already happened, then the function is a no-op. That way, if one step fails, the job can try again without duplicating changes. If you're looking at that and thinking, "that sure looks a lot like a workflow," keep reading!

Managing customers and taking payments is where Stripe integration starts. After that we rely on webhooks to manage licenses.

Handling Webhooks

After a subscription is created or canceled, Stripe sends a webhook to notify us. Much like payment processing, it is critical that we are resilient to failures and eventually grant or revoke a license. Granting a license sounds simple enough, but it involves four separate steps that either touch the database or make external calls.

Like payment processing, the steps must be idempotent—we don't want to create multiple licenses or deliver the same email repeatedly if something fails. However, unlike payment processing, we model webhook handling as a workflow.

Here you can see how the webhook controller's create/2 function matches on the webhook type and inserts a corresponding workflow:

def create(conn, %{"data" => %{"object" => data}, "type" => type}) do
  insert_workflow(type, data)

  send_resp(conn, 200, "")
end

defp insert_workflow("customer.subscription.created", data) do
  args = %{data: data}

  workflow =
    WorkflowWorker.new_workflow()
    |> WorkflowWorker.add(:hex, HexWorker.new(args))
    |> WorkflowWorker.add(:license, LicenseWorker.new(args))
    |> WorkflowWorker.add(:welcome, WelcomeWorker.new(args), deps: [:hex, :license])
    |> WorkflowWorker.add(:notify, NotifyWorker.new(args), deps: [:hex, :license])

  Oban.insert_all(ObanPrivate, workflow)
end

The workflow coordinates generating a legacy hex license, building a self-hosting license, delivering a welcome email with the new information, and notifying us that we have a new subscriber. There are similar workflows for cancellation and deletion, each of which is decomposed and simple to test in isolation.

Where the Magic Happens

Using Oban to build the licensing site is essential to walking in the shoes of our customers. We're running the same software that our customers are. That means we run into the same rough spots that they do, and when something goes wrong, we're plagued by the same bugs they are—with a rich incentive to expedite a fix. It's a fortunate situation; how many products can go all-in on themselves?

Thanks to everybody that hammers on the demo or signs up for a subscription. You're helping us refine Oban!

Batch Worker

The Batch worker was the original worker bundled with Pro. It allows applications to coordinate the execution of tens, hundreds or thousands of related jobs in parallel. Batch workers can define optional callbacks that execute as a separate job when any of these conditions are matched:

• all jobs in the batch are attempted at least once
• all jobs in the batch have completed successfully
• any jobs in the batch have exhausted retries or been manually cancelled
• all jobs in the batch have either a completed or discarded state

The concept is simple enough. How about an example?

Batch Example

Batches are ideal for map/reduce style operations where you need to parallelize many jobs across separate nodes and then aggregate the result.

Imagine you have a service that thumbnails images and then archives them on demand. While you could do all of the thumbnailing and archiving in a single job, it wouldn't scale horizontally across nodes and it'd lose all progress when the node restarts. Instead, you can model processing as a batch where each job thumbnails a single image and a callback generates the final archive.

We'll define a batch thumbnailer with callbacks for when the entire batch is completed or retries are exhausted:

defmodule MyApp.Workers.BatchThumbnailer do
  use Oban.Pro.Workers.Batch, queue: :media

  alias MyApp.{Account, Media}

  @impl true
  def process(%Job{args: %{"uuid" => uuid, "url" => url}}) do
    with {:ok, path} <- Media.download_original(url),
         {:ok, path} <- Media.generate_thumbnail(path) do
      Media.upload_thumbnail(uuid, path)
    end
  end

  @impl Batch
  def handle_completed(%Job{args: %{"batch_id" => "batch-" <> account_id}}) do
    paths = Account.all_thumbnails(account_id)

    with {:ok, file_name} <- Media.create_archive(paths) do
      Media.upload_archive(account_id, file_name)
    end
  end

  @impl Batch
  def handle_exhausted(%Job{args: %{"batch_id" => "batch-" <> account_id}}) do
    with {:ok, account} <- Account.fetch(account_id) do
      Mailer.notify_archive_failure(account.email)
    end
  end
end

The process/1 function handles the mundane task of generating thumbnails for each image. The handle_completed/1 and handle_exhausted/1 callbacks are where the magic happens after all the thumbnailing is executed, as shown in this flow diagram:

A batch is created through new_batch/1,2, which takes a list of args and outputs a matching list of changesets ready for insertion. Typically the batch_id is an auto-generated UUID, but here we're providing a value that bakes in the account_id to simplify our callbacks.

alias MyApp.Account
alias MyApp.Workers.BatchThumbnailer

account_id
|> Account.all_images()
|> Enum.map(&Map.take(&1, [:uuid, :url]))
|> BatchThumbnailer.new_batch(batch_id: "batch-#{account_id}")
|> Oban.insert_all()

The thumbnailer we've built only defines a couple of the available callbacks. Other callbacks give more nuanced control over post-processing and batch management. Take a look at the Batch Guide to explore the other callbacks and see how to insert very large batches.

Chunk Worker

Chunks are the most recent worker addition, and by far our favorite worker name. A chunk worker executes jobs together in groups based on a size or a timeout option, e.g. when 1000 jobs are available or after 10 minutes have ellapsed. Multiple chunks can run in parallel within a single queue, and each chunk may be composed of many thousands of jobs. Combined, that makes for a massive increase in job throughput.

Aside from a massive increase in the possible throughput of a single queue, chunks are ideal as the initial stage of data-ingestion and data-processing pipelines.

Chunk Example

A chunk is unique among Oban workers because it receives a list of jobs which it operates on at the same time. That enables operations that span large amounts of data based on a naturally spaced stream of events. Sounds like a great fit for real-time ETL (extract, transform, and load) data-pipelines!

Pretend that our business handles thousands of disparate operations every minute, and we want to pass that data through our ETL pipeline as it flows in. A key part of our transformation is deduplicating and aggregating—something we need to perform in batches (not those batches).

We'll define a worker that waits for a chunk of 10,000 available jobs or 10 minutes, whichever is first:

defmodule MyApp.Workers.Transformer do
  use Oban.Pro.Workers.Chunk, size: 10_000, timeout: :timer.minutes(10)

  alias MyApp.{Events, Warehouse}

  @impl Chunk
  def process(jobs) do
    aggregated =
      jobs
      |> Stream.map(& &1.args)
      |> Stream.map(&Events.fetch_data/1)
      |> Stream.dedup_by(&Events.duplicate?/1)
      |> Stream.transform([], &Events.aggregate/2)
      |> Enum.to_list()

    with {:error, reason} <- Warehouse.insert(aggregated) do
      {:error, reason, jobs}
    end
  end
end

Assuming our various Events functions handle the data fetching, duplicate checking, and aggregation logic, this is all we need to process groups of events. The chunk worker fetches jobs from the database in a single call, passes them to process/1 as a list, and then tracks them based on the return value.

When inserting data into the Warehouse fails, all of the jobs are flagged as having errored and can be retried again later. Likewise, if the node crashes or somebody trips over the power cord we have a guarantee that the chunk will run again.

The Chunk worker draws on aspects of GenStage, Flow, and Broadway, but because it is implemented in Oban it has the persistence and reliability of a database backed queue. See the Chunk Guide for more usage and error handling details.

Workflow Worker

Workflows are the most powerful worker abstraction provided with Pro, and they have the dubious honor of the most redundant "worker" name. They enable fast, reliable, and inspectable execution of related tasks. Within a workflow, jobs compose together based on explicit dependencies that control the flow of execution. Essentially, workflows are a directed acyclic graph of jobs.

Where a batch or chunk needs homogeneous worker modules (all the same type of job), a workflow can span any combination of worker modules. Dependencies between the jobs are evaluated before the jobs are inserted into the database and then Oban does the rest, enforcing ordered execution even across multiple nodes.

Workflow Example

Workflows are ideal when there are dependencies between jobs, where downstream jobs rely on the success or side-effects of their upstream dependencies.

For this example we'll look at a video ingestion pipeline. As users upload videos we want to process and analyze them before sending a notification that processing is finished. Processing involves a number of jobs that are CPU intensive, call to functions outside the BEAM, or make network calls—all things that are slow and error prone. It would be a shame if we made it through most of the work only to fail on the last step! Instead, let's split the steps up into distinct jobs that we can scale and retry independently.

Overall we have the following workers that we'll pretend all exist and work in isolation: Transcode, Transcribe, Indexing, Recognize, Sentiment, Topics and Notify. Some jobs must run sequentially while others may run in parallel. The execution graph should look like this:

Translating that into code, here's what the Transcode.process_video/1 function would look like:

defmodule MyApp.Workers.Transcode do
  use Oban.Pro.Workers.Workflow

  alias MyApp.Workers.{Transcribe, Indexing, Recognize}
  alias MyApp.Workers.{Sentiment, Topics, Notify}

  def process_video(video_id) do
    args = %{id: video_id}

    new_workflow()
    |> add(:transcode, new(args))
    |> add(:transcribe, Transcribe.new(args), deps: [:transcode])
    |> add(:indexing, Indexing.new(args), deps: [:transcode])
    |> add(:recognize, Recognize.new(args), deps: [:transcode])
    |> add(:sentiment, Sentiment.new(args), deps: [:transcribe])
    |> add(:topics, Topics.new(args), deps: [:transcribe])
    |> add(:notify, Notify.new(args), deps: [:indexing, :recognize, :sentiment])
    |> Oban.insert_all()
  end

  # ...
end

Notice that we start a workflow and then declare jobs with a name and an optional set of dependencies. No, you aren't imagining things, it does look a lot like an Ecto.Multi.

We kick it all off by passing a video_id, which we'll pretend is the id of a persisted video record with a URL and a handful of other attributes we need for processing:

{:ok, _jobs} = MyApp.Workers.Transcode.process_video(video_id)

Once everything is inserted we're guaranteed that the transcode job runs first, and only after it succeeds will transcript, indexing and recognize work, and so on. All of a workflow's coordination happens behind the scenes and you can focus on making the workers do what your business needs them to.

For more details, options, and the full API see the Workflow Guide.

Making Difficult Workflows Simple

We've covered a lot of concepts in the worker examples! To recap, our cast of composable workers are:

• Batch — Track execution of related jobs and run callbacks based on group state
• Chunk — Accumulate jobs by size or time and process them all at once from a single function
• Workflow — Define dependencies between jobs and execute them in a strict order

Our goal is to make complex job interaction simple for you by offloading all the complexity to Pro. Hopefully, you've identified some solutions that are helpful to you, your clients or your business.

Pro has much more to offer, which we'll explore it in future posts. In the meantime, check out the full list of features or peruse the guides to learn more.

How and Why We're Self-Hosting

Self-hosting hex packages is now possible thanks efforts by the Hex team, and Wojtek Mach in particular. After dashbit shut down the bytepack project, a platform for delivering software products to developers, the team open sourced most of the underlying tech. The last project that they open sourced was mix hex.registry, which made self-hosting a package repository practical.

Oban Web and Oban Pro are paid products that require a license to access. Currently (or historically, depending on when you read this), they are hosted as private packages served directly through the official Hex servers. The Hex servers are fast, stable, fronted by a CDN, and relied on by the entire Elixir ecosystem. That sounds great, right? Why would we want to switch to our own servers?

Well, hosting our own packages is desirable for a few key reasons:

• Primarily, it enables us fine grained control over managing licenses and the ability to differentiate between products. Limiting access to one product or another isn't possible through Hex's private packages since that really isn't what they are meant for.
• As a consideration, it doesn't violate the Hex team's terms of use, which stipulates that user accounts may only be one person. Until now, the team has graciously allowed us to use private hex for distribution, and we don't want to overstay our welcome.

Our Schmancy Implementation

Along with the mix hex.registry introduction post on the Dashbit blog there is an official guide on how to self-host a package repository. We used that as a starting point and then modified it to suit our needs.

The hex.registry mix task generates a set of static files that can be hosted anywhere and fetched by the hex client. The official guide walks through serving them using Plug.Static with authentication via Plug.BasicAuth. That solution is simple and worked wonderfully for us initially, but there were a couple of downsides:

• Providing repository files directly from the server would require us to store everything in our application's priv directory, which would bloat the git repository over time and didn't seem like an elegant solution.
• We have Pro customers all over the globe, from Hong Kong to Australia, Brazil and Norway. Serving packages from a data center in middle of the United States isn't ideal—and a significant step backwards from private hex hosting.

That lead us to a slightly more complex, yet ultimately more robust solution—instead of serving files from our server, or even streaming them back from external storage, we redirect requests to a signed, temporary URL on CloudFront. While developing this redirect flow we discovered and fixed a small bug in hex, so be sure to run mix hex.local for the latest hex release before attempting a redirect based flow.

Once we worked out which files hex requests and how to securely sign redirect URLs, the overall flow was rather simple:

1. Publish new packages to a local copy of the package repository and then sync it to a private S3 bucket.
2. As package requests come in we route them to a plug that checks the auth-key against active licenses and records some light tracking information about the version and client.
3. Authenticated requests are then redirected to a short-lived CloudFront URL that expires after a few minutes.

If you hand-wave over license fetching, package authorization, and URL signing, the entire process fits into a single Plug's call/2 function:

def call(conn, _opts) do
  with [license_key] <- get_req_header(conn, "authorization"),
       {:ok, license} <- Accounts.fetch_license_by_key(license_key) do
    if package_allowed?(conn.path_info, license.product) do
      signed_url =
        ["registry" | conn.path_info]
        |> Path.join()
        |> Signer.sign()

      Controller.redirect(conn, external: signed_url)
    else
      conn
      |> send_resp(403, "package not allowed")
      |> halt()
    end
  else
    _ ->
      conn
      |> send_resp(401, "unknown or incorrect license key")
      |> halt()
  end
end

That's all there is to it behind the scenes! It's easily maintained with a couple of mix tasks and extremely lightweight. For the security minded, license fetching has optimizations to prevent timing attacks or brute force discovery of license keys.

Using the Self-Hosted Oban Repository

Adding a self-hosted repo is negligibly more complex than authenticating a private hex organization. The mix hex.repo command takes care of adding the registry, verifying the public/private key pair, and verifying the auth-key (license) all in a single command:

mix hex.repo add oban https://oban.pro/repo \
  --fetch-public-key ${OBAN_KEY_SHA} \
  --auth-key ${OBAN_API_KEY}

With a proper public key fingerprint (OBAN_KEY_SHA) and auth-key (OBAN_API_KEY) set in the environment, that command will add a new local package repo. You can verify the name and settings with mix hex.repo list:

$ mix hex.repo list

Name   URL                    Public key         Auth key
hexpm  https://repo.hex.pm    SHA256:O1LOYhHFW4  6d37f61cc0
oban   https://oban.pro/repo  SHA256:/BIMLnK8NH  12e3671cc1

Note: This example is modified for space, and to obfuscate actual keys

Now you specify the oban repo for the :oban_web and :oban_pro packages, where previously you'd use organization: "oban".

  {:oban_web, "~> 2.6", repo: "oban"},
  {:oban_pro, "~> 0.7", repo: "oban"},

If you're an existing license holder, don't worry: the old hosting will stay active for a while so that you can transition when you're ready. We'll give plenty of warning before we stop supporting private hex hosting.

Recursive (Not Redundant)

Since oban.pro both uses Web/Pro and serves Web/Pro, we actually fetch the private packages from our running server instance while deploying a new instance. There's a beautiful recursion to it!

There's more recursion to come in a future post when we share how we use Oban to handle payments, coordinate licenses and run the Web demo.

Many thanks to the Hex Team, Dashbit, and Wojtek for all the groundwork they laid to make self-hosting possible. This enables a new era for indie developers in Elixir—now we have all the tools necessary to maintain, prepare and serve our own hex packages securely.