Oban Pro v0.13.2

Changelog for Oban Pro v0.13

This release includes numerous engine and telemetry improvements which require Oban v2.14+

enhanced-structured-workers

 🏗️ Enhanced Structured Workers

Structured workers are extended with support for type validation, type casting, enum validation, nested structures, and required field checks at arbitrary depths. The following example demonstrates type checking, required annotations, enums, and nested structures:

use Oban.Pro.Worker, structured: [
  id: {:*, :id},
  name: {:*, :string},
  mode: ~w(enabled disabled paused)a,
  data: [total: {:*, :float}, notes: :string]
]

On new/ keys and types are validated, and errors bubble up to prevent insertion:

StructuredWorker.new(%{id: "not-an-id", mode: "unknown"}).valid?
# => false (invalid id, invalid mode, missing name)

StructuredWorker.new(%{id: "123", mode: "enabled"}).valid?
# => false (missing name)

StructuredWorker.new(%{id: "123", name: "NewBiz", mode: "enabled"}).valid?
# => true

The args, which are stored as JSON in the database, are then cast prior before passing to process/1:

# {"id":123,"name":"NewBiz","mode":"enabled","data":{"parent_id":456}}

%MyApp.StructuredWorker{
  id: 123,
  name: "NewBiz",
  mode: :enabled,
  data: %{parent_id:456}
}

Existing users of structured workers don't worry—the legacy keys and required fields are automatically translated to the new syntax.

See the Oban.Pro.Worker docs to learn more about structured options.

consistent-module-docs

 📗 Consistent Module Docs

Documentation for all plugins, and most extensions, is now in moduledocs rather than guides. The move exposes function and callbacks docs, typespecs, and retains the guide's overview for the module.

Take a look at some of the plugin docs for a taste:

v0-13-2-2023-03-01

 v0.13.2 — 2023-03-01

bug-fixes

 Bug Fixes

  • [SmartEngine] Only track global partitions with active jobs.

    With a large number of unique workers or args, a globally partitioned queue would slowly accumulate tracked partitions with a zero count. Those "dead" partitions bloated the producer row and caused the job fetching query to balloon to an unusable number of clauses.

  • [SmartEngine] Apply timeout option to the entire insert_all transaction.

    Timeout values were only propagated down to inner transactions, not the outer wrapping transaction. When inserting multiple batches worth of jobs (e.g. 30,000) the outer transaction could timeout. Now the timeout options are correctly passed to the outer transaction as well.

  • [DynamicQueues] Persist pause/resume changes across restarts.

    Previously, scaling messages were handled, but pause/resume were ignored. Now pausing/resuming all instances of a queue is persisted, while pausing, resuming, or scaling a single queue is still ignored.

  • [Testing] Set drain_jobs limit using default with_recursion value.

    The :with_recursion option defaults to true, but the limit was set prior to the default being applied. Now drain_jobs/1 operates as advertised without a with_recursion: true option.

enhancements

 Enhancements

  • [Batch] Forward batch_callback_meta to callback workers.

    If it works for args, why not meta? Batch callback uniqueness checks are now scoped to the batch_id and callback keys.

v0-13-1-2023-02-02

 v0.13.1 — 2023-02-02

bug-fixes-1

 Bug Fixes

  • [SmartEngine] Safely ignore new jobs when applying unique replacements.

    For new unique jobs, there aren't any duplicates to update!

v0-13-0-2023-01-25

 v0.13.0 — 2023-01-25

enhancements-1

 Enhancements

  • [SmartEngine] Remove database backed running_ids tracking to minimize database churn.

    Tracking ids on the producer record in the database is no longer necessary for global coordination and it causes excessive vacuum load, particularly for AWS Aurora databases.

  • [DynamicPruner] Remove transaction wrapping each pruning run.

    Partial progress is better than no progress, and each sub-transaction has it's own timeout applied.

  • [Workflow] Halt incomplete workflows with :cancel rather than :discard

    Discarding based on a {:discard, _} return value is deprecated and cancelling has identical functionality.

bug-fixes-2

 Bug Fixes

  • [DynamicCron] Correctly handle paused: false in a crontab update.

    Previously, change tracking ignored false because it was the default value.

  • [SmartEngine] Clean up registred producers after queues terminate.

    The registry cleans up after register/3, but it doesn't ever clean put_meta/3 values. Not only does that cause bloat over time for dynamic queues, it also makes it look as if processes are alive when they aren't.

  • [SmartEngine] Ignore errors caused when recording paused on shutdown.

    An application's shutdown sequence could cause queue shutdown to raise an error, preventing graceful shutdown.

deprecations

 Deprecations

  • [Reprioritizer] Renamed to DynamicPrioritizer for consistency.

For changes prior to v0.13 see the v0.12 docs.