Settings Changelog for Oban Pro v0.13

This release includes numerous engine and telemetry improvements which require Oban v2.14+

enhanced-structured-workers

🏗️ Enhanced Structured Workers

Structured workers are extended with support for type validation, type casting, enum validation, nested structures, and required field checks at arbitrary depths. The following example demonstrates type checking, required annotations, enums, and nested structures:

use Oban.Pro.Worker, structured: [
  id: {:*, :id},
  name: {:*, :string},
  mode: ~w(enabled disabled paused)a,
  data: [total: {:*, :float}, notes: :string]
]

On new/ keys and types are validated, and errors bubble up to prevent insertion:

StructuredWorker.new(%{id: "not-an-id", mode: "unknown"}).valid?
# => false (invalid id, invalid mode, missing name)

StructuredWorker.new(%{id: "123", mode: "enabled"}).valid?
# => false (missing name)

StructuredWorker.new(%{id: "123", name: "NewBiz", mode: "enabled"}).valid?
# => true

The args, which are stored as JSON in the database, are then cast prior before passing to process/1:

# {"id":123,"name":"NewBiz","mode":"enabled","data":{"parent_id":456}}

%MyApp.StructuredWorker{
  id: 123,
  name: "NewBiz",
  mode: :enabled,
  data: %{parent_id:456}
}

Existing users of structured workers don't worry—the legacy keys and required fields are automatically translated to the new syntax.

See the Oban.Pro.Worker docs to learn more about structured options.

consistent-module-docs

📗 Consistent Module Docs

Documentation for all plugins, and most extensions, is now in moduledocs rather than guides. The move exposes function and callbacks docs, typespecs, and retains the guide's overview for the module.

Take a look at some of the plugin docs for a taste:

• DynamicCron
• DynamicLifeline
• DynamicPrioritizer
• DynamicPruner
• DynamicQueues

v0-13-3-2023-03-13

v0.13.3 — 2023-03-13

enhancements

Enhancements

• [Testing] Apply Chunk and Workflow optimizations directly in drain_jobs/2.

  Testing chunks or workflows with drain_jobs directly rather than through run_chunk/2 or run_workflow/2 missed out on imporant timing optimizations. Now, those optimizations are made while draining jobs for more predictable testing.

• [Batch] Add handle_retryable/1 batch callback.

  The new handle_retryable/1 callback triggers the first time a job in the batch fails with retries available.

bug-fixes

Bug Fixes

• [Batch] Ensure handle_cancelled/1 callback is always triggered.

  Generative batch property tests didn't hit less frequent situations like one-or-more cancelled jobs. That meant tests couldn't catch that cancelled handling was never hit! Now all batch tests are declarative and cover every callback state.

• [Batch] Prevent batch callbacks from triggering other callbacks.

  In certain situations a combination of job events caused a race condition where a batch callback triggered inserting another callback. For batches with custom args or meta this prevented the callback from passing along custom attributes, which caused failures.

v0-13-2-2023-03-01

v0.13.2 — 2023-03-01

enhancements-1

Enhancements

• [Batch] Forward batch_callback_meta to callback workers.

  If it works for args, why not meta? Batch callback uniqueness checks are now scoped to the batch_id and callback keys.

bug-fixes-1

Bug Fixes

• [SmartEngine] Only track global partitions with active jobs.

  With a large number of unique workers or args, a globally partitioned queue would slowly accumulate tracked partitions with a zero count. Those "dead" partitions bloated the producer row and caused the job fetching query to balloon to an unusable number of clauses.

• [SmartEngine] Apply timeout option to the entire insert_all transaction.

  Timeout values were only propagated down to inner transactions, not the outer wrapping transaction. When inserting multiple batches worth of jobs (e.g. 30,000) the outer transaction could timeout. Now the timeout options are correctly passed to the outer transaction as well.

• [DynamicQueues] Persist pause/resume changes across restarts.

  Previously, scaling messages were handled, but pause/resume were ignored. Now pausing/resuming all instances of a queue is persisted, while pausing, resuming, or scaling a single queue is still ignored.

• [Testing] Set drain_jobs limit using default with_recursion value.

  The :with_recursion option defaults to true, but the limit was set prior to the default being applied. Now drain_jobs/1 operates as advertised without a with_recursion: true option.

v0-13-1-2023-02-02

v0.13.1 — 2023-02-02

bug-fixes-2

Bug Fixes

• [SmartEngine] Safely ignore new jobs when applying unique replacements.

  For new unique jobs, there aren't any duplicates to update!

v0-13-0-2023-01-25

v0.13.0 — 2023-01-25

enhancements-2

Enhancements

• [SmartEngine] Remove database backed running_ids tracking to minimize database churn.

  Tracking ids on the producer record in the database is no longer necessary for global coordination and it causes excessive vacuum load, particularly for AWS Aurora databases.

• [DynamicPruner] Remove transaction wrapping each pruning run.

  Partial progress is better than no progress, and each sub-transaction has it's own timeout applied.

• [Workflow] Halt incomplete workflows with :cancel rather than :discard

  Discarding based on a {:discard, _} return value is deprecated and cancelling has identical functionality.

bug-fixes-3

Bug Fixes

• [DynamicCron] Correctly handle paused: false in a crontab update.

  Previously, change tracking ignored false because it was the default value.

• [SmartEngine] Clean up registred producers after queues terminate.

  The registry cleans up after register/3, but it doesn't ever clean put_meta/3 values. Not only does that cause bloat over time for dynamic queues, it also makes it look as if processes are alive when they aren't.

• [SmartEngine] Ignore errors caused when recording paused on shutdown.

  An application's shutdown sequence could cause queue shutdown to raise an error, preventing graceful shutdown.

deprecations

Deprecations

• [Reprioritizer] Renamed to DynamicPrioritizer for consistency.

For changes prior to v0.13 see the v0.12 docs.