Unique Jobs#
Unique jobs lets you specify constraints that prevent enqueuing duplicate jobs. These constraints only apply when jobs are inserted. Uniqueness has no bearing on whether jobs are executed concurrently.
Uniqueness is based on a combination of job attributes using the following options:
period — How long from a job’s scheduled time until it is no longer considered a duplicate. Can be an integer number of seconds, a
timedelta, or a period string such as"5 minutes"or"7 days". Supported units aresecond,minute,hour,day,week, andmonth(always 30 days). Defaults toNone, where jobs are considered duplicates for as long as they’re retained in the database (see Job Maintenance).fields — The fields to compare when evaluating uniqueness. The available fields are
args,queue,worker, andmeta. Defaults to["queue", "worker", "args"]. It’s recommended that you leave the default fields, otherwise you risk unexpected conflicts between unrelated jobs.keys — A specific subset of the
argsormetato consider when comparing against historic jobs. This allows a job with multiple key/value pairs in its arguments to be compared using only a subset of them. Keys requireargsormetain the fields.group — A named group of job states that are checked for duplicates:
all- All states, includingcancelledanddiscardedincomplete- Jobs that haven’t finished processing:scheduled,available,executing,retryable, andsuspendedscheduled- Onlyscheduledjobs (useful for “debouncing”)successful- All states exceptcancelledanddiscarded(the default)
Configuring Uniqueness#
The simplest form of uniqueness considers a job a duplicate for as long as a matching job exists in the database, unless it was cancelled or discarded:
@worker(queue="default", unique=True)
class MyWorker:
async def process(self, job):
pass
Uniqueness can also be set, overridden, or disabled for an individual job:
# Jobs are unique for 5 minutes
await MyWorker.enqueue({"id": 1}, unique={"period": "5 minutes"})
# A timedelta works as well
await MyWorker.enqueue({"id": 1}, unique={"period": timedelta(minutes=5)})
# Skip uniqueness for this job entirely
await MyWorker.enqueue({"id": 1}, unique=False)
Here’s a more complex example which uses multiple options:
@worker(
unique={
# Jobs should be unique for 2 minutes after they're scheduled
"period": "2 minutes",
# Don't consider the whole args field, just the url key within args
"keys": ["url"],
# Consider a job unique across all states, including cancelled/discarded
"group": "all",
# Consider a job unique across queues; only compare the worker and url key
"fields": ["worker", "args"],
}
)
Options are validated when the job is built, so an unknown option, field, or group raises an error immediately rather than at insert time.
Tip
Oban checks uniqueness with a transactional advisory lock and a query for matching jobs. It does not rely on unique constraints in the database, which leaves it prone to race conditions in some circumstances.
Oban Pro automatically enforces uniqueness with unique constraints, which provides strong uniqueness guarantees.
Uniqueness vs Concurrency#
Understanding the distinction between uniqueness and concurrency is crucial for designing efficient processing pipelines. While these concepts may seem related, they operate at different stages of a job’s lifecycle.
Uniqueness operates at job insertion time. When a job is marked as unique, Oban checks whether an identical job already exists before inserting a new one.
When it applies - During job insertion
What it prevents - Duplicate jobs from being inserted
What it doesn’t affect - Which jobs run concurrently
A common misunderstanding is that unique jobs run one at a time or in sequence. This isn’t true—uniqueness only prevents duplicate insertions. Once unique jobs are in the queue, they’ll run according to the queue’s concurrency settings.
For example, given the following configuration that allows 10 concurrent email jobs:
[queues]
emails = 10
These 10 unique jobs will all run concurrently:
await oban.enqueue_many(
EmailWorker.new({"user_id": id}, unique=True) for id in range(10)
)
To restrict the number of jobs that run at once you must set concurrency accordingly.
Detecting Conflicts#
When unique settings match an existing job, enqueue still returns a Job. However, the job
isn’t inserted, and you can detect the conflict by checking conflicted in the job’s extra. If
there was a matching job the value is True; otherwise it is False.
You can use the conflicted value to customize responses after insert:
job = await MyWorker.enqueue({"id": 1}, unique=True)
if job.extra["conflicted"]:
print("Job already exists")
else:
print(f"Job inserted with ID: {job.id}")
Conflict detection works the same way for bulk inserts with enqueue_many. Each job is checked
individually, including against other jobs earlier in the same batch:
jobs = await oban.enqueue_many(
MyWorker.new({"id": 1}, unique=True),
MyWorker.new({"id": 1}, unique=True),
)
[job.extra["conflicted"] for job in jobs] # [False, True]
Note
A job is also marked as conflicted when another process is inserting an identical unique job at the same moment. In that case one insert wins and the others are reported as conflicts.
Specifying Fields and Keys#
The fields option determines which high-level job attributes Oban will consider when checking
for uniqueness, including args, queue, worker, and meta.
When args or meta are included in the fields list, the keys option provides additional
specificity by allowing you to designate particular keys within the dictionary for comparison,
rather than comparing the entire dictionary.
Let’s see this with an example:
# This compares the entire args dictionary
await MyWorker.enqueue(
{"url": "...", "user_id": 1},
unique={"fields": ["worker", "queue", "args"]},
)
# This compares only the url key within the args dictionary
await MyWorker.enqueue(
{"url": "...", "user_id": 1},
unique={"keys": ["url"], "fields": ["worker", "queue", "args"]},
)
In the second example, the uniqueness check only looks at the url key within the args
dictionary because keys is specified. Jobs with the same url but different user_id values are
considered duplicates.
Without keys, the entire dictionary must match exactly. A job with args {"id": 1} is distinct
from one with {"id": 1, "tenant": None}, and empty args are distinct from non-empty args.