Erlang Notifier

Erlang Notifier#

Oban uses notifications to wake queues when jobs are inserted, pause and scale queues, cancel jobs, and deliver signals and relay results. By default, these messages go through PostgreSQL’s LISTEN/NOTIFY. The Erlang notifier replaces that with direct pub/sub, where each OS process connects to the others and sends notifications straight to them. Python processes discover each other automatically, so you don’t need to configure a list of peers.

Note

Erlang and Elixir are not required to use this notifier. It runs in pure Python, and a cluster of Python processes works entirely on its own.

The notifier uses the Distributed Erlang wire protocol, which makes it compatible with Elixir nodes running Oban.Notifiers.PG. Python and Elixir processes can share one cluster.

Why Avoid PostgreSQL Pub/Sub?#

Using PostgreSQL for pub/sub is convenient, but it has several drawbacks:

  • Connection pooling — LISTEN needs a dedicated session, which transaction pooling can’t provide.

  • Payload limits — NOTIFY caps each payload at 8000 bytes, which is a limitation for Relay notifications and metrics.

  • Database load — every notification is a statement against the database, and NOTIFY serializes committing transactions.

  • Latency — notifications make an extra trip through the database.

Configuration#

Enable the notifier in oban.toml by setting type = "erlang" in [notifier]:

[notifier]
type = "erlang"

Every process in the cluster shares a secret cookie that authenticates connections. Set it in the environment:

# Use a real random value shared across nodes:
export OBAN_ERLANG_COOKIE="shared-secret-cookie"

The notifier also accepts RELEASE_COOKIE, which takes precedence if both variables are set. Cookies are only read from the environment, never from oban.toml.

Programmatic Usage#

When you construct Oban directly, select the notifier by setting type in the notifier dict:

from oban import Oban

oban = Oban(pool=pool, notifier={"type": "erlang"}, queues={"default": 10})

Oban passes its pool and prefix to the notifier. Set the cookie through OBAN_ERLANG_COOKIE or RELEASE_COOKIE as described under Configuration.

Naming and Networking#

Every process has a distribution node name, which defaults to oban_<pid>@<hostname>. Each process needs a unique name. If it finds another process using the same name, it logs a warning and won’t connect to it.

The listener port is ephemeral by default. To allow notifier traffic through a firewall, set a fixed port or port range and allow it through your firewall or network policy. It’s good practice to restrict access to the listener even though connections require a cookie.

[notifier]
type = "erlang"
port = [9100, 9155]

Connecting to Elixir#

Elixir nodes are discovered through EPMD, the Erlang port mapper. Set hosts to a stable hostname for the cluster, such as a Kubernetes headless service or an internal DNS record:

[notifier]
type  = "erlang"
hosts = ["oban-workers.internal"]

The notifier queries each host through EPMD (port 4369), connects to the nodes it finds, and discovers additional nodes through connected peers. It forgets a node after several discovery passes if the node is absent from EPMD and has no active connection.

Without hosts, the notifier scans EPMD on 127.0.0.1, which covers an Elixir node running on the same host.

nodes pins exact peers and disables EPMD discovery. Use it for local tests or fixed topologies, not production. Include a port in the host portion to reach a fixed distribution port:

[notifier]
type  = "erlang"
nodes = ["worker_1@127.0.0.1:9010"]

Note that Python initiates all connections to Elixir. Elixir nodes never connect to Python on their own. Python must reach at least one Elixir node through local EPMD, hosts, or nodes before the processes can exchange notifications.

Names and Authentication#

Erlang distribution is strict about node names. Short-name clusters (--sname) must use short node names, and long-name clusters (--name) must use long names, so override the default to match:

[notifier]
type = "erlang"
node = "obanpy@app-3.internal"

The Python node name may need to be allowed by the Elixir cluster when :net_kernel.allow/1 is used, and the cookie must match the cluster’s Erlang cookie. The Oban prefix becomes the :pg group, so it must match the Elixir cluster’s Oban prefix.

Required Ports#

When using EPMD discovery, Python must be able to reach both EPMD (4369) and the distribution ports of the Elixir nodes. In production, configure a fixed distribution port range for the Elixir cluster and allow it through firewalls or network policies.

If the Python notifier connects only to Elixir peers, you can disable its listener by passing "port": None in the notifier dict.

Options#

Options go in the [notifier] table or the notifier dict passed to Oban. Environment variable equivalents are listed where available. The security cookie is the only exception since it comes from the environment.

  • type (OBAN_NOTIFIER) — Set to "erlang" to use the Erlang notifier.

  • node (OBAN_ERLANG_NODE) — The process’s distribution node name. Defaults to oban_<pid>@<hostname>.

  • host — Listener bind address. Defaults to 0.0.0.0. A specific address is also advertised to other processes.

  • port (OBAN_ERLANG_PORT) — Listener port or port range. Defaults to an ephemeral port.

  • hosts (OBAN_ERLANG_HOSTS) — EPMD seed hosts for finding Elixir nodes, comma-separated in the environment. Defaults to 127.0.0.1.

  • nodes (OBAN_ERLANG_NODES) — Exact nodes to connect to, comma-separated in the environment. Disables hosts discovery.

Protocol Compatibility and Limitations#

The notifier uses the OTP 23+ distribution handshake, targets OTP 27+, and handles both the OTP 29 :pg message shapes and the older ones.

Python connects as a hidden node and doesn’t register in EPMD or appear as a visible node. It supports only the notifier protocol, without remote shell, RPC, distributed monitors, :global, global name registration, or general OTP coordination.

Connections are plain TCP, authenticated by the cookie but not encrypted. Clusters that run distribution over TLS (-proto_dist inet_tls) aren’t supported, and their nodes will refuse connections from Python. Keep notifier traffic on a private network, as you would for standard Erlang distribution.

Troubleshooting#

If processes don’t receive each other’s notifications, check that they share a cookie, that each has a unique node name, and that the advertised host:port is reachable between them.

A connection to Elixir rejected with not_allowed usually means the cookie is wrong or the Python node is not allowed by :net_kernel.allow/1.

If no Elixir nodes are discovered, confirm that EPMD is reachable from Python:

nc -vz worker-1.internal 4369

If EPMD is reachable but handshakes fail, check that the distribution port returned by EPMD is also reachable. Elixir nodes commonly use dynamic distribution ports unless a range is configured.

If notifications arrive from Python to Elixir but not from Elixir to Python, confirm that the Oban prefix matches on both sides. The prefix becomes the :pg group name, so a Python notifier using public will not receive notifications from an Elixir Oban instance configured with another prefix.