# Multi-Process

Oban Pro provides multi-process job execution, allowing CPU-intensive jobs to run in parallel across
multiple worker processes. This bypasses Python's Global Interpreter Lock (GIL) and enables true
parallelism for compute-heavy workloads.

## Why Multi-Process?

Python's GIL prevents multiple threads from executing Python bytecode simultaneously. While asyncio
handles I/O-bound concurrency efficiently, CPU-intensive tasks block the event loop and become a
bottleneck.

Multi-process execution solves this by running jobs in separate Python processes, each with its own
GIL and asyncio event loop. This provides:

- **Parallelism** for CPU-bound work across multiple cores
- **Concurrency** within each process for I/O-bound work

Jobs within each process are still async, so a single process can efficiently handle many
concurrent I/O operations (API calls, database queries) while also benefiting from parallel CPU
execution across workers.

This makes multi-process execution ideal for:

- **Data processing** — parsing, transforming, or aggregating large datasets
- **Machine learning** — model inference or batch predictions
- **Cryptographic operations** — hashing, encryption, or signature verification
- **Scientific computing** — numerical simulations or statistical analysis

## Enabling Multi-Process

Executing with multi-process requires using `obanpro` rather than regular `oban`. By default, it
will create a process for each CPU core:

```bash
obanpro start
```

Alternatively, you can set a fixed number of processes with the `--processes` flag:

```
obanpro start --processes 4
```

Or use the `OBAN_PRO_PROCESSES` environment variable:

```bash
export OBAN_PRO_PROCESSES=4
obanpro start
```

Each process runs its own asyncio event loop. Jobs are distributed across processes, with each
process handling its share of concurrent jobs _across all queues_.

## Concurrency vs Parallelism

Queue limits control concurrency, or the total number of jobs that can execute at once. This works
identically to standard Oban, where a queue with `limit=20` can run up to 20 jobs concurrently.
With multi-processing, jobs are distributed between native processes allowing CPU-bound work to
run simultaneously on multiple cores.

The number of processes does *not* multiply concurrency. For example, with 4 processes and a queue
limit of 20, you still get 20 concurrent jobs while utilizing 4 CPU cores for parallel work.

```bash
# 20 concurrent jobs distributed across 4 parallel processes
obanpro start --processes 4 --queues "default:20"
```

### Recommended Configuration

For CPU-intensive workloads, tailor the number of processes to available cores:

```bash
obanpro start --processes 8 --queues "heavy:16"
```

For mixed workloads, use moderate queue limits with enough processes:

```bash
obanpro start --processes 4 --queues "default:20,heavy:8"
```

```{tip}
Unix systems use copy-on-write memory and processes share the parent's memory pages until
modified, so loading large read-only data (like ML models) before starting reduces total memory
usage. However, Python's reference counting can trigger copies when objects are accessed, so
memory savings vary. For memory-constrained environments, favor fewer processes with higher queue
limits.
```

## Database Connections

Database connections can't be shared across processes, so each worker process opens its own
connection pool in addition to the main process. The total number of connections is therefore
`(processes × pool_max_size) + main_pool_max_size`, and that total must stay within your
PostgreSQL `max_connections`.

To keep the count manageable, worker pools default to a maximum size of 5 rather than inheriting
the main pool's size. Setting `--pool-max-size` (or `OBAN_POOL_MAX_SIZE`) applies to worker pools
as well, so size it with the multiplier in mind:

```bash
# 4 workers × 4 + 10 for the main process = 26 connections
obanpro start --processes 4 --pool-max-size 4
```

Worker queries through `Workflow` and `Oban.get_instance()` use the worker's own pool
automatically, so no setup hook is required for database access.

## Setup and Teardown Hooks

Worker processes may need to initialize per-process resources like ML models or external clients.
Use the `--setup` and `--teardown` options to specify initialization functions:

```bash
obanpro start --setup myapp.worker_setup --teardown myapp.worker_cleanup
```

The setup function runs once when each worker process starts:

```python
# myapp.py
model = None

async def worker_setup():
    global model
    model = load_model("model.bin")

async def worker_cleanup():
    global model
    model = None
```

```{note}
Database access doesn't belong in a setup hook. Each worker process opens its own Oban connection
pool automatically (see [Database Connections](#database-connections)), so `Workflow` and
`Oban.get_instance()` work inside workers without any extra setup.
```

```{note}
Setup and teardown functions must be async and importable by path (e.g., `myapp.worker_setup`).
```

## Performance Comparison

Multi-process execution provides significant speedups for CPU-bound work. In a benchmark of 100
jobs with "heavy" CPU work (100k SHA-256 hash iterations for each job), the speedup is apparent:

| Configuration                                       | Time     | Speedup  |
| --------------------------------------------------- | -------- | -------- |
| Single process (`oban start`)                       | ~2,200ms | 1x       |
| Multi-process with 4 workers (`obanpro start -p 4`) | ~790ms   | **2.8x** |

The speedup scales with the number of processes, up to the number of available CPU cores. Note
that the benchmark isn't representative of maximum throughput as it includes database pool
creation and process instantiation.