# Multi-Process

Oban Pro provides multi-process job execution, allowing CPU-intensive jobs to run in parallel across
multiple worker processes. This bypasses Python's Global Interpreter Lock (GIL) and enables true
parallelism for compute-heavy workloads.

## Why Multi-Process?

Python's GIL prevents multiple threads from executing Python bytecode simultaneously. While asyncio
handles I/O-bound concurrency efficiently, CPU-intensive tasks block the event loop and become a
bottleneck.

Multi-process execution solves this by running jobs in separate Python processes, each with its own
GIL and asyncio event loop. This provides:

- **Parallelism** for CPU-bound work across multiple cores
- **Concurrency** within each process for I/O-bound work

Jobs within each process are still async, so a single process can efficiently handle many
concurrent I/O operations (API calls, database queries) while also benefiting from parallel CPU
execution across workers.

This makes multi-process execution ideal for:

- **Data processing** — parsing, transforming, or aggregating large datasets
- **Machine learning** — model inference or batch predictions
- **Cryptographic operations** — hashing, encryption, or signature verification
- **Scientific computing** — numerical simulations or statistical analysis

## Enabling Multi-Process

Executing with multi-process requires using `obanpro` rather than regular `oban`. By default, it
will create a process for each CPU core:

```bash
obanpro start
```

Alternatively, you can set a fixed number of processes with the `--processes` flag:

```
obanpro start --processes 4
```

Or use the `OBAN_PRO_PROCESSES` environment variable:

```bash
export OBAN_PRO_PROCESSES=4
obanpro start
```

Each process runs its own asyncio event loop. Jobs are distributed across processes, with each
process handling its share of concurrent jobs _across all queues_.

## Concurrency vs Parallelism

Queue limits control concurrency, or the total number of jobs that can execute at once. This works
identically to standard Oban, where a queue with `limit=20` can run up to 20 jobs concurrently.
With multi-processing, jobs are distributed between native processes allowing CPU-bound work to
run simultaneously on multiple cores.

The number of processes does *not* multiply concurrency. For example, with 4 processes and a queue
limit of 20, you still get 20 concurrent jobs while utilizing 4 CPU cores for parallel work.

```bash
# 20 concurrent jobs distributed across 4 parallel processes
obanpro start --processes 4 --queues "default:20"
```

### Recommended Configuration

For CPU-intensive workloads, tailor the number of processes to available cores:

```bash
obanpro start --processes 8 --queues "heavy:16"
```

For mixed workloads, use moderate queue limits with enough processes:

```bash
obanpro start --processes 4 --queues "default:20,heavy:8"
```

```{tip}
Unix systems use copy-on-write memory and processes share the parent's memory pages until
modified, so loading large read-only data (like ML models) before starting reduces total memory
usage. However, Python's reference counting can trigger copies when objects are accessed, so
memory savings vary. For memory-constrained environments, favor fewer processes with higher queue
limits.
```

## Setup and Teardown Hooks

Worker processes may need to initialize resources like database connections or ML models. Use the
`--setup` and `--teardown` options to specify initialization functions:

```bash
obanpro start --setup myapp.worker_setup --teardown myapp.worker_cleanup
```

The setup function runs once when each worker process starts:

```python
# myapp.py
import asyncpg

pool = None

async def worker_setup():
    global pool
    pool = await asyncpg.create_pool("postgresql://localhost/mydb")

async def worker_cleanup():
    global pool
    if pool:
        await pool.close()
```

```{note}
Setup and teardown functions must be async and importable by path (e.g., `myapp.worker_setup`).
```

## Performance Comparison

Multi-process execution provides significant speedups for CPU-bound work. In a benchmark of 100
jobs with "heavy" CPU work (100k SHA-256 hash iterations for each job), the speedup is apparent:

| Configuration                                       | Time     | Speedup  |
| --------------------------------------------------- | -------- | -------- |
| Single process (`oban start`)                       | ~2,200ms | 1x       |
| Multi-process with 4 workers (`obanpro start -p 4`) | ~790ms   | **2.8x** |

The speedup scales with the number of processes, up to the number of available CPU cores. Note
that the benchmark isn't representative of maximum throughput as it includes database pool
creation and process instantiation.