fix(poller): use SKIP LOCKED for horizontal scalability

[nicolasburtey]

Summary

• Add SKIP LOCKED to the FOR UPDATE clause in the job polling query

Problem

Immediate: pollers block each other

The poll_jobs query uses FOR UPDATE OF je to atomically claim pending jobs. Without SKIP LOCKED, concurrent poller instances block on each other's row locks instead of claiming non-overlapping job sets. Adding more poller replicas increases lock contention rather than throughput.

Deeper: permanent jobs pin slots to the primary forever

Several downstream consumers (lana-bank) use the job crate for permanent daemon-style jobs — infinite loop {} inside run() that only exit on shutdown:

Job	Pattern	Holds slot for
fetch-provider-price	loop { fetch(); sleep(300s) }	Forever
self-custody-balance-sync	loop { sync(); sleep(60s) }	Forever
eod-process-manager	loop { run phases; await children }	Hours (EOD)
sync-reports	loop { poll dagster; sleep(10s) }	Until run completes

Each of these holds a CurrentJob (which owns a PgPool clone) and counts against the JobTracker running count. Combined with min_jobs = 30 as the polling threshold, a handful of permanent jobs permanently consume slots.

The keep-alive handler writes UPDATE job_executions SET alive_at = ... WHERE poller_instance_id = ... every job_lost_interval / 4 (75s by default). The lost-job detector reclaims jobs whose alive_at has drifted. Both are writes to job_executions, so:

1. All job infrastructure is write-bound to the primary — polling, keep-alive, lost-job detection, state transitions
2. Permanent jobs hold tracker slots indefinitely, reducing capacity for transient work
3. You can't offload to a read replica because the job lifecycle (poll → claim → keep-alive → complete/reschedule) is inherently a write-side state machine on job_executions

This means horizontal scaling requires more primaries or partitioned write capacity, not just read replicas. SKIP LOCKED is the minimum fix to let multiple writers coexist efficiently.

Solution

FOR UPDATE OF je SKIP LOCKED is the standard Postgres pattern for competing consumers:

• Poller A locks rows 1-10
• Poller B skips those locked rows and claims rows 11-20 instead
• Both proceed concurrently without blocking

This is safe because:

1. Skipped rows are already being claimed by another poller — they'd be running by the time B's transaction could see them anyway
2. The queue_id ordering constraint in the candidates CTE still ensures per-queue serialization
3. The min_wait CTE operates on the eligible CTE (pending jobs only), so skipped-but-locked rows don't affect wait time

What this doesn't fix (future work)

• Permanent jobs consuming tracker slots — these could be modeled differently (e.g., not count against min_jobs, or use a separate "daemon" pool)
• Single-pool contention — callers share one PgPool across HTTP + jobs + infra; a dedicated job pool would isolate load
• Read replica offloading — job read-only business logic (e.g., price fetching, report polling) could use a replica pool, but the job lifecycle itself must stay on the primary

Test plan

• Existing tests pass (no behavioral change for single-poller deployments)
• Under concurrent pollers, jobs distribute across instances instead of serializing

🤖 Generated with Claude Code