"Job Scheduler" — Distributed Background Job Platform
A distributed job scheduling platform that runs background jobs across a worker cluster with priorities, per-queue concurrency limits, retries with backoff, a dead-letter queue, cron-based recurring jobs, and a live WebSocket dashboard. Atomic claiming with FOR UPDATE SKIP LOCKED guarantees no job ever runs twice, and advisory-lock leader election keeps exactly one scheduler active with automatic failover. Six hand-rolled data structures power the scheduling layer — a binary min-heap for priority dispatch, Kahn's algorithm for DAG execution plans, three-colour DFS for cycle rejection, union-find for independent job clusters, a retry delay queue, and Deficit Round Robin for fair queuing — covered by 156 unit tests and CI.
- My role
- Solo — backend, scheduling algorithms, dashboard, tests
- Built for
- Personal project · Jul–Sep 2026
“The most important line of code is a carefully worded
FOR UPDATE SKIP LOCKED.”
The Brief
A job scheduler has one promise that matters more than every feature: a job must never run twice by accident, and it must never be lost when a worker dies halfway through.
Most tutorials reach for Redis or a broker. I wanted to see how far correctness could go on the one dependency the system already needed — Postgres — and where the database genuinely runs out of expressiveness.
The Approach
Claiming is a single UPDATE … WHERE id IN (SELECT … FOR UPDATE OF j SKIP LOCKED): many workers poll the same queue, each locks rows the others aren’t touching, and dependency gating happens in the same statement. Per-queue concurrency is a hard bound across the fleet — the count and the claim share one transaction behind a per-queue advisory lock — and a Postgres advisory lock elects a single scheduler leader, with automatic failover when the leader’s connection drops.
Where SQL can’t express the decision, six structures are implemented by hand and wired into the running scheduler: a binary min-heap for cross-queue priority dispatch, Kahn’s algorithm for DAG execution plans, three-colour DFS to reject dependency cycles, union-find for independent job clusters, a min-heap delay queue for retries, and Deficit Round Robin so one saturated queue can’t starve the rest.
How It Works
- 01APIExpress · JWT · project RBAC
- 02Postgresjobs table, SKIP LOCKED claim
- 03Workersmin-heap dispatch, heartbeats
- 04Schedulerleader-elected tick, cron, reaper
- 05DashboardWebSocket push + polling fallback
Key Decisions
Postgres over a broker
One dependency instead of two. The trade-off is a latency floor from polling and the jobs table as the throughput ceiling — acceptable at this scale, with LISTEN/NOTIFY as the next step.
Lock only what you claim
The claim uses FOR UPDATE OF j, not a plain FOR UPDATE: it joins each job’s dependency only to read it, and locking those rows too would let two jobs that share a dependency block each other’s claims for no reason.
At-least-once, honestly
If a worker dies, the reaper requeues after the heartbeat timeout, so a job can run twice in a crash. Exactly-once isn’t achievable in general; handlers are expected to be idempotent.
A push that carries no data
WebSocket events are just “go refresh jobs/queues/workers”. The client re-fetches through the authenticated REST routes, so the socket never duplicates per-row access checks — and polling stays on as a fallback.
- Polling sets a latency floor;
LISTEN/NOTIFYwould remove idle polls at larger scale. - Guarantees are at-least-once — job handlers must be idempotent.