Distributed Job Scheduler System Design: Cron at Scale Without Double Runs

Your product emails each customer a usage report at 09:00 in their time zone, syncs CRM accounts every 15 minutes and expires trials at midnight. On one server, cron handles it. Add a second server and the report goes out twice. Add a lock, let its holder hang during a deploy, and one morning the report goes out zero times.

This post designs a distributed job scheduler that starts millions of jobs on time without letting a duplicate do damage, from schedule storage and leases to daylight saving time, misfires, scaling and existing tools.

Requirements and the numbers that matter

Assume a multi-tenant SaaS product:

• 10 million active schedules, most hourly or daily.
• 240 million runs a day if the average schedule fires 24 times: about 2,800 per second (240,000,000 / 86,400).
• Runs start within seconds of their due time and last up to 30 minutes.
• 30 days of run history: 240 million rows a day at about 200 bytes is 48 GB a day, 1.4 TB a month.

People write 0 * * * *, so due times pile up on round minutes. If 3 million hourly schedules use minute 0, 3 million runs fall due at once, and dispatching them within a minute takes 50,000 per second, 18 times the average. Jenkins added the H cron symbol for this: it hashes the job name to a stable minute. Spread internal jobs the same way.

Data model: jobs and runs

jobs holds each definition and its next due instant; job_runs holds one row per occurrence.

CREATE TABLE jobs (
  job_id       uuid PRIMARY KEY,
  cron_expr    text NOT NULL,          -- '0 9 * * *'
  time_zone    text NOT NULL,          -- 'America/New_York'
  misfire      text NOT NULL DEFAULT 'run_once',
  partition_id smallint NOT NULL,      -- hash(job_id) % 256
  next_run_at  timestamptz NOT NULL,   -- a UTC instant
  paused       boolean NOT NULL DEFAULT false
);
CREATE INDEX jobs_due ON jobs (partition_id, next_run_at) WHERE NOT paused;

CREATE TABLE job_runs (
  job_id       uuid NOT NULL,
  scheduled_at timestamptz NOT NULL,
  attempt      int NOT NULL DEFAULT 0,
  status       text NOT NULL,          -- queued, running, retrying, succeeded, dead
  lease_until  timestamptz,
  PRIMARY KEY (job_id, scheduled_at)   -- the run key
) PARTITION BY RANGE (scheduled_at);

(job_id, scheduled_at) is the run key: one occurrence, and the anchor for every later safeguard. Daily partitions make retention a partition drop; PostgreSQL allows the primary key because it includes the partition column.

Finding due jobs

Each scheduler polls the partitions it owns:

SELECT job_id, next_run_at
FROM jobs
WHERE partition_id = ANY($1) AND NOT paused
  AND next_run_at <= now()
ORDER BY next_run_at
LIMIT 500
FOR UPDATE SKIP LOCKED;

In the same transaction it inserts a job_runs row per job and moves next_run_at forward. SKIP LOCKED skips rows another transaction holds instead of waiting, a use PostgreSQL documents for queue-like tables. After the commit, later pollers no longer see the row as due: each occurrence is dispatched once.

Publishing to the queue after the commit is a dual write, so let the new run row double as an outbox record, as in the transactional outbox pattern. Polling has alternatives:

Leases, heartbeats and idempotent runs

From the queue on, delivery is at least once: a consumer can crash before acknowledging, and a relay can publish twice. So the executor claims the run first:

UPDATE job_runs
SET status = 'running', attempt = attempt + 1,
    lease_until = now() + interval '30 seconds'
WHERE job_id = $1 AND scheduled_at = $2
  AND (status IN ('queued', 'retrying')
       OR (status = 'running' AND lease_until < now()))
RETURNING attempt;

No row back means a live lease elsewhere or a finished run: drop the message. The executor heartbeats every 10 seconds, so a 30-second lease survives one lost heartbeat; if it dies, the lease expires and the owning scheduler republishes the run.

A paused executor can wake up after its lease expired and keep writing. The returned attempt is a fencing token, as in Martin Kleppmann's analysis of distributed locks: results are written WHERE attempt = $3, so a stale attempt cannot overwrite a newer one. Targets outside your database get the run key as an idempotency key.

That is the honest guarantee: dispatch once per run key, execution at least once, effects once only for idempotent jobs. The Kubernetes CronJob documentation says the same, because a CronJob can create two Jobs or none; its controller even names each Job after the scheduled time, so a duplicate create fails.

Retries with backoff and a dead-letter state

• Classify errors. Timeouts, 5xx and 429 are retryable; validation errors go straight to dead.
• Back off with full jitter: delay = random(0, min(cap, base * 2^attempt)). With a 10-second base, four retries wait up to 20, 40, 80 and 160 seconds, and the fifth failure is final. If a whole target is down, retry budgets and circuit breakers keep retries from piling on.
• Keep the dead state useful: the payload, the last error, an alert and a replay that reuses the run key.
• Cap concurrency per tenant, as a rate limiter does, so one tenant's burst cannot starve the rest.

If the 09:00 run is still retrying at 10:00, an overlap policy decides: skip, queue, run both or cancel the old one. Kubernetes calls it concurrencyPolicy (default Allow); Temporal schedules default to Skip.

Cron semantics: time zones, DST and misfires

Store each cron expression with an IANA zone such as America/New_York, never a fixed offset, and compute the next run in local time before converting it to UTC; adding 24 hours drifts twice a year. Use UTC for infrastructure jobs and local time where people expect a clock time, as with quiet hours in a notification system.

Daylight saving time breaks local time: in New York, 2:00 to 2:59 AM did not exist on 8 March 2026, and 1:00 to 1:59 AM happened twice on 1 November 2026. Schedulers disagree about both:

The Kubernetes row comes from its cron library, which documents that spring-forward jobs do not run, and from issue 121594, a 02:00 Berlin job that ran twice. Choose a rule and test it.

A misfire is a run that started late because the scheduler was down or overloaded. A per-job policy applies when a scheduler claims a late row:

Quartz counts a run as misfired after 60 seconds by default. Kubernetes has two traps: a startingDeadlineSeconds below 10 can skip runs because the controller checks every 10 seconds, and after an outage it starts only the most recent missed run, so earlier ones are lost (past 100 misses it also records a TooManyMissedTimes warning).

Scaling the scheduler

Three ways to run several schedulers:

1. Single leader. One instance holds a lease (etcd, ZooKeeper, a Kubernetes Lease) and polls everything. It is simple but capped at one node, and failover waits out the lease: 15 seconds by default in kube-controller-manager, home of the CronJob controller.
2. Competing pollers on one table with SKIP LOCKED: no coordinator, but one hot index. Quartz clusters through a database lock and warns it degrades beyond about three nodes.
3. Partition ownership. Split jobs into 256 logical partitions, far more than schedulers; each scheduler leases its partitions in etcd and polls only those. Consistent hashing moves only a share when a node joins or dies, and etcd keeps leases consistent through Raft.

Ownership is an optimization, not the guarantee: if a paused old owner polls once more after a handover, the row lock, the advanced next_run_at and the run key stop it. Partitions also become the unit of database sharding.

Existing tools and what each is good at

Build your own only for millions of tenant-defined schedules.

The distributed job scheduler diagram

How to read it:

• Left to right, a run moves toward execution, from the Job API and the schedule store.
• The frame is the scheduler tier: three workers split 256 partitions using leases from etcd and publish due runs.
• Executor workers lease each run in run history, call targets with an idempotency key and record the result.
• Dashed edges are failure paths: retries with backoff, and the dead letter queue after the fifth failure.
• The Markdown block lists the misfire policies and sizing numbers.

Drawing job scheduler diagrams clearly

• Write the run key on the canvas. (job_id, scheduled_at) explains duplicate protection better than any arrow.
• Show ownership: partitions and where leases come from. One box labeled scheduler hides the hardest part.
• Dash the failure paths and label their triggers.
• Put timings on the canvas: lease, heartbeat and retry cap decide recovery time.

The post on turning requirements into a design diagram applies the same habits to a URL shortener.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Container from the C4 Model pack for the Job API and a Postgres node from the Data pack for the schedule store.
3. Add a Frame from the Core pack holding three Container nodes for the schedulers, and a Container above it for the etcd coordinator.
4. Add a Queue from the Data pack for runs, a Container for the executors, a second Postgres for run history, a Queue for dead letters and an External System from the C4 Model pack for target services.
5. Draw labeled edges and dash the retry and dead-letter paths.
6. Add a Markdown block from the Core pack with the misfire policies, and short Text notes such as the claim query and the run key.

A Mermaid sketch of the pipeline becomes a first draft with the Mermaid to editable diagram tool.

The example uses 13 diagram nodes, within the Free plan.

FAQ

How do you prevent a scheduled job from running twice? Use the job ID plus the scheduled time as a unique run key. Claim due jobs and advance their next run time in one transaction, lease runs before executing them, and pass the run key to targets as an idempotency key.

What does FOR UPDATE SKIP LOCKED do? It locks the returned rows and skips rows other transactions hold, so several workers can poll one table without waiting on each other. PostgreSQL has had it since 9.5.

How should a job scheduler handle daylight saving time? Store an IANA time zone with each schedule, compute next runs in UTC, and decide what happens in the missing and repeated hours, because tools differ: a Kubernetes CronJob can even run twice when clocks go back.

What is a misfire in job scheduling? A run that missed its start time, usually because the scheduler was down or overloaded. The misfire policy decides whether to run once now, skip to the next slot or catch up on every missed run.

Is Kubernetes CronJob enough for a distributed scheduler? For a few hundred batch jobs on a cluster you already run, yes. For millions of tenant schedules, no: each schedule is a Kubernetes object, each run a Job with Pods, and timing is approximate.

Start free — one seat, no credit card.