Distributed Job Scheduling and Durable Workflows.

Picture a single server that wakes up every night and pays out your merchants. It works fine, right up until the power flickers at 02:00 and the server is asleep when the payout was due. Nobody runs the job. You find out at 09:00, when a merchant asks where their money is.

So you do the obvious thing and add a second server, so one covers for the other. Except now both wake at 02:00, both read the same list of jobs, and both pay the merchant. You have gone from paying zero times to paying twice, and you have not written any business logic yet.

That little story is the whole problem. Once a scheduled job matters, running it on more than one machine is surprisingly hard, because you have to guarantee three things at once:

• Every job fires. None get silently skipped.
• Each job fires exactly once. No accidental double-runs.
• Each job finishes, even if the machine running it dies halfway through.

Each needs a different trick: one to decide which machine runs the job, one to make it count once, one to make it finish after a crash. Skip any and you are back to paying twice.

Why running jobs on many machines is hard

On paper the goal is tiny, small enough to write on a napkin. Then you start building, and it grows a tail. You need a shared place for the machines to agree on what already ran, so you add a database. You need to stop two machines grabbing the same job, so you add a lock. The machine holding the lock crashes mid-job, so you add a timeout that lets someone else take over. The crashed machine wakes up and finishes its write anyway, corrupting things, so you need to reject that late write. Jobs fail, so you need retries. The retries all fire at once and flatten the thing they were calling, so you spread them out. One job fails forever and clogs everything, so you need somewhere to dump it. The machine is down for an hour, so you have to decide what happens to the runs you missed.

Every one of those you will hand-build without the right tools, and every one a proper scheduler and durable workflow engine already handle. The rest of this post is how they do it, and where the sneaky bugs hide.

The claim: many machines run, one wins each job

The simplest design is one machine that checks the database every second and runs whatever is due. It has two opposite problems: one copy is a single point of failure, and two copies for safety both run every job. No version of "one machine checking" is both safe and reliable.

The way out is a claim. Every machine may check for due jobs at once, but the database lets only one "own" each job. This query does the heavy lifting:

UPDATE scheduled_jobs
SET    status='claimed', claimed_by=$worker, claimed_at=now(),
       fence_token = fence_token + 1
WHERE  id IN (
  SELECT id FROM scheduled_jobs
  WHERE  status='idle' AND next_fire_at <= now()
  ORDER  BY next_fire_at
  FOR UPDATE SKIP LOCKED
  LIMIT  100
)
RETURNING id, fence_token, cron_expr, next_fire_at;

The magic words are FOR UPDATE SKIP LOCKED: when a machine hits a job another machine has already grabbed, it skips past it instead of waiting. So ten machines can run this same query at the same instant, each walks away with a different handful of jobs, and none step on each other.

Without it you would write the tempting version: check if the job is free, and if so, run it. But between the check and the run, a second machine does the same check, also sees it free, and both run the job. SKIP LOCKED closes that gap by letting the database pick the winner. This is how mature schedulers like Quartz and Airflow do it, and you can add machines without ever creating a "boss" to bottleneck on.

The winner marks the job taken, runs it, works out when it runs next, and puts it back in the pool. One run, one clean handoff. If a machine crashes mid-job, another takes it over rather than losing it.

One small detail quietly decides correctness. Base the next run time on the job's scheduled time, not the clock reading when it finished. If a job due at 12:00 finishes at 12:00:40 because it was slow, and you schedule the next run from "now," it drifts 40 seconds late, the next run drifts further, and eventually it skips slots. Anchor every run to the original schedule and slow jobs stay on the grid.

Firing exactly once: label the job by its scheduled time

The claim stops two healthy machines from double-running a job. It misses a nastier case, the one that actually wakes you up: the failover double-run.

Machine A grabs the 12:05 payout and starts. Then A freezes for a moment (a memory cleanup pause, a network hiccup) and its lease, the lock that expires if it stops checking in, runs out. The system assumes A is dead, so Machine B takes the job over and runs it. Then A un-freezes, oblivious, and finishes running it too. Two machines, one payout, paid twice. The claim worked perfectly and you still double-paid.

So the real protection lives one level deeper, in whatever records "this payout happened," and it hinges on one choice: what do you call this run? Its identity is its planned time, not the moment a machine picked it up. Both A and B must label it "the 12:05 payout" identically, so the recording layer sees one thing and keeps one. Label by the exact clock time it started and A calls it "12:05:00.31" while B calls it "12:05:00.74": two labels, two payouts. Always label by the scheduled slot.

That handles identical writes. It does not stop A's late write landing after B already did the work. A simple expiring lock cannot: it says when it is safe to hand over, not that the old machine's in-flight write will not arrive late and clobber things. That late zombie write is a classic headache, and the fix is a fencing token, a counter that ticks up by one on every claim. Whoever holds the job stamps that number on every write, and storage remembers the highest it has seen and refuses anything lower.

A claims the job, gets token 7, then freezes.
Lease expires. B claims it, gets token 8, writes the payout stamped 8.
A wakes up, tries to write the payout stamped 7.
Storage has already seen 8, so it rejects 7.

B's write stands, A's bounces off, the payout happened once. Step back and the pattern is clean: the claim dispatches a job at most once, retries run it at least once, and labeling by the slot so duplicates collapse makes the result happen effectively once. Do not chase a world where dispatch is perfectly once and no machine ever double-fires. Over an unreliable network that is provably impossible (the Two Generals Problem). Make firing safe to repeat instead.

Retries, backoff, and the traffic jam you cause yourself

When a job fails you want to try again, but naive retries create two new problems. They keep hammering something already struggling, and if every failed job retries on the same rhythm, they all come back at once and knock it over again as it recovers.

The first question is not how to retry but whether to, so sort failures into two buckets. Temporary ones (a timeout, a "server busy," a dropped connection) might work next attempt, so wait and retry. Permanent ones (bad input, an invalid request, a business rule that says no) give the identical rejection forever. Mix the buckets and you get the worst of both: giving up on a job that needed one more try, or retrying a hopeless job forever and jamming the queue behind it. (Payments folks know this as "remember a declined card, but never remember a temporary glitch.")

For temporary failures, wait longer and longer between tries, up to a ceiling, with a hard cap on attempts and a dead-letter queue (a holding pen for jobs that never succeed) at the end. The piece people forget is the randomness, and it is not optional:

wait = random_between(0, min(cap, base * 2 ** attempt))
# base=1s, cap=120s  ->  0-1s, then 0-2s, then 0-4s, ... never more than 120s

Growing waits spread one job's retries. They do nothing to spread different jobs from each other. Ten thousand jobs that all failed against one outage and all wait "1s, then 2, then 4" come back in perfect lockstep and rebuild the exact jam that caused it. The randomness breaks up the crowd, making sure ten thousand jobs do not pick the same moment. When a job burns through its attempts, move it to the dead-letter queue, record it, and page a human if money is involved. Do not let it loop forever.

Durable workflows: replaying history to survive a crash

A scheduled job is a single action. A workflow is a chain of them in order, carrying state along the way: authorize a payment, hold the inventory, wait a day, charge the card, send the receipt. It might run for minutes or months, and you have to accept up front that the machine running it will die somewhere in the middle. Not a rare risk to guard against, a certainty to design around.

The tempting fix, a "current step" column plus a pile of if-statements, falls apart the moment a crash lands between two steps and you cannot tell whether the payment went through. Durable workflow engines (Temporal, Cadence, AWS Step Functions) solve this with replay, which is clever enough to slow down for.

You write the workflow as normal code. Every step that touches the real world (charging a card, calling an API) is wrapped as an activity, and the engine writes down each activity's result in a durable log as it happens. If the machine crashes, a fresh machine re-runs your code from the top, but it does not re-do the activities: wherever the log already has a result, the engine hands back the saved answer instead of calling out again. So it races through everything that already happened in milliseconds, arrives exactly where the crash hit, and carries on for real.

class ProcessOrderImpl : ProcessOrderWorkflow {
    // Each activity is retried independently, at-least-once, so it must be idempotent.
    private val activities = Workflow.newActivityStub(
        OrderActivities::class.java,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(30))
            .setRetryOptions(RetryOptions.newBuilder().setMaximumAttempts(5).build())
            .build(),
    )

    override fun process(order: Order): Result {
        val auth = activities.authorizePayment(order)
        activities.reserveInventory(order)

        // A durable timer, not Thread.sleep. It is saved, and survives a restart.
        Workflow.sleep(Duration.ofHours(24))

        return activities.capturePayment(auth)
    }
}

This reads like plain step-by-step code, which is the point. If the machine dies during reserveInventory, a new machine replays authorizePayment from the log without charging the card again, and picks up where the old one fell over. And that 24-hour wait is not a machine blocked for a day, it is a saved timer that uses no resources while it waits and still fires on time if the server restarts on day two.

The one rule you cannot break: keep the workflow predictable

Replay only lands in the right place if your workflow code makes the same decisions the second time. This trips up every team new to this, so be blunt about it. Inside workflow code you cannot read the current time, use a random number, call a database or API directly, read a global that might change, or use a real sleep. Every one of those can give a different answer next time, and the instant it does, your code takes a different path than the log recorded, the engine notices the mismatch, and it stops with an error (on older setups it quietly corrupts your data instead).

The trap in one line: you write if today is Monday inside a workflow, it passes every test, then three weeks later a replay runs on a Tuesday, takes the other branch, and no longer matches a history recorded on a Monday. The engine gives you safe versions of time, randomness, and timers that record the value on the first run and reuse it on every replay. The whole rule is one sentence: anything that touches the outside world goes in an activity, anything that only decides stays in the workflow.

This is also why you cannot casually edit a running workflow. Rename a step, reorder two, add one, and every in-flight workflow fails on its next replay, because the code no longer matches its recorded history. Changing workflow code is more like a database migration than a normal edit, and there is a versioning feature for doing it safely. Stripe runs its financial workflows on this and built an entire team just to enforce that discipline.

That history log is not infinite either. Every step, timer, and message adds to it, and there is a ceiling (around 50,000 entries) because a longer history is slower to replay. A workflow that loops forever hits the wall, so there is an escape hatch that ends the current run and starts a fresh one with the same identity, a summarized state, and an empty history. Think of it as gracefully "starting a new page," well before the limit, to keep replays fast.

Sagas: undo steps that survive a crash

The trickiest workflow is the one that has to reverse itself. You authorized the payment and reserved the inventory, then the final charge fails after all its retries. Now you owe two undo actions, in reverse: release the inventory, then cancel the authorization. That pattern (a chain of steps, each with a matching undo, run backwards when something fails) is a saga.

The hard part is not writing the undo steps, it is guaranteeing they run when the machine crashes mid-undo. A hand-rolled version that dies halfway through cleanup leaves the card on hold and the stock reserved for an order that will never ship, with nothing left running that even knows there is cleanup to finish.

A durable workflow gives you this for free, and it is the single best reason to use one. Each forward step, on success, records its undo action into the same durable history. If a later step fails for good, the workflow walks back through the recorded undos in reverse, and if the machine crashes during cleanup, a new machine resumes it from history, because the list of undos is as durable as everything else. The hold always gets released in the end. One catch: undo steps also run at-least-once, and "cancel the authorization" can even race the authorization succeeding late, so the undos must be safe to repeat too. A saga is only as safe as its undo steps are repeatable.

Where scheduling breaks

Every failure above has a name, and knowing the names is how you catch them in review instead of at 02:00.

Two machines both think they are in charge. A network split can convince two machines each is the sole leader, and both run every job. An expiring lock alone will not save you, because the old leader's writes are still on their way when the new one starts. You need fencing tokens on every write, plus a lock backed by a proper coordination system rather than a plain timestamp two machines can each misread.

Clock drift fires jobs early, late, or twice. A machine whose clock jumps ahead fires early; one whose clock slips back can fire the same job twice. Decide which slot a run belongs to using one source of truth for time (the database's own clock, say), and label runs by that slot so duplicates collapse. Never trust one machine's local clock for correctness.

Missed runs during downtime. The scheduler was down 10:00 to 10:30 and three jobs were due. On recovery, do you run all three, run one, or skip to the next time? No universal answer, so decide per job and write it down: a "send the monthly statement" job runs once to catch up, a "refresh the cache" job skips ahead, because running twelve stale refreshes back to back is pointless.

Checking the database every second stops scaling. Asking "what is due now?" every second is fine with a few thousand jobs and falls over at tens of millions, because you re-scan a giant table on every tick. The fix is a smarter structure (a timing wheel) that only looks at the jobs due right now. This is how systems like Kafka and Netty juggle millions of timers.

Losing the history. If the workflow history or the job table gets wiped (an eviction, a restart without saving, an accidental delete), in-flight workflows cannot resume and pending jobs vanish. Treat this data like a bank ledger: same care, same backups. Never keep it somewhere that can silently forget.

When to use all this, and when not to

This machinery is not free, so be honest about when it earns its keep. Reach for a distributed scheduler or durable workflow when the job has to survive a machine dying, when running it twice or dropping it costs real money or sends real emails or holds real inventory, when the work is long or multi-step and has to pick back up after a crash, or when you want retries, backoff, and undo logic handled for you instead of hand-rolled.

Skip it when none of that applies. A simple request-and-response API gains nothing from the bookkeeping. A workflow engine is not a firehose for millions of events a second. Do not put fast, CPU-heavy work behind a queue that adds delay. And do not wrap a throwaway job that can just restart from scratch, because the whole point is protecting state that is painful to lose, and a job with no such state pays for a guarantee it never uses.

Real systems layer these rather than picking one. A durable workflow runs the overall process, each real-world step inside it is a safely-repeatable activity, events going out are published reliably to a message queue, and a scheduler kicks the whole thing off on a timer. Each layer turns "at least once" into "effectively once" for its own hop.

Running scheduled jobs across many machines comes down to three promises: every job fires, it fires only once, and it finishes through a crash. Each has its own trick. Letting the database hand each due job to one machine (SKIP LOCKED) gives you a single winner and no bottleneck. Labeling each run by its scheduled time, plus a counter that rejects stale late writes, stops a failover from double-paying. Retrying with growing, randomized waits and a dead-letter queue keeps retries from rebuilding the outage that triggered them. And a durable workflow turns a crash-prone multi-step process into something that replays its own history to recover, undoes itself safely when a step fails, and asks one thing in return: keep the deciding code predictable, and push everything that touches the outside world into activities. You cannot get "exactly once" free over a flaky network. You build "effectively once," one safely-repeatable step at a time.