A long job and a hot reload
Some of the work this platform does takes ten minutes. A model builds a page. That build is not a function call that returns. It is a run of many steps that takes five, ten, fifteen minutes to finish. That length is the whole story, because it broke something I did not expect. It broke the way I work.
The fifteen-minute problem
When a job takes fifteen minutes, an engineer waiting on it has two options. Sit and watch it, or go do something else. The second is the only sane choice. Nobody bills fifteen minutes of staring at a progress line.
But going to do something else means editing code. And this codebase runs on hot reload. Every service restarts itself the instant a file changes, which is what makes the iteration loop fast. It is also what made the long jobs impossible to live with. The job ran inside the service process. The moment I changed a line anywhere, the reload restarted the process and took the running job down with it. So the real choice was worse than it looked. Sit idle for fifteen minutes, or kill the job I was waiting on the second I touched a file. The fastest part of the dev loop had become the part that killed the job.
The fastest part of the dev loop had become the part that killed the job.
Durability, in one word
A durable job is one the process does not own. Its state lives somewhere the process cannot take with it when it dies.
So the job moved out of the process and into a queue backed by Redis. The work is still done by a worker in the service. But the worker no longer holds the job in memory. It claims the job from the queue, takes a lock on it, and checkpoints its progress as it goes. The lock is sized for the work. A job that runs ten minutes holds its lock for ten. If the worker stops answering, because the process was restarted or killed, the lock goes stale within a minute and the job returns to the queue for another worker to pick up where the checkpoint left off.
The effect on the dev loop is the entire point. I can edit code while a job runs. The reload restarts the service and the old worker dies mid-job. A new one reclaims the job and carries it forward. The timing is tuned so that an interrupted run is picked up quickly enough to survive a restart or two. The fifteen minutes are the platform's to spend now, not mine to sit through.
The same fix, two more times
Durability was worth it for the dev loop alone. It happened to pay off twice more.
In production the failure is not a hot reload but everything else. A service restarts under load. An instance is recycled. A network call blips at the wrong second. To a job held in memory, all of those are the same as the reload. All of them lose the work. To a durable job, they are the same as the reload too, which is to say survivable. The job is reclaimed and resumed. Nothing that took ten minutes has to start over because something twitched in the ninth.
And because the queue is shared rather than owned by one process, the work scales sideways. Any worker on any instance can claim any job from the pool. When there is more work than one instance can get through, the answer is another instance, not a rewrite. The queue does not care how many workers read from it.
The ground an agentic job stands on
A long model job is not one call. It is a chain of steps with state carried between them. That state is governed by a contract, a fixed shape the job has to hold from the first step to the last. A contract is only as good as its ability to survive the run. If a restart wipes the state halfway through, the contract it was enforcing is gone with it.
Durability is what lets the contract hold. Because the state is checkpointed into the queue, the contract survives every restart the job meets. Because the queue is shared, it holds the same way whether one worker is running the job or ten are running ten of them. Stable across failures, and the same under load. The durable queue is the ground the agentic runtime stands on. Without it there is no contract worth enforcing, because nothing is keeping the contract alive long enough to matter. I would not build the AI work any other way now.
What I would tell someone
Earlier I wrote that slow work belongs in a job, and that Postgres alone is enough to be the queue. That is still true, for the jobs it was true about, the fire-and-forget kind that either happens or gets retried. This is the case where enough runs out. A job that runs for ten minutes, carries state all the way, has to survive a restart, and needs to spread across instances is a different problem. It earns a queue built for exactly that.
The part I did not expect is where the value showed up first. Durability is sold as a production virtue, a thing you add so the system holds up under real load. It is that. But it paid off first in the dev loop, not in production, by giving me the fifteen minutes back to keep working while the platform worked. When the jobs get long enough, durability stops being an operations feature and becomes what lets you build at all.
Have questions?
If you're interested in my work or Giant Context, contact me!