When a job takes fifteen minutes, an engineer waiting on it has two options. Sit and watch it, or go do something else. The second is the only sane choice. Nobody bills fifteen minutes of staring at a progress line.

But going to do something else means editing code. And this codebase runs on hot reload. Every service restarts itself the instant a file changes, which is what makes the iteration loop fast. It is also what made the long jobs impossible to live with. The job ran inside the service process. The moment I changed a line anywhere, the reload restarted the process and took the running job down with it. So the real choice was worse than it looked. Sit idle for fifteen minutes, or kill the job I was waiting on the second I touched a file.

The reload looked like a local annoyance and it was not one. A hot reload kills a process at an arbitrary point in a run. So does a service restarting under load, an instance being recycled, and a network call that blips at the wrong second. To a job holding its state in memory those are one event, and every one of them loses the work. The only difference is frequency. In production it happens occasionally. On my machine every file change is a reload, so it happened on every save.

So the reload did the work of a test I never wrote. It reproduced a production failure hundreds of times over a working day, for free, and it was telling me the job system was fragile rather than telling me to stop editing code. Building for the reload meant building for every other way a process dies.

A long model job is not one call. It is a chain of steps with state carried between them. That state is governed by a contract, a fixed shape the job has to hold from the first step to the last. If a restart wipes the state halfway through, the contract it was enforcing is gone with it.

Durability is what lets the contract hold. Because the state is checkpointed into the queue, the contract survives every restart the job meets. Without that, there is no contract worth enforcing, because nothing keeps the state around long enough for the contract to apply to it.

And because the queue is shared rather than owned by one process, the work scales sideways. Any worker on any instance can claim any job from the pool. When there is more work than one instance can get through, I add an instance. The queue does not care how many workers read from it.

Earlier I wrote that slow work belongs in a job, and that Postgres alone is enough to be the queue. That is still true, for the jobs it was true about, the fire-and-forget kind that either happens or gets retried. A job that runs for ten minutes, carries state all the way, has to survive a restart, and needs to spread across instances is a different problem, and Postgres alone stops being enough at that point.

Hot reload was breaking my jobs in a way I could have worked around by not editing code while a job ran. Reading it as a real failure instead produced a job system that survives arbitrary process death. Production does not announce its restarts either. The dev cadence came back with it. I would not build the AI work any other way now.