The cache stored results rather than work. For each test file it computed a hash, and if a passing result existed for that hash it skipped running the test and replayed the pass.

The hash was computed from the bundled test file. Suites that boot the whole platform reach it through a dynamic import the bundler is told not to follow, so the application, its routes and every declaration in them were invisible to the key. The key moved only when the test file's own text changed.

One pass recorded, then replayed on every run afterwards. The pipeline restored the cache between runs, so a result recorded once survived indefinitely.

The guard that enforces object-level authorization was green in the pipeline for an unknown length of time. Run cold it failed with a hundred and eighty undeclared identifiers, and the core of the platform was zero of a hundred and twenty-eight files declared. That is the subject of its own post, and it is what made this one urgent.

It had happened before. A rename shipped three broken test files reporting green, because the rename changed route source rather than the test files, so their hashes never moved and they were never re-run. The response then was a step that clears the cache for changed packages, which over-fires on most pushes and still cannot see the surface the boot suites test.

Deleting it was one commit and it cost almost nothing in hit rate, which is worth explaining because it is the reason the decision was easy.

I push about twice a day in large batches. Measured against real history, a single commit touches a foundational package sixty-four percent of the time, a batch of five ninety-five percent, and a batch of eleven every time. So any honest cache, one that actually invalidates when its inputs change, would hit approximately never at this cadence. The cache was only fast because it was dishonest.

What it had been hiding, apart from the guard, was the real cost of the suite. Cold, the pipeline took about eighteen minutes of wall time and sixty-five minutes of compute, and it ran twice before a deploy. That number had never been anybody's problem, because nobody had ever paid it.

The slowest job was the application unit suite at eighteen minutes, and it set the floor for everything else because the jobs run in parallel.

I had it filed as transform-bound. Splitting the tests across shards had produced no improvement, and the obvious reading of that is that every shard rebuilds the same module graph, so splitting the files does not split the dominant cost. A note in the repository said bigger runners did not help either.

Both readings were wrong, and the measurement that settled it is simple. Add up the time each phase spent and compare it to the wall-clock. They matched almost exactly, which means nothing was running in parallel at all.

Each worker carried a six-gigabyte heap. The runner had seven gigabytes. Exactly one worker fit, so a suite designed to run in parallel ran one file at a time, and had been doing so for as long as anyone had been measuring it.

The fix was a larger runner and a cap of four workers, which is four times six gigabytes inside thirty-two. Eighteen minutes became five. No test changed. Phase-sum against wall-clock now shows nearly four times real parallelism.

The earlier note that bigger runners do not help was true when it was written, measured on a box too small to fit a second worker. Same lever, different conditions, opposite answer. Write the conditions next to the rejection, or the next person reads it as a settled fact.

Five other approaches were investigated properly and rejected. Each is worth a line, because the reasons are more useful than the conclusions.

A dependency-aware build cache. It works technically, and the concern that killed it earlier turned out to be false. It dies on hit rate: at a batch cadence every batch invalidates, so an honest cache degenerates into always running, which one line of configuration already achieves without a migration.

A Rust bundler under the test runner. Measured cold against the stock one: eleven percent slower. The transform phase it speeds up was eleven seconds of a nine hundred and fifty second run. The cost was module evaluation inside each worker, which a faster bundler does not touch.

Turning off per-file isolation. A genuine hundred-times win, and unusable. The shared module registry means a per-file mock silently does nothing when an earlier file already loaded the module, and the expensive tests are the ones that rely on those mocks most. Central fixes recovered a hundred and fifty-six of a hundred and eighty-eight failures, and the remainder was cross-file pollution that only re-isolating would fix.

Pre-bundling the component library. Eleven times faster on my machine, flat in the pipeline, and it made another suite six minutes worse. Applied, measured live, pulled. My machine has many cores and the runner has two, so the local number was measuring a bottleneck the runner does not have.

And sharding, covered above, plus a different test runner rejected on the grounds that it is partly fast because it does less of what this codebase needs.

The one that did land alongside the runner fix was a native typechecker, which took that job from seven and a half minutes to under five.

The whole sequence was one thing causing the next. The cache was deleted for correctness, which made the suite slow. The slowness was now real, so it had to be measured rather than tolerated. Measuring it found a memory limit nobody had looked for, and fixing that took more time off the pipeline than the cache had ever saved.

None of it would have happened while the cache was there, because there was nothing to fix. Eighteen minutes was a number that only appeared on a cold run, and cold runs were rare enough to be somebody else's problem.

The generalisation is not about caching. Any mechanism that makes a slow thing tolerable also removes the reason to make it fast, and the cost of that is invisible for exactly as long as the mechanism works. This one had the additional property of being wrong, which is what forced the issue.

The pipeline is now honest, cold, and about twice as fast as the version that was cheating. A green push flows through to production with nothing to press. And the remaining ceiling is a job whose real cost I can finally see.