The cache stored results rather than work. For each test file it computed a hash, and if a passing result existed for that hash it skipped running the test and replayed the pass.

The hash was computed from the bundled test file. Suites that boot the whole platform reach it through a dynamic import the bundler is told not to follow, so the application, its routes and every declaration in them were invisible to the key. The key moved only when the test file's own text changed.

One pass recorded, then replayed on every run afterwards. The pipeline restored the cache between runs, so a result recorded once survived indefinitely.

The guard that enforces object-level authorization sat green in the pipeline for an unknown length of time. Run cold, it failed immediately. That is the subject of its own post, and it is what made this one urgent.

It had happened before. A rename shipped three broken test files reporting green, because the rename changed route source rather than the test files, so their hashes never moved and they were never re-run. The response then was a step that clears the cache for changed packages, which over-fires on most pushes and still cannot see the surface the boot suites test.

The pipeline went red every week or so. A test would fail, I would fix it, and the next run was green. That is what a healthy pipeline looks like, and it is why I never went looking.

Those failures were real, and they were all coming from newly written tests.

The design was to move the heavy work off my machine. CI runs the light suites first and exits early on a failure, so the compute sits next to the repository and the laptop stays free to keep working. Cadence depends on that, and it holds only if the tests in the pipeline are the tests being run.

A component enters the suite when a test is written for it, and as the codebase grew I was targeting new components locally. A newly written test has no recorded pass, so the cache had nothing to replay and the test ran for real. New tests were the only tests running cold, and new tests are the ones most likely to fail.

So the weekly red came from the newest and least settled corner of the suite, and I read it as proof that the whole thing was running. Everything older was replaying a pass recorded months earlier. The signal that would have contradicted that is an old test going red, which is the one thing the cache made impossible.

Deleting the cache was one commit, and losing it cost almost nothing in hit rate.

I push about twice a day in large batches. Measured against real history, a single commit touches a foundational package sixty-four percent of the time, a batch of five ninety-five percent, and a batch of eleven every time. So any honest cache, one that actually invalidates when its inputs change, would hit approximately never at this cadence. The cache was only fast because it was dishonest.

What it had been hiding, apart from the guard, was the real cost of the suite. Cold, the pipeline took about eighteen minutes of wall time and sixty-five minutes of compute, and it ran twice before a deploy to production.

The slowest job was the application unit suite at eighteen minutes, and it set the floor for everything else because the jobs run in parallel.

I had it filed as transform-bound. Splitting the tests across shards had produced no improvement, and the obvious reading of that is that every shard rebuilds the same module graph, so splitting the files does not split the dominant cost. A note in the repository said bigger runners did not help either.

The time each phase spent, added up, matched the wall-clock almost exactly, which means nothing was running in parallel at all. Both readings were wrong.

Each worker carried a six-gigabyte heap. The runner had seven gigabytes. Exactly one worker fit, so a suite designed to run in parallel ran one file at a time, and had been doing so for as long as anyone had been measuring it.

The fix was a larger runner and a cap of four workers, which is four times six gigabytes inside thirty-two. Eighteen minutes became five. No test changed. Phase-sum against wall-clock now shows nearly four times real parallelism.

The earlier note that bigger runners do not help was true when it was written, measured on a box too small to fit a second worker. A rejection recorded without its conditions reads as a settled fact to whoever finds it next.

A dependency-aware build cache. It works technically, and the concern that killed it earlier turned out to be false. It dies on hit rate: at a batch cadence every batch invalidates, so an honest cache degenerates into always running, which one line of configuration already achieves without a migration.

A Rust bundler under the test runner. Measured cold against the stock one: eleven percent slower. The transform phase it speeds up was eleven seconds of a nine hundred and fifty second run. The cost was module evaluation inside each worker, which a faster bundler does not touch.

Turning off per-file isolation. A genuine hundred-times win, and unusable. The shared module registry means a per-file mock silently does nothing when an earlier file already loaded the module, and the expensive tests are the ones that rely on those mocks most. Central fixes recovered a hundred and fifty-six of a hundred and eighty-eight failures, and the remainder was cross-file pollution that only re-isolating would fix.

Pre-bundling the component library. Eleven times faster on my machine, flat in the pipeline, and it made another suite six minutes worse. Applied, measured live, pulled. My machine has many cores and the runner has two, so the local number was measuring a bottleneck the runner does not have.

And sharding, covered above, plus a different test runner rejected on the grounds that it is partly fast because it does less of what this codebase needs.

The one that did land alongside the runner fix was a native typechecker, which took that job from seven and a half minutes to under five.

Each step caused the next. The cache was deleted for correctness, which made the suite slow. The slowness was now real, so it had to be measured rather than tolerated. Measuring it found a memory limit nobody had looked for, and fixing that took more time off the pipeline than the cache had ever saved.

None of it would have happened while the cache was there. There was nothing to fix, and a weekly red to prove it. Eighteen minutes was a number that only appeared on a cold run, and cold runs were rare enough to be somebody else's problem.

Any mechanism that makes a slow thing tolerable also removes the reason to make it fast, and the cost of that stays invisible for exactly as long as the mechanism works. This one had the additional property of being wrong, which is what forced the issue.

The pipeline is now honest, cold and about twice as fast as the version that was cheating. A green push flows through to production with nothing to press, and the remaining ceiling is a job whose real cost I can finally see.