Evaluation-driven development in YAML files
Building a non-deterministic system is not a build-it-once job. You tune it. Change a prompt, a token budget, a retrieval step, run it against real cases, read what came out, decide whether it moved the right way, and do it again. That loop is where an AI feature actually gets made. Increasingly it is not one person running it by hand. It is one or more agents working a surface across many sessions, which changes what the loop needs, because the knowledge it produces is fragile.
The knowledge that does not survive
What did I try. What was the theory. Did the last change help, or did it only feel like it helped. Which run was the good one. What was different about it. That is the real substance of building an AI feature. By default it lives in exactly two places, the head of whoever is working and the scrollback of a chat with whatever agent they are working alongside. Both are gone the moment the session ends or the agent process crashes.
So you solve the same problem twice, because the record of solving it the first time evaporated. With a second agent, or a second person, on the same surface, there is no shared account of what has already been tried. The bottleneck in engineering with agents is not the agent. It is that the reasoning does not persist past the agent.
Context as files
Evaluation-Driven Development is the tooling I built for that. It is almost aggressively unclever. It is context as files.
Every investigation is a small YAML document that lives in the repository, versioned next to the code it concerns. Not a dashboard, not a database, a file:
One investigation, as a file
name: Builder copy architecture, a page reads as one voicestatus: activepriority: p1created: 2026-05-05problem: > Generated section headlines drift in register across a page. The hero reads confident, the features read tentative, the CTA reads generic. The page does not sound like one writer.theory: > The copy step has no page-level voice pass. Derive a single voice directive once per page and thread it into every section prompt, instead of each section inventing its own tone.refs: - packages/ai/src/shared/builder/stages/creative/copy/architecture.tsfixture: run the 12-page creative fixture, read every headline set aloudruns: - 2026-05-05T14:02 FAIL a6bc5142 # directive too weak, features still tentative - 2026-05-05T16:20 PASS 3d17f0e1 # directive moved ahead of section prompts, holdsThe problem is the investigation. The theory is the current best explanation. The refs point at the code in play. The runs are an append-only log, one line each, of every attempt, a timestamp, a verdict, the commit, and the signal it produced.
The primary reader of that file is not me. It is the next agent or human to pick up the work. They open the story and know the problem, the current theory, and every dead end already walked, without reading a line of chat.
The runs are append-only
The rule that makes it work is that the run log is only ever added to. A run is never edited and never deleted. A failed attempt is not a mistake to clean up. It is often the most valuable line in the file, because it is the day the next person would otherwise spend rediscovering that the same idea does not work.
That runs against the instinct to tidy history into a clean story. Here the messy history is the asset. Five recorded dead ends are what turn the sixth attempt from a guess into an informed move. How the log stays honest across commits is a discipline of its own, for another time.
Why it survives the crash
Because a story is a file in the repo, everything the loop learned is durable by default. The agent working it can crash. The session can close. The machine can die. The story is still in .edd/active/ at its last committed state. The next session reads it and resumes. Two agents can work two stories in the same repo, each reading the other's record, without colliding.
This is the same shape the product's own builder uses on its model calls, turned around to face the engineering. A model call in the pipeline retries the transient failure, audits its output before anything trusts it, and traces what happened so no failure dies in silence. An investigation does the same at human speed. It runs, evaluates, and writes the result down where it cannot be lost. The story is the trace. The runs are the audit history. The file on disk is what makes the work durable across the crash and the handoff.
Runs the work
Retry the transient failure
Run the investigation
Checks the result
Audit the output before trusting it
Evaluate what came out
Keeps a record
Trace so no failure dies in silence
Write the result down in the story
Traceable, shareable, measurable
Living in the repo buys more than durability. The work becomes traceable the way the code is, every investigation versioned and attributable, in the same history as the change it produced. It becomes shareable without a handoff. Another developer clones the repo, sees every active and blocked story, and picks one up where it stands. And because a story is structured rather than prose, the whole set of them is data. You can read the shape of the work across all of it, how fast stories close and which ones are stuck, which files show up in story after story, where the effort actually goes. A folder of stories is not just a memory. It is a queryable record of how the system got built.
The point
Engineering with agents is only as good as the memory it runs on. An agent is capable and forgets everything the instant its session ends. The leverage is not a smarter agent. It is giving the work a memory the agent cannot take with it when it goes. Context as files is that memory, cheap and durable and versioned with the code. The next hand to touch the problem starts where the last one left off, not where the last one began.
Building something like this
I'm Jesse. I build platforms end to end, and I'm open to work. If this is the kind of engineering you need, get in touch.