#Method
#Architecture
#AI & Agents
Model infrastructure is unreliable, and not only at the edges. Calls time out, come back empty or truncated, or arrive well-formed and wrong. A pipeli...
Jesse James Richard
Read previous|
May 20, 2026Evaluation-driven development in YAML files
|
6 min read
#Method
#AI & Agents
#Architecture
Building a non-deterministic system means tuning it. Change a prompt, a token budget, a retrieval step, run it against real cases, read what came out, decide whether it moved the right way, and do it again. That loop is where an AI feature gets made. Increasingly it is run by one or more agents working a surface across many sessions rather than by one person by hand, which changes what the loop needs, because the knowledge it produces is fragile.
The knowledge that does not survive
What did I try. What was the theory. Did the last change help, or did it only feel like it helped. Which run was the good one. What was different about it. Those answers are the substance of building an AI feature. By default they live in exactly two places, the head of whoever is working and the scrollback of a chat with whatever agent they are working alongside. Both are gone the moment the session ends or the agent process crashes.
So you solve the same problem twice, because the record of solving it the first time evaporated. With a second agent, or a second person, on the same surface, there is no shared account of what has already been tried. The bottleneck is that the reasoning does not persist past the agent that did it.
Context as files
Evaluation-Driven Development is the tooling I built for that. It is almost aggressively unclever. It is context as files.
Every investigation is a small YAML document that lives in the repository, versioned next to the code it concerns. Not a dashboard, not a database, a file:
One investigation, as a file
name: Builder copy architecture, a page reads as one voicestatus: activepriority: p1created: 2026-05-05problem: > Generated section headlines drift in register across a page. The hero reads confident, the features read tentative, the CTA reads generic. The page does not sound like one writer.theory: > The copy step has no page-level voice pass. Derive a single voice directive once per page and thread it into every section prompt, instead of each section inventing its own tone.refs: - packages/ai/src/shared/builder/stages/creative/copy/architecture.tsfixture: run the 12-page creative fixture, read every headline set aloudruns: - 2026-05-05T14:02 FAIL a6bc5142 # directive too weak, features still tentative - 2026-05-05T16:20 PASS 3d17f0e1 # directive moved ahead of section prompts, holdsThe problem is the investigation. The theory is the current best explanation. The refs point at the code in play. The runs are an append-only log, one line each, of every attempt, a timestamp, a verdict, the commit, and the signal it produced.
The file is written for the next agent or human to pick up the work rather than for me. They open the story and know the problem, the current theory, and every dead end already walked, without reading a line of chat.
The runs are append-only
The rule that makes it work is that the run log is only ever added to. A run is never edited and never deleted. A failed attempt is often the most valuable line in the file, because it is the day the next person would otherwise spend rediscovering that the same idea does not work.
That runs against the instinct to tidy history into a clean story. Five recorded dead ends are what turn the sixth attempt from a guess into an informed move.
Why it survives the crash
Because a story is a file in the repo, everything the loop learned is durable by default. The agent working it can crash. The session can close. The machine can die. The story is still in .edd/active/ at its last committed state. The next session reads it and resumes. Two agents can work two stories in the same repo, each reading the other's record, without colliding.
The product's own builder uses the same pattern on its model calls, turned around to face the engineering. A model call in the pipeline retries the transient failure, audits its output before anything trusts it, and traces what happened so no failure dies in silence. An investigation does the same at human speed. It runs, evaluates, and writes the result down where it cannot be lost.
The model-call loop | The engineering loop | |
|---|---|---|
Runs the work | Retry the transient failure | Run the investigation |
Checks the result | Audit the output before trusting it | Evaluate what came out |
Keeps a record | Trace so no failure dies in silence | Write the result down in the story |
Traceable, shareable, measurable
Living in the repo buys more than durability. The work becomes traceable the way the code is, every investigation versioned and attributable, in the same history as the change it produced. It becomes shareable without a handoff. Another developer clones the repo, sees every active and blocked story, and picks one up where it stands. And because a story is structured rather than prose, the whole set of them is data. I can query how fast stories close, which ones are stuck, which files show up in story after story, and where the effort goes.
The dead ends are worth keeping
An agent is capable and forgets everything the instant its session ends. Context as files is the memory it cannot take with it, cheap and durable and versioned with the code. A story whose run log is mostly failures is worth more than one with a single clean pass, because those failures are the days nobody has to spend again. The next hand to touch the problem opens the file and starts from the last line.
Hiring an early engineer, or building something like this
Remote, Pacific time, full-time or contract. Get in touch.