The rules file that made things worse

In 2026 two research groups measured the same artifact — the instruction file you put at the root of your repo so coding agents behave — and reached opposite conclusions. One found agents 28.6% faster with it. The other found that machine-written ones cost 23% more and slightly reduced success. Both measured carefully. The reconciliation is a single rule, and it is not the one the tooling recommends.

Andriy Kudelya · 2026-07-26 · 8 min

Start with the uncomfortable part. We build a product that has a model write instruction files for agents. There is now a peer-reviewable measurement suggesting that doing exactly that, the obvious way, makes agents slightly worse and meaningfully more expensive. That is worth putting at the top of our own lab rather than waiting for somebody else to.

So: what was actually measured, why the two results disagree, and what survives.

Two verdicts on one file

The second study is the more thorough of the two: four agents rather than one (Claude Code with Sonnet-4.5, Codex with GPT-5.2 and with GPT-5.1 mini, Qwen Code with Qwen3-30b-coder), two benchmarks, and a genuinely new dataset built only from repositories that ship a real context file — so the “no file” condition means deleting the one the maintainers actually wrote.

A note on the numbers, because they are already being misquoted, including once by me. The −2% and +23% are the AGENTbench row for generated files. On SWE-bench Lite the same comparison is −0.5% and +20%. The paper's introduction summarises differently — −3% for generated files, +4% for human ones — and those figures don't reconcile arithmetically with the per-dataset ones. Quote the per-dataset measurements; treat −3%/+4% as the authors' framing rather than a result.

It isn't that the agents ignore the file

The tempting explanation is that agents skim these files and carry on. Measurably false. When a context file mentions uv, the agent uses it 1.6 times per task; when it doesn't, fewer than 0.01 times. Repository-specific tooling: 2.5 times versus under 0.05. The instructions are read and obeyed.

Which makes the result worse, not better. The file works exactly as designed, and the outcome is still negative. So the cost has to be coming from somewhere other than disobedience.

It is. With generated files present, GPT-5.2 spent 22% more reasoning tokens on the same tasks. The agent is doing more thinking, more searching, more testing — and arriving at roughly the same answer, later, for more money. Every additional requirement is a thing that must be held in mind and satisfied. A file full of requirements does not make the task clearer. It makes it bigger.

The instructions weren't ignored. They were obeyed, and obeying them was the expense.

The experiment that explains both results

Here is the part that reconciles the two papers, and it is a single ablation almost buried in section 4.2. The researchers deleted every piece of documentation from the repositories — all the .md files, the docs/ directory, the example code — and re-ran the same comparison.

With nothing else to read, the machine-generated context files improved performance by 2.7%, and beat the human-written ones.

So the file was never the problem. Competition was. In a repository with a decent README, package manifests and a docs folder, a generated instruction file mostly restates things the agent would have found anyway — and charges for the privilege in tokens and attention. Strip the alternatives away and the same file becomes genuinely useful.

That also dissolves the disagreement between the two studies. The first measured well-maintained repositories' own hand-written files on small, well-scoped tasks, and found real savings. The second measured machine-generated files competing against documentation that already existed, and found real waste. Neither result is wrong. They are measurements of different amounts of redundancy.

An instruction file's value is inversely proportional to what the agent could have discovered without it.

The most popular section is the one that does nothing

The single most common ingredient in these files is a codebase overview — a tour of the directory tree. Every generation prompt asks for one. In the study, 100% of Sonnet-4.5-generated files included an overview; 99% for GPT-5.2, 95% for Qwen.

The researchers then measured whether it helps, in the most direct way available: how many steps before the agent first touches a file that the real fix actually modified. The answer is that the presence of a context file does not reduce that number — the figure caption notes it is “generally lower without” one. Agents navigate repositories perfectly well by reading them.

Worth noting who already knew this: Anthropic's own initialisation prompt warns against listing components that are easily discoverable. Its output still contained an overview 100% of the time. A model asked to produce a document will produce the document's conventional sections, and “the conventional sections” is not the same set as “the sections that help.”

What survives

The paper's own prescription is unusually blunt for an academic abstract: “unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements.” Its named example of a minimal requirement is the specific tooling a repository uses — which is precisely the thing the instruction-following data showed actually changes behaviour.

Which gives a test you can apply line by line, without a benchmark:

One question, asked of every line: could the agent have known this without me? If yes, the line is not neutral. It is a cost with no matching benefit, and now there is a number attached to it.

What we changed here

Two things, the day we read the papers.

Prowrit's readiness review — the fresh-context pass that grades every workflow before you see it — gained a defect it hunts for by name: content the agent could already infer. Overviews of things it can see. Rules that are somebody's default. Lines that repeat the prompt they sit next to. It fails the right size aspect and names the lines to cut, because a review that says “looks good” about a bloated file is worse than no review.

And every skill file the studio produces now states its own size on the card — lines and words, plainly. Restraint that isn't visible reads as thinness. Nine lines is not us running out of things to say; it is the finding above, applied.

The honest caveats, since this whole note is about not overclaiming. The thorough study is Python-heavy, and its authors say so: much of what a context file might explain about Python tooling is already in the models' training. It sampled each configuration once. Its unit tests were partly model-generated. And it does not cite the efficiency study at all, so the tension between them is still unreconciled by anyone but readers like us. The direction of the effect is well supported. The exact percentages will move.

None of which changes the rule, because the rule doesn't depend on the size of the effect. Every line in a file an agent must read competes for the same attention as the code it describes. Write the part it couldn't have worked out. Delete the rest.

Sources: efficiency result — “On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents” (arXiv:2601.20404) · success and cost results, the documentation-removal ablation, the overview measurement, and the instruction-following and reasoning-token figures — Gloaguen, Mündler, Müller, Raychev & Vechev, “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” (arXiv:2602.11988; code at eth-sri/agentbench) · what effective files contain across 2,500+ repositories — GitHub engineering. Section references are to arXiv:2602.11988v1. Where the two papers disagree we have said so rather than picked one; the reconciliation offered here is ours, not theirs.