I was scrolling Hacker News last week and stumbled on a paper that made me stop and read the comments: Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study (arXiv:2605.20049). 139 points on HN, 200+ comments, the kind of thread where senior engineers start sharing their real workflows.
The author is Olivier Schmitt from SonarSource. The headline finding is exactly what you'd guess from the title: clean code doesn't make coding agents smarter, but it does make them less wasteful. The numbers are concrete — 7-8% fewer tokens, 34% fewer file revisitations. Not vibes, not theory. 660 controlled trials on Claude Code.
Let me walk through what the paper actually does, why the HN thread matters more than the paper itself, and what this means for the rest of us shipping AI-coded projects in 2026.
What the Study Actually Did
The setup is the interesting part. The question sounds simple: does the cleanliness of a codebase affect how an AI coding agent performs? But the experimental design is genuinely clever.
You can't just take two existing codebases and compare them — they'd differ in language, framework, dependencies, age, architecture, and a hundred other things. You'd never know if the agent's behavior change was due to "cleanliness" or just "the second repo was a Rails project and the first was a Java monolith."
So Schmitt built minimal pairs — six pairs of repositories where each pair shares architecture, dependencies, and external behavior, but differs on static analysis rule violations. The two sides of a pair are behaviorally equivalent (same test suite passes on both), but they look completely different under the hood.
Two directions of pair construction:
- Slopify (going dirty): Start with a clean codebase, run a multi-agent pipeline that inlines helpers, duplicates logic, pads files with dead code, occasionally merges modules into single files. The result "plausibly reflects what the codebase might have looked like without it."
- Vibeclean (going clean): Start with a messy codebase, run a multi-agent pipeline that mechanically resolves SonarQube rule violations. Delete dead code, dedupe string literals, replace legacy idioms, extract god structures into named helpers.
The paper then wrote 33 coding tasks across the six pairs, with three task types:
- 13 cognitive-hotspot tasks (work concentrated in dense single-method or single-class complexity)
- 14 multi-module tasks (work that crosses module boundaries)
- 6 calibration tasks (deliberately cleanliness-insensitive, as a sanity check)
Each task runs 10 times per side = 660 trials. Agent is Claude Code with Claude Sonnet 4.6, default tool set, containerized sandbox, fixed temperature, with a median-threshold outlier filter that drops ~9.7% of trials.
The Core Finding: Same Results, Different Process
The headline result, in one sentence:
Cleanliness doesn't change whether the agent finishes the task. It changes how the agent gets to the finish.
Dataset-level deltas, cleaner relative to messier:
- Pass rate: 0.913 vs 0.921, -0.9 percentage points. Basically the same.
- Input tokens: -7.1%
- Output tokens: -8.5%
- Reasoning characters: -11.1%
- Conversation messages: -7.0%
- Files read: +3.2% (slightly higher)
- File revisitations: -33.8% (the killer number)
- Lines edited: -3.2%
Pass rate didn't budge, which means cleanliness doesn't help the agent be more correct. But the operational footprint shrank. The most striking single effect is file revisitation: on cleaner code, the agent goes back to files it's already touched a third less often.
The paper has a great case study explaining this. On commons-bcel, the bytecode disassembly logic was originally two parallel god methods, each a several-hundred-line switch over JVM opcodes. The cleanup pipeline replaced both with thin dispatchers delegating to ~10 named helpers each. On the messier variant, the agent dives into the 250-line switch, gets confused, and keeps re-reading files. On the cleaner variant, the agent greps dispatch_<opcode> and lands on the right helper first try. Same code change, dramatically different navigation cost.
This is the "name things well" advice, except now it has a citation.
But the Paper Has Real Methodological Holes
The top-voted HN comment cut straight to the weak point:
"They used Opus 4.6 to synthetically produce 'degraded' or 'cleaned' code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output."
Two distinct critiques here:
-
The "clean" and "messy" baselines are AI-generated. The Vibeclean pipeline is itself an AI agent cleaning up a codebase. There's no guarantee that an AI agent's idea of "clean" matches what a senior engineer would call clean. The paper doesn't claim to test that, but it doesn't control for it either.
-
They only test pass rate on the task's hidden tests. A cleaner-side and messier-side solution can both pass the test, while breaking different unrelated tests elsewhere in the repo. The paper doesn't track regressions on tests outside the task scope.
The author replied in the comments, honestly:
"What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know."
That's an admirably candid admission for an arXiv author. The paper's limitations section does mention this, but it's a real caveat on the results.
Another commenter pointed out the obvious: SonarSource makes SonarQube. The paper uses SonarQube as the cleanliness proxy. Not a fatal flaw — the rules are public, the methodology is documented — but worth keeping in mind when reading the conclusions. The "cleaner" condition happens to align exactly with what SonarQube measures, which means the experiment is testing "does SonarQube's definition of clean matter to agents" rather than "does clean code in some abstract sense matter."
My read after the paper + the comments: the direction is correct (cleanliness helps), the data is real (660 trials is a lot), but the magnitude should be taken with a grain of salt. The 7-8% number is real on these six pairs with this one agent — whether it generalizes to GPT-5.6 or Gemini or your specific codebase is conjecture.
The HN Comments Are Better Than the Paper
This is the part that actually changed how I work. The paper tells you that cleanliness matters. The HN comments tell you how to make it matter.
Pattern 1: LEGACY CODE comments + a rules document
"I have the agent inject comments that mention that this particular code is legacy and must not be used as a reference, should not be cleaned up, etc. If you have a document that lists all of the reasons not to use or touch some code, the comments can simply be references to it.
// LEGACY CODE, per docs/legacy_rules.md §14, §19"
I tried this in a legacy Python codebase I was working on. The problem is exactly what they describe: LLM agents pick up bad habits from whatever they see in context first. The fix is to mark the bad patterns explicitly and tell the agent not to learn from them. A // LEGACY CODE comment + a linked rules doc is a lighter-weight solution than rewriting the legacy code. It's basically giving the agent a "do not reproduce" sticker on each bad pattern.
Pattern 2: Pre-commit hooks are the real moat
"You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding."
Multiple commenters converged on the same insight: linter > prompt. LLM output is non-deterministic. Two runs of the same prompt produce different code. Linters are deterministic — if you can write a rule, the linter will catch violations every time. So the engineering ROI is on linter coverage, not on prompt tuning.
A specific tool that came up several times: tach (from Hatch). It's a Python tool that enforces import boundaries using the AST. You declare which modules can import from which, and CI fails if a forbidden import shows up. The argument: agentic coding makes bad import patterns easier to introduce (LLMs love coupling everything), and only a deterministic boundary check stops them at scale.
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
| 9 | |
| 10 | |
| 11 | |
| 12 | |
| 13 | |
| 14 | |
| 15 | |
| 16 | |
| 17 | |
| 18 | |
| 19 | |
When the agent's commit fails pre-commit, it can read the error and try again. Hooks become a teaching signal, not just a CI gate.
Pattern 3: A direct counter-case
The paper found a task where the messier variant beat the cleaner one — genie/cluster-active-job-limit, +8% tokens on the cleaner side. The reason: the cleanup pipeline extracted helpers around the focal logic but kept the original logic intact. The cleaner variant spread the work across more methods, forcing the agent to navigate a wider surface area without any gain in grep-ability. Over-abstraction is its own form of mess.
The takeaway: SonarQube tells you when a function is too long. It does not tell you whether splitting it makes the codebase easier to navigate. Cleanup pipelines that mechanically apply rules can produce code that's worse for agents, not better.
Pattern 4: The funny one
"Have you tried telling it: 'Write perfect code, make no mistakes' I use this one in my Ralph Harness all the time, it's a classic! It's not that it can't do that, it's just that you haven't told it to!"
A commenter pointing out that "write perfect code" is the AI version of telling a junior dev "just don't introduce bugs." Funny, but it surfaces a real point: a lot of agentic prompt engineering is theater. Linters enforce. Prompts suggest. Pick the deterministic tool when you can.
What the Paper Doesn't Tell You
The limitations section is honest about what wasn't measured. A few that matter in practice:
One model, one harness. All numbers are Claude Sonnet 4.6 in Claude Code. No GPT-5.6, no Gemini, no Cursor, no Aider. The mechanism (less code structure confusion = less re-reading) plausibly transfers, but the magnitude is unverified. The author explicitly says: "the transfer is conjecture in this paper."
Tokens, not dollars. Tokens are a cost proxy. Actual dollar cost depends on cache state, model version, queue delays, and provider pricing — none of which are properties of the codebase. A 7-8% token reduction may map to a different (larger or smaller) dollar reduction depending on configuration.
No cleanliness scan of agent output. Does clean code in = clean code out? The paper doesn't run SonarQube on the agent's final state. This is the most important follow-up question and it's open.
Short horizon. Each task is a single round-trip. The 7-8% saving is per-task. The interesting question is whether it compounds over months of agent work on a codebase, or whether the codebase drifts and erases the gains. SlopCodeBench (a different paper) shows agent-written code grows to 2.3x human-maintained code length over time, so the drift direction is the wrong one.
Huge trial-to-trial variance. This one jumped out at me. The paper notes: "The most expensive trial in a group typically costs around 2.5× the cheapest, and roughly 72% of groups have a ratio above 2×." Per-task, a 10% delta is often agent noise. The dataset-level numbers hold because they pool hundreds of trials, but if you're trying to measure "did my refactor make things faster" with a single agent run, you're measuring nothing.
"Cleaner" and "messier" are within-pair labels. The paper doesn't claim that commons-bcel or ckan are messy repositories. It claims the within-pair comparison is informative. That's the right way to read it.
Three Related Papers Worth Reading
While I was in the literature, three other papers gave useful context.
SlopCodeBench (Orlanski et al., 2026): measures how coding agents degrade over long-horizon iterative tasks. The headline result is that agent-written code drifts to 2.3x human-maintained baseline length after many iterations. Complementary to the cleanliness paper: this one is about agent output drift, the cleanliness paper is about agent input quality. Together they suggest cleanliness governance has to be bidirectional — clean what the agent reads, monitor what the agent writes.
Tokenomics (Salim et al., MSR 2026): dissects ChatDev on GPT-5 and finds that the code review phase alone consumes 59.4% of total tokens, with input tokens being 53.9% of the total. This explains why file revisitation matters so much — review is a high-revisit activity, and every revisit re-loads file contents into context. If cleanliness cuts revisits 34%, the review-heavy phase of agent work saves disproportionately more than the 7-8% dataset average.
SWE-Effi (Fan et al., 2025): introduces resource-aware metrics for SWE-bench and finds that failed tasks consume 4x more tokens than successful ones. Failures don't bail out early — they thrash. This is the case against treating cleanliness as a magic bullet: if the task is fundamentally hard and the agent fails, the 7-8% saving doesn't matter because the 4x failure multiplier dominates. Cleanliness is an amplifier, not a force multiplier.
How I'm Using This in My Own Projects
I changed four things after reading the paper and the HN thread:
- AGENTS.md now carries quality baselines, not vague advice. Previously I had "write clean code" in the AGENTS.md, which is theater. Now it has actual numbers pulled from SonarQube weekly reports — current issue density, target issue density, coverage target. LLMs respond to numbers, not slogans.
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
-
Pre-commit hooks are mandatory, not optional. I previously ran them locally but didn't enforce them in CI. After seeing the HN thread I added a CI job that fails if pre-commit doesn't pass. The agent now has to learn to pass the hooks, which it does in 2-3 attempts.
-
Legacy code gets tagged, not rewritten. I have a 6-year-old Python service that I'm not going to refactor. I added a
// LEGACY CODEheader at the top of each problematic file with a link todocs/legacy_rules.mdthat explains what the legacy patterns are and why. The agent now leaves them alone, which is the goal. -
I run SonarQube weekly and watch the trend line. Not the absolute number — the absolute number on an old codebase is depressing. The trend is what matters. Is the new-code issue density going up or down week over week? That's the metric.
The combination: clean input via pre-commit (deterministic enforcement), clean input via AGENTS.md (LLM guidance), clean output via SonarQube trend (continuous monitoring). The 7-8% from the paper is just the floor of what this combo gets you.
FAQ
Q1: Is "clean code" subjective?
Not in this paper. Cleanliness is operationalized as SonarQube rule violations: function length, nesting depth, identifier length, duplicate lines, cognitive complexity, comment coverage. The paper measures concrete rule counts, not vibes.
Q2: Does that mean if I get SonarQube to zero issues, the agent will be maximally efficient?
No. The paper has a counter-case where the cleaner variant cost 8% more tokens. Over-abstraction spreads logic across more files, which makes the agent hop more. SonarQube catches pattern violations, not structural optimality.
Q3: Should I change my coding style to accommodate agents?
Probably not. The overlap between "human-readable code" and "agent-readable code" is large. If your variable names are descriptive, your functions are short, your abstractions are reasonable, you're probably fine. The things agents hate (anonymous 50-line lambda chains, magic numbers, circular dependencies) are the things experienced humans also hate.
Q4: Will agent-written code get worse over time?
Probably yes, based on SlopCodeBench. LLM default style is verbose-but-shallow, because that style dominates the training data. Linters catch the worst of it, but they don't stop drift. Continuous monitoring matters more than one-time cleanup.
Q5: Will GPT-5.6 / Gemini 2.5 show the same effect?
Mechanism suggests yes, magnitude is unmeasured. Cleanliness affects navigation and revisitation, not reasoning. Any agent that reads files to understand code will show some version of the effect.
Q6: Is the 7-8% number stable across codebases?
No. The paper shows per-task deltas ranging from -47% to +44%, with a median of -4.5%. The 7-8% is the dataset-level average of 27 tasks, 540 trials, with substantial per-task noise. For a specific project, the number could be different.
Closing
There's an irony in the fact that SonarSource's study on "code cleanliness matters for AI agents" generated 200+ comments, and the most-upvoted responses were all about linters, pre-commit hooks, and deterministic enforcement — not about prompts or model selection. The HN consensus was effectively: the answer to LLM code quality is the same as the answer to human code quality 20 years ago, just with a different lint configuration.
That's also the honest summary of the paper. Cleanliness is a real, measurable factor in agent performance, but it's not a magic factor. It's the same factor it has always been for human developers, with a slightly different cost structure: agents accumulate tokens, humans accumulate frustration.
Next thing I want to test: does tach (the import boundary tool) actually do what the HN comments claim, or is it another piece of agentic-coding theater? A few weeks of using it on a real project, and I'll write up the results.
Got questions? Drop them in the comments.
- Written July 6, 2026. Data: arXiv:2605.20049 v1 + HN discussion thread + my own SonarQube integration on a 50k-LOC Python + TypeScript project. Paper: arXiv:2605.20049. HN thread: item?id=48798815.*