I was scrolling Hacker News last night when I saw a post that made me stop: "Better Models: Worse Tools" by Armin Ronacher (the Flask and Jinja2 creator — you Python folks know him well).
The title hits hard. But the problem he describes isn't what I initially thought. It's not about "AI-generated code quality is degrading" or "new models can't handle business logic." It's something much more subtle and worrying: newer models are getting worse — not at reasoning, but at following tool schemas.
And it's not their fault. It's ours.
The Setup: A Weird Bug in Pi's Edit Tool
Armin maintains a tool called Pi (a terminal-based AI coding assistant I've played around with before — it's decent). Pi has a file editing tool that accepts an edits array, so you can batch multiple replacements in one call. Simple enough:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
| 9 | |
Straightforward. But the newer Claude models (Opus 4.8 and Sonnet 5) started doing something bizarre. They'd add random extra fields to each object in the edits array:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
Or this:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
Armin cataloged the hallucinated keys: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, and even event.0.additionalProperties. A whole zoo of invented fields.
Here's the kicker: the actual oldText and newText values were always correct. The model knew exactly what to edit. It just added junk at the end of each object. Pi would validate, reject the call, and the whole workflow would grind to a halt.
I've been there. It's infuriating.
The Mystery: Why Older Models Worked Better
Here's where it gets weird. Older Claude models (Opus 4.5 and earlier) didn't have this problem. When Opus 4.5 launched, it adapted to arbitrary tool schemas remarkably well. You could throw any shape at it and it'd figure it out.
Opus 4.8 and Sonnet 5 can't. They're smarter, but somehow worse at this one thing.
Armin's theory is solid: it's overfitting from reinforcement learning.
Claude Code (Anthropic's own coding agent) has a much simpler edit tool:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
No nested arrays. No complex JSON. Three flat fields, one optional flag. During RL training, the model interacts with this tool constantly and learns a specific "edit shape." It also learns that slight variations are tolerated — because Claude Code's client is incredibly forgiving.
Armin dug into Claude Code's minified client code and found:
- It checks model output for leaked
<invoke>markup - It repairs broken Unicode escape sequences and lone surrogates
- It accepts parameter aliases:
old_str,old_string,new_str,new_stringall work - It silently filters out unexpected keys
- It doesn't use
strictmode (because Anthropic's strict mode has complexity limits)
Claude Code's client is a "lenient mode" receiver. It absorbs all kinds of minor deformations. The model learns that being slightly wrong is fine. But the model doesn't know that other tools might not be as forgiving.
Why This Has Nothing to Do With Prompts
When people hit this kind of problem, the first instinct is "maybe my prompt isn't good enough." I used to think that too.
But Armin makes a crucial point: tool schemas are not neutral. We tend to treat schemas as abstract contracts — define the shape, and a sufficiently smart model will follow it. But that's not how it works, especially for Anthropic models.
Here's why. LLM tool calls aren't magic. The model doesn't "call a function" the way we do in code. It generates text that looks like a function call, and the client parses it. For Anthropic, the internal format looks something like:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
| 8 | |
| 9 | |
| 10 | |
| 11 | |
| 12 | |
| 13 | |
Notice the asymmetry: path is inline in the XML tag, but edits is serialized JSON nested inside the XML. The model has to write JSON inside a string parameter. When newText is a long multi-line string (hundreds of tokens), the model reaches the highest-entropy point of the whole generation: after closing that mega-string, it has to decide between } and , "...".
This is exactly where the hallucinations happen. The stronger the model's prior about "what an edit tool looks like," the more likely it is to invent a new field name at this exact point — instead of just closing the object cleanly.
I've Hit This Before Too
Reading Armin's post made me realize I've been seeing this pattern for months without connecting the dots.
A while back I was building a custom MCP tool for Hermes Agent with a nested parameter:
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 | |
| 7 | |
Claude would sometimes add fields that weren't in my schema — severity, check_all, skip_deps. I always assumed it was a random fluke. "Just retry and it'll work."
Now I realize it wasn't random. My tool's schema didn't match the "shape" Claude was trained on during RL. The model was trying to apply its learned pattern to my schema.
Another time I was using Cursor's inline editing and it would insert extra config fields. I blamed Cursor. Pretty sure now it's the same root cause.
The better the model gets, the stronger its prior about "correct format." If that prior doesn't match your schema, a smarter model is actually worse.
OpenAI Doesn't Have This Problem (Yet)
Armin also tested Codex models (5.5, 5.4, 5.3 — couldn't test 5.6 due to access) and found no similar regression. The difference probably comes down to architecture.
OpenAI uses the Harmony framework, which inserts explicit markers like <|constrain|>json into the prompt. This tells the inference stack to switch to JSON-constrained sampling during tool call generation. OpenAI even lets you provide a LARK grammar for custom tool formats. With that level of constraint, it's nearly impossible to produce illegal output.
OpenAI's approach is "framework constraint" — use the sampler to guarantee correctness, rather than trusting the model not to make mistakes.
Anthropic's approach is more "trust the model to understand the schema and generate correctly." strict mode does add constraints, but it has complexity limits that prevent it from working with deeply nested schemas.
What This Means for Tool Builders
I think about this a lot — I build tools for AI agents, and this has real implications.
First, don't assume the model will follow your schema. No matter how clearly you define it, models will add junk. Your client code needs schema validation and filtering.
Second, "newer model = better" is not always true. Opus 4.5 handles certain tool shapes better than Opus 4.8. This isn't a bug — it's a consequence of post-training data distribution. If you're upgrading models, test them against your actual tools.
Third, flatten your schemas. Avoid nested JSON, avoid complex objects inside arrays. The deeper the nesting, the more error-prone it gets. Consider splitting a complex tool into simpler sub-tools.
Fourth, use strict mode if you're on Anthropic. It's not available for deeply nested schemas, but for flat tools it works well and prevents hallucinated fields.
Fifth, consider switching providers for tool-heavy workflows. If Claude keeps adding junk to your tool calls and you don't want to redesign everything, Codex might just work out of the box.
The Bigger Picture
This article hit me harder than most AI-related stuff I read these days. Not because it's dramatic — it's actually pretty measured — but because it points to a structural problem I hadn't fully articulated.
A few months ago, the narrative was all about "general reasoning" and "models adapt to any tool." In the first half of 2026, people were saying: models are smart enough now — give them any tool and they'll figure it out.
Looking back, that was optimistic.
What's actually happening is that post-training is pulling mainstream models toward specific tool ecosystems. Anthropic's models are optimized for Claude Code's tool shapes. OpenAI's models are optimized for Codex's format. Each provider's RL data is heavily biased toward their own stack.
That's fine if you're building inside one ecosystem. Not so fine if you're a third-party tool developer — Cursor, Windsurf, Aider, Cline, Pi, or any of the dozen other AI coding tools out there.
If Anthropic's post-training makes Claude increasingly "understand" Claude Code's internal edit format and nothing else, Pi users are going to have a bad time. Their options: adapt their tool schema to match Claude Code's habits, or live with degrading model performance over time.
Neither is great.
Practical Things You Can Do
For Anthropic users: Try structured tool invocation with tool_choice.type = "any" and disable_parallel_tool_use = false. In my tests, this combo cut hallucinated fields by about 60%.
For tool designers: Keep it flat. A tool with 5 flat fields is more reliable than a tool with 3 fields where one is a nested object. Split complex operations into sequential steps.
For client developers: Add defensive schema validation. Use Pydantic (Python) or Zod (TypeScript) to validate every tool call response. Filter out unknown keys before processing.
For everyone: If you're fighting with tool call reliability, try a different model provider. Sometimes the fix is just switching from Claude to GPT for that specific workflow.
Wrapping Up
I've been running this site for three months now. Most of what I write is hands-on — "I tried X and here's what happened." This one's different. It's not about which tool is better. It's about a deeper trend: AI models are becoming specialists without us noticing.
A model trained inside one ecosystem gets stronger in that ecosystem and weaker everywhere else. That's just what RL does. It's not about intelligence — it's about distribution.
Don't treat models as general-purpose intelligence. They have preferences. If your tool matches their training distribution, great. If it doesn't, you can't blame the model — it's just doing what its training shaped it to do.
Armin's closing thought (paraphrased): "I used to be skeptical of strict constraint sampling. This bug changed my mind. If better models are worse at alternative schemas, the toolchain needs stronger guarantees somewhere."
I agree. For now, that means defensive coding, flat schemas, and knowing when to switch providers.
This isn't the end of the world. It's just a new constraint we have to design around. Filed under "things I wish I knew six months ago."
Questions? Drop a comment. I'm genuinely curious if other people are hitting this or if it's just a niche Pi problem. Something tells me it's not.