The prompt carries judgment. The code carries the things that must never happen.
My prompt told the agent never to open a pull request without a committed change to the manifest. It opened one anyway.
Here’s how. The fixer agent I’ve been building hit a string of 404s from an API it was calling. It retried, got nowhere, and instead of stopping it went ahead and opened an empty PR. The rule it broke was right there in the system prompt, stated in plain English. It read the rule. It broke the rule.
I could have written a sterner prompt. All caps, more emphasis. I’ve done that before and it holds until the next weird state the model hasn’t seen. Instead I moved the rule out of the prompt and into the tool. The function that opens a PR now compares the two refs and refuses when the branch matches the base. There is no empty PR the model can open, because the code won’t produce one. I reran the whole thing clean and the fixer bailed the way it should have the first time: package not in the manifest, zero PRs.
That’s the pattern I’ve settled on. The prompt carries judgment. The code carries the things that must never happen.
What moved into code
The constraints I pushed down are the ones that would be a real problem if the model ignored them even once.
- The agent works on one repo and one branch, both fixed when it starts and read from the database. The model never picks either.
- It can edit the manifest and nothing else. The edit tool has a path allowlist and throws on anything that looks like source code.
- Every PR it opens is a draft. The draft flag is hardcoded. No code path marks a PR ready for review, so it can’t.
- It never auto-fixes a high-severity finding, and it works one repo at a time. Both are refused before any work goes out.
- On a red CI result, the code decides what happens next, not the model. Same failure twice, or more than six attempts, and the run halts and the card gets marked blocked.
None of that lives in the prompt anymore. The prompt still does real work. It decides which override to reach for, when to narrow the scope, what the smallest safe change is. That’s judgment, and the model is genuinely good at it. But judgment and hard limits are different jobs. Judgment can be wrong on a given call and you recover. A hard limit that fails once opens the empty PR, touches the file it shouldn’t, or ships the thing nobody approved. You don’t want that in a paragraph the model is free to reinterpret when it hits a state you never tested.
This is the same instinct behind harness discipline at team scale: the leverage isn’t a smarter prompt, it’s the structure around the model that holds regardless of what the model does.
The over-correction is the same bug, inverted
Here’s the part I didn’t see coming, and it turned out to be the more useful half.
Once the hard limits were in code, I got confident and pushed the same instinct into the triage rubric. Be conservative. When you’re not sure which repo a security ticket belongs to, don’t guess, escalate to a human.
Safe, right. I ran it against 50 real security tickets from our backlog. All 50 landed in escalate-to-a-human. Every one. The gate worked exactly as written and it was useless. I had built a system whose only output was “ask me.”
The tickets weren’t even ambiguous. Most of them named the repo in the title. The agent had the answer in front of it. What it didn’t have was permission to act on an inference, because I’d told it never to guess, and reading a repo name out of a title counts as guessing when your rule is that absolute.
That’s the trap. A deny-by-default gate with no way to say yes doesn’t remove the work, it relocates the whole pile onto a person. It’s the same failure as an agent that fixes everything in sight, just flipped, and it’s easier to miss because it looks responsible. Nothing broke. Nothing shipped either.
Both halves or neither
So the rule I actually run on now has two parts, and you need both.
Put the constraints that must never break into code, where the model can’t talk itself past them. Then put real effort into the inference layer that lets the gate say yes when it’s safe, or the gate blocks everything and hands you back your own backlog. Confinement and inference aren’t opposites. One keeps the agent from doing damage. The other keeps it from doing nothing. Ship one without the other and you’ve built either a loose cannon or an expensive way to generate a to-do list with your name on every line.
Prompts are where you tell the model how to think. Code is where you decide what it can’t do, and what it’s finally allowed to do without you. Get that line right and the agent earns its keep. Get it wrong in either direction and you’ve built a sophisticated way to make more work for yourself.