Harnessing the Agent: Feedback Loops as a Leash
“You’re absolutely right - I missed that edge case.”
“This time all the edge cases really are covered.”
Are they?
If you use coding agents daily, you’ve had this conversation. I’ve had it more times than I’d like to admit. It’s the thread I pulled on in my talk at BRAVE AI Community Kraków #2 - and this post is the written version of it, plus the parts of the Q&A that stuck with me long after the pizza was gone.
The short version: the bottleneck isn’t what AI can build. It’s whether we can trust what it built. And trust doesn’t come from a better prompt - it comes from feedback loops wired into the process.
The context: a large organisation, not a side project
Quick bit of context, because it shapes everything below. For roughly ten years I’ve been building large-scale backend systems - gaming and ad tech. Right now I lead a team at a leading gaming company, working on services used by tens of millions of daily active users.
That means the boring, important stuff is non-negotiable. Quality gates. Compliance. A real pull request culture. Code that has to be good, not just green.
And the most important rule of all: you are responsible for the code you ship. AI doesn’t change that. If anything, it makes accountability harder to trace - the agent changes who writes the code, not who answers for it.
Four stages of AI maturity
To frame where we are, I like a simple maturity model for building software with AI.
- Stage 1 - Niche. Copy a function from ChatGPT, paste it, tweak it. Marginal speed-up - maybe faster, maybe not, hard to tell.
- Stage 2 - Utility. Agents as assistants. New, better tools - but plugged into the old process. We do things the same way, just a bit faster.
- Stage 3 - Competitive. The agent builds whole modules. Our job shifts to programming the process of building software - the SDLC as code.
- Stage 4 - Ubiquity. The future baseline. Everyone does it this way, there are books on how to do it right, and nobody can imagine working otherwise - the way nobody programs with punch cards today.
Most teams I talk to are somewhere between Stage 2 and Stage 3. Many think they’re at 3 because they have Claude or Cursor in their IDE. They’re at 2.
The jump from 2 to 3 isn’t a better model or a better prompt. It’s a different workflow - one with a feedback loop and a verification layer for what the agent produces. Without that, you’re at 1.1x, convinced you’re doing agentic coding.
The case: a legacy service with zero tests
So what does that look like in practice?
The case was a complex Java backend service. Lots of endpoints, lots of edge cases, lots of consumers. And with an API that many people depend on, you can assume one thing - somebody, somewhere, relies on the behaviour you never meant to ship. Any change is a potential regression.
We needed scenario tests. A lot of them - exercising the API end to end, against the service running in a container. And we had zero test infrastructure to start from.
The original estimate for covering it properly was measured in months.
The naive way
The naive approach is easy to picture. Tell the agent: “write tests”. It writes tests. They look fine. The PR gets approved.
But what about all the edge cases? How do we actually know the target is properly tested?
That’s the real question - and it isn’t about capability. The agent can absolutely write the tests. The problem is that nobody can tell what it skipped.
Why agents skip things
Part of it is physical. You’ve probably heard of the smart zone / dumb zone - models do much better with what’s early in a fresh context, and degrade as the context fills up. They forget what they planned, forget what they already did, drift away from the original task. Compaction silently throws things out. And past that boundary, the agent has no way to verify its own work.
The other part is what I call the AI multiplier. An agent accelerates beautifully - but it can’t brake before a corner. If the plan is wrong and nobody checks it before execution, the agent will execute it anyway. Confidently, at full speed, to completion.
That’s not about bad prompting. It’s that an unverified plan now gets implemented - and replicated - much faster. Garbage in, garbage out, at a pace you can’t keep up with fixing.
The fix: structure, phases, and an external trace
The answer is structure: clear phases in the development lifecycle, each with a concrete output, so the agent knows exactly what “done” means at every step.
The key principle - we can’t rely on the agent remembering. We have to force an external, verifiable trace out of every phase. Something that lives outside the context window and that the agent can’t lose.
Then the next question: can we make the agent check that trace itself?
That’s where the two loops come in.
Loop 1 - Accountability
Here’s the uncomfortable truth about reviewing AI-written tests. Even when the PR is reasonable - one endpoint, ten test cases, not a thousand lines - judging whether the edge cases are covered requires the reviewer to build the whole mental model of the service in their head.
At that point we’re back at Stage 2. The agent is just a typing accelerator, because the mental model still lives in the human.
And to be clear, the agent isn’t being malicious. It simply has no reason - and no mechanism - to check that what it did matches what it planned.
So we gave it one:
- Plan first. Before writing any code, the agent analyses the service and documents every planned test case in a plain markdown file. Each case is a short entry - name, category, and a unique tag. No essays.
- Implement against the plan. Only then does it write the tests, each one carrying its tag.
- Verify mechanically. A simple script compares the two. A tag in the docs without a matching test - build fails. A test without a matching tag in the docs - an orphan - build fails.
We deliberately didn’t use the model for that check. No point burning tokens asking an LLM whether the IDs match - a script does it deterministically, every time.
The result: the agent can’t say “done” without proof. Not because it was told to be careful - because the build won’t let it through.
The plan also becomes the place where human review pays off most. Reviewing a list of test cases - does this case make sense, do we even want to test this? - is far cheaper than reverse-engineering intent from a few thousand lines of test code. And an unreviewed plan is exactly how you end up with a mountain of pointless tests.
How do we know the plan is complete?
Loop 1 proves the agent did what it planned. It says nothing about what it never planned at all.
Enter JaCoCo - a code coverage tool for the JVM. It shows exactly which branches of the code were exercised and which weren’t: green where the tests reached, red where they didn’t.
Here’s the funny part. When I wrote tests myself, coverage reports were never my primary tool. I had the mental model. When I added an endpoint, I knew how it behaved and what I wanted to test - I didn’t need a report to tell me.
The agent doesn’t have that mental model. But JaCoCo produces XML - and AI is genuinely better at reading that kind of structured report than most engineers, who (honestly) just skip it.
Loop 2 - Coverage
JaCoCo runs alongside the scenario tests, against the live container. While the system under test handles the scenarios, JaCoCo records which branches were actually hit.
Then we hand the report to the agent. It parses the XML and finds the gaps - a legacy flow nobody exercised, an exception handler that never fired, an odd edge case in response mapping.
Each gap becomes a newly documented test case - which goes straight back into Loop 1: tagged, implemented, verified, re-run.
In effect, we gave the agent eyes on its own blind spots. Not by asking it to review its own work - by giving it an external signal that points at exactly what’s missing.
Together: accountable AI
Put the two together:
- Loop 1 - the agent proves it did what it said it would do.
- Loop 2 - the agent checks whether what it planned actually covers what the service does.
That’s the leash from the title. Accountability woven into the process - not bolted on from the outside.
The outcome: work originally estimated in months landed in a few weeks. But the speed isn’t what I’m proudest of. What mattered was that I could actually trust that the tests behind it were really checking the service.
And yes - every bit of it still went through code review. Large organisation, code quality, accountability. Believe me, that was a lot of generated code to read.
The twist: the tests were never the goal
Good test coverage is nice. But it wasn’t why we built any of this.
The real goal was to rewrite the service. Old framework, new framework - and the API had to remain 100% compatible. We wanted AI to do the migration. But how could we possibly trust it?
That’s what the scenario tests really were. Not a deliverable - a behavioural contract of the original service.
The agent rewrites the service. The new service is tested as a black box against the same 500+ scenarios. Green across the board means compatible. Failures point straight back to the agent - which behaviour drifted, what to fix.
Same principles as the small loops. Just a much bigger one. Everything we learned building them - external traces, mechanical verification, structured feedback - is exactly what makes the big loop credible.
What the Q&A taught me
The questions afterwards were, honestly, sharper than parts of the talk. A few I keep coming back to.
“Coverage tells you the code ran - not that the test is any good.” Someone suggested mutation testing - deliberately breaking the code and checking whether the tests notice. It’s a fair point. In our case the scenario tests had a narrow job - API compatibility - but the general gap is real: the harness catches missing coverage, not weak assertions. A test that passes while checking nothing is still worthless.
“When do you stop?” That one connects to something I’d seen days before. I was running Codex as a PR reviewer and it fell into a loop. By the third iteration I suspected something was off. Around the thirtieth, it concluded on its own that the task couldn’t be done - you can’t satisfy every context at once. Automation needs a hard stop. We don’t want 100% coverage; sometimes it’s not even possible. Leaving an agent alone “for a week and see what happens” didn’t work for us - you still have to supervise, and know when to say “enough”.
“What slipped through?” Things you didn’t anticipate always do. Which is why the person opening the PR still signs off on it - and has to genuinely understand it. A PR that gets rubber-stamped defeats the point. Tests are also a forgiving example; for production code, the bar is higher and the story is less rosy.
“We say what needs to be done in the morning. Two hours later there are 2,000 lines of code. And our job becomes only review.” This one hit hardest. Someone in the audience put it bluntly - at some point we’ll have to decide what’s worth reviewing and what we let go. My take: you can’t drop review anywhere without trust first - that’s what the loops are for. But once that trust infrastructure exists, the question becomes unavoidable. Our current process - write code, review code - was built for human-scale output. To really reach Stage 3, the process itself probably has to change.
“In five years, will Java be what assembler is today?” Another attendee answered for me: specs are what will remain - code is already the assembler. I don’t know. But if the code is the output and the process is the work, then designing the process - the plans, the contracts, the loops that verify the output - becomes the engineering that matters most.
And a thought experiment I’m still chewing on. If an AI diagnoses correctly 95% of the time and the average doctor 90%, which doctor will bet their judgement against the machine? Developers will face the same temptation. I don’t think the answer is to blindly defer to the better average. It’s to understand why the AI decided what it did - which, again, is only possible when the reasoning leaves a trace you can check.
The challenge
Here’s what I left the room with.
Don’t start with two loops and a coverage pipeline. Start smaller:
Pick one AI task you already run. Ask yourself how you know the agent did what it was supposed to - and did it well. Do you check manually? Do you know exactly where it tends to cut corners?
Then make it verify that one thing itself - with something external: a script, a tool, a build gate - and automate it.
One external check doesn’t catch every problem. But it makes problems visible - and right now, AI failures are invisible by default.
The question was never what AI can build. It’s whether we can trust it - and whether everything is actually covered.
Thanks to the BRAVE AI Community and Remitly for hosting, and to everyone who asked questions. The slides are here.