What we learned from 1,594 AI-generated tickets

This week we ran our internal AI swarm — the multi-agent engine we use to build and maintain Axonris itself, not Forge’s own runtime — against our own codebase. The swarm generated 1,594 tickets: code fixes, refactors, new features, security patches. We expected most of them to get processed. Instead, 87% silently disappeared.

The silent drop

Our deduplication logic — the part of swarm_manager.py that prevents the same fix from being attempted twice — was marking 1,388 out of 1,594 tickets as “duplicates” and skipping them. No alarm, no log line, no dashboard warning. They just vanished.

This is the kind of bug that doesn’t crash anything. It doesn’t throw an error. It just quietly makes your system do almost nothing while looking like it’s working fine.

How we found it

We didn’t find it through monitoring (we didn’t have the right monitoring). We found it because the numbers didn’t add up — too few completed tasks relative to how many were being created. One manual count later, the gap was obvious.

What we changed

Three things, in order:

1. Funnel metrics. We now track a drop rate at every stage of the pipeline: created → dispatched → attempted → completed. If any stage silently swallows more than a configurable threshold of tickets, the dashboard shows a warning. Trust is good; metrics are better.

2. Fast-fail for local models. Our local AI (Qwen 14B on Ollama) was getting 6 attempts per ticket before giving up. Our data showed it either succeeds on the first try or never succeeds — the extra 5 attempts were pure waste. We cut it to 3.

3. Task-based routing. We discovered that local models handle RESEARCH tasks at 100% success rate but fail at code-writing tasks 94% of the time — not because they’re not smart enough, but because they can’t reliably format JSON tool calls. So we stopped asking them to. RESEARCH stays local (free). Code changes route to DeepSeek (cheap, reliable). Hard problems escalate to Claude.

The takeaway

The lesson isn’t “AI is unreliable.” The lesson is that the plumbing around AI is where most failures hide. The models themselves worked fine. The routing, deduplication, error handling, and observability around them is where 87% of our work silently vanished. If you’re building AI-powered tools, instrument the pipeline, not just the model.

We’re sharing this because we think the honest version of how AI tools get built is more useful than the polished version. If you want to follow along as we figure this out in public, check the blog or join the waitlist.