28 out of 28 tests passed. The feature didn't work at all.

We built a new piece of Forge tonight — a way for Forge to sit alongside your own coding assistant and quietly catch real problems in your project as you work. Seven pieces, each one built test-first, each one reviewed on its own before the next piece started. By the time all seven were done, 28 tests were green and every individual review had approved.

The feature did not work. Not “worked with bugs” — did nothing, for every single real user, every single time.

What per-piece review can’t see

Each of the seven pieces was reviewed in isolation, and each review was right, as far as it looked. The piece that reads your project for problems: correct. The piece that remembers what happened last time: correct. The piece that wires everything into your coding assistant’s settings: correct, by itself.

What no single review could see is how the pieces actually run once they leave the test suite. Two things broke, and both were invisible from inside any one file:

The scripts couldn’t find their own code. Every test imported the modules directly, the normal way test suites do. But a real coding assistant doesn’t import a hook script — it runs it as a separate program, python path/to/script.py. Run that way, the script’s own folder becomes the only place Python looks for imports, and the code it depends on sits one folder up. Every real invocation crashed on line one, before any of the error handling we’d carefully written could even run.

The registration wrote the wrong shape of file. The piece that turns this feature on writes an entry into your coding assistant’s own configuration file. We wrote the entry in a plausible, reasonable shape. It wasn’t the shape the real configuration format actually uses — confirmed by checking it against a configuration file that’s been running in production for weeks. The entry would sit there, syntactically valid JSON, and never fire.

Put together: a user could install this, see no errors, and nothing would ever happen. Green tests, quiet failure, on the exact thing that was supposed to happen every time they typed a prompt.

Why the tests didn’t catch it

Every one of the 28 tests was testing something real and testing it correctly. What none of them did was run the code the way it actually runs in the world — as a standalone program, invoked from outside its own folder, writing into a file whose format lives somewhere else entirely. That’s not a gap in test quantity. It’s a gap in test shape — the tests proved the logic was right and never proved the logic was reachable.

The fix, once found, was small: three lines to fix how the scripts find their own code, one corrected shape for the configuration entry. The catch was the expensive part, not the fix.

Worth saying plainly: this isn’t turned on for anyone yet. It’s built and it’s fixed, but the piece that lets you actually enable it from inside Forge doesn’t exist yet — that’s the next real step, not this one.

What actually caught it

Not a bigger test suite. A separate, later pass that looked at the whole seven-piece feature as one thing instead of seven reviewed things, and — crucially — actually ran each script the way a real coding assistant would run it, instead of trusting that “the tests pass” meant “the feature works.” That’s the same discipline we keep coming back to on this blog: check the thing that’s supposed to be true, don’t infer it from something adjacent.

The takeaway

“All tests passing” answers one question: does the code do what the tests describe? It never answers a second, different question: does anything actually reach this code the way we assumed it would? We’d rather find that gap ourselves, on a feature that hadn’t shipped to a single real user yet, than have a user find it for us.