Our AI debugger's confidence numbers were inverted. Here's the graph.

Two fixes came back from the same call last week, ranked by confidence like they always are. The higher-ranked one failed the moment we ran it against the real bug. The lower-ranked one passed. We'd been trusting that ranking, the same way anyone reading a DebugAI response trusts it: a bigger number means the model is more sure. So we stopped trusting it and started checking.

Two weeks ago we found three bugs in our own eval harness before we'd let it grade a single real result. This is what the harness found once we trusted it, and what we built because we couldn't unsee it.

What the numbers said#

We bucket every fix by its reported confidence, then run it against the real bug and record whether it actually passed. Here's the split from the run we're standing behind:

Read that twice. Fixes the model was most sure about passed less often than a coin flip. Fixes in the 80-90% band, still a band most people would read as "pretty confident," passed exactly none of the time in this run. The number was going up while the odds of it being right were not.

Note: "Passed" here means the fix, applied to a clean copy of the source, made the actual test pass. Not "looked reasonable." Not "the model sounded sure." Ran, on real code, and worked.

Same shape, three times#

The first sign of this wasn't the table above. It was a single bug, checked by hand, where two ranked fixes came back at 92% and 85% confidence and both of them failed. That one case could have been noise. So we went looking for more of it, on two separate runs of the harness we'd built specifically to check.

All three looks found the same shape: confidence going up did not mean correctness going up. Then we rebuilt the harness from scratch in a new home, ran it a fourth time on a fresh 11-bug corpus, and got the table above. Same shape again, on code the earlier runs had never touched.

Four separate looks, at different times, on different bugs, with different versions of the checking tool. If this were a one-off measurement error, at least one of those runs should have come back clean. None did.

What we built because of it#

A confidence number that doesn't track correctness isn't just unhelpful, it's actively misleading, because it looks exactly like a helpful one. So instead of asking the model to grade its own homework, we built a grader that isn't the model.

For two classes of bug where the check is cheap and deterministic, parse errors and import errors, DebugAI now runs the proposed fix through a real mechanical check before it ever reports a confidence number:

• Parse check: does the fixed code actually parse. Not "does it look like valid syntax to a language model," a real parser, sandboxed, with a hard CPU limit.
• Import check: does the import the fix relies on match a file DebugAI actually retrieved for this request. Worth being precise about what this claims: it means the import resolves against the context we pulled, not that the package is installed or the function exists at runtime. We say exactly that, nowhere more.

Fix: if the check fails, confidence gets capped at 15, no matter what the model originally reported. min(fix.confidence, 15) when verified comes back false. The model doesn't get a vote on its own grade once we've actually run the check.

Both checks run sandboxed, with a five-second budget shared across all candidate fixes for a request. If a check can't finish in time, it doesn't guess. It reports that it didn't finish.

Note: a check that didn't finish and a check that failed are different claims, and we don't let them collapse into each other. verified: null means "we didn't check," either because this bug class isn't covered yet or because the five-second budget ran out. verified: false means "we checked, and it failed." Rounding null up to false, or down to a silent pass, would make the label lie in exactly the way the original confidence number did.

Seeing it happen for real#

This isn't theoretical. The first time the check fired against a real request, this is what came back:

{
  "rank": 1,
  "confidence": 15,
  "verified": false,
  "verification_reason": "import(s) not found in retrieved context: billing.pricing (checked against files DebugAI retrieved, not installed packages)"
}

The model didn't know the exact name of the function it needed to import. Rather than confidently inventing one, it generated a placeholder and flagged that it needed more context. The old behavior would have reported whatever confidence the model felt like reporting on a guess. Instead, the check ran, found the import didn't resolve against anything we'd actually retrieved, and the confidence landed exactly where min(fix.confidence, 15) puts it: 15. Not a dramatic catch, the model was already being honest here, but it's the first real proof the wiring works end to end, on a live request, not a fixture.

What this doesn't fix yet#

Parse and import checks cover exactly two bug classes. The bucket table above, 0% at 80-90% confidence, 38% at 90%+, was measured on a corpus that's mostly outside those two classes, and outside those classes, nothing has changed. The inversion is still there for TypeError, RuntimeError, and everything else that needs a live environment or real data to actually verify, not just parse.

Type-class checking, the kind that would catch more of what's left, is built behind the same interface and not shipped. It's gated on whether it can run inside the five-second budget without becoming the slow part of every response. We're not claiming this closes the gap. We're claiming it closes it for the two classes cheap enough to check honestly today, and that the rest stays visibly unverified rather than quietly assumed fine.

Every fix DebugAI returns now carries one of three states: true, false, or null. True means we ran a real check and it held. False means we ran it and it failed, and the confidence number reflects that whether the model likes it or not. Null means we didn't check, and we say so instead of letting a high number stand in for an answer we don't have.

FAQ#

Q: Does this mean DebugAI's fixes are usually wrong?

A: No. It means the confidence number attached to a fix wasn't a reliable signal of whether that specific fix would work, for the classes we measured. The fixes themselves aren't graded here, the correlation between "how sure the model sounded" and "was it right" is. Those are different questions, and conflating them is exactly the mistake this whole feature exists to stop making.

Q: Will this cover more bug classes over time?

A: Parse and import were the two cheap enough to check mechanically without adding real latency. Type checking is built and waiting on a latency decision. Anything that needs live execution against real state, most runtime errors, needs a different approach than a five-second sandboxed check, and we're not going to claim we've solved that until we actually have.

DebugAI reads your codebase, not just your stack trace, and hands you a fix you can apply in one click. Now some of those fixes come with a number we checked ourselves, and the ones we haven't checked yet say so.